Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention
Authors: Shuang Wu
Youtian Lin
Feihu Zhang
Yifei Zeng
Yikang Yang
yajie bao
Jiachen Qian
Siyu Zhu
Xun Cao
Philip Torr
Yao Yao
Venue: NeurIPS 2025 poster ()
Keywords: AIGC
3D Generation
Diffusion
Primary area: deep_learning
TL;DR: a novel approach to enable training high resolution sparse voxel diffusion model for 3D generation

Abstract: Generating high-resolution 3D shapes using volumetric representations such as Signed Distance Functions (SDFs) presents substantial computational and memory challenges. We introduce Direct3D-S2, a scalable 3D generation framework based on sparse volumes that achieves superior output quality with dramatically reduced training costs.
Our key innovation is the Spatial Sparse Attention (SSA) mechanism, which greatly enhances the efficiency of Diffusion Transformer (DiT) computations on sparse volumetric data. SSA allows the model to effectively process large token sets within sparse volumes, significantly reducing computational overhead and achieving a 3.9$\times$ speedup in the forward pass and a 9.6$\times$ speedup in the backward pass.
Our framework also includes a variational autoencoder (VAE) that maintains a consistent sparse volumetric format across input, latent, and output stages. Compared to previous methods with heterogeneous representations in 3D VAE, this unified design significantly improves training efficiency and stability.
Our model is trained on public datasets, and experiments demonstrate that Direct3D-S2 not only surpasses state-of-the-art methods in generation quality and efficiency, but also enables training at 1024³ resolution using only 8 GPUs—a task typically requiring at least 32 GPUs for volumetric representations at $256^3$ resolution, thus making gigascale 3D generation both practical and accessible. Project page: https://www.neural4d.com/research-page/direct3d-s2.

Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention
Shuang Wu
Youtian L.n
Feihu Zhang?
Yifei Zeng
Yikang Yang'
Yajie Bao²
Jiachen Qian²
Siyu Zhu
Xun Cao
Philip Torr
Yao Yao-t
"Nanjing University
Dream
3 Fudan University
Uniieeriity of Oxford
[wushuang, (aanyyouuaan@@smaallel nju. edu. cn
{ yaoyao Anju. edu. cn
Abstract
Generating high-resolution 3D shapes using volumetric representations such as Signed Distance Functions (SDFs) presents substantial computational and mem- ory challenges. We introduce Direct3D-S22, a scalable 3D) generation framework based on sparse volumes that achieves superior output quality with dramatically reduced training costs. Our key innovation is the Spatial Sparse Attention (SSA) mechanism, which greatly enhances the efficiency of Diffusion Transformer (DiT) computations on sparse volumetric data. SSA allows the model to effectively pro- cess large token sets within sparse volumes, significantly reducing computational overhead and achieving 3,9x speedup in the forward pass and a 9.6x speedup in the backward pass. Our framework also includes a variational autoencoder (VAE) that maintains consistent sparse volumetric format across input, latent, and output stages. Compared to previous methods with heterogeneous representa- tions in 3D VAE, this unified design significantly improves training efficiency and stability. Our model is trained on public datasets, and experiments demonstrate that Direct3D-S2 not only surpasses state-of-the-an methods in generation quality and efficiency, but also enables training at 1024 resolution using only GPUs, task typically requiring at least 32 GPUs for volumetric representations at 256 resolution, thus making gigascale 3D generation both practical and accessible. Project page: http://wwwww..lo.oooo///eeeeceee
Introduction
Generating high-quality 3D models directly from text images offers significant creative potential, enabling rapid 3D content creation for virtual worlds, product prototyping, and various real-world applications. This capability has garnered increasing attention across domains such as gaming, virtual reality, robotics, and computer-aided design.
Recently, large-scale 3D generative models based on implicit latent representations have made notable progress. These methods leverage neural fields for shape representation, benefiting from compact latent codes and scalable generation capabilities. For instance, #DShape2Vecsed [46] pioneered diffusion-basee shape synthesis by using a Variational Autoencoder (VAE) [13] to encode 3D shapes into latent vecset, which be decoded into neural SDFs or occupancy fields and rendered via Marching Cubes [23]. The latent vecset is then modeled with diffusion process to generate
"quual contribution.
t Cooresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 1: Mesh generation results from our method on different input images. Using our high- resolution SS-VAE and SS-DiT, we are able to generate detailed and complex 3D shapes. The meshes show fine geometry and high visual quality, demonstrating the strength of our approach for high-resolution 3D generation.
diverse 3D shapes. CLAY [48] extended this pipeline with Diffusion Transformers (DiT) [29], while TripoSG [17] further improved fidelity through rectified flow transformers and hybrid supervision. However, implicit fatent-based methods often rely on VAEs with asymmetric 3D representations, resulting in lower training efficiency that typically requires hundreds of GPUs.
Explicit latent methods have emerged as compelling alternative implicit ones, offering better interpretability.. simpler training. and direct editing capabilities, while also adopting scalable architec- tures such as DiT [29]. For instance, Direct3D [38] proposes use tri-plane latent representations to accelerate training and convergence. XCube [31] introduces hierarchical sparse voxel latent diffusion for 1024 sparse volume generation, but only restricted to millions valid voxels, limiting the final output quality. Trellis [39] integrates sparse voxel representations of 256 resolution, with the rendering supervision for the VAE training. In general, due high memory demands, existing explicit-latee methods are limited in output resolution. Scaling to 1024 with sufficient latent tokens and valid voxels remains challenging, as the quadratic cost of full attention DiT renders high-resolution training computationally prohibitive.
To address the challenge of high-resolution 3D shape generation, we propose Direct3D-S2. a unified generative framework that utilizes sparse volumetric representations. At the core of our approach is a novel Spatial Sparse Attention (SSA) mechanism, which significantly improves the scalability of diffusion transformers in high-resolution 3D shape generation by selectively attending to spatially important tokens via learnable compression and selection modules. Specifically, we draw inspiration from the key principles of Native Sparse Attention (NSA) [45], which integrates compression, selection, and windowing to identify relevant tokens based on global-local interactions. While NSA is designed for structurally organized ID sequences, it is not directly applicable unstructured sparse 3D data. To adapt it, we redesign the block partitioning to preserve 3D spatial coherence and revise the core modules to accommodate the irregular nature of sparse volumetric tokens. This enables efficient processing of large token sets within sparse volumes. We implement custom Triton [36] GPU kernel for SSA, achieving 3.9x speedup in the forward pass and 9.6x speedup in the backward pass compared to FlashAttention-22 at 10243 resolution.
Our framework also includes a VAE that maintains consistent sparse volumetric format across input, latent, and output stages. This unified design eliminates the need for cross-modality translation, commonly seen in previous methods using mismatched representations such as point cloud input, ID

---

vector latent, and dense volume output, thereby improving training efficiency, stability, and geometric fidelity. After the VAE training, the DiT with the proposed SSA will be trained on the converted latents, enabling scalable and efficient high-resolution 3D shape generation.
Extensive experiments demonstrate that our approach successfully achieves high-quality and efficient gigascale 3D generation, milestone previously unattainable by explicit 3D latent diffusion methods. Compared prior native 3D diffuuion techniques, our model consistently generates highly detailed 3D shapes while significantly reducing computational costs. Notably, Direct3D-S2 requires only 8 GPUs train on public datasets [8, 9, 20] at a resolution of 1024", in stark contrast prior state-of-the-an methods, which typically require 32 or more GPUs even for training at 256 resolution.
2 Related work
2.1 Multi-view Generation and 3D Reconstruction
Large-scale 3D generation has been advanced by methods such as [15, 21, 22, 41], which employ multi-view diffusion models [37] trained on 2D image prior models like Stable Diffusion [32] to generate multi-view images of 3D shapes. These multi-view images are then used to reconstruct 3D shapes via generalized sparse-view reconstruction models. Follow-up works [19, 26, 35, 42, 47] further improve the quality and efficiency of reconstruction by incorporating different 3D repre- sentations. Despite these advances, these methods still face challenges maintaining multi-view consistency and shape quality. The synthesized images may fail to faithfully represent the underlying 3D structure, which could result in artifacts and reconstruction errors. Another limitation is the reliance on rendering-base While this avoids the need for direct 3D supervision (e.g., meshes), it adds significant complexity and computational overhead to the training process. Rendering- based supervision can be slow and costly, especially when scaled to large datasets.
supervision, such as Neural Radiance Fields (NeRF)[27] DMT.{333..
2.2 Large Scale 3D Latent Diffusion Model
Motivated by recent advances in Latent Diffusion Models (LDMs) [32] in 2D image generation, several methods have extended LDMs to 3D shape generation. These approaches broadly fall into two categories: vecset-based methods and voxel-based methods. Implicit vecset-based methods, such as 3DShape2 Vecset [46], Michelangelo [50], CLAY [48], and CraftsMan3D [16], represent 3D shapes using latent vecset and reconstruct meshes through neural SDFs or occupancy fields. However, implicit methods are typically constrained by vecset size: larger vecset lead to more complex mappings to the 3D shape and require longer training times. In contrast, voxel-based methods, such as XCube [31] and Trellis [39], employs voxel grids as latent representations, providing more interpretability and easier training. Nevertheless, voxel-based methods face limitations in latent resolution due to cubic growth in GPU memory requirements and high computational costs associated with attention mechanisms. To address this issue, our work specifically targets reducing the computational overhead of attention mechanisms, thereby enabling the generation of high-resolution voxel-based latent representations that were previously infeasible.
2.3 Efficient Large Tokens Generation
Generating large tokens efficiently is challenging problem, especially for high-resolution data. Native Sparse Attention (NSA) [45] addresses this by introducing adaptive token compression that reduce the number of tokens involved in attention computation, while maintaining performance comparable to full attention. NSA has been successfully applied to large language models [45, 30] and video generation [34], showing significant reductions in attention cost. In this paper, we extend token compression to 3D data and propose a new Spatial Sparse Attention (SSA) mechanism. SSA adapts the core ideas of NSA but modifies the block partitioning strategy to respect 3D spatial coherence. We also redesign the compression, selection, and window modules to better fit the properties of sparse 3D token sets. Another line of work, such as linear attention [12], reduces attention complexity by approximating attention weights with linear functions. Variants of this technique have been applied in image [40, 52] and video generation [25] to improve efficiency. However, the absence of non-linear similarity can lead to significant decline in the performance of the model.
3

---

SS-VAE
Reconstruction Loss
SS-VAE Encoder
SS-VAE Decoder
3D Assets SDF
Multi-Ress Rett-Resllution Sparse SDF
Multi-Resolution Latents
Multi-Resolutinn Sparse SDF
SS-DIT
Marching Cube
Image Condition
DINOV2
Decoding
Mesh Extraction
Limear
Spatial
Linear
Cross Attention
MLP
Block
SS- SS-DIT
Attention
Sparse
Marching
Norm
Norn
Noised Latents
Mesh
Figure 2: The framework of our Direct3ID-S2. We propose fully end-to-end sparse SDF VAE (SS- VAF), which employs a symmetric encoder-decoder network to efficiently encode high-resolution sparse SDF volumes into sparse latent representations z. Then we train an image-conditioned diffusion transformer (SS-DiT) based on z, and design a novel Spatial Sparse Attention (SSA) mechanism that significantly improves the training and inference efficiency of the DiT.
Sparse SDF VAE
While variational autoencoders (VAEs) have become the cornerstone of image generation by compressing pixel representations into compact latent spaces for efficient diffusion training, their extension to 3D geometry faces fundamental challenges. Unlike images with standardized pixel grids, 3D representations lack unified structure, while meshes, point clouds, and implicit fields each require specialized processing. This fragmentation forces existing 3D VAEs into asymmetric architectures with compromised efficiency. For instance, prominent approaches [6, 48, 49] based vecset [46] encode the input point cloud into a vector set latent space before decoding it into SDF field, while Trellis [39] and XCube [31] rely on differentiable rendering post-hoc neural kernel surface reconstruction [11] ] to bridge their latent spaces to usable meshes. These hybrid pipelines introduce computational bottlenecks and geometric approximations that limit their scalability to high-resolution 3D generation. In this paper, we propose a fully end-to-end sparse SDF VAE that employs symmetric encoding-decoding network encode high-resolution sparse SDF volumes into a sparse latent representation, improving training efficiency while maintaining geometric precision.
Given mesh represented as signed distance field (SDF) volume V with resolution Re (e.g.. 1)243), the SS-VAE first encodes into latent representation z = E(V), then reconstructs the SDF through the decoder V = D(z). Direct processing of dense R² SDF volumes proves computationally prohibitive. To address this, strategically focus valid sparse voxels where absolute SDF values fall below threshold T:
= ((x, s (xx)))](((x)) < <
where s(x,) denotes the SDF value position x,
3.1 Symmetric Network Architecture
Our fully end-to-end SDF VAE framework adopts a symmetric encoder-decoder network architecture, as illustrated in the upper half of Fig 2. Specifically, the encoder employs hybrid framework combining sparse 3D convolution networks and transformer networks. We first extract local geometric features through a series of residual sparse 3D CNN blocks interleaved with 3D mean pooling operations, progressively downsampling the spatial resolution. We then process the sparse voxels as variable-length tokens and utilize transformer layers to capture global contextual information between the valid voxels. Inspired by Trellis [39]. the feature each valid voxel is augmented with positional encoding based its 3D coordinates before being fed into multi-heas self-attention layers. This

---

Sparse Compression
Compress
Pooling
k
Compressed Attantion
cmpp
Query
Linear
Spatial
Blockwise
Selection
Top-n
Selection
Selected Attention
Gated Output
t slc
Key Value
Linear
Window
Windaw
h
tuin
Window Attention
t win
Input Tokens
Blocks
Inactive Blocks
Attention Score
Figure 3: The framework our Spatial Sparse Attention (SSA). We partition the input tokens into blocks based on their 3D coordinates, and then construct key-value pairs through three distinct mod- ules. For each query token, we utilize sparse 3D compression module to capture global information, while the spatial blockwise selection module selects important blocks based on compression attention scores to extract fine fin-graiied features, and the sparse 3D window module injects local features. Ultimately, we aggregate the final output of SSA from the three modules using predicted gate scores.
hybrid design outputs sparse latent representations reduced resolution ( )s where f denotes the downsampling factor. The decoder of our VAE adopts symmetric structure with respect to the encoder, leveraging transformer layers and sparse 3D CNN blocks to progressively upsample the latent representation and reconstruct the SDF volume V.
3.2 Training Losses
The decoded sparse voxels V contain both the input voxels Vin and additional valid voxels Venta We enforce supervision on the SDF values across all these spatial positions. To enhance geometric fidelity, we impose additional supervision the active voxels situated near the sharp edges of the mesh, specifically in regions exhibiting high-curvature variations on the mesh surface. Moreover, the term KL-divergence regularization imposed on the latent representation z to constrain excessive variations in the latent space. The overall training objective of our SS- VAE is formulated as:
L.
Σ
||s(x) - -(xx c C {in, ext, sharp).
(2)
[Val
(x, ((x)))VVVV
Listal
Ede
No Le + AKI. LKI.
where Ain Aext Asturp and ÀgL denote the weight of each term.
3.3 Multi-resolution Training
To enhance training efficiency and enable our SS- VAE to encode meshes across varying resolutions, utilize the multi-resolution training paradigm. Specifically, during each training iteration, randomly sample target resolution from the candidate set {256' 3843, 5123, 10243), J. then trilinearly interpolate the input SDF volume the selected resolution before feeding into the SS-VAE.
4 Spatial Sparse Attention and DiT
Through our SS-V VSE framework, 3D shapes can be encoded into latent representations z. adopt multi-stage generation pipeline, where the sparse structure of is generated first, followed by the
5

---

generation of the sparse latent representations z. For the first stage. we employ the same Structure Flow Model Trellis [39]. the stage of generating the sparse latent representations z, to ensure efficient generation of high-resolution meshes, we propose spatial sparse attention that substantially accelerates both training and inference processes. Furthermore, we introduce a sparse conditioning mechanism to extract the foreground region of the input images, thereby reducing the number of conditioning tokens. The architecture of the DiT is illustrated the lower half of Fig 2.
4.1 Spatial Sparse Attention
Given input tokens k, v E RNxd where N denotes the token length, and d represents the head dimension, the standard full attention is formulated as:
N Σ
Pr.V 1j=1 Pr.
= Atm k. v)
Paj
vi
EE NO.
(4)
As the resolution SS-VAE escalates, the length of input tokens grows substantially, reaching over 100k at a resolution of 10243, leading to prohibitively low computational efficiency in attention operations. Inspired by NSA (Native Sparse Attention) [45], we proposes Spatial Sparse Attention mechanism, which partitions key and value tokens into spatially coherent blocks based on their geometric relationships and performs blockwise token selection achieve significant acceleration.
A naive implementation involves treating latent tokens z as a ID sequence and partitioning it into fixed-length blocks based token indices, analogous to NSA. However, this approach suffers from two critical limitations: On the one hand, tokens within the same block may not be spatially adjacent in 3D space, despite sharing contiguous indices. On the other hand, due to the sparse voxel structure, blocks with identical indices across different samples occupy divergent spatial regions. These issues collectively lead to unstable training convergence. To resolve these challenges, we propose partitioning blocks based on 3D coordinates. As illustrated in Fig 3, we divide the 3D space into subgrids of size m². where active tokens from sparse voxels residing in the same subgrid are grouped into one block. Our Spatial Sparse Attention comprises three core modules: sparse 3D compression, spatial blockwise selection, and sparse 3D window. The attention computation proceeds as follows:
O = wi Attn qr, ke v.
(.mp)) + wi Attn qr, ktl: vgle) w. win AyttnAttteea qr. k.vin, viii)
(5)
where k, and v. represent the selected key and value tokens in each module for query q., respectively. wi the gating score for each module, obtained by applying linear layer followed by sigmoid activation to the input features.
Sparse 3D Compression. After partitioning input tokens into spatially coherent blocks based on their 3D coordinates, we leverage a compression module to extract block-level representations of the first incorporate intra-block positional encoding for each token within
input tokens. Specifically, a block of size mmp then employ sparse 3D convolution followed by sparse 3D mean pooling to compress the entire block:
cmp = 5(k, + )E( kc
where k, cmma denotes the block-level key token, PE(-) is absolute position encoding, and 5(-) represents operations of sparse 3D convolution and sparse 3D mean pooling. The sparse 3D compression module effectively captures block-level global information while significantly reducing the token count, thereby enhancing computational efficiency.
Spatial Blockwise Selection. The block-level representations only contain gaarse-rrained informa- tion, necessitating the retention of token-level features to enhance the fine details in the generated 3D shapes. However, the excessive number of input tokens leads to computationally inefficient operations if all tokens are utilized. By leveraging the sparse 3D compression module, we compute the attention scores Semp between the query q and each compression block, subsequently selecting all tokens within the top-k blocks exhibiting the highest scores. The resolution Male of the selection blocks must be both greater than and divisible by the resolution memp of the compression blocks. The relevance score sile for a selection block is aggregated from its constituent compression blocks. GQA (Grouped-Que Quurryy-uurruuuuu Attention) [4] is employed to further improve computational efficiency, the attention

---

scores of the shared query heads within each group are accumulated as follows:
h2
amp, 2.4
PE cap h=1
where cemp denotes the set of compression blocks within the selection block, and ha represents the number of shared heads within a group. The top-k selection blocks with the highest stle scores are selected, and all tokens contained within them are concatenated form k and vz", which are used to compute the spatial blockwise selection attention. We implement the attention kernel using Triton [36]. with a key challenge arising from the sparse 3D voxel structure where the number of tokens per block varies. To address this, we pre-compute the index C. of the first token in each block as an input to the kernel and sort the input tokens based on their block indices, ensuring that all tokens belonging to the same block are contiguous in memory. We describe the spatial blockwise selection attention forward pass in Algorithm. I of the appendix.
Sparse 3D Window. In addition to sparse 3D compression and spatial blockwise selection modules, we further employ an auxiliary sparse 3D window module to explicitly incorporate localized feature interactions. Drawing inspiration from Trellis [39], we partition the input eokenncontaining voxels into non-overlapping windows of size m For each token, we formulate its contextual computation by dynamically aggregating active tokens within the corresponding window form kjiin and vi followed by localized self-attention calculation exclusively over this constructed token subset.
Through the modules of sparse 3D compression, spatial blockwise selection, and sparse 3D window, corresponding key-value pairs are constructed. Subsequently, attention calculations are performed for each module, and the results aggregated and weighted according to gate scores to produce the final output of the spatial sparse attention mechanism.
4.2 Sparse Conditioning Mechanism
Existing image-to-3D) models [16, 38, 48] typically employ DINO-v2 [28] to extract pixel-level features from conditional images, followed by cross-attention operation with noisy tokens to achieve conditional generation. However, for majority of input images, more than half of the regions consist of background, which not only introduces additional computational overhead but may also adversely affect the alignment between the generated meshes and the conditional images. To mitigate this issue, we propose sparse conditioning mechanism that selectively extracts and processes sparse foreground tokens from input images for cross coss-atteenion computation. Formally, given an input image I, the sparse conditioning tokens e are computed as follows:
C= == Linear(/ [EniNo (T))
(EDiOO (Z))),
(8)
where EDINO is the DINO-v2 encoder, f(-) denotes the operation of extracting the foreground tokens based on the mask, PE(-) is the absolute position encoding, and Linear(-) represents a linear layer. Then perform cross attention using the finalized sparse conditioning tokens e and the noisy tokens.
4.3 Rectified Flow
We employ rectified flow objective [10, 18] to train generative model. Rectified flow defines forward process as linear trajectory between data distribution and standard normal distribution:
x(t) (1 t)xo + le,
where is the noise, and denotes the timestep. Our generative model is trained predict the velocity field from noisy samples to the data distribution. The training loss is formulated using conditional flow matching, formulated as follows:
LCPM = Ex,...c llo(xe, c, - (e - -x0)1
(10)
where vo is the neural networks.
Experiments
5.1 Datasets
Our Direct3D-S2 is trained on publicly available 3D datasets including Objaverse [9], Objaverse- XL [8], and ShapeNet [5]. Due to the prevalence of low-quality meshes in these collections,
7

---

Table 1: Quantitative comparisons of meshes generated by different methods in the image-to-3D task.
Methods
ULIP-2 1
Uni3D f
OpenShape f
Trellis [39]
0.2825
0.3755
0.1732
Hunyuan3D 2.0 [49]
0.2535
0.3738
0, 0,1699
TripoSG [17]
0,2626
0.3870
0.1728
Hi3DGen [44]
0.2725
0.3723
0.1689
Ours
0.3111
0.3931
0.1752
Input
Trellis
Hunyuan3D 2.0
TripoSG
Hi3DGen
Ours
Figure 4: Qualitative comparisons between other image-to-3l methods and our approach.
curated approximately 452k assets through rigorous filtering for training. Following prior approach [48] in geometry processing, we first convert the original non-watertight meshes into watertight ones, then compute ground-truth SDF volumes that serve as both input to and supervision for our SS-Ve VAE, For training our image-conditioned DIT, we render 45 RGB images per mesh at 1024 x 1024 resolution with random camera parameters. The camera configuration space is defined follows: clevation angles ranging from 10º to 40", azimuth angles spanning (0", 180%), and focal

---

Forward Time
Backward Time
Flash Attesioo
Flash Atcation
Speriall Spanse Altontinn
Spriial Speree Attention
800
0.7X
1.4X
2.3X
2.5%
3.9X
1.4X
2.5%
3.8X
6.3X
9.6X
Tints (ms)
(ms)
1500
1200
800
200
400
88
16k
32k
64k
121K
168
32k
64K
128K
Number
Figure 5: Comparison of the forward and backward time of our SSA and FlashAttention-2.
lengths varying between 30mm and 100mm. To compare the geometric fidelity of meshes generated by Direct3D-S2 and other methods, we established challenging benchmark comprising highly detailed images sourced from professional communities including Neural4D [3], Meshy [2], and CivitAI [1]. The quantitative assessment employs ULIP-2 [43], Uni3D [51] and OpenShape [20] metries to measure shape-image aligmment between generated meshes and conditional input images, enabling systematic comparison with state-of-the-art 3D generation methods. For the ablation studies, we conduct qualitative comparisons on this benchmark, and perform quantitative evaluations on subset from the Objaverse dataset that does not overlap with the training set. We employ the FID metric to more rigorously evaluate the geometry quality of generated meshes, which requires GT meshes. We render paired normal maps for both the generated meshes and GT meshes from the same viewpoints, then calculate the Normal-FID between them to assess model performance. The implementation details of our Direct3D-S2 provided in the appendix.
5.2 Quantitative and Qualitative Comparisons
To empirically validate the effectiveness of our Direct3D-S2 framework, we conducted comprehensive experiments against state-of-the-ane image-to-3I) approaches. The quantitative results reported in Tab 1, where it is evident that our Direct3D-S2 outperforms the other approaches across three metrics, indicating that the meshes produced by Direct3D-S2 achieve better alignment with the input images. Moreover, we present qualitative comparisons in Fig 4. Although the other methods generate overall satisfactory results, they struggle to capture finer structures due resolution limitations, as illustrated by the railings of the house and surrounding branches of trees in the first row. In contrast, thanks to our proposed SSA mechanism, our Direct3D-S2 is capable of generating high- nigh-resolution meshes, achieving superior quality even for these intricate details. We provide additional qualitative comparisons in the appendix.
5.3 Ablation Studies
Runtime of Different Attention Mechanisms. We implemented custom Triton [36] GPU kernel for SSA. And we compare the forward and backward execution times of our SSA with those of FlashAttention-22 [7] across various token counts, using the implementation from Xformers [14] for FlashAttention-2. The comparison results are shown in Fig 5, which indicate that our SSA achieves comparable speeds to FlashAttention-2 when the token count is low; however, as the token count increases, the speed advantage of our SSA becomes more pronounced. Specifically, when the token count reaches 128k, the forward and backward speeds of our SSA are 3.9x and 9.6x faster than those Flash Attention 2, respectively, demonstrating the efficiency of our SSA.
Table Quantitative comparisons of meshes generated using different attention mechanisms. For Full Attention, we apply a latent packing strategy to downsample the tokens.
Methods
Normal-FID 1
Full Attention NSA
76.58
71.93
SSA (Ours)
69.31

---

Input
Full Attention
NSA
SSA(OurS)
Figure 6: Ablation studies of our proposed SSA mechanism.
Effectiveness of SSA. We conduct ablation studies to validate the robustness of SSA. Given the insufficient geometric fidelity at /56 3843 resolutions, which do not adequately reflect the model's precision, and prohibitive computational costs at 10243 resolution, we perform experiments at 5123 resolution. We establish three comparative configurations: 1) Full attention: directly training the DiT with full attention proves to be inefficient. Therefore, following Trellis' latent packing strategy [39], we group latent tokens within 2 local regions to reduce the token count before feeding them into the DiT blocks. 2) NSA: process latent tokens as ID sequences with fixed-length block partitioning, disregarding spatial coherence. 3) Our proposed SSA. We report the quantitative results in Tab 2, where SSA outperforms the other attention mechanisms. The qualitative results are illustrated in Fig 6. It is evident that the Full Attention variant produces meshes with high-frequency surface artifacts, attributed to its forced packing operation that disrupts local geometric continuity. The NSA implementation exhibits training instability due positional ambiguity in block partitioning, resulting in less smooth meshes. In contrast, our SSA not only preserves the details of the meshes, but also yields smoother and more organized surface, thereby demonstrating the effectiveness of our proposed SSA mechanism.
Additional ablation studies and the evaluation of our SS-VAE are provided in the appendix.
6 Conclusion
In this work, we presented a novel framework for high-resolution 3D shape generation, dubbed Direct3D-S2. The key contribution of our approach is the design of Spatial Sparse Attention (SSA) mechanism, which significantly accelerates the training and inference speed of DiT. The integration of fully end-to-end symmetric sparse SDF VAE further enhances training stability and efficiency. Extensive experiments demonstrate that our Direct3D-S2 outperforms existing state-of-the-a image- to-3D methods in generation quality, while requiring only GPUs for training.
Acknowledgment. This work is supported by National Natural Science Foundation of China (62472213), Gusu Innovation & Entrepreneurship Leading Talents Program (ZXL2024361), and computing resources from DreamTech
10

---

References
[1] Civitai. https://civttai.com
[2] Meshy. https:/wwww.meshh.at
[3] Neural4d. https:/www.neuraald.oo
[4] Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305. 13245, 2023.
[5] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository.
arXiv preprint arXiv:1112.03012. 2015.
[6] Rui Chen, Jianfeng Zhang, Yixun Liang, Guan Luo, Weiyu Li, Jiarui Liu, Xiu Li, Xiaoxiao Long, Jiashi Feng, and Ping Tan. Dora: Sampling and benchmarking for 3d shape variational auto-encoders. arXiv preprint or Xiv:2212. 17008, 2024.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint
(tt::2307.08691 2023.
[8] Mau Deitke, Ruoshi Liu, Matthew Wallingford. Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan. Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects.
In NeurlPS, 2023.
[9] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsami, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
[10] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICMI., 2024.
[11] Jiahui Huang, Zan Gojeic, Matan Atzmon, Or Litany, Sanja Fidler, and Francis Williams. Neural kernel surface reconstruction. CVPR, 2023.
[12] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In ICML, 2020.
[13] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[14] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: modular and hackable transformer modelling library. haagg//bbboooo/ccccoooeeeeeeecee
2022.
[15] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In ICLR, 2024.
[16] Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, and Xiaoxiao I Long. Craftsman: High-fidelity mesh generation with 3d native generation and interactive geometry refiner. arXiv preprint arXiv:2405. 149999 2024.
[17] Yangguang L., Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan I Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang. Wanli Ouyang, et al. Triposg: High-fideliyy 3d shape synthesis using large-scale rectified flow models. arXiv preprini arXiv:2502. 066688 2025.
[18] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint wwXiv:2210.0274: 2022.
[19] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In CVPR, 2024.
[20] Minghua Liu, Ruoxi Shi, Kaiming Kuang. Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. In NewrIPS, 2023.
[21] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 scconds without per-shape optimization. In NeurlPS, 2024.
[22] Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9970-9980, 2024.
[23] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pages 347-353. 1998.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:171.05100,.22 2017.
11

---

[25] Kaiyue Lu, Zexiang Liu, Jianyuan Wang. Weixuan Sun, Zhen Qin, Dong Li, Xuyang Shen, Hui Deng. Xiaodong Han, Yuchao Dai, et al. Linear video transformer with feature fixation. arXiv preprint arXiv:2210.08164 2022.
[26] Yuanxun Lu, Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Xun Cao, and Yao Yao. Direct2. 5: Diverse text-to-3d generation via multi-view 2.5 d diffusion. In CVPR, 2024.
[27] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[28] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.00 30.00719,, 2023.
[29] William Pechles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4 4195-4205,
2023.
[30] Piotr Piçkos. Róbert Csordás, and Jürgen Schmidhuber. Mixture of sparse attention: Content-based
learnable sparse attention via expert-choice routing. arXiv preprint arXiv:2505.0033 2025,,
2025.
[31] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Samja Fidler, and Francis Williams. Xcube:
Large-scale 3d generative modeling using sparse voxel hierarchies. In CVPR, 2024.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution
image synthesis with latent diffusion models. In CVPR, 2022.
[33] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NewrIPS, 2021.
[34] Xin Tan, Yuctao Chen, Yimin Jiang, Xing Chen, Kun Yan, Nan Duan, Yibo Zhu, Daxin Jiang, and Hong Xu. Dsv: Exploiting dynamic sparsity to accelerate large-scale video dit training. arXiv preprint
mmXiv:2502.07590 2025.
[35] Jiaxiang Tang. Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large
multi-view gaussian model for high-resolution 3d content creation. In ECCV, 2024.
[36] Philippe Tillet, Hsiang- -Tuung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019.
[37] Peng Wang and Yichun Shi. Imagedream: image-prompp multi-view diffussin for generation. arXiv preprint or Xiv:2312. 02201,2111222 2023.
[38] Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Torr, Xun Cao, and Yao Yao. Direct3d: Scalable image-to-3d to-3d generation via 3d latent diffusion transformer. In NewrlPS, 2024.
[39] Jianfeng Xiang. Zelong Lv. Sicheng Xu, Yu Deng, Ruicheng Wang. Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.0ff
2024.
[40] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li. Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion 2024.
transformers. arXiv preprint arXiv:2410.1066
[41] Jiale Xu, Weihan Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024.
[42] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng. Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. In ECCV, 2024.
[43] Xue, Ning Yu. Shu Zhang, Artemis Panagopoulou, Junnan Li. Roberto Martin-Martin, Jiajun Wu. Caiming Xiong. Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scalable multimodal pre- pre-training for 3d understanding. In CVPR, 2024.
[44] Chongjie Ye, Yushuang Wu, Ziteng Lu, Jiahao Chang, Xiaoyang Guo, Jiaqing Zhou, Hao Zhao, and Xiaoguang Han. Hiddgen: High-fidelity 3d geometry generation from images via normal bridging. arXiv preprint 2r0iv:2503. 22236, 2025.
[45] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, lean Wang, Zhiping Xiao, ct al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502. / 1089, 2025.
[46] Biao Zhang. Jiapeng Tang. Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions On Graphics (TOG), 42(4):1-16.
2023.
[47] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In ECCV, 2024.
[48] Longwen Zhang. Ziyu Wang. Qixuan Zhang, Qiwei Qiu, Anqi Pang. Haoran Jiang, Wei Yang. Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-qualit 3d assets. ACM Transactions Graphics (TOG), 43(4): 20, 2024.
12

---

[49]
Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang. Sheng Zhang, Xianghui Yang, al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured assets generation. arXiv preprint arXiv:2501. 1200, 2025.
[50] Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng. Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image-tex aligned latent representation. In NeurlPS, 2023.
[51] Junsheng Zhou, Jinsheng Wang, Baorui Ma. Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation scale. arXiv preprint arXiv:2310.066 06773, 2023.
[52] Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xinggang Wang. Dig: Scalable and efficient diffuusin models with gated linear attention. arXiv preprint urXiv:2405.188: 18428, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explicitly delineate the contributions and scope of our approach in both the abstract and the introduction in Section. 1 of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discusses the limitations of the work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed descriptions of the experimental setup, including the algorithms used, parameter settings, datasets, and evaluation metrics. Additionally, it outlines the procedures followed to obtain the results, ensuring that other researchers can replicate the experiments and verify the findings. We mention this in Section. 5 and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
15

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
We will release the code after the completion of the review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
conoooccy)))) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/ wtc/public///e htalliic/giies///eee SCae//Coommmiiiccc iindabbissse ieiiionooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and testing details in Section. and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Repeated training runs of large transformer models strain available GPU resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide comprehensive information on the type of computing workers, memory, and time of execution required for each experiment. We mention this in Section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https htt///nerripppp wttppppp...ppppppeeeeeeeee cc/public/ Ceallic//Ehiieeee csQuidel lnes?
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics, and we confirm that our research adheres to all outlined ethical guidelines, including considcrations for fairness, transparency, and respect for all participants involved.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models that pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of all assets used in the paper are properly credited. The licenses and terms of use for these assets are explicitly mentioned and have been respected in accordance with their respective requirements.
Justification:
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper includes comprehensive documentation for all new assets introduced. This documentation is provided alongside the assets, ensuring that users have access to detailed descriptions, usage guidelines, and any necessary supporting information.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The paper does not include the full text of instructions given to participants, screenshots, or details about compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: We request several volunteers to participate simple anonymous question- naire for user study, with no potential risks involved for the volunteers.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... wcc//onfeeeecce//22//11111 for what should or should not be described.
20

---

Appendix
A Algorithm of Spatial Sparse Attention
In our Triton-basee implementation of the spatial blockwise selection attention kernel, two key challenges arise within sparse 3D voxel structures: 1) the number of tokens varies across different blocks, and 2) tokens within the same block may not be contiguous in HBM. To address these, we first sort the input tokens based on their block indices, then compute the starting index C' of each block as kernel input. In the inner loop, C dynamically governs the loading of corresponding block tokens. The complete procedure of forward pass formalized Algorithm 1.
Algorithm 1 Spatial Blockwise Selection Attention Forward Pass
Require: q E RR.Nx(ha.xh.)x00 k E R
Nx number of key/value heads hke number of the shared heads hg, number of the selected blocks T, indices of the selected blocks the number of divided key/value blocks No. index of the first token in each block C E R No+11 block size Bs. Divide the output o E R x xh,)xd into (N, hkv) blocks, each of size h2 d. Divide the logsumexp
E NNxx x
X into (N, kk,)) blocks, each of size htg
2: Sort all tokens within q, k and v according to their respective block indices.
3: for 1 to do
4: for f = 1 to lake do (0),,xd Rh,xd, logsumexp 4, (0)n, R and m
5: Initialize Or,
(-inf)), ER. Load qt.h E RR It E RRT from HBM on-chip SRAM.
6:
7:
for = to do
Load the starting token index b, = cap.) and ending token index b. c of the I (5) block from HBM to on-chip SRAM.
8:
1.6
9:
for = b, to be by B& do
Load k,, V; E IR B, from HBM on-chip SRAM.
11:
Compute Sthh qr,kk E A.
12:
Compute (4)
i. = max (m rowmax (s) E R
max(m
Compute Pi, e m Rb,
(6)
14:
Compute Og, = c me - O + Pi
Compute 4. = m log end for
(i)
log(e".. h. (aa-mi rowsum(p my.h = m,,
(4)
17:
end for
Compute Ot, Me.A -1. Or.h-
Write o,, and 4 to HBM as the (t, h) block of and respectively. 20: end for
19:
21:
end for
22: Return the output o and the logsumexp l.
B Implementation Details
Table 3: The training configuration of our SS-DiT.
Resolution
Token Count
Learning Rate
Batch Size
Training Time
2563
~2058
le-4
8 x 8
days
3843
x5510
le-4
8 x 8
days
5123
R 106555
5e-5
8 x 8
days
3
#=45904
2c-5
2 x 8
day
21

---

VAE. We utilize active voxels from volumes with SDF values less than 7 1
128 imputs to the SS- VAE. The downsampling factor for the encoder is set 8, and the channel dimension of the latent representation z is configured to 16. The weights for the various losses are set as: Ain 1.0. - We employ the Adam [24] optimizer with an initial learning rate of le 4. To enhance training efficiency, we first conduct multi-resolution training using SDF volumes at three resolutions of {256', 384", 5122) over a period of one day on A 100 GPUs, with batch size of per GPU. Subsequently, we fine-tune the SS- SS- for one additional day at 1024 resolution with a learning rate of lE 5 with a batch size of per GPU.
Aex 1e Ashurp 1.0, and λKI.
DiT. Our SS-DiT comprises 24 layers of DiT blocks with hidden dimension of 1024. We employ crouped-Quers Attention (GQA) [4] with group number set to 2, where each group contains 16 attention heads. The hidden dimension of each head is configured as 32. For the spatial sparse attention (SSA) mechanism, we configure the resolution of the compression blocks to Memp
the resolution of the selection blocks to Male 8, and the size of the sparse 3D windows Miin 8. We utilize DINO-v2 Large [28] to extract features from conditional images, with input images having a resolution of 518 x 518. For the DiT, increases the resolution from 256 to 1024 to accelerate convergence. Tab presents the average latent token count, learning rate, batch size, and training duration settings at different resolutions. We employ the Adam W optimizer and trained the model for a total of 7 days on 8 A100 GPUs. For the 1 resolution,
implement progressive training strategy that gradually
further filtered 68k high-fidelity 3D assets for training. Additionally, similar to Trellis [39], we trained an extra DiT to predict the indices of the sparse latent tokens z, which was trained for days 8 A100 GPUs
C User Study for Image-to-3D Generation
Trellis
Huanyuan3D 2.0
TripoSG
Hi3DGen
Ours
4.5
4.21
4.17
3.52
3.5
3.25
3.33
3.04
3.07
2.93
3
2.81
2.78
2.5
2
Image Consistency
Overall Quality
Figure 7: User study for image-to- to-3 generation.
We conducted user study with 40 participants evaluating 75 unfiltered meshes generated by Direct3D-S2 and other image-to-3D methods. Participants scored each output using two criteria: image consistency and overall geometric quality, with scores ranging from (poorest) 5 (excellent). As shown in Fig 7, Our Direct3D-S2 demonstrates statistically superiority over other approaches across both evaluation metrics.
D Comparison of VAE
To validate the reconstruction quality of our SSEVAE, we curated challenging validation from the Objaverse [9] dataset, comprising meshes with complex geometric structures. Qualitative and quantitative comparisons with competing methods are shown in Fig 8 and Tab It can be observed our SS- VAE achieves superior reconstruction accuracy at 512 resolution. and demonstrates markedly improved performance on complex geometries at 10248 resolution. Notably, thanks to our fully end-to-end SDF reconstruction framework, SS VAE requires only 2 days of training on 8 A 100 GPUs, significantly fewer than competing methods that typically demand at least 32 GPUs for equivalent training durations.
22

---

Trellis
XCube
Dora
Ours (512*)
Ours (1024
GT
Figure 8: Qualitative comparisons of VAE reconstruction results. Note that we used latent token
length of 4096 during the inference of Dora [6].
Table 4: Quantitative comparisons of meshes reconstructed by different VAE.
Methods
Chamfer Distance +
Fl (0.005) t
F1 (0.001) 1
Trellis [39]
28.60
88.77
11.57
Xcube [31]
15.42
99.79
31.85
Dora [6]
15.12
99.98
31.86
Ours (512)
13.59
99.98
41.11
Ours (1024)
7.91
100.0
70.19
E
Additional Ablation Studies
Table 5: Quantitative comparisons of our method at various resolution.
Resolution
Normal-FID t
256
132.97
384
87.10
512
69,31
1024
46.44
Results of Image-to-3D Generation Different Resolution. We present the generation results of our Direct3D-S2 across four resolutions (2563, 3843, 5123, 10243) in Fig and in Tab 5. The results demonstrate that increasing resolution progressively improves mesh quality. At lower resolutions 256 and 3843, the generated meshes exhibit limited geometric details and misalignment with input images. At 5123 resolution, the meshes display significantly enhanced high-frequency geometric details. Further increasing the resolution to 1024 yields meshes with sharper edges and improved alignment with input image details.
Effect of Each Module in SSA. We validated the effect of the three modules in SSA at resolution 5123, with the results presented Fig 10 and Tab When using only the sparse 3D window module (win), the generated meshes exhibited detailed structures but suffered from surface irregularities due to the lack of global context modeling. Introducing the sparse compression module (win+cmp) showed minimal performance changes, which is reasonable as this module primarily serves to obtain
23

---

Input
256
384
512
1024
Figure 9: The visualization results of our Direct3D-S2 for image-to-3D generation across four resolutions: (256S, 384 512 1.
Input
win
win+emp
emp+sle
win+tmp+sse
Figure 10: Ablation studies for the three modules of SSA, where win, cmp, slc denote the sparse 3D window, sparse 3D compression, and spatial blockwise selection modules, respectively.
Table 6: Ablation results for different modules of SSA at 512 resolution.
Modules
Normal-FID 1
win
74.10
win+emp cmp+slc
77.68
70.05
win+emp+sle
69.31
the attention scores for the blocks. After incorporating the spatial blockwise selection module (win+cmp+sle), the model can focus on the most important regions globally, resulting notable improvement in mesh quality. We also observed that not utilizing the window (emp+slc) did not

---

Input
w/o sparse conditioning
w/ sparse conditioning
Figure 11: Ablation studies for sparse conditioning mechanism.
result significant drop in model performance, but slowed convergence, demonstrating that local feature interaction contributes to more stable training and enhances convergence speed.
Table 7: Ablation results for sparse conditioning at 512 resolution.
Methods
Normal-FID
Ours w/o sparse conditioning
71.49
Ours w/ sparse conditioning
69.31
Effect of Sparse Conditioning Mechanism. We perform ablation experiments validate the effect of the sparse conditioning mechanism at 5123 resolution. As demonstrated in Fig ann Tab 7, the exclusion of nor-foregr conditioning tokens through sparse conditioning enables the generated meshes to achieve significantly better alignment with the input images.
F Additional Qualitative Comparisons for Image-to-3D Generation
We provide more qualitative comparisons with both open-source and closed-source models in Fig 12 and Fig 13.
G Limitations
Our proposed spatial sparse attention mechanism achieves significant speed improvements over FlashAttention-2. However, the forward pass exhibits notably smaller acceleration ratio compared to the backward pass. This discrepancy primarily stems from the computational overhead introduced by top-k sorting operations during the forward pass. We acknowledge this limitation and will prioritize optimizing these operations in future work.
25

---

Image
Trellis
Hunyuan-2.0
TripoSG
Hi3DGen
Ours
Figure 12: Qualitative comparisons between other open-source image-to-3D methods and our approach.
26

---

Image
Model N
Model M
Model R
Model
Ours
Figure 13: Qualitative comparisons between other open-source, closed-source image-to-3D models and our approach. Note that all the closed-source model the default setting of their web app. Best viewed with zoom-in.
27


Reviews:
[{"review_id": "cVYd9QvoLt", "reviewer": "Reviewer_WHua", "summary": "This paper addresses the challenge of generating high-resolution 3D shapes in an efficient and scalable manner. Prior methods based on explicit voxel representations suffer from significant computational overhead, particularly in the attention modules of Diffusion Transformers (DIT), as the voxel grid resolution increases.\n\nTo overcome this limitation, the authors propose Spatial Sparse Attention (SSA), a novel mechanism that significantly reduces computational cost while maintaining high-quality 3D generation. SSA combines three key components: sparse 3D compression, spatial blockwise token selection, and sparse 3D window attention.\n\nFurthermore, the paper introduces Sparse SDF VAE, a unified variational autoencoder architecture where both the input and output are represented as sparse Signed Distance Fields (SDFs).\n\nExperimental results demonstrate that the proposed framework not only achieves superior computational efficiency but also improves the visual quality of image-conditioned 3D generation.", "questions": "1. Clarification on performance attribution: It would be helpful to clarify whether the performance gains primarily come from the proposed network architecture or from operating at higher resolutions. For example, Trellis operates at a resolution of 256³, and the results of the proposed method at the same resolution (as shown in Appendix Figure 3) appear relatively less compelling. A direct comparison at equal resolutions, both qualitatively and quantitatively, would help to better isolate the contribution of the proposed design.\n\n\n2. SSA component ablation: In the ablation studies of the SSA module, it would be interesting to see results when only cmp or only slc (if available) is used. This could provide further insight into the individual contribution of each component within SSA.\n\n\n3. Potential for text-to-3D extension: Given the structured and scalable nature of the proposed method, I am curious whether it could be extended to support text-to-3D asset generation. If applicable, any preliminary results or discussion on this direction would be appreciated.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed Spatial Sparse Attention (SSA) mechanism presents a novel approach to scaling attention in 3D space. By effectively adapting Local Sparse Attention (LSA) to volumetric data, the method addresses the core inefficiency in high-resolution 3D generation with a well-motivated and technically solid solution.\n\n2. The architecture successfully handles gigascale 3D generation, and its design remains scalable to even larger scenes due to its efficient sparse formulation.\n\n3. The qualitative results are visually compelling, demonstrating detailed and structurally coherent 3D outputs that surpass prior methods in terms of geometric complexity and realism.\n\n4. The paper is clearly written, and the methodological components are effectively illustrated with informative and well-organized figures, which aid in understanding the proposed architecture and its contributions.\n\n\nWeaknesses\n1. All ablation studies are qualitative. A quantitative analysis of the proposed components would better clarify the contribution and effectiveness of each module.\n\n2. The Sparse SDF VAE is only evaluated qualitatively. Quantitative comparisons are necessary to demonstrate its advantages in training efficiency and geometric fidelity.\n\n3. Minor: Redundant mention of ULIP-2 [42], Uni3D [50], and OpenShape [20] in Lines 256 and 264. Please revise to avoid repetition.", "was_revised": true, "final_justification": "I would like to thank the authors for their comprehensive ablation studies and insightful comparisons.\nThe method’s impressive visual quality, coupled with its computational efficiency, makes it a valuable contribution to the community.\nI firmly believe that this work meets the standards of NeurIPS and deserves acceptance.", "rebuttal": "Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:** Quantitative results of ablation studies.\n\n**[A1]:** Thanks for your valuable suggestions. We perform quantitative comparisons on a subset from Objaverse dataset, which comprises 500 meshes disjoint from the training set. Unlike the approach in our paper which uses metrics such as Uni3D and ULIP-2 to measure semantic similarity between generated meshes and input images, we employ the FID metric to more rigorously evaluate the geometry quality of generated meshes, which requires GT meshes. We render paired normal maps for both the generated meshes and GT meshes from the same viewpoints, then calculate the Normal-FID between them to assess model performance. The results are as follows: \n\n(1) Generation results of our Direct3D-S2 at various resolutions:\n\n| Resolution  |Normal-FID ↓|\n------------ | :----: | \n| 256       | 132.97 |\n| 384 | 87.10 | \n| 512   | 69.31 |\n| 1024  | **46.44** |\n\n(2) Ablation results for different modules of SSA at 512³ resolution:\n| Modules  | Normal-FID ↓|\n------------ | :----: | \n| cmp       | 304.48 |\n| slc       | NaN   |\n| win       | 74.10 |\n| win+cmp | 77.68 | \n| cmp+slc   | 70.05 |\n| win+cmp+slc  | **69.31** |\n\n(3) Comparative results of SSA against Full Attention and NSA at 512³ resolution:\n\n| Methods  | Normal-FID ↓|\n------------ | :----: | \n| Full Attention | 76.58 |\n| NSA | 71.93 | \n| SSA (Ours)   | **69.31** |\n\n(4) Ablation results for sparse conditioning at 512³ resolution:\n\n| Methods  | Normal-FID ↓|\n------------ | :----: | \n| Ours w/o sparse conditioning | 71.49 |\n| Ours w/ sparse conditioning  | **69.31** |\n\n\n**[Q2]:** Results when only cmp or only slc is used in SSA.\n\n**[A2]:** Thanks for your suggestions. We have added supplementary experiments using only cmp or slc, with results presented in **[Q1]\\(2)**. When using only the cmp module, which provides exclusively coarse-level features, the model's performance significantly degrades. When using only the slc module, the model consistently collapses (with loss diverging to NaN) after several hundred training iterations due to the absence of the cmp module which learns block-level attention scores. Consequently, the slc module must operate in conjunction with the cmp module.\n\n**[Q3]:** Quantitative comparisons of Sparse SDF VAE.\n\n**[A3]:** Thanks for your suggestions. We present the quantitative comparisons of VAE below. The  evaluation metrics includes Chamfer Distance and F1 score with thresholds of 0.005 and 0.001. The metrics are multiplied by 10,000, 100 and 100, respectively. Regarding training efficiency, Dora requires 32 A100 GPUs for two days of training, while Trellis necessitates 64 A100 GPUs. Other compared methods do not report training time. In contrast, our Sparse SDF VAE requires only 8 A100 GPUs for one day to train at 512³ resolution, and merely an additional day to progress to 1024³ resolution.\n\n| Methods  | Chamfer Distance↓| F1 (0.005)↑ | F1 (0.001)↑ | \n------------ | :----: |  :----: |  :----: | \n| Trellis | 28.60 | 88.77 | 11.57 |\n| Xcube | 15.42 | 99.79 | 31.85 |\n| Dora   | 15.12 | 99.98 | 31.86 |\n| Ours (512) | 13.59 | 99.98 | 41.11 |\n| Ours (1024) | **7.91** | **100.00** | **70.19** |\n\n**[Q4]:** Clarification on performance attribution and results at 256³ resolution.\n\n**[A4]:** Thanks for your comments. We acknowledge that at 256³ resolution, our method performs worse than Trellis, an expected outcome for two reasons:\n\n(1) Trellis' VAE employs a 53-dimensional FlexiCubes parameters at 256³, whereas our Sparse SDF VAE uses only 1D SDF representations at the same resolution.\n\n(2) Trellis' VAE is supervised by high-resolution normal maps rendered from GT meshes, while we use 256³ SDF grids as supervision signals, which provide significantly inferior geometric fidelity compared to mesh-derived normals.\n\nHowever, our approach offers critical advantages over Trellis:\n\n(1) **Model scalability**: Trellis’ VAE requires rendering normal maps via FlexiCubes representations for supervision, demanding prohibitive GPU memory (e.g., out-of-memory errors at 512³). Our Sparse SDF VAE scales seamlessly to 1024³ resolution.\n\n(2) **Training efficiency**: Trellis’ asymmetric VAE (point cloud → latent → FlexiCubes with normal map supervision) exhibits high training complexity, requiring 64 A100 GPUs for several days to converge. Our symmetric encoder-decoder architecture (SDF grid → latent → SDF grid) reduces optimization difficulty, converging from 256³ to 1024³ in just 2 days on 8 A100 GPUs.\n\n(3) **Long-token modeling**: Trellis’ DiT uses full attention, becoming prohibitively slow with long tokens. Our proposed SSA substantially improves the training and inference efficiency.\n\n(4) **Progressive quality gain**: While 256³ SDF grids offer limited quality, this resolution serves merely as a warm-up phase to save training time. Quality progressively improves at 512³, with 1024³ SDF grids approaching GT mesh fidelity, enabling significant performance gains at higher resolutions.\n\nIn summary, Trellis achieves strong 256³ reconstruction quality but suffers from inefficiency and 256³ resolution limitations. Our Direct3D-S2 framework is explicitly designed to overcome this, enabling scalable VAE resolution (up to 1024³) and introducing SSA for efficient DiT training/inference with ultra-long tokens.\n\n**[Q5]:** Potential for text-to-3D extension.\n\n**[A5]:** Thanks for your valuable suggestions. While our framework could technically enable text-to-3D generation by substituting image conditioning with text conditioning during training and inference, this approach faces fundamental data scalability limitations since text-to-image/video models typically require billions of training samples whereas available 3D mesh datasets are orders of magnitude smaller (∼1M total, reduced to ∼100k usable samples after quality filtering). This severe data scarcity prevents training sufficiently generalizable text-to-3D models, evidenced by Trellis’ text-to-3D variant exhibiting significantly inferior stability and generalization versus its image-to-3D counterpart. Consequently, state-of-the-art 3D generation works (e.g., Hunyuan3D 2.0 [1], Craftsman [2], Triposg [3], Hi3DGen [4]) predominantly adopt text-to-image-to-3D pipelines leveraging pre-trained text-to-image models, and notably in our evaluation cases where a substantial portion of images sourced from communities were synthesized by text-to-image models, our Direct3D-S2 achieves compelling results, demonstrating seamless compatibility with such text-to-image-to-3D workflows.\n\n**[Q6]:** Redundant mention of ULIP-2, Uni3D, and OpenShape in Lines 256 and 264.\n\n**[A6]:** Thanks for your suggestions. We will reorganize these sentences in the final version.\n\n[1] Zhao, Zibo, et al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv 2025.\n\n[2] Li, Weiyu, et al. CraftsMan3D: High-fidelity Mesh Generation with 3D Native Diffusion and Interactive Geometry Refiner. CVPR 2025.\n\n[3] Li, Yangguang, et al. Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models. arXiv 2025.\n\n[4] Ye, Chongjie, et al. Hi3dgen: High-fidelity 3d geometry generation from images via normal bridging. ICCV 2025."}, {"review_id": "FqLIMDkUPv", "reviewer": "Reviewer_uJi9", "summary": "The paper introduces Direct3D-S2, a novel framework for scalable 3D generative modeling. The authors propose a fully end-to-end symmetric sparse SDF VAE that maintains a consistent sparse representation across encoding, latent, and decoding stages using direct 3D supervision. The authors also introduce a new spatial sparse attention mechanism that allows for efficient processing of large-scale 3D data. Experiments on public datasets demonstrate that Direct3D-S2 outperforms state-of-the-art methods in both geometric fidelity and training efficiency, making it promising for scaling 3D generative models.", "questions": "I would recommend the authors to consider the following questions:\n\n1. Could the authors provide more comprehensive numerical ablation results to demonstrate the model's scalability across different resolutions?\n\n2. Would it be possible to include more generation results on in-the-wild images to better demonstrate the model's practical applicability?\n\n3. Could the authors include additional discussion regarding the frequency of unfaithful detail generation and potential future solutions to address this issue?\n\n4. The 256 resolution results in Figure 3 appear to be significantly worse than those of Trellis, which also operates at 256 resolution. Given that the proposed encoding and generation framework is expected to achieve better performance at the same resolution, could the authors provide additional analysis to explain this discrepancy?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The authors propose Spatial Sparse Attention (SSA) which enables efficient modeling for large-scale 3D generative models.\n- The authors propose a fully end-to-end symmetric sparse SDF VAE that maintains a consistent sparse representation using direct 3D supervision.\n- The generation results are impressive, providing significant detail improvement over the state-of-the-art methods.\n\n**Weaknesses:**\n- The paper lacks numerical ablation studies across different resolutions in both VAE and generation to demonstrate the model's scalability in terms of resolution.\n- Although the generation results are impressive, some unfaithful details are present in the generated results (e.g., first row in Figure 4, the model generates two circles in the zoom-in area, which is only one circle in the ground truth image).\n- The paper lacks generation results on in-the-wild images.", "was_revised": true, "final_justification": "Thanks authors for their rebuttal. Most of my concerns have been addressed! I vote for accept.", "rebuttal": "Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:** Lacks numerical ablation studies across different resolutions in both VAE and generation.\n\n**[A1]:** Thanks for your valuable suggestions. We present the numerical ablation studies of VAE across different resolutions below. The evaluation metrics includes Chamfer Distance and F1 score with thresholds of 0.005 and 0.001. The metrics are multiplied by 10000, 100 and 100, respectively.\n\n| Resolution  | Chamfer Distance↓| F1 (0.005)↑ | F1 (0.001)↑ | \n------------ | :----: |  :----: |  :----: | \n| 256 | 30.86 | 85.61 | 9.93 |\n| 384 | 20.89 | 98.46 | 19.43 |\n| 512 | 13.59 | 99.98 | 41.11 |\n|1024 | **7.91** | **100.00** | **70.19** |\n\nWe perform quantitative comparisons of generation across different resolutions on a subset from Objaverse dataset, which comprises 500 meshes disjoint from the training set. Unlike the approach in our paper which uses metrics such as Uni3D and ULIP-2 to measure semantic similarity between generated meshes and input images, we employ the FID metric to more rigorously evaluate the geometry quality of generated meshes, which requires GT meshes. We render paired normal maps for both the generated meshes and GT meshes from the same viewpoints, then calculate the Normal-FID between them to assess model performance. The results are as follows: \n\n| Resolution  |Normal-FID ↓|\n------------ | :----: | \n| 256 | 132.97 |\n| 384 | 87.10 | \n| 512   | 69.31 |\n| 1024  | **46.44** |\n\nThere quantitative results demonstrate the scalability of our Direct3D-S2 in terms of resolution.\n\n**[Q2]:** The 256 resolution results in Figure 3 appear to be significantly worse than those of Trellis.\n\n**[A2]:** Thanks for your comments. We acknowledge that at 256³ resolution, our method performs worse than Trellis, an expected outcome for two reasons:\n\n(1) Trellis' VAE employs a 53-dimensional FlexiCubes parameters at 256³, whereas our Sparse SDF VAE uses only 1D SDF representations at the same resolution.\n\n(2) Trellis' VAE is supervised by high-resolution normal maps rendered from GT meshes, while we use 256³ SDF grids as supervision signals, which provide significantly inferior geometric fidelity compared to mesh-derived normals.\n\nHowever, our approach offers critical advantages over Trellis:\n\n(1) **Model scalability**: Trellis’ VAE requires rendering normal maps via FlexiCubes representations for supervision, demanding prohibitive GPU memory (e.g., out-of-memory errors at 512³). Our Sparse SDF VAE scales seamlessly to 1024³ resolution.\n\n(2) **Training efficiency**: Trellis’ asymmetric VAE (point cloud → latent → FlexiCubes with normal map supervision) exhibits high training complexity, requiring 64 A100 GPUs for several days to converge. Our symmetric encoder-decoder architecture (SDF grid → latent → SDF grid) reduces optimization difficulty, converging from 256³ to 1024³ in just 2 days on 8 A100 GPUs.\n\n(3) **Long-token modeling**: Trellis’ DiT uses full attention, becoming prohibitively slow with long tokens. Our proposed SSA substantially improves the training and inference efficiency.\n\n(4) **Progressive quality gain**: While 256³ SDF grids offer limited quality, this resolution serves merely as a warm-up phase to save training time. Quality progressively improves at 512³, with 1024³ SDF grids approaching GT mesh fidelity, enabling significant performance gains at higher resolutions.\n\nIn summary, Trellis achieves strong 256³ reconstruction quality but suffers from inefficiency and 256³ resolution limitations. Our Direct3D-S2 framework is explicitly designed to overcome this, enabling scalable VAE resolution (up to 1024³) and introducing SSA for efficient DiT training/inference with ultra-long tokens.\n\n**[Q3]:** Some unfaithful details are present in the generated results.\n\n**[A3]:** We sincerely appreciate you highlighting this observation. The surface details occasionally manifesting in our generated meshes but absent in input images stem from a dataset limitation: a subset of textureless meshes in Objaverse renders geometrically ambiguous conditional images. This creates a supervision mismatch during training, where input images lack discernible details while target meshes retain them, causing the model to potentially generate unfaithful surface details misaligned with the conditioning image. To resolve this, we will filter textureless meshes from the training set and retrain the DiT model. The updated model will also be publicly released. We deeply value your insight, which significantly improves our method’s robustness.\n\n**[Q4]:** The paper lacks generation results on in-the-wild images.\n\n**[A4]:** Thank you for your suggestion. Our test cases collected from professional communities encompass diverse image types, including samples generated by text-to-image models, images drawn by artists, and in-the-wild images, on which our method consistently demonstrates robust performance. Due to current conference policy prohibiting PDF attachments or external links during rebuttal, we will include additional results on in-the-wild images in the final version."}, {"review_id": "QtpbQBmXn6", "reviewer": "Reviewer_y1dk", "summary": "The paper proposes a novel pipeline, Direct3D-S2 for 3D mesh generation.\nOn the high level, the pipeline is based on a latent diffusion architecture (VAE + latent DiT).\nThe paper proposes SSA (Spatial Sparse Attention) in the DiT to speed up the attention layer which originally has quadratic complexity for high-resolution 3D information that translates into many tokens.\nThe generated meshes are detailed with bumps and dips and have crisp-clear edges on the geometry, showing the effectiveness and advantage of high 3D resolution.", "questions": "The work is in general good to me but there are important missing links in the whole pipeline making it hard to understand. I am glad to increase the score if the points in the weaknesses are clarified.", "limitations": "The limitation section in appendix mainly describes the limitation of SSA, which I think is interesting and significant but should not be dominating whole paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The implementation of SSA is non-trivial and could benefit the 3D community overall as a general mechanism for harvesting sparsity that naturally exist in 3D data.\n+ The generated mesh geometry is visually appealing and has high resolution.\n+ The paper is well-organized.\n\nWeaknesses:\n- It is unclear in the paper how the sparse structure is generated at inference time. This point confuses me from the beginning to the end when I am reading the paper. By \"sparse structure\" I mean where non-zeros locate, so sparsity can be utilized to speedup processing. There is a separate stage to generate sparse structure in the previous work TRELLIS. The lack of information about the source of this structure during generation makes the understanding of data structure of SSA hard.\n- The speed of generation is unclear. Though the attention is sped up much by SSA, the huge number of tokens may still limit the throughput with the computation of feed forward MLPs in transformer. The speed of generation is important for practical application.\n- The NeurIPS checklist item 8 \"Experiments compute resources\": I did not find \"comprehensive information on the type of computing workers, memory, and time of execution required for each experiment\"  in Section. 5 as claimed by the item. Missing memory and type of GPU for training (though I do find the information in appendix), and absolutely no information about data curation.", "was_revised": true, "final_justification": "The original concerns are addressed well by the rebuttal. The paper needs revision to include the information in the rebuttal, but with the discussions I am on the positive side.", "rebuttal": "Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:** It is unclear in the paper how the sparse structure is generated at inference time.\n\n**[A1]:** Thank you for raising this point. Our Direct3D-S2 employs an additional dense 3D VAE and DiT trained in a manner consistent with Trellis to predict indices of sparse latent tokens within voxel grids, which constitute the **sparse structure**. This is noted in Lines 29-30 of the supplementary material. We acknowledge this may cause confusion and incomprehension during paper reading, and we sincerely apologize for this oversight. In the final version, we will restructure the method section to include explicit descriptions of the sparse structure generation process, thereby avoiding any misunderstanding. We deeply appreciate your suggestion.\n\n**[Q2]:** The speed of generation is unclear.\n\n**[A2]:** Thanks for your comments. Within the DiT architecture, the self-attention module exhibits O(N²) complexity for N tokens, while other components such as feed-forward MLPs maintain linear complexity O(N). At high resolutions where the number of tokens N become substantial, self-attention accounts for over 90% of the model’s runtime. Therefore, although SSA only accelerates the self-attention module, it still leads to a significant improvement in the overall generation speed. In the table below, we compare the time required to denoise a single step using SSA versus full attention:\n\n| Methods  | Average Inference Time (One Step) ↓|\n------------ | :----: | \n| Ours (512) w/ Full Attention | 2.06s | \n| Ours (512) w/ SSA | 1.13s |\n| Ours (1024) w/ Full Attention   | 18.34s |\n| Ours (1024) w/ SSA  | 5.14s |\n\nFor 512³ resolution, around 30 denoising steps are needed, while for 1024³ resolution, around 15 steps suffice. Predicting the sparse structure takes approximately 5 seconds. Therefore, excluding post-processing steps such as mesh simplification, generating a 512³ resolution mesh takes about 0.5 minute, and a 1024³ resolution mesh takes around 2 minutes, which meets the requirements for practical applications.\n\n**[Q3]:** Experiments compute resources.\n\n**[A3]:** Thanks for your comments. As documented in the Implementation Details section of supplementary material, we have already specified the type of GPU and training durations. For further clarity, we provide additional experiments compute resources in the table below:\n\n| Type of GPU  | GPU Memory for Training (512³) | GPU Memory for Inference (512³) | GPU Memory for Training (1024³) | GPU Memory for Inference (1024³) |\n:----: | :----: | :----: | :----: | :----: |\n| NVIDIA A100 | ~24G | ~11G | ~40G | ~20G |\n\nNote that we trained our Direct3D-S2 using 8 GPUs, and the table above lists the GPU memory usage per single card for training.\n\n**[Q4]:** No information about data curation.\n\n**[A4]:** Thanks for your comments. In fact, we described the data processing pipeline in Section 5.1 in the paper. Here, we provide a more detailed explanation of the data curation process to avoid any misunderstanding:\n\n(1) **Data Filtering**. We first render all meshes from the ShapeNet, Objaverse, and Objaverse-XL datasets from six fixed viewpoints. We then randomly select several thousand 3D models and manually annotate whether each one is of high quality. For each mesh, we extract CLIP features from its six rendered images and concatenate them. Based on this, we train a lightweight binary classification network. Using this network, we filter out 452k high-quality meshes from the datasets for training.\n\n(2) **Geometry processing**. We follow the same strategy as CLAY [1]: for each non-watertight mesh, we first compute a UDF grid, and then use ray tracing to determine the visibility of each grid point. For invisible interior points, we assign negative UDF values. The resulting mesh is made watertight via marching cubes. We then extract its SDF grid for training.\n\n(3) **Rendering conditional images**. We rendered 45 RGB images per mesh at 1024x1024 resolution with random camera parameters. The camera configuration space is defined as follows: elevation angles ranging from 10° to 40°, azimuth angles spanning [0°, 180°], and focal lengths varying between 30mm and 100mm.\n\n**[Q5]:** Additional limitations.\n\n**[A5]:** Thank you for your suggestions. Another limitation of our method is that, since the final output of our model is a sparse SDF grid, we need to assign a default SDF value to the remaining grid points before applying marching cubes. As a result, the extracted mesh is hollow and requires post-processing, similar to Trellis, to remove internal structures. This issue could also be addressed by implementing a sparse marching cubes algorithm, which we leave as future work.\n\n[1] Zhang, Longwen, et al. Clay: A controllable large-scale generative model for creating high-quality 3d assets. TOG 2025."}, {"review_id": "DPLiJHHoPd", "reviewer": "Reviewer_GMc2", "summary": "This work introduces Direct3D-S2, a scalable 3D generation framework based on sparse volumes. At the core of the approach is a novel Spatial Sparse Attention mechanism which significantly speeds up both forward and backward pass, facilitating the scalability and efficiency of sparse structure-based 3D generative models. Experimental results demonstrate the superior performance of the proposed method.", "questions": "In line 144 it is mentioned that the VAE decoder outputs both input voxels and additional valid voxels. What does “additional valid voxels” refer to?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The proposed Spatial Sparse Attention mechanism, which is an adaptation of Native Sparse Attention on 3D data, effectively reduces the computation cost of sparse structure-based 3D generative models.\n- The qualitative results are compelling, with fine-grained geometry details.\n\n### Weaknesses\n\n- The modifications in the proposed Spatial Sparse Attention based on Native Sparse Attention are predictable and mostly engineering, with lack of novelty.\n- Line 229~230 claim that using all regions of conditional images for cross attention “may adversely affect the alignment between the generated meshes and the conditional images”, however, it misses the supporting evidences. It would be useful to provide several examples showcasing such misalignment.\n- As for the ablation on proposed Sparse Conditioning mechanism, it contains only two cases in Fig. 5, which implies potential cherry-picking. Due to the randomness of sampling process, the favor might can not be held under another random seed. More comparison cases or a quantitative results should be provided to demonstrate the constant benefits of the mechanism.", "was_revised": true, "final_justification": "Thanks for the authors' comment, and all my concerns are well addressed. By the detailed explanation of authors, I acknowledge that the contribution of proposed SSA is not trivial. It effectively overcomes the computational bottleneck in sparse structured 3D generative modeling at high resolution. While I maintain the view that the proposed \"Sparse Conditioning Mechanism\" appears somewhat contrived, overall, I believe the practice of this paper can advance the field of 3D generation.  I will raise my final rating.", "rebuttal": "Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:**  Novelty of the proposed SSA.\n\n**[A1]:**  Thank you for your comments. The main contribution of our SSA is to address the scalability and efficiency limitations in training with ultra-long tokens in 3D data, where the number of tokens increases cubically with resolution. The SSA is also well acknowledged by all other three reviewers: Reviewer WHua noted, **\"the method addresses the core inefficiency in high-resolution 3D generation with a well-motivated and technically solid solution\"**. Reviewer uJi9 said our approach is **\"a novel framework for scalable 3D generative modeling\"**, giving it a significance score of 4 (excellent). Reviewer y1dk commented that **\"the implementation of SSA is non-trivial and could benefit the 3D community overall as a general mechanism for harvesting sparsity naturally existing in 3D data\"**.\n\nAlthough our core idea of accelerating the attention mechanism via block partitioning is inspired by NSA, NSA is only applicable to structurally organized 1D sequences, as it partitions tokens based on their sequential order. This approach loses critical information when applied to unstructured tokens, such as sparse 3D voxels, and suffers from two key limitations. First, even though tokens within the same block may share consecutive indices, they are not necessarily spatially adjacent in 3D space. Second, due to the nature of sparse voxel structures, blocks with the same index across different samples may correspond to entirely different spatial regions. These issues collectively lead to unstable training convergence and a loss of detail in the generated meshes. To address this, we propose a novel method that partitions tokens into blocks based on their 3D spatial locations, followed by blockwise token selection. As shown clearly in Figure 6 of the paper, meshes generated using SSA exhibit smoother surfaces and more fine-grained details. We believe the novelty of SSA is well supported, as also highlighted by Reviewers WHua, uJi9 and y1dk.\n\n\n**[Q2]:** Sparse Conditioning Mechanism for improved alignment.\n\n**[A2]:** Thanks for your valuable suggestions. We have presented a qualitative comparison in Figure 5 of the supplementary material between results generated using cross-attention over the full input images and those using sparse conditioning. The regions highlighted in red indicate areas where the meshes generated by the former does not align well with the input images. After applying sparse conditioning, the generated results are corrected in these regions. To avoid potential cherry-picking, we perform quantitative comparisons on a subset from Objaverse dataset, which comprises 500 meshes disjoint from the training set. Unlike the approach in our paper which uses metrics such as Uni3D and ULIP-2 to measure semantic similarity between generated meshes and input images, we employ the FID metric to more rigorously evaluate the geometry quality of generated meshes, which requires GT meshes. We render paired normal maps for both the generated meshes and GT meshes from the same viewpoints, then calculate the Normal-FID between them to assess model performance. The results are as follows:\n\n| Methods  | Normal-FID ↓|\n------------ | :----: | \n| Ours (512³) w/o sparse conditioning | 71.49 |\n| Ours (512³) w/ sparse conditioning  | **69.31** |\n\nIt can be observed that the sparse conditioning module brings a certain level of performance improvement, demonstrating its effectiveness. However, we would like to clarify that the main performance gains and primary contributions of our method come from the proposed fully symmetric sparse SDF VAE and the SSA mechanism, which significantly improve both training and inference efficiency. These advances overcome the resolution limitations of previous methods and enable the generation of 1024³ resolution meshes.\n\n**[Q3]:** What does \"additional valid voxels\" in line 144 refer to.\n\n**[A3]:** Thank you for pointing this out. Our Sparse SDF VAE encodes sparse SDF grids of resolution N³ into sparse latent tokens at a lower resolution of (N/8)³, and then decodes them back to N³ sparse SDF grids. For each sparse latent token, the decoder upsamples it to 8³ grids. However, only a subset of these 8³ grid points correspond to the original input sparse SDF grids, and the remaining grid points fall into the category of extra valid voxels. We apologize for the confusion and misunderstanding this may have caused. In the final version, we will include a clear explanation of extra valid voxels to improve clarity."}]

Author rebuttal:

Meta-review: All five reviewers recommend accepting this paper (2 “Accept” and 2 “Borderline Accept”). The authors have provided a detailed rebuttal to successfully resolve most of the reviewers’ concerns. The reviewers mentioned that the pipeline of the proposed 3D mesh generation method is novel, the qualitative results of this paper are visually appealing, and the method’s computational efficiency is high. The AC agrees with the reviewers that the paper is interesting and deserves to be published in NeurIPS 2025.
Decision: Accept (poster)