Title: Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: AIGC

3D Generation

Diffusion
Abstract: Generating high-resolution 3D shapes using volumetric representations such as Signed Distance Functions (SDFs) presents substantial computational and memory challenges. We introduce Direct3D-S2, a scalable 3D generation framework based on sparse volumes that achieves superior output quality with dramatically reduced training costs.
Our key innovation is the Spatial Sparse Attention (SSA) mechanism, which greatly enhances the efficiency of Diffusion Transformer (DiT) computations on sparse volumetric data. SSA allows the model to effectively process large token sets within sparse volumes, significantly reducing computational overhead and achieving a 3.9$\times$ speedup in the forward pass and a 9.6$\times$ speedup in the backward pass.
Our framework also includes a variational autoencoder (VAE) that maintains a consistent sparse volumetric format across input, latent, and output stages. Compared to previous methods with heterogeneous representations in 3D VAE, this unified design significantly improves training efficiency and stability.
Our model is trained on public datasets, and experiments demonstrate that Direct3D-S2 not only surpasses state-of-the-art methods in generation quality and efficiency, but also enables training at 1024³ resolution using only 8 GPUs—a task typically requiring at least 32 GPUs for volumetric representations at $256^3$ resolution, thus making gigascale 3D generation both practical and accessible. Project page: https://www.neural4d.com/research-page/direct3d-s2.


Peer reviews:
comments: SUMMARY:
This work introduces Direct3D-S2, a scalable 3D generation framework based on sparse volumes. At the core of the approach is a novel Spatial Sparse Attention mechanism which significantly speeds up both forward and backward pass, facilitating the scalability and efficiency of sparse structure-based 3D generative models. Experimental results demonstrate the superior performance of the proposed method.

STRENGTHS AND WEAKNESSES:
### Strengths

- The proposed Spatial Sparse Attention mechanism, which is an adaptation of Native Sparse Attention on 3D data, effectively reduces the computation cost of sparse structure-based 3D generative models.
- The qualitative results are compelling, with fine-grained geometry details.

### Weaknesses

- The modifications in the proposed Spatial Sparse Attention based on Native Sparse Attention are predictable and mostly engineering, with lack of novelty.
- Line 229~230 claim that using all regions of conditional images for cross attention “may adversely affect the alignment between the generated meshes and the conditional images”, however, it misses the supporting evidences. It would be useful to provide several examples showcasing such misalignment.
- As for the ablation on proposed Sparse Conditioning mechanism, it contains only two cases in Fig. 5, which implies potential cherry-picking. Due to the randomness of sampling process, the favor might can not be held under another random seed. More comparison cases or a quantitative results should be provided to demonstrate the constant benefits of the mechanism.

QUESTIONS:
In line 144 it is mentioned that the VAE decoder outputs both input voxels and additional valid voxels. What does “additional valid voxels” refer to?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel pipeline, Direct3D-S2 for 3D mesh generation.\nOn the high level, the pipeline is based on a latent diffusion architecture (VAE + latent DiT).\nThe paper proposes SSA (Spatial Sparse Attention) in the DiT to speed up the attention layer which originally has quadratic complexity for high-resolution 3D information that translates into many tokens.\nThe generated meshes are detailed with bumps and dips and have crisp-clear edges on the geometry, showing the effectiveness and advantage of high 3D resolution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The implementation of SSA is non-trivial and could benefit the 3D community overall as a general mechanism for harvesting sparsity that naturally exist in 3D data.\n+ The generated mesh geometry is visually appealing and has high resolution.\n+ The paper is well-organized.\n\nWeaknesses:\n- It is unclear in the paper how the sparse structure is generated at inference time. This point confuses me from the beginning to the end when I am reading the paper. By \"sparse structure\" I mean where non-zeros locate, so sparsity can be utilized to speedup processing. There is a separate stage to generate sparse structure in the previous work TRELLIS. The lack of information about the source of this structure during generation makes the understanding of data structure of SSA hard.\n- The speed of generation is unclear. Though the attention is sped up much by SSA, the huge number of tokens may still limit the throughput with the computation of feed forward MLPs in transformer. The speed of generation is important for practical application.\n- The NeurIPS checklist item 8 \"Experiments compute resources\": I did not find \"comprehensive information on the type of computing workers, memory, and time of execution required for each experiment\"  in Section. 5 as claimed by the item. Missing memory and type of GPU for training (though I do find the information in appendix), and absolutely no information about data curation.\n\nQUESTIONS:\nThe work is in general good to me but there are important missing links in the whole pipeline making it hard to understand. I am glad to increase the score if the points in the weaknesses are clarified.\n\nLIMITATIONS:\nThe limitation section in appendix mainly describes the limitation of SSA, which I think is interesting and significant but should not be dominating whole paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Direct3D-S2, a novel framework for scalable 3D generative modeling. The authors propose a fully end-to-end symmetric sparse SDF VAE that maintains a consistent sparse representation across encoding, latent, and decoding stages using direct 3D supervision. The authors also introduce a new spatial sparse attention mechanism that allows for efficient processing of large-scale 3D data. Experiments on public datasets demonstrate that Direct3D-S2 outperforms state-of-the-art methods in both geometric fidelity and training efficiency, making it promising for scaling 3D generative models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The authors propose Spatial Sparse Attention (SSA) which enables efficient modeling for large-scale 3D generative models.\n- The authors propose a fully end-to-end symmetric sparse SDF VAE that maintains a consistent sparse representation using direct 3D supervision.\n- The generation results are impressive, providing significant detail improvement over the state-of-the-art methods.\n\n**Weaknesses:**\n- The paper lacks numerical ablation studies across different resolutions in both VAE and generation to demonstrate the model's scalability in terms of resolution.\n- Although the generation results are impressive, some unfaithful details are present in the generated results (e.g., first row in Figure 4, the model generates two circles in the zoom-in area, which is only one circle in the ground truth image).\n- The paper lacks generation results on in-the-wild images.\n\nQUESTIONS:\nI would recommend the authors to consider the following questions:\n\n1. Could the authors provide more comprehensive numerical ablation results to demonstrate the model's scalability across different resolutions?\n\n2. Would it be possible to include more generation results on in-the-wild images to better demonstrate the model's practical applicability?\n\n3. Could the authors include additional discussion regarding the frequency of unfaithful detail generation and potential future solutions to address this issue?\n\n4. The 256 resolution results in Figure 3 appear to be significantly worse than those of Trellis, which also operates at 256 resolution. Given that the proposed encoding and generation framework is expected to achieve better performance at the same resolution, could the authors provide additional analysis to explain this discrepancy?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of generating high-resolution 3D shapes in an efficient and scalable manner. Prior methods based on explicit voxel representations suffer from significant computational overhead, particularly in the attention modules of Diffusion Transformers (DIT), as the voxel grid resolution increases.\n\nTo overcome this limitation, the authors propose Spatial Sparse Attention (SSA), a novel mechanism that significantly reduces computational cost while maintaining high-quality 3D generation. SSA combines three key components: sparse 3D compression, spatial blockwise token selection, and sparse 3D window attention.\n\nFurthermore, the paper introduces Sparse SDF VAE, a unified variational autoencoder architecture where both the input and output are represented as sparse Signed Distance Fields (SDFs).\n\nExperimental results demonstrate that the proposed framework not only achieves superior computational efficiency but also improves the visual quality of image-conditioned 3D generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed Spatial Sparse Attention (SSA) mechanism presents a novel approach to scaling attention in 3D space. By effectively adapting Local Sparse Attention (LSA) to volumetric data, the method addresses the core inefficiency in high-resolution 3D generation with a well-motivated and technically solid solution.\n\n2. The architecture successfully handles gigascale 3D generation, and its design remains scalable to even larger scenes due to its efficient sparse formulation.\n\n3. The qualitative results are visually compelling, demonstrating detailed and structurally coherent 3D outputs that surpass prior methods in terms of geometric complexity and realism.\n\n4. The paper is clearly written, and the methodological components are effectively illustrated with informative and well-organized figures, which aid in understanding the proposed architecture and its contributions.\n\n\nWeaknesses\n1. All ablation studies are qualitative. A quantitative analysis of the proposed components would better clarify the contribution and effectiveness of each module.\n\n2. The Sparse SDF VAE is only evaluated qualitatively. Quantitative comparisons are necessary to demonstrate its advantages in training efficiency and geometric fidelity.\n\n3. Minor: Redundant mention of ULIP-2 [42], Uni3D [50], and OpenShape [20] in Lines 256 and 264. Please revise to avoid repetition.\n\nQUESTIONS:\n1. Clarification on performance attribution: It would be helpful to clarify whether the performance gains primarily come from the proposed network architecture or from operating at higher resolutions. For example, Trellis operates at a resolution of 256³, and the results of the proposed method at the same resolution (as shown in Appendix Figure 3) appear relatively less compelling. A direct comparison at equal resolutions, both qualitatively and quantitatively, would help to better isolate the contribution of the proposed design.\n\n\n2. SSA component ablation: In the ablation studies of the SSA module, it would be interesting to see results when only cmp or only slc (if available) is used. This could provide further insight into the individual contribution of each component within SSA.\n\n\n3. Potential for text-to-3D extension: Given the structured and scalable nature of the proposed method, I am curious whether it could be extended to support text-to-3D asset generation. If applicable, any preliminary results or discussion on this direction would be appreciated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}