Title: FRACTAL3DGEN: COARSE-TO-FINE 3D GENERATION VIA FRACTAL HIERARCHICAL TRANSFORMERS
Primary area: generative models
Keywords: 3D shape generation, autoregressive, fractal architecture, voxel pruning, Hausdorff dimension

Abstract: Autoregressive generation models have demonstrated strong performance across a wide range of applications. In the context of 3D shape generation, several approaches have attempted to adopt this paradigm by flattening the 3D representation into a serialized one-dimensional sequence and predicting it sequentially, token by token. These methods are inherently inefficient due to their strictly sequential generation process and are often inadequate in exploiting hierarchical self-similar representations during generation. Inspired by the success of FractalGen in image generation, we propose Fractal3DGen, a hierarchical fractal-autoregressive framework that leverages self-similarity across multiple scales for 3D shape generation.  Specifically, the 3D shapes, represented by sparse SDF grids, are encoded into a compact, low-resolution sparse voxel latent space for more efficiency. Our generation process then employs a divide-and-conquer strategy to efficiently capture the intrinsic hierarchical structure of 3D shapes. Furthermore, due to the spatial sparsity of 3D shapes, we introduce an adaptive pruning mechanism during generation to promptly halt the processing of empty regions at higher scales, significantly improving both training and inference efficiency while reducing memory consumption. Extensive experiments on ShapeNet demonstrate the effectiveness of Fractal3DGen. Fractal3DGen achieves the lowest average FID against all baselines, with over 15% improvement across five categories. Moreover, it provides a 1.85× speedup,  reducing the average inference time per shape from 54.1s to 29.3s compared with OctGPT. Beyond benchmark evaluations with class conditioning, we further explore text- and image-conditioned 3D generation, demonstrating the generalizability of our approach.

TL;DR: Fractal3DGen introduces a fractal-based autoregressive architecture for efficient and high-quality 3D shape generation, outperforming existing methods in speed and quality.