High-quality Text-to-3D Character Generation with SparseCubes and Sparse Transformers.
Authors: Jiachen Qian
Hongye Yang
Shuang Wu
Jingxi Xu
Feihu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: anime avatar
diffusion
transformer
large reconstruction model
Primary area: generative models
TL;DR: We introduce a novel, efficient, sparse differentiable mesh representation method, termed SparseCubes, alongside a sparse transformer network designed to generate high-quality 3D anime characters.

Abstract: Current state-of-the-art text-to-3D generation methods struggle to produce 3D models with fine details and delicate structures due to limitations in differentiable mesh representation techniques. This limitation is particularly pronounced in anime character generation, where intricate features such as fingers, hair, and facial details are crucial for capturing the essence of the characters.

In this paper, we introduce a novel, efficient, sparse differentiable mesh representation method, termed SparseCubes, alongside a sparse transformer network designed to generate high-quality 3D models. Our method significantly reduces computational requirements by over 95% and storage memory by 50%, enabling the creation of higher resolution meshes with enhanced details and delicate structures. We validate the effectiveness of our approach through its application to text-to-3D anime character generation, demonstrating its capability to accurately render subtle details and thin structures (e.g. individual fingers) in both meshes and textures.

Published as a conference paper at ICLR 2025
HIGH-QUALITY TEXT-TO-3D CHARACTER GENERA- TION WITH SPARSECUBES AND SPARSE TRANSFORM- ERS
Jiachen Qian', Hongye Yang', Shuang W Jingxi Xu', Feihu Zhang Dream'Tch, 2Najjing University
ABSTRACT
Current state-of-the-art text-to-3 generation methods struggle produce 3D models with fine details and delicate structures due to limitations in differen- tiable mesh representation techniques. This limitation is particularly pronounced in anime character generation, where intricate features such as fingers, hair, and facial details are crucial for capturing the essence of the characters.
In this paper, we introduce a novel, efficient, sparse differentiable mesh repre- sentation method, termed SparseCubes, alongside sparse transformer network designed to generate high-quality 3D models. Our method significantly reduces computational requirements by over 95% and storage memory by 50%, enabling the creation of higher resolution meshes with enhanced details and delicate struc- tures. We validate the effectiveness of our approach through its application text-to-3D anime character generation, demonstrating its capability to accurately render subtle details and thin structures (e.g.individual fingers) in both meshes and textures.
INTRODUCTION
Recent advancements in 3D generation have primarily concentrated transforming textual descrip- tions into 3D models. Techniques such as DreamFusion (Poole (0222) and subsequent method- ologies employ score distillation sampling iteratively refine an implicit Neural Radiance Field (NeRF) (Mildenhall all 2020) representation, facilitating the creation of detailed 3D models. In a similar vein, [ang et all (20245) introduce implicit 3D Gaussians for enhancing 3D model genera- tion.
One-2-3-45++ (Lw al. 2IPZR) and XCube(Ben 20244 discretize 3D spaces into voxel grids and uses diffusion to learn and optimize 3D meshes. Similarly, DMTet (Shen et all (02I) and Flexicubes (Shen el all, 2)XE) employ a grid-based method, but they advance it further by using densely packed cubes within the grid. Unlike traditional voxels, these cubes incorporate controllable geometric features such as Signed Distance Fields (SDFs) (Park et all, 2019), enabling DMTet and Flexieubes generate smoother and more accurate mesh surfaces.
Recently, large reconstruction models (Hong et all, 2023: Wang et all, 2023 all 2023a have been introduced for single multi-view based 3D reconstruction and generation. These models typically encode input images as tokens and train a powerful transformer architecture to reconstruct the objects using implicit representations like NeRF (Triplanes) eLi all 2023a; Hong et all, 2023),
3D Gaussians (Tang et all 20244) or DMTe//lexicube (Peng et all 2024: Xuletal 2024).
However, key limitation of these methods lies in their struggle to capture fine details and intricate structures. This is primarily due to the challenges associated with the differentiable representation of the meshes. Character generation exemplifies this issue, particularly with elements such as hands, hair, and facial features, which are critical for capturing character's 's essence (as shown in Fig. MI. Unfortunately, compressing the 3D space into implicit representations, such as 2D Triplanes (Li
et all (02%a) (Fig. La (a)), 3D Gaussian TTag et all (024E) (Fig. II often leads to significant number of floating-point artifacts on the recovered 3D meshes. To mitigate this issue, existing
"Corresponding author.

---

Published as a conference paper at ICLR 2025
(a) Instant3D
(b) LGM
(c) Our method
(d) Ground Truth
Figure I: Visualization of the generation results with other methods. All algorithms used the same set multi-view images as input to generate the current avatar. (a) Instant3D (Li et all 20238)
produces blurred textures and overly smoothed meshes. (b) L.M (Tang all 2024a) provides relatively clear rendered images but fails to provide detailed geometric structures. (c) Our method can generate high-quality results, producing clear textures and detailed geometric structures even in intricate areas such as the face and hands. (d) Ground truth.
methods often employ strong smoothing filters, which unfortunately remove crucial details in the process.
Voxel-based methods (Schwarz et all 2022: etll 20244) offer way represent 3D space but often suffer from voxel effects on the final mesh surfaces (Fig. ( While increasing the resolution can mitigate these artifacts, it comes at significant cost. Memory usage and computational require- ments grow cubically, making high-resolution approaches impractical. Methods such as DM'eet and Flexicubes address this issue by incorporating Signed Distance Fields (SDFs) and geometric infor- mation into each cube, enabling the extraction of high-quality meshes. However, this approach also relies on dense, high-resolution space, which leads to high memory and computational demands, especially when aiming for very detailed reconstructions generations.
We propose a novel approach address the challenges of high-resolution 3D mesh generation: the Differentiable SparseCubes representation. This representation enables the efficient learning of de- tailed 3D meshes and textures with significantly lower memory and computational costs compared to other methods. To leverage this new representation, we introduce 3D Sparse Cube Transformer architecture specifically designed for processing SparseCubes. The combination of these innova- tions ensures superior preservation of geometric details, ultimately leading to the generation of high- quality meshes.
The effectiveness of our SparseCubes representation and Sparse Cube Transformers is demonstrated in the task of text-to-3D character generation. As illustrated in Fig. MI our method excels in pro- ducing high-quality meshes and textures that feature exceptional details and intricate structures, particularly in crucial areas such as faces and hands.
RELATED WORK
2.1 3D REPRESENTATION
201: Henzler all all XIT Lunz et all Previous methods (Wi 2016 Cadelha et
2020: Smith & Meger, 2017) employ voxels to represent 3D objects. However, due to the high memory footprint of voxel-based representations, alternative 3D representations such as point clouds (Achlioptas et all 2013 Yang et al] 2019; Zhou et all 2021: Mo all 2019), occupancy networks (Mescheder at all 2019, Signed Distance Fields (SDFs) (Park 2019), and octrees (lb= ing et al, 2023) have been explored. Recently, Neural Radiance Fields (NeRF) (Mildenhall et all 2020) have emerged as a groundbreaking implicit 3D representation, widely used 3D reconstruc- tion (Zhang et al, 2022, Müller et all 2022: Barron all 2022 212A) Hu et.ll 2023: Li et all 20256) and generation (Poole et all 2022: Lm et al. 2023, Chan et all 2022). Instead of purely using multi-layer perceptrons (MLPs) to implicitly represent the scene, TensoRF (Chen et all, 2022)
2

---

Published as a conference paper at ICLR 2025
and Strivec (Gaooetal. 2023) utilize tensor grids. Additionally, 3D Gaussian Splatting (3D GS) has been proposed as an alternative 3D representation to NeRF, demonstrating impressive quality and speed. Liao et all (2018) apply the Marching Cubes (MC) algorithm to extract surface meshes from implicit representations. DM'Tet (Shen et 2021) represents a shape using a Signed Distance Field (SDF) encoded with deformable tetrahedral grid and directly utilizes explicit supervision on the surface, thus generating high-quality shapes with arbitrary topology. FlexiCubes (Shen et all, 2023) utilize dense regular cubes to capture the geometry.
2.2 3D GENERATION
Score Distillation Sampling (SDS) based Methods. As diffusion models have achieved remark- able results in the text-to-image task, there is an emerging trend of employing pretrained 2D text- to-image diffusion models to perform text-to-3D synthesis. Dreamfusion (Poolc all, (0222 intro- duces score distillation sampling (SDS) loss, which transfers pretrained 2D image-text diffusion models to 3D object synthesis without any 3D data. Magic3D (Lin et all. 2023) combines the hash grid representation (Miller Mi (022) with the image diffusion prior and refines the model in a coarse-to-fine fine manner. Zero-1-to-3 (Liu et all, (023b) fine-tunes a pretrained diffusion model to learn controls over camera parameters and employs Score Jacobian Chaining (SJC) (Wang et all 2023a) to optimize a 3D representation with priors from texl-to-image diffusion models. Dream- Booth3D (Raj et all 20255 first fine-tunes the DreamBooth (Ruzz et all 2023) model and uses SDS loss to optimize the NeRF, then renders multi multi-view results from the NeRF and translates them into pseudo multi-view images. These images are used to fine-tune both the DreamBooth model and the NeRF. Make-it-3D (Tang et all 20233 first applies SDS loss to leverage 2D diffusion priors to gener- ate a coarse model with plausible geometry, then refines the texture of the model using both visible regions in the reference image and the diffusion prior. Realfusion Mlelas-Kyrizı al, (IPK) uses single-image textual inversion as a substitute for alternative views and normal vector regularization to avoid low-level artifacts on surfaces. NeRDi (Deng et all 2023) concatenates text embeddings from an image caption network and Textual Inversion (Gal et all, R022), using the Pearson correla- tion as a geometric regularization term. Fantasia3D (Chen et all 2023) disentangles geometry and appearance modeling and introduces the spatially-varying Bidirectional Reflectance Distribution Function. ProlifieDreamer (Wang et all MIZET) introduces Variational Score Distillation (VSD), exhibiting better generation results than SDS. Magic123 (Qran et all 2012)) incorporates both 2D and 3D priors into the 3D generation process and optimizes the model in a coarse-to-fin to-fine manner. DreamGaussian (Tang all, 20244) combines 3D Gaussian splatting (Kerbl et all 2023) and SDS loss for efficient 3D content generation.
Reconstruction based Methods. LRM (Hong et all 2023) takes an image as input and adopts a large transformer- based encoder-decoder architecture for learning 3D representations of objects. PF-LRM (Wang et ali, 20233) employs large transformer model to predict camera poses and re- construct 3D objects from a few unposed images. DMV3D (Ku al 2023) uses LRM as the multi-view denoiser, aiming to predict a clean Triplane representation from noisy multi-view im- ages. Instant3D eLl 20223) first generates four-view images simultancously via a fine-tuned text-to-image diffusion model and then regresses a Triplane-base these images using a transformer- sansformer--bsed model. CRM (Wang et all. 2024) designs a Convolutional Reconstruction Model and integrates geometric priors using six orthographic images. LGM (Tang atlll 20225) combines 3D Gaussian splatting with a proposed asymmetric U-Net to facilitate high- resolution training. Our method utilizes similar transformer network. While, we implement more efficient SparseCubes representation and Sparse 3D transformer for high-quality 3D generation.
(Chan ct all 2022) NeRF from
METHOD
To realize a text-to-3D pipeline, we first convert the text into multi-view images. Due to the limited amount of 3D character training data, we fine-tune text-to-image model based on PIxART-E. E (Chen stall (022)). We then train multi-viess diffusion model similar to ImageDream (Wang & Shi, 2023). but with higher resolution (512).
Our contribution lies in the proposed SparseCubes (Sec. [.TI and Sparse Cube Transformer (Sec. that can convert four image views into high-quality textured 3D character models. As illustrated
3

---

Published as a conference paper at ICLR 2025
in Fig. D, the proposed method contains two parts: 1) a low-resllutio coarse mesh proposal network used to sparsify the cubes in the 3D space, and 2) a high-resolution Sparse Cube Transformer used to further extract accurate meshes and textures.
Since learning and optimization in 3D spaces require cubic computations and memory costs, existing methods like LRM, Instant3D, and InstantMesh al.encode 2D) features and optimize the Triplanes, and then interpolate to achieve the 3D object. The compression of 3D information to 2D sometimes leads skew artifacts and over-smoothness of 3D meshes, illustrated in Fig. and Fig. E We instead utilize 3D transformers that can preserve the 3D geometric information and produce better meshes.
3.1 SPARSECUBES
Although current differentiable 3D representations, such as NeRF and Gaussian Splatting, can render high-quality RGB images, extracting high-quality meshes from them remains challenging.
Schwarz et all (2022) and Ren et all (20244) use 3D voxels to extract meshes but result in only coarse meshes with voxel effects on mesh surfaces (Fig. G) Using high-resolution voxels can reduce these artifacts, but the memory and computational costs grow cubically. Similarly, DM'Tet (Shen et all 2021) and FlexibleCube (Shen et all 202K) can produce smooth meshes. However, limited by the 3D resolution, they cannot preserve details and delicate structures (e.g..fingers.
To achieve high-quality meshes, we specifically design a new differentiable 3D representation, named SparseCubes. As illustrated in Fig. E (a), implement both geometric features and tex- ture features into one cube. For the geometric feature. similar to FlexiCubes (Shen et all 2023), use 25-dimensional feature represent the mesh. The first dimension represents the SDF of each vertex, while the second to fourth dimensions represent the deformation of each vertex. The remaining 21 dimensions consist of three sets of weights defined within each cube. These weights are used to adjust the location of the crossing point along each edge, the dual vertex inside each face, and the midpoint, respectively.
Different from other methods (Shen et all, 2021; 2023) that use a separate implicit representation to interpolate RGB texture information in order reduce memory and computational costs, we directly implement a 30 30-dimensional feature vector into each cube to represent the texture information. This means both geometry and texture can be optimized simultaneously.
We observe that the vast majority (> 95%) of cubes are empty, with only small portion containing mesh faces or vertices. Therefore, we do not use dense cubes but instead select small subset (less than 5%) to save memory and computational costs. The sparsification process is shown in Fig. D (b). We design a criterion to extract SparseCubes from these dense cubes. For a cube, if the SDF values of all eight corners of cube are either greater than 0 or less than 0, it indicates that the cube is entirely outside or inside the mesh respectively. To select cubes contain faces, we use the following criterion:
T
T(SDF; > 0) > 0,
PEW
where wa represents the set of all corners of the cube. To compensate for the information loss due to sampling, we select not only the cubes that meet the above criterion but also their neighboring cubes.
3.2 NETWORK ARCHITECTURE
As illustrated in Fig. 2. we follow the large reconstruction model to build our transformer network. Our network comprises three components. The first the Coarse Proposal Network (CPN), which takes images from four viewpoints and learnable cubes input and predicts coarse cubes around the 3D avatar. The second component involves a sparsification operation, which reduces the number of invalid cubes, thereby significantly enhancing computational efficiency. The last component is the Sparse Cube Transformer (SCT), which decodes the cube features into the corresponding 3D avatar.
Coarse Proposal Network We use DINO (C CCrron et all 2021) to encode the multi-view images into pose-aware tokens (similar to Li et all (2023))) ). Unlike LRM (Hong et al, 2025) and In- stant3]) (Li ELetall all (023a), we use 3D cubes instead of Triplanes to represent the scene. We first
4

---

Published as a conference paper at ICLR 2025
Proposal Network
Spursification
Sganse
Learnalle
SDF
Transforer Black
Geometry
Decony
Sampling
Texture
Alts
Local Alta
MLP
Self Attn
Propugation
Devany
Local Atta
Sparsification
Sparse Transformer
Figure 2: Illustration of the proposed pipeline. Given the learnable cubes and multi-view images, the coarse proposal network predicts the cubes that enclose the avatar. A sparsification operation is applied to reduce the number of invalid cubes. The Sparse Cube Transformer takes these valid cubes as input and decodes the geometric structure and texture of the 3D avatar.
initialize cubes with learnable 3D shape of 323 1024. Then, the cubes reshaped into sequence of learnable tokens. The DINO image tokens are embedded into the 3D cubes through cross-attention, followed by self-attention and MLP layers.
The number of tokens has a significant impact on the generated meshes. High-resolution cubes with tokens can produce better meshes. In the coarse proposal network, 323 to reduce memory and computational costs. After sixteen 3D transformer blocks, we extract the coarse meshes from the 3D cubes. To improve the extracted mesh resolutions, we use two lightweight deconvolution layers (with stride of 2) to upsample the cube representation before the mesh extraction. Note that the lightweight upsampling block is used after the transformer and only before the mesh extraction. This makes it memory and computationally efficient.
cubes of
Sparsification As illustrated in Fig. Z. based on the SDF values of these cubes (Eq. W., we trans- form the dense cubes into sparse ones, preserving only valid cubes. To avoid inaccurate meshes and missing parts in the coarse proposal meshes, We also propagate the valid voxels with its neighbor- ing cubes. This propagation is repeated twice to select each neighborhood and its neighborhood's neighborhoods. In this way, total of Mt cubes are selected, where M& < M and M is the total number of cubes. In our experiments, M is set to 643, and Mj is about 12K, accounting for less than 5% of the total number of cubes.
Sparse Cube Transformer To generate a high-quality 3D avatar from the Mj valid cubes, we design Sparse Cube Transformer (SCT) (illustrated in Fig. D). The M₁ SparseCubes and the DINO image tokens are fed into transformer decoder to refine the SparseCubes. The decoder contains three kinds of transformer blocks. After the first and the second transformer blocks, we implement upsampling layers to improve the cube resolution to 2563.
For the first part, we implement sixteen transformer blocks similar to Instant3D((Li all 2023a). Then, we implement the first upsampling layer and employ eight transformer blocks to optimize the cubes at higher resolution 128 In the cross-attention mechanism, the SparseCubes serve as the query sequence, while the image tokens from DINO act the key-value sequence. Instead of the original self-attention, which is memory and computationally expensive, we implement self- attention in minimal 3D region (consisting of 23 consecutive cubes). Such self-ateention layer can reduce the memory and computation costs by more than 1000 times. can also help generate fine details and delicate structures in local region after upsampling.
5

---

Published as a conference paper at ICLR 2025
Furthermore, we implement another upsampling and refinement block (consisting of one deconvolu- tion layer and four transformer blocks). This refinement block is realized at a resolution 256 The number of tokens increases significantly, even though we use SparseCubes. Therefore, we remove the cross-attention layer from the transformer block and only implement the self-attention layer in a minimal region of 43. We observe that there is little loss in mesh and texture quality, but our method achieves a 10 times acceleration with 100 times less memory usage.
3.3
IMPLEMENTATION DETAILS
Training Loss
We optimize our model by minimizing the following loss function:
18t) + Cipips (I:. 1!a) (Canage (fe. (
L
(2)
2ml
Amask (amass(M,,, M..h) + Mepth (De, D. Anormal (N;, NT))..
where V is the number of image views, L. M.. Dj and N₂ are the rendered image, mask, depth map and normal of each viewpoint, respectively, and 15th M D and Ñ are their corresponding ground truths. Limuge. Edepth and nommal are L2 loss and mmskk is an L1 loss. During the training of the Coarse Proposal Network, set Xipips and Amask and Adepth and Anomaa of the Sparse Cube Transformer, we set Aipips and Anomaal to 1, Amack to 8, and Adepth to 20.
For the training
Learning Strategies and Computational Resources The whole network can be trained in an end- to-end manner. However, for faster training, we first optimize the coarse proposal network using Eq. 2 for 121 hours with 32 A100 GPUs (30k iterations). The batch size is 5 and the learning rate is 4e-4 with cosine decay. After that, we warm up the Sparse Cube Transformer with a learning rate of 4e-4. We also use the upsampled output of the coarse proposal network as the learning target and train the Sparse Cube Transformer with L2 loss for 14k iterations. Finally, start to optimize the Sparse Cube Transformer using the same loss as the coarse proposal network with a smaller learning rate of 5e-5 and batch size of Learnable positional embeddings are used in both stages. The shapes of the positional embeddings in the first and second stage are 32 x 1024 and 64 x 1024, respectively. During the training phase, the positional embeddings for the second stage are initialized by upsampling the positional embeddings used in the first stage. After getting the relative spatial positions of valid SparseCubes, we can select valid tokens from the learnable positional embeddings as the input of the second stage. LRM (Hong et all 2023) requires 9,000 A10 GPU hours for training, while our algorithm requires a total of 3,800 A100 GPU hours, less than half of LRM's requirement.
Datasets and Augmentations We use 20K anime characters to train our models. Limited by the available amount of 3D data, we find it difficult to generate high-quality representations for some uncommon characters (c.g., characters with wings). We observe that most of our character data is centered within the cube, with only a small portion deviating from the center due to large tails or wings. For this reason, position embeddings that are not centered within the cuhe did not receive effective optimization, resulting in poor generation outcomes for characters with large tails or wings. Therefore, we designed a data augmentation method to randomly displace character data in the front, back, left, and right directions. This ensures that position embeddings not centered within the cube also receive effective optimization, leading to significantly improved results.
To further improve the texture accuracy of our generated mesh, we employed texture projeeronnnioon strategy similar to that in Peng all (2024). Since our generated mesh is highly accurate and best matches the input multi-view images, there are very few conflicting areas during the back-projection process.
Multi-view Back Projection
4 EXPERIMENTS
In this section, we conduct both qualitative and quantitative experiments to compare our method with existing state-of-the-art 3D generation methods.

---

Published as a conference paper at ICLR 2025
(a) Instant3d
(b) L.GM
(c) InstantMesh
(d) Ours
(c) GT
Figure 3: Qualitative Comparisons the LVIS subset of the Objaverse dataset (Deitke atlll 2023)
Each of the columns shows the results method (from left to right: Instant3D (Li all 2023)),
LGM (Tang et arl 20zem) InstantMesh (Xu all 2022), Ours and ground truth). Our method is able to generate significantly more detail for general 3D objects.
Table Comparisons on ground truth single image 3D. To ensure the fairness of the evaluation, we fine-tune all the methods used for evaluation on our dataset of 20K anime characters. The test set includes data of 30 randomly selected anime characters. We calculate PSNR and SSIM on eight full-body images and four headshot images from different perspectives. For DreamGaussian and InstantMesh, we use the provided pre-trained model without fine-tuning.
Method
PSNR (%)f
SSIM CD (10 5) (%)
Time (s)
DreamGaussian (Tag et (0246)
15.87
0.8263
184.96
180
InstantMesh (Xu all (024))
14.76
0.8460
492.48
9
Instant3D CLetal (1265
23.19
0.9106
19.42
15
Instant3D with DM'ect (Peng ct all (112))
20.40
0.8915
48.26
10
LGM (Tang et al. 2(228))
24.05
0.9213
28.49
52
Ours
24.99
0.9283
5.90
20
4.1 COMPARISONS WITH STATE-OF-THE-AT
We compare our method against recent techniques capable of generating 3D meshes from a single image. The comparison results are shown in Fig. B and Fig. R All input images are produced using the mnn-tuned PIXART-E. (Chen entire body (upper) and specific body parts (lower). The face and hand regions are rendered using Blender (Community, 2018) to allow for detailed visual comparisons. Our method produces 3D) meshes with superior visual effects and high-quality details. The meshes we generate exhibit vivid and lifelike facial features, while those generated by other methods often appear blurry. Additionally, our meshes feature distinct and well-separated fingers, whereas meshes generated by other methods frequently show merged hands.
((22)). For each generated mesh, we visualize both the
We train Instant3D and LGM on our training set under the same settings and then compare our method with them. We also compare method with DreamGaussian (Tang et all 20247) and InstantMesh (Xu et all 2024) by using the provided pre-trained models.
Comparisons on the anime avatar dataset Tab. I presents the comparison results. To ensure fair comparison we do not use the texture back-projection strategy for any of the algorithms when calculating these metrics. Our algorithm achieved the best PSNR and SSIM, indicating that the generated results of our method have the highest visual quality. addition, our method signifi- cantly outperforms other approaches in terms of Chamfer Distance (CD) loss. Compared with the second-be bestt method, Instant3D, our method reduces Chamfer Distance error by about 70%. indicat- ing that approach produces much higher mesh quality and better details. As shown in Tab. M our algorithm is also very fast in terms generation speed. We also compare our method with an SMPL--based approach, TeCH Huang al], 2024). The result is shown on Fig. of the Appendix.
7

---

Published as a conference paper at ICLR 2025
young git with part hun grean syes and tlue and while achool unitorm.
She has diei butlik wings, extending y from book
gi
opea and light complesinn dressed a pastel colree outi..
(a) DreanGaussian
(b) Instant3D
(C) LGM
(d) Ours
Figure 4: Qualitative Comparisons. First column: We use the text prompts to generate the input image. Each of the remaining columns shows the results of a method (from left to right: Dream- Gaussian (Tang et al, 20248), Instant3l) all 2023a). LGM (Tang et all 20245), and Ours). To compare the quality of details and delicate structures, also show zoomed-in views of the hand and face regions.
Table 2: Comparisons with other methods the LVIS subset of the Objaverse dataset (Deitke all 21R3) The test set includes data from 30 randomly selected objects.
Method
PSNR (%)
SSIM CD (10-5) 5) (%)
InstantMesh (Xu ala 21224)
23.67
0.9129
71.32
Instant3D (L cl a (()4))
23.16
0.9097
81.09
Instant3D with DMTet (Peng all 2025
22.00
0.9063
230.64
LGM (Lag ct all (1AAR
24.70
0.9389
116.14
Ours
26.78
0.9452
36.86
Comparisons the realistic human dataset We test our method on the RenderPeople dataset (rem, 2018) to demonstrate that it can generate high-quality, realistic-style characters. The experimental results in the Tab. E of the Appendix demonstrate the effects of our method. Our Sparse Transformers achieve the best PSNR for evaluating the rendering qualities and the best CD loss for evaluating the mesh qualities.
Comparisons on the LVIS subset of the Objaverse dataset Since most algorithms (Tang et all 20221 Xu all 2024: Liet all 20233 Tang et all 20245), are designed for the general 3D object generation task. in order to ensure fair comparison with these methods, the LVIS subset of the Objaverse dataset et all 2023). Tab. shows the quantitative results. Our method outperforms other methods across all metries, demonstrating its ability generate high-
train method on
8

---

Published as a conference paper at ICLR 2025
(a) DreamGaussian
(b) Instant3D
(C) LGM
(d) Ours
Figure 5: Mesh Qualitative Comparisons. Characters from Fig. y Additional results provided
in the Appendix. Our method far outperforms tfate-of-the-as the-art methods in mesh quality.
Table 3: Comparisons between SparseCubes and Dense Cubes representations. We report the speed and memory usage during the training phase with batch size of 1.
Type Resollution PSNR* SSIM |CD(10 ) MEM(GB) Time per iter (s) Number of faces (K)
Dense
21.34
0.8987
19.88
Dense
128
24
340
50
Dense
256
80
2850
170
Sparse
64
21.08
0.8970
13.87
8
1
20
Sparse
256
24.99
0.9283
5.90
37
5
170
quality 3D objects. The generated results our method have the highest visual quality, as shown in Fig. B and Fig. K.
SparseCubes versus Implicit Representation Instant31 (Lietal. 2023a) and LGM (Tang et all (02E%) encode the 3D space into an implicit NeRF (TriPlanes) or 3D Gaussian and utilize trans- formers to learn the meshes and textures. As shown in Tab. M. Fig. Ml and Fig. I2, methods with implicit representations use strong smoothness to remove floating artifacts; as result, fine details and delicate structures missing in the predicted results. Our SparseCubes and 3D transformers learning, which can better preserve geometric features and generate high- quality meshes with exceptional detail and intricate structures, particularly in crucial areas such as faces and hands.
3D representation
SparseCubes versus DMTet/FlexiCuber DMTet and FlexiCubes use dense cubes as 3D mesh representations. Instant3D with DMTet and InstantMesh with FlexiCubes use dense 3D cubes that require much more memory and computational costs than our SparseCubes. As shown in Tab. MI they can only with low-resolution (1283) cube representation. The generated results do not have a large Chamfer Distance error. Additionally, we observe that InstantMesh and Instant3D with TMt generate many holes in light-colored regions (as shown in Fig. of the Appendix).

---

Published as a conference paper at ICLR 2025
Table Analysis of data augmentation. For faster training and evaluation, we use the model with resolution of 64.
Random displacement
PSNR
SSIM
CD (10-5)
19.81
0.8856
53.8
22.11
0.8997
46.1
SparseCubes verse Voxels Voxel-based methods that discretize 3D space as and (Ren all 2023: Schwarz 2022) tend to generate meshes with voxel effects (as illustrated in Fig. B) Our SparseCubes can produce smooth mesh surfaces with intricate structures.
4.2 ABLATION STUDY
SparseCubes versus Dense Cubes Tab. B presents the comparison results between using Spar- seCubes and Dense Cubes representations. We use the same number of transformer blocks and compute the memory usage and speed with batch size of one.
At the same resolution of 64, the PSNR SSIM of the sparse model are comparable to those of the dense model, indicating that the degradation in visual quality introduced by our sparsification operation negligible. Mcanwhile, the Chamfer Distance the sparse model much better than that of the dense model, demonstrating that the proposed sparsification enables the model to focus more on the valid positions, thus resulting in more accurate geometry. addition, the sparse model 40 times faster than the dense one. When increasing the resolution to 256, our sparse model achieves far better accuracy in both textures and meshes.
Employing Dense Cubes with resolution of 256 significantly increases GPU memory consumption (by 2 times) and training time (by more than 100 times), thereby hindering the acquisition of timely training outcomes. Training with resolution of 128 is also difficult it consumes 64 times more computational resources than our Sparse Cube Transformer.
Note that, the FlashAttention eCal 202 in the xFormers (Letaudeux eLa 202 reduce the
memory costs in training transformers at the expense of increased runtime. The actual memory costs of the dense models (for both resolutions of 128 and 64) are much higher than those our Sparse Cube transformers.
Random Displacement We conduct further analysis assess the impact of random displacement on the performance of our algorithm. Experiments are performed using a model with a resolution of 64. The results in Tab. indicate that data augmentation method significantly improves the quality the generated results.
CONCLUSION
In this work, introduced SparseCubes, novel and efficient sparse differentiable mesh represen- tation method, along with a sparse transformer network for generating high-quality 3D characters. Our approach addresses the limitations of current state-of-the-art 3D generation methods, which struggle to capture fine details and delicate structures crucial for applications such as anime avatar generation. Through extensive testing and validation, demonstrated that SparseCubes signifi- cantly reduce computational requirements and achieves high-quality and highly detailed 3D mesh generation.
Limitations and Future work Limited by the amount of available 3D data and the diversity of text descriptions, the fine-tuned text-to-image model currently doesn't 'tt always perfectly match the textual descriptions. We believe this can be improved by fine-tuning more powerful diffusion transformer models (Lletal 2024). Additionally, providing more diverse and longer text prompts boost the alignment. As for future work, we will explore ways to further increase the resolution of the SparseCubes (e.g., 1 with millions of mesh faces) achieve hyperfine structures in 3D meshes.
also
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our work shares similar concerns with other generative modeling efforts, particularly regarding the potential for misuse in creating misleading content. Additionally, biases present in the training datasets can be reflected in the generated outputs. As with any powerful creative tool, there is risk of misuse. Generated characters could be used inappropriately to create harmful content. is essential to develop and enforce ethical guidelines and content moderation strategies to mitigate such risks.
REFERENCES
Renderpeople. In Btttp::/rennerreccplec. peppppppppecommmdpppppppppppp people.ppeeee....o..... 2018.
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for point clouds. In ICML, 2018.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In ICCV, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in supe-supe vision transformers. In ICCV, 2021.
Eric Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awan 3d generative adversarial networks. CVPR, 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. (ixart-/sigmal Weak-to-strong training of diffusion trans- former for 4k text-to-image generation. arXiv preprint, 2024.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. ICCV, 2023.
Blender Online Community. Blender- 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL pttp:
Tri Dao, Daniel Y. Fu, Stefano Ermon. Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In NeurIPS, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. CVPR, 2023.
Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guides diffusion as general image priors. In CVPR, 2023.
Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects. In 3DD,20017
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint, 2022.
Quankai Gao, Qiangeng Xu, Hao Su, Ulrich Neumann, and Zexiang Xu. Strivee: Sparse tri-vector radiance fields. In ICCV, 2023.

---

Published as a conference paper at ICLR 2025
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversar- ial rendering. In ICCV, 2019.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image 3d. arXiv preprint. 2023.
Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In ICCV, 2023.
Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. In 3DV, 2024.
Moritz Ibing, Gregor Kobsik, and Leif Kobbelt. Octree transformer: Autoregressive 3d shape gen- eration on hierarchieally structured sequences. In CVPR, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ToG, 2023.
Benjamin
Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https: /ttp:://qtthupe com hamm//aacoookeeeeeeeoao 2022.
rme
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model. arXiv preprint, 2023a.
Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In CVPR, 2023b.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang. Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li. Chen Zhang. Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang. Meng Chen, Jie Liu, Zheng Fang. Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin, Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu, Xiao Xiao, Yan Chen, Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, and Qinglin Lu. Hunyuan-dit: A powerful multi-resolluton diffusion transformer with fine-grained chinese understanding, 2024.
Yiyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. CVPR, 2018
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image to 3d object. In CVPR, 2023b.
Sebastian Lunz, Yingzhen Li, Andrew Fitzgibbon, and Nate Kushman. Inverse graphics gan: Learn- ing to generate 3d shapes from unstructured 2d data. arXiv preprint, 2020.
Luke Melas Kyriazi,, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. CVPR, 2023.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2025
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Struc- turenet: Hierarchical graph networks for 3d shape generation. Siggraph Asia, 2019.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. TOG, 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
Yang Peng, Jia-Peng Zhang, Meng-Hao Guo, Yan-Pei Cao, and Shi-Min Hu. Charactergen: Efficient 3d character generation from single images with multi-view pose canonicalization. arXiv preprint, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint, 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR, 2024.
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. Dreambooth3d: Subject-driven text-to-3d generation. In ICCV, 2023.
Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hicrarchies. In CVPR, 2024.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d- aware image synthesis with sparse voxel grids. 2022.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NeurlPS, 2021.
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojeic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient- based mesh optimization. TOG, 2023.
Edward J Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In Conference on Robot Learning, 2017.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint, 2024a.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. ICLR, 2024b.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make- it-3d: High-fidelity 3d creation from a single image with diffusion prior. In ICCV, 2023.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023a.
Peng Wang and Yichun Shi. Imagedream: proge-prompt multi-view diffusion for 3d generation. arXiv preprint, 2023.
Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexi- ang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. In ICLR, 2023b.
13

---

Published as a conference paper at ICLR 2025
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurlPS, 2023c.
Zhengyi Wang, Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint. 2024.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning proba- bilistic latent space object shapes via 3d generative-adversaria modeling. In NeurIPS, 2016.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from single image with sparse-view large reconstruction models. arXiv preprint, 2024.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint, 2023.
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV, 2019.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint, 2020.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. ICCV, 2021.
A APPENDIX / SUPPLEMENTAL MATERIAL
Ablation study on other parameters Since training large-scale transformer models costs signif- icant GPU resources, ablation all kinds of parameter settings (e.g.,the balance weights of the loss function and the learning rate) is impossible. For the remaining parameters, we adjust them only in a fine-tuning manner. Specifically, we fine-tune the model with different parameter settings (e.g., balance weights) for just 1000 iterations (about 6 hours) to discover the best settings by observ- ing its convergence speed and CD loss of the meshes.
(a) Voxels in 256
(b) Our SparseCubes 256
Figure 6: Comparison of mesh results between our SparseCubes (at a resolution 2563) and voxels. (a) There are clear voxel effects in the mesh produced by voxelization. (b) Our SparseCubes can produce high-quality meshes the same resolution.
More results
14

---

Published as a conference paper at ICLR 2025
!!
(a) Instant3D with DMTet
(b) InstantMesh
(c) Ours
Figure 7: Illustration of holes in light-colored regions using Instant3D with DM'Tet and InstantMesh (without fine-tuning)
Instant3d
InstantMesh
(d)
(e)
Figure 8: Results of normal on the LVIS subset of the Objaverse dataset (Deitke ct al. 202%) Each of the columns shows the results of method (from left right: Instant31D (Li all (
LGM (Tang et al,, 20245), InstantMesh (Xu all 2024), Ours and ground truth). Our method detail for general 3D objects.
able to generate significantly
Table 5: Comparison on RenderPeople ren (2018) dataset.
Method
PSNR (%)
CD (10 (10-3) (%)
Instant3D Li all (2(255)
18.79
15.00
Instant3D with DMTet Peng et all (21124)
18.32
34.81
LGM Tang all (21244
20.02
11.78
Ours
25.93
9.60
15

---

Published as a conference paper at ICLR 2025
Tech
ImageDream
Ours
Figure 9: Compairsons with the shapes of Tech (Huang et all 2022)) and ImageDream (Wang Shi 2023). The TeCH method (Huang et al, 2022) heavily depends on the SMPL model. However, the 3D character data includes diverse hairstyles and outfits that the SMPL model cannot adequately represent. Consequently, the characters generated by TeCH exhibit significantly poor quality.
16

---

Published as a conference paper at ICLR 2025
Charactergen
Figure 10: Comparisons with CharacterGen (Peng et all 2024). We have conducted comparison between CharacterGen and our method, as shown Tab. II Specifically, we implemented Instant31) with DMTet, the approach adopted by CharacterGen. Here we utilize the weights released by Char- acterGen for further evaluation. The results clearly demonstrate that our method significantly out- performs CharacterGen, producing meshes with finer details and textures that are noticeably sharper and more realistic.
Coarse Proposal Network
sparse transformer
Figure 11: Comparisons between the coarse proposal network and the sparse transformer. We com- pared the quality of the finger meshes generated by the coarse proposal network and the sparse transformer. It is evident that the sparse transformer significantly improves the quality of the fingers in the generated meshes.
17

---

Published as a conference paper at ICLR 2025
Wonder3D
OpenLRM
Ours
Figure 12: Visual Qualitative Comparisons. Extra comparisons of Fig.
18

---

Published as a conference paper at ICLR 2025
Figure 13: More results
(a) DreamGaussian
(b) Instant3D
(C) LGM
(d) Ours
Figure 14: Additional mesh results from Fig. P
19


Reviews:
[{"review_id": "whcQPzN5aH", "reviewer": "Reviewer_mvv6", "summary": "The paper introduces a novel method to produce detailed 3D generation with less computational cost and training time. Specifically, the authors propose a sparse differentiable mesh representation called SparseCube to enable more efficient modeling of 3D subjects. Correspondingly, a sparse transformer network is designed to replace tri-plane to accommodate the proposed 3D representation. Through extensive evaluations, the novel 3D representation is demonstrated to be better than DMTet, implicit representation, and voxels. Meanwhile, the experiments showcase the best performance of the proposed method and the effectiveness of each component.", "questions": "Considering the above strengths and weaknesses, I currently still lean toward a positive rating. I would like to hear from the other reviewers and the authors during the rebuttal.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper introduce a novel 3D representation, called SparseCube. It's capable of capturing the details while largely decreasing the computational cost and training time.\n\n+ The proposed method is demonstrated to perform better than existing techniques.\n\n+ Extensive experiments have been provided to demonstrate SparseCube's superiority over DMTet, voxels, and implicit representations.", "weaknesses": "- The paper would benefit from further refining the paper structures and polishing the writings.\n\n- The geometry quality in Fig. 1 and Fig. 5 is not good.\n\n- What are the results and comparisons in general 3D avatars like \"people wearing daily outfits\" or movie characters like \"Iron Man\"?\n\n- There are no comparisons with human-specified methods like Human-LRM or more recent 3D generative methods like 3DTopia, 3DTopiaXL, etc.\n\n- The design of the features within SparseCube is not well-justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gBnDWA0bCx", "reviewer": "Reviewer_KMF3", "summary": "This paper introduces SparseCubes, a new method for improving text-to-3D generation of anime characters. SparseCubes uses a sparse mesh representation that cuts computational needs by over 95% and storage by 50%, allowing for better detail in 3D models. The authors demonstrate its effectiveness in accurately rendering delicate features in anime character generation.", "questions": "1. Is the entire training process conducted in stages, or is it an end-to-end process?\n\n2. The results on the real human dataset only include quantitative metrics and lack qualitative results for comparison.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper presents the SparseCubes representation, which can capture fine details like fingers and hair.\n\n2. A corresponding transformer network is designed for the SparseCubes representation\n\n3. A significant efficiency improvement compared to previous methods", "weaknesses": "Method:\n\n1. High-resolution voxels have already been used to generate smooth meshes in DMTet and FlexibleCube. Improving efficiency and increasing resolution through a sparse design is a straightforward idea, which somewhat affects the novelty of the work.\n\nExperiments:\n\n1. The paper lacks a comparison of results using FlexibleCube as a post-processing step.\n\n2. The experimental results are somewhat confusing: First, the quantitative improvements in PSNR and SSIM on the anime avatar dataset are quite small, and an explanation for this is needed. Additionally, the qualitative comparisons provided for the LVIS subset make it difficult to discern clear differences.\n\n3. The paper lacks a comparison with the current state-of-the-art method, CharacterGen[1], including both qualitative and quantitative comparisons.\n\n4.  For the qualitative results of the anime avatar dataset, the main text focuses on only a few selected IDs. It would be beneficial to include comparisons for more IDs in the experiments.\n\n[1] Charactergen: Efficient 3d character generation from single images with multi-view pose canonicalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2cfKt0WEZa", "reviewer": "Reviewer_ozD8", "summary": "This paper targets to solve the high-resolution mesh generation problem with text prompts. \nIt argues the current text-to-mesh methods are mainly bottlenecked by the cubical memory usage to generate high-resolution meshes,\nwhich leads to the limitation of the mesh resolution, and the quality of the generated meshes.\nThis problem is believed to be in particular severe for human-like/character-based meshes, like fingers, hair, fur, etc.\n\nTo solve this problem, this paper introduce the sparseCube and the sparse transformer. \nIt first generates a low-resolution (32x32x32) grid to obtain a mesh in a differentiable manner, and then sparsifies the grid to get a sparse grid.\nFinally, it uses a sparse transformer to generate the high-resolution mesh from the sparse grid.\nThis paper compares the proposed method on objaverse and human character datasets, and demonstrating the effectiveness of the proposed method.", "questions": "1. I am a little bit confused about the relationship between 3D character generation and sparse CUBE/Transformer. For sparse cube/transformer, it seems they can work on any 3D content generation without any specific constraints like character. So I am curious why this paper mainly argues about 3D character generation? I can understand 3D character has some very thin structures like hair, fingers, but I still think generating high-quality 3D mesh would be a common interest for 3D generation, whatever the objective is. \n2. Just for clarification, the renderer (from 3D mesh to image) is the same as FlexiCubes or not?\n3. For sparse transformer, do you apply position encoding (more specifically, the 3D location/index of occupied cubes) ? If so, how do you do that?\n4. It would be helpful to evaluate the results just after the coarse proposal network, and after prunning, to show how much gains we get from the sparse Transformer as a refinement stage.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The overall pipeline is sound and reasonable. Utilizing the sparsity of voxel grid, we can pursue the high-resolution meshes (using FlexiCubes), leading to high-resolution 3D assets generation. \n2. This paper is well written: the story and method is easy to follow and understand.\n3. Results look great.", "weaknesses": "Overall, the contribution of this project is OK but mild. \nThis paper utilizes the idea of Flexicubes to train a model to predict the 3D voxels for a mesh, and renders the 2D images from the 3D mesh in a differentiable manner, which is a still in the scope of Flexicubes.\nThe main contribution of this paper should be utilizing a sparse transformer to do refinement or upsampling for the 3D voxels after pruning. \nThis is a sound idea while not novel. \n\nSome weakness: \n\n1. I think there are some baselines missing. In particularly, XCUBE [XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies]. Although XCUBE uses diffusion model for generation and its output voxel grids are supervised by 3D loss, but it also highlights a sparse voxel grid solution (a novel data structure and companying network to process it). I am curious can you use XCUBE's sparse voxel grid method (fvdb) to do the same thing as yours? And compare the speed, memory and performance difference? I am very curious about how the proposed method. is compared again the XCUBE's backbone. Also, SparseConv-Unet might be another reasonable  backbone to compare against the Sparse Transformer. \n\n2. In table 1, the proposed method takes 20 s to generate, which is slower than InstantMesh. I am curious as a feed-forward model, why it is slower? And what's the time using ratio of each component?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)