Tetrahedron Splatting for 3D Generation
Authors: Chun Gu
Zeyu Yang
Zijie Pan
Xiatian Zhu
Li Zhang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: text-to-3D
tetrahedron splatting
volumetric rendering
Primary area: generative_models

Abstract: 3D representation is essential to the significant advance of 3D generation with 2D diffusion priors. As a flexible representation, NeRF has been first adopted for 3D representation. With density-based volumetric rendering, it however suffers both intensive computational overhead and inaccurate mesh extraction. Using a signed distance field and Marching Tetrahedra, DMTet allows for precise mesh extraction and real-time rendering but is limited in handling large topological changes in meshes, leading to optimization challenges. Alternatively, 3D Gaussian Splatting (3DGS) is favored in both training and rendering efficiency while falling short in mesh extraction. In this work, we introduce a novel 3D representation, Tetrahedron Splatting (TeT-Splatting), that supports easy convergence during optimization, precise mesh extraction, and real-time rendering simultaneously. This is achieved by integrating surface-based volumetric rendering within a structured tetrahedral grid while preserving the desired ability of precise mesh extraction, and a tile-based differentiable tetrahedron rasterizer. Furthermore, we incorporate eikonal and normal consistency regularization terms for the signed distance field to improve generation quality and stability. Critically, our representation can be trained without mesh extraction, making the optimization process easier to converge. Our TeT-Splatting can be readily integrated in existing 3D generation pipelines, along with polygonal mesh for texture optimization. Extensive experiments show that our TeT-Splatting strikes a superior tradeoff among convergence speed, render efficiency, and mesh quality as compared to previous alternatives under varying 3D generation settings.

Tetrahedron Splatting for 3D Generation
Chun Gu Zeyu Yang' Zijie Pan'
Xiatian Zhu
Li Zhang'*
'ccooo of Data Science, Fudan University
UUiiversity of Surrey
htfps://fudanzvvv github.io/tet- ttttttitttttteee -el.ttt--ptlinnggggeee
DSLR photo of the Imperial State Crown England
"a DSLR photo of candelabra with candles red velvet tablecluth"
a DSLR phoso of reust turkey platter
DSLR photo of steaming backet full of dumplings
crupting volcano,
photo origmii crne
zoomed photo adurable cotaage with thanched moo"
"Wedding
photo of recliner bald bagle carved
photo pigzon reoding
Figure 3D assets generated by our proposed TeT-Splatting
Abstract
3D representation is essential to the significant advance of 3D generation with 2D diffusion priors. As a flexible representation, NeRF has been first adopted for 3D representation. With density-based volumetric rendering, it however suffers both intensive computational overhead and inaccurate mesh extraction. Using a signed distance field and Marching Tetrahedra, DMTet allows for precise mesh extraction and real-time rendering but limited in handling large topological changes in meshes, leading to optimization challenges. Alternatively, 3D Gaussian Splatting (3DGS) is favored in both training and rendering efficiency while falling short mesh extraction. In this work, we introduce novel 3D representation, Tetrahedron Splatting (Te--Sllttingg) that supports easy convergence during optimization, precise mesh extraction, and real-time rendering simultaneously. This is achieved by integrating surface-based volumetric rendering within structured tetrahedral grid while preserving the desired ability of precise mesh extraction, and a tile-based differentiable tetrahedron rasterizer. Furthermore, we incorporate eikonal and normal consistency regularization terms for the signed distance field to improve
Li Zhang (lizhangfd@ fudan.edu.cn) is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generation quality and stability. Critically, our representation can be trained without mesh extraction, making the optimization process easier to converge. Our TeT- Splatting can be readily integrated in existing 3D generation pipelines, along with polygonal mesh for texture optimization. Extensive experiments show that our TeT- Splatring strikes superior tradeoff among convergence speed, render efficiency, and mesh quality as compared to previous alternatives under varying 3D generation settings.
1 Introduction
Table I: Comparison of different representations for 3D generation.
Representation
NcRF [28]
3DGS
DM1ct [40]
Tel TeT-Splatting (Ours)
Precise mesh extraction
Easy convergence
Real-time rendering
Representative method
Dream Pusion [32].
DreamGaussion [46]. Fantasia B3.
Ours
MD [18]
GSGEN [5]
RichDreamer [34]
Automatic 3D content generation is revolutionizing fields such as virtual reality, augmented reality, video games, and industrial design. This technology significantly enhance user experiences and streamline creative processes for reducing time demands and simplifying the complexities associated with creating high-quality 3D assets.
3D representations (e.g., Neural Radiance Field [28] (NeRF)) play an essential role in recent advance- ments in 3D generation, along with the Score Distillation Sampling (SDS) technique objective [32] for exploiting off-the-shelf 2D diffusion models [11, 37, 38, 1]. Although serving as a pioneer representation, NeRF is significantly limited due to its intensive computational demands, particularly when paired with high-resolution 2D diffusion models. Moreover, its density- based volumetric rendering struggles with accurate mesh extraction, which is crucial for practical applications.
By utilizing a signed distance field and Marching Tetrahedra for differentiable mesh extraction, DMTet [40] enables efficient high-resolution rendering and precise mesh extraction, overcoming the limitations of the NeRF approach. cases, it becomes a favored choice [18, 33, 3]. However, DMTet is limited in its ability to handle large topological changes in meshes, as it can only backpropagate to the zero-level set of the signed distance field, constraining its geometry convergence during optimiza- tion. As workaround, two-stage 3D generation pipeline has been adopted that initially utilizes NeRF for rapid geometry convergence and then transitions to DMTet for detailed refinement [18]. However, transitioning from NeRF to DMTet often results in degradation of quality, the strengths of each representation are not fully leveraged throughout the entire optimization process.
Alternatively, recent methods have introduced 3D Gaussian Splatting [13] (3DGS) into the optimiza- tion process, significantly enhancing efficiency. For example, DreamGaussian [46] utilizes 3DGS but acknowledges that meshes directly generated from 3DGS can be blurry, and the mesh extraction process often results in unsatisfactory surfaces with visible holes [45]. Moreover, the text-to-3D process with 3DGS suffers from instability due to its unstructured nature and the densification process.
In this work, introduce TeT-Splatting, novel all-round 3D representation that integrates surface- based volumetric rendering into the tetrahedral grid, while preserving precise mesh extraction through Marching Tetrahedra. It supports easy convergence during optimization, precise mesh extraction. and real-time rendering simultaneossly. enabling high-fidelity 3D generation effectively (Figure l). Drawing inspiration from 3DGS [13], we design a tile-based fast differentiable rasterizer for real- time rendering, efficiently handling the alpha- blending of projected 2D splats from 3D tetrahedra. These splats are blended based on opacity values derived from the signed distance field within each tetrahedron as in NeuS [49]. To further increase efficiency, we include pre- filtering process to remove nearly transparent tetrahedra, reducing the number of tetrahedra necessary for splatting. Moreover, we introduce eikonal and normal consistency regularization terms to refine the signed distance field, which helps stabilize the optimization process and prevents the common issue of debris in the optimization with DMTet. In Table I we compare the features of different 3D representations.

---

Our contributions are fourfold: (i) Introducing a novel 3D representation, TeT-Splatting that inte- grates synergistically volumetric rendering into tetrahedral grid: (ii) Designing fast differentiable rasterizer for tetrahedra; (iii) Forming generic two-stage 3D generation pipeline that initially lever- ages TeT-Splatting for geometry optimization, and then transitions it to polygonal mesh for texturing; (iv) Extensive evaluations demonstrating the superior tradeoff of our method among easy convergence, real-time rendering, and precise mesh extraction over alternative representations (InstantNGP [29], DMTet [40], and 3DGS [13J) under a variety of settings with different diffusion priors.
2 Related work
3D representation Since the introduction of Neural Radiance Field (NeRF) [28], NeRF has become a foundational technique in the field of 3D reconstruction. It employs volumetric rendering to enable 3D optimization with only 2D supervision. Despite its significance, NeRF faces major issues, such as slow rendering speeds and high memory usage. To address these problems, several research[43, 39, 29, 2, 13] have developed novel variants of the radiance field, focusing faster training and rendering and using less computing resources. Diverging from the path of NeRF, DMTet [40] has introduced an approach based Marching Tetrahedra and surface rendering by differentiable rasterization [14], offering much faster rendering speed. Recently, 3D Gaussian Splatting [13) (3DGS) has unified NeRF-like alpha-blending with tile-based rasterization, achieving high performance in both quality and rendering speed. In this paper, our proposed TeT-Splating takes inspiration from the structured tetrahedral grid in DMTet [40] and incorporates tile-based rasterization from 3DGS [13], utilizing tetrahedra for splatting. TeT-Splatting achieves high converge and rendering speed while preserving precise mesh extraction through Marching Tetrahedra.
3D generation The data-driven 2D diffusion models [11, 37, 38, 1] have demonstrated unprece- dented success in image generation. However, the transition to direct 3D generation [31, 12, 10, 24, 8, 58, 6, 52, 19, 15, 45] faces formidable challenges, as this research line often fails to gener- ate high-quality 3D assets limited by the lack of training data. To circumvent these issues, some works [20, 41, 23, 42, 21, 25, 50] train 2D diffusion models to make them have 3D awareness. However, discrete and sparse 2D images still cannot offer sufficient 3D information. In this context, DreamFusion [32] first introduced score distillation sampling (SDS) loss to leverage 2D diffusion priors for 3D generation. Subsequent studies [47, 57, 59, 54, 51, 17, 26, 44, 48] have aimed to improve the SDS loss, enhancing both the fidelity and stability of 3D generation. Moreover, several efforts [55, 5, 56,4 4, 16, 42, 34, 22] have been made to improve the quality and multi-view consistency of 3D models by integrating wider array of diffusion priors. Despite these advancements, some methods are hindered by the significant computational demands due to the usage of NeRF [28], which limits the effective use of righ-resolution diffusion priors. Additionally, other mesh-based models [3, 16, 34] encounter issues with instability and slow convergence due to the nature of surface rendering. By contrast, our TeT-Splattin facilitates the use of high-resolution diffusion priors and ensures efficient updates, thanks to its volumetric rendering and tile-based differentiable rasterizer.
3 TeT-Splatting
3.1 Deformable tetrahedral grid
In this section, we will start with introduction the deformable tetrahedral grid, which is the geometric primitive for the proposed representation. The deformable tetrahedral grid is first employed in DefTet [7] and then extended in DMTet [40] to approximate the implicit surface by assigning each vertex an SDF value. Specifically, this structure considers a tetrahedral mesh composed of N vertices and K tetrahedra, denoted as (Vr, T', where V/- {v,, E 1.....N] signifies the positions of vertices, and T = [t 1..... K}. with each t; representing the indices (ax. bb, Ck. dy) of four vertices (Vaz. Ubu VeL vd,) ) that form a tetrahedron. Utilizing the SDF value associated with each vertex Un. denoted by fn. a signed distance field is established by interpolating the SDF values within each tetrahedron. DMTet [40] has developed a method for mesh extraction from the tetrahedral grid by assigning one or two triangles to each tetrahedron that intersects the zero-level set of the signed distance field, known as Marching Tetrahedra (MT). Employing a differentiable triangular rasterizer, it attains a remarkable rendering speed while maintaining minimal memory consumption.
3

---

Differentiable Tetrahedron Splatting
Geometry Optimization
Prompt
Pre-filter
TeT-Splatting
Sa
Tet grid
Normal/ Depth/ Opacity
Spren
Steee
Texture Refinement
Prompt
Sdt
MT
SDS
Rasterization
Normal
Depth
Opacity
Tet grid
Albodo PBR
TeT-Spllating
3D Generation
Figure 2: Left: An overview SeT-Splatting To produce the final renderings, we first pre-filter and remove nearly transparent tetrahedra, then project the remaining ones into 2D splats. These are blended based on opacity values derived from the SDF values at specific pixel intersections. Right: TeT-Splatting for 3D generation. We employ TeT-Splatting in the initial stage of the 3D generation pipeline and subsequently transition polygonal mesh for texture optimization.
However, a particular limitation of the MT is that only the parameters associated with tetrahedra intersecting the zero-level set of the signed distance field can be updated during optimization. This restriction poses challenges in managing large topological changes and often causes the optimization stuck in the undesired shape in the early stage. In contrast, NeRF is less affected by such instability thanks to its volumetric nature. Many prior works [18, 33] have employed NeRF in 3D generation. These works typically adopt two-stage pipeline that starts with the volumetric representation to swiftly achieve a coarse model with low-resolution diffusion priors and then transitions to polygonal mesh for further refinement with high-resolution diffusion priors. However, these approaches are often hindered by the slow optimization and inaccurate geometry brought by volume rendering. The inaccurate geometry would lead to obvious degradation after mesh extraction.
3.2 Differentiable tetrahedron splatting
In this work, we present unified representation that combines the precise mesh extraction via the tetrahedral grid and the efficient optimization of volumetric rendering. Inspired by 3D Gaussian Splatting [13] (3DGS), we also integrate the tile-based rasterizer into our framework facilitate real-time rendering. 3DGS enhances rendering efficiency through rasterization and ensures efficient optimization via alpha-blending by projecting 3D Gaussians 2D splats followed by fast alpha- blending. However, 3DGS relies on unstructured 3D Gaussians as rendering primitives, necessitating carefully designed densification processes and learning rates manage the highly noisy SDS loss. In contrast, the tetrahedral grid is structured while its vertices can only deform in local region and are connected with neighbors form tetrahedra. We explore treating tetrahedron rendering primitive of the splatting process perform alpha-blending. Moreover, we can directly extract polygonal mesh through Marching Tetrahedra from the tetrahedral grid, while the mesh extracted [46] from 3DGS may result in unsatisfactory surface with visible holes.
Next, we will elaborate on how realize differentiable tetrahedron splatting through alpha-biending. Consider a pixel on the image plane, along with its corresponding ray in 3D space. To perform alpha-blending, we need first determine the intersected tetrahedra between the ray and the tetrahedral grid. For single tetrahedron with vertices (Va, Ug Ve, va) and SDF values (fa, fo. fe. fa), we can project the vertices onto the image plane, resulting four overlapped triangles that form a 2D tetrahedron splat. Intersection with the tetrahedron is equal to the intersection with the four triangles. The position and SDF value an intersection point can calculated using barycentric coordinates (see Appendix A for details). Different from 3DGS [13], we consider the opacity of tetrahedra instead of Gaussians. Note that a ray can only have two intersection points with a tetrahedron, we denote their SDF values frcy and feeut in depth order. Then the opacity of the tetrahedron can be derived

---

DMTTe
TeT Spttttgtntigg
Tel:Speeting (MT)
iter
100 iter
200 iter
500 iter
1000 iter
1500 iter
2000 iter
2500 iter
3000 iter
Latent code
Optimization progress
Normal
Figure 3: Normal map comparison during optimization 3D generation. We utilize DMTet and TeT-Splatting as 3D representations in the geometry modeling stage of the RichDreamer [34]. The first two rows show normal maps obtained from DMTet and TeT-Splatting during optimization. TeTT-Slaating achieves more stable and smooth optimization, while DM'Te becomes fragmented initially and gets stuck in an undesirable shape. The third row shows the normal maps of meshes extracted from the signed distance field of TeT--Slaating via Marching Tetrahedra [40] (MT). As optimization progresses, TeT-Splatting's behavior aligns with rendering through MT.
in NeuS [49] manner:
a = max (Po(fpor)) & +s((Dexu) .0).
(I)
4, (pree)
where 4 (x) = (1 +1+6 (1+e= +----=--1 and the s value controls the steepness the conversion. Following Voxurf [53]. update manually for each iteration i: i/ Stati + Satat The final normal map N,
depth map D and opacity map O are derived by alpha- aiphabblending sequentially ordered tetrahedra
from front to back:
IN, D2 = Σ Toi 2i, 1;,1}, 1}, T. = II (1-0)) #=1
iEN
where n denotes the per-tetrahedron normal and z; denotes the average depths of four vertices. Pre-filtering To conserve computational resources, the tetrahedra with low opacity will be filtered. Depending on different intersection points, the opacity of tetrahedron can take different values. We can establish the upper bound Eq. with the maximum and minimum SDF values of four vertices. Tetrahedra with Omar less than a predefined threshold T₁ 255 are filtered to ensure that only tetrahedra with significant enough contribution to the alpha- blendbbending are included in the subsequent splatting process.
opacity, denoted as Ommaa by replacing Sprea and Spent in
Per-tetrahedron normal As discussed in Section 3.1, the tetrahedral grid establishes a signed distance field by interpolating the SDF values within each tetrahedron. This interpolation is linear combination of the SDF values of four vertices. Correspondingly. the barycentric coordinates of an arbitrary point with respect to the four vertices of the tetrahedron exhibit linear correlation with its spatial position. This ensures that the gradient g of the SDF within the tetrahedron results in constant vector (see Appendix A for details). The normal vector of the tetrahedron is thus obtained by normalizing this gradient.
Relationship between DMTet and TeT-Splatting During optimization, DMTet employs Marching Tetrahedra extract polygonal mesh from the tetrahedral grid and subsequently renders through triangular rasterization [14]. Consequently, only limited number of tetrahedra are involved in each single rendering process. In contrast, TeT-Splatting employs volumetric rendering, which allows all visible tetrahedra within the view frustum that have sufficient weight in alpha- bigha-bbeendng to contribute to the final renderings. Moreover, the rendering process Te7-Splatting is fully differentiable, enabling single optimization step to influence a significantly larger number of parameters compared to DMTet. Figure presents a comparative analysis convergence speeds between DMTet and TeT-Splatting within the same Text-to-3D pipeline. As observed, TeT-Splatling achieves rapid convergence, whereas DMTet exhibits slower topological changes and gets stuck an undesirable shape. Furthermore, the inverse standard deviation in Eq. increases, the curve of 4, becomes steeper, causing
5

---

to approach under conditions that Sprev 0 and Speat < 0 and to approach O otherwise. This behavior (Figure 3) aligns with the rendering process DMTet [40] through Marching Tetrahedra, where only the tetrahedra intersecting the zero-level set of the signed distance field are visible.
3.3 Fast differentiable rasterizer for tetrahedra
We implement tile-based differentiable rasterizer for tetrahedra with custom CUDA kernel building upon the framework of 3DGS [13]. Similarly, we begin by dividing the screen into tiles and culling tetrahedra that do not overlap with the view frustum. We then replicate the tetrahedra based on the number of tiles they overlap and sort them by their tile and the average depth of each tetrahedron's vertices, using a fast GPU radix sort [27]. Note that the per-tile sorting in 3DGS is not equivalent to per-pixel ordering. Differently, we maintain short resorting window [36] of size N for each pixel to re-sort the primitives based on the results of per-tile sorting using the insertion sort. Due to the structured nature of the tetrahedral grid, we find that the sorting error is almost eliminated with a window size of 5 under grid resolution of 256. The operations after re-sorting for alpha-blending are the same as in 3DGS, except for the computation a, which we have already given in Eq. 1.
4 3D generation with TeT-Splatting
In this section, we introduce our 3D generation pipeline and discuss various settings, aiming at validating the effectiveness of TeT-Splatting in 3D generation. As shown in Figure our pipeline is divided into two stages: first get detailed geometry with TeT-Splatring and then transition to polygonal mesh through Marching Tetrahedra [40] for texture optimization. We begin by describing our overall 3D model (Section 4.1), and then detail the regularizations (Section 4.2) and diffusion priors (Section 4.3) used in our experiments.
4.1 3D modeling
Geometry stage We employ hash grid [29] Pg with parameter Og to encode the signed distance field and deformation which allows each vertex in tetrahedral grid to deform in certain range. 4 is initialized to a spherical shape. Given randomly sampled camera, our tetrahedron rasterizer produces renderings of the normal map, depth map, and opacity map.
Texture stage Given the well-optimized signed distance field from the geometry stage, we convert it
into a polygonal mesh through Marching Tetrahedra. To texture the polygonal mesh we employ the physically based rendering (PBR) pipeline proposed by Nvdiffree [30]. Please refer to [30, 34] for details. We use another hash grid 4, with parameter O, to encode the spatially varying materials of the surface: albedo, roughness, metallic, and bump. Finally, given specific environment lighting and randomly sampled camera, we can obtain the renderings of the albedo map and PBR map.
4.2 Regularization
Eikonal loss To ensure a proper signed distance field, we employ an eikonal term that regularizes the SDF gradient g in each tetrahedron: Lek = Σx (l1g 1)².
Normal consistency loss Inspired by the normal consistency loss for triangle meshes, adapt this approach to tetrahedra. While we have designed per-tetrahedron normal, we project these tetrahedral normals onto vertices and enhance the consistency of the signed distance field by regularizing the cosine similarity between normals of adjacent vertices connected by edges: Lox -; (1 cos (Ne,1 M); where ex and ez represent the vertices forming edge es.
4.3 Diffusion priors
To validate the capability of TeT-Splatting for 3D generation, employ two types of diffusion priors: the vanilla RGB-based diffusion priors and the rich diffusion priors proposed in RichDreamer [34].
Vanilla RGB-based diffusion priors Vanilla RGB-based diffusion models represent diffu- sion models that can generate RGB images from a given prompt. For both geometry and tex- ture stages, utilize SDS loss to leverage 2D diffusion priors from Stable Diffussin [37]:

---

in Ice orean suntaa"
DSLR
stndk of panaakes
K photo
hour
38 min
15 min
min
min
2 min
min Ours 133 FPS
Input
Magic3D 13 FPS
Fantasia3 166 FPS
DreamGaussian 145 FPS
Figure 4: Qualitative comparison 3D generation using vanilla RGB-based diffusion priors. We present visual comparisons of the rendered RGB maps and color maps from various 3D generation methods. The methods, arranged from left right, are: Magic3D, Fantasia3D, DreamGaussian, and Ours. The comparison conducted across two tasks, text-to-3D and image-to-3D. with results shown from top bottom, respectively. Additionally, for each method, provide the training time and the rendering speed (FPS) for the first stage of the process.
VeLsDS E [c(t)(cs((:y))) c) 87 0z where w(t) weighting function, 2 denotes the VAE latent code, and (NV E) represents the noise estimated by the UNet cg-
Rich diffusion priors The sole use of vanilla diffusion priors often leads to issues such as multi-face Janus problem, domain gap between image diffusion model and normal map while using normal maps as input of diffusion models, and inaccuracies in material decomposition. To this end, we utilize the rich diffusion priors proposed in RichDreamer [34] to handle high-fidelit, 3D generation. Specifically, for geometry optimization, we combine a vanilla Stable Diffusion with a Normal-Deph diffusion model, which generates multi-view normal and depth maps from given text prompt, represented as: LSDS LSD Nammaa + CND ND- For texture optimization, we combine a vanilla SI) with Depth-conditioned Albedo diffusion model, capable of producing multi-view albedo maps CSD from a given text prompt, represented as: LSDS = LSDS-RGE RGB + LSDS Albeda'
CAlbedo
In summary, the final loss function for the geometry stage is defined as: Lgoo LSDS + AeikCaik AncCne. For the texture stage, the loss function simplifies to: Lie = LSDS-
Experiment
In this section, we assess the efficacy of TeT-Splatting across two distinct tasks: 3D generation employing vanilla RGB-based diffusion priors and text-to-3D with rich diffusion priors. In Section 5.1, a qualitative evaluation to demonstrate the superiority of TeT-Splatting relative to other representations. To substantiate TeT. Splatting's proficiency in handling high-fidelity generations, we conduct experiments with advanced rich diffusion priors in Section 5.2. Section 5.3 encompasses series of ablation studies aimed at validating the representation and pipeline. The details of inplementation and experimental setting can be found in the Appendix B.
generation for both text-to-3D and image-to-3D modalities conducted
7

---

5.1 Results with vanilla RGB-based diffusion priors
Before
After
Before
After
Before
After
Input
Magic3D
DreamGaussian
Ours
Figure 5: Visualization of normal maps before and after mesh exportation. Note that the normal maps of DreamGaussian [46] are derived from its depth maps.
Focusing on illustrating the effectiveness of the proposed representation, we primarily compare to three competitors: Magic3D [18], Fantasia3D [3] and DreamGaussian [46]. All these competitors employ a two-stage optimization pipeline and leverage Stable Diffusion with SDS loss, but utilize three different representative 3D representations in their initial stages: Instant-NGP [29] (a fast version of NeRF), DMTet [40], and 3DGS [13], respectively. We adapt the diffusion priors of all methods to Stable Zero-1-to-3 [20] for fair comparison in the image-to-3D task, adding identical MSE alignment loss. The qualitative evaluations, as shown in Figure 4, illustrate our method's ability generate more detailed and compact meshes in relatively short time. In Figure report the rendering speed (FPS) of the first stage, at rendering resolution of 512x512. Although TeT-Splatting operates a lower FPS compared DMTer (Fantasia3D) and 3DGS (DreamGaussian), it still achieves real-time rendering. Importantly, this lower FPS does not adversely affect the overall generation process, as the primary bottleneck generation speed lies with the diffusion model.
also
We also conduct a comparison of the mesh extraction with Magic3D and DreamGaussian, visualized in Figure 5. It reveals that the meshes extracted from Magic3I often do not faithfully replicate the geometries from the first stage due to the imprecise threshold for converting densities to SDF values, and the extracted mesh in Dream contrast, our method maintains high quality with negligible degradation after mesh extraction.
result in unsatisfactory surfaces with visible holes. In
5.2 Results with rich diffusion priors
"a DSLR photo of poreelain dragon'
ta DSLR photo cup full of pens and pencils"
photo of turtle standing hind logs, wearing top hat"
11 hours PromifieDreamer
hour MVDream
2 hours
1.5 hours
RichDreamer
Ours
Figure 6: Qualitative comparison on Text-to-3D with rich diffusion priors. We also report the total training time of each method.

---

DMTel
TeT TeT-Splatiig
Tametuls
Typowriter
Fagle
Tamntuls
Typourierr
Figure 7: Normal map comparison between DMTet [40] and TeT-Splatting in the early training iterations.
DSLR photo of a barbecue grill cookingsausages and burger patties"
"a DSLR photo of a barbecue grill cookingsausages and burger patties"
Figure 8: Visualization of the rendered normal, albedo, and PBR map from the generated 3D assets in the second stage.
Our approach is also compatible with state-of-the-art diffusion priors. In this part, evaluate TeT--plattinn on the text-to-3D task, equipped with rich diffusion priors from RichDreamer [34]. key distinction between
approach and RichDreamer is the of TeT-Splatting the 3D representation during the geometry stage, We evaluate our method against two SOTA competitors, ProlificDreamer [5]] and RichDreamer [34]. As illustrated Figure our method is capable of handling high- figh-fidelity 3D generation, achieving superior geometric quality with considerably reduced generation times compared these competitors.
Additionally, we present visualizations of the normal maps from early training iterations in Figure 7. The results from RichDreamer are fragmented at the early iterations due the of DMTet, which may harm subsequent optimization and slow convergence. In contrast, TeT-Splatting demonstrates rapid and smooth convergence. We employ the same quantitative evaluation method as RichDreamer to assess the quality of geometry and texture.
In Table 2, report the Geometry CLIP [35] score and Appearance CLIP score. Notably, Rich- Dreamer's prompt list, comprising 113 objects used for scoring, is not publicly available. Conse- quently, we calculate our scores using alternative set of prompts (see Appendix B for details). Our method outperforms RichDreamer in terms of CLIP scores and significantly reduces the time required for geometry optimization (40 min vs 70 min).
In Figure we present the decomposed albedo maps of generated 3D assets. Guided by the Depth-conditioned Albedo diffusion model, we achieve natural albedo maps.
5.3 Ablation
Under the settings of rich diffusion priors [34], conduct ablation studies to evaluate our method.

---

wio Eikonal Eikonal
wio consistency
Resolution
(a) Eikonal loss
(b) Normal consistency loss
(c) Grid resolution
Figure 9: Ablation studies eikonal loss, normal consisteney loss and the resolution of the tetrahedral grid.
Table 2: CLIP score comparison. Results marked with "*" are taken from RichDreamer [34]. Since RichDreamer [34] did not release their prompt list (1 objects), we use our own prompt list (183 objects) for evaluation. See Appendix B for more details.
Prolitidrreamer [ST]
MVDream [42]
RichDreamer [34]
RichDreamer
Ours
Geometry CLIP score"
23. 2
24.8003
25.3820
23.0143
23.1641
Appearance CLIP score
31,8022
28.7331
31.7099
29.2198
29.4197
Eikonal loss We assess the role of eikonal loss in 3D generation by comparing 3D assets generated with and without it, illustrated in Figure 9a. Models created without cikonal loss tend to develop into undesirable shapes. This issue arises because the SDF values rapidly reach extreme levels and get trapped in local minima when eikonal loss is not applied.
Normal consistency loss Additionally, assess the importance of normal consistency loss. Fig- ure 9b demonstrates that applying normal consistency loss results in more compact models. This loss smoothing prior that helps prevent the surface of the model from becoming fragmented.
Tetrahedral grid resolution We investigate the effects of tetrahedral grid resolution on model performance by conducting experiments at resolutions of 128 and 256. Higher resolution yields more detailed geometries, as shown in Figure 9c.
6 Limitations
TeT-Splatting struggles with modeling figh-frequency features, such texture, because it uses tetrahedra as rendering primitives, which limits the final output by the resolution the tetrahedral grid. Therefore, transition it to polygonal mesh for enhanced texture optimization. The rendering speed of our implemented rasterizer, although operating in real-time, is slower than that of 3DGS. Additionally, using only pre-filter operation might not fully leverage TeT--Slaatinn's potential in rendering quality and speed. A similar densification process as in 3DGS could improve this, which leave for future work.
7 Conclusion
In this study, we introduce Tetrahedron Splatting (TéT-Splatting), a novel all-round 3D representation that integrates volumetric rendering within a structured tetrahedral grid while preserving precise mesh extraction through Marching Tetrahedra. Equipped with newly designed tile-based fast differ- entiable tetrahedron rasterizer, TeT-Splatting achieves real-time rendering. As showcase, we integrate TeT-Splatting in common 3D generation pipeline with polygonal mesh for texture optimization. Extensive experiments under varying 3D generation settings demonstrate Te7-Splattin''s superiion superiority in producing high-fidelity 3D content compared to other 3D representations.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China (Grant No. 62 126000 and 62376060) and Natural Science Foundation of Shanghai (Grant No. 22ZR 1405500).
10

---

References
[1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Tmxae diffusion models with an ensemble of expert denoisers. arXiv preprint, 2022. 2.3
[2] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022. 3
[3] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023, 3, 15
[4] Yang Chen, Yingwei Pan, Yehao Li, Ting Yao, and Tao Mei. Control3d: Towards controllable text-to-3d generation. In ACM MM, 2023.
[5] Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. CVPR, 2024. 3,
[6] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In CVPR, 2023. 3
[7] Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. NeurlPS, 2020.
[8] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In NeurlPS, 2022. 3
[9] Yuan-Chen Guo, Ying-Tiat Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song- Hai Zhang. threestudio: A uni- fied framework for 3d content generation. https: /ttp://gitttt com/threestudle -tw.eeessttooo-ooeeettt. threestudio, 2023. 6, 15
[10] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint, 2023. 3
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 2,
[12] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint, 2023.
[13] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023. 2, 3, 4, 6, 8, 15
[14] Samuli Laine, Janne Hellsten, Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM TOG, 2020. 3,5
[15] Ming Li, Pan Zhou, Jiawei Liu, Jussi Keppo, Min Lin, Shuicheng Yan, and Xiangyu Xu. Instant3d: Instant text-to-3d generation. arXiv preprint, 2023. 3
[16] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in
2d diffusion for consistent text-to-3d. In ICLR, 2024.
[17] Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching. In CVPR, 2024. 3
[18] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023. 2, 4, 8, 15
[19] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45; Any single image to 3d mesh in 45 seconds without per-shape optimization. In NeurlPS, 2024. 3
[20] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image 3d object. In ICCV, 2023. 3, 8
[21] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Learning to generate multiview-consistent images from single-view image. In CVPR, 2024. 3
11

---

[22] Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for relightable text-to-3d generation. arXiv preprint, 2023. 3
[23] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint, 2023. 3
[24] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp. Tsung-Y Yi Lin, Ming-Yi Yu Liu, Sanja Fidler, and James Lucas, Att3d: Amortized text-to-3d object synthesis. In ICCV, 2023. 3
[25] Yuanxun Lu, Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Xun Cao, and Yao Yao. Direct2. 5: Diverse text-to-3d generation via multi-view 2.5 diffusion. arXiv preprint, 2023. 3
[26] Yiwei Ma, Yijun Fan, Jiayi Ji, Haowei Wang, Xiaoshuai Sun, Guannan Jiang, Annan Shu, and Rongrong Ji. X-dreamer: Creating high-quality 3d content by bridging the domain gap between text-to-2d 2d and text-to-3d 3d generation. arXiv preprint, 2023.
[27] Duane G Merrill and Andrew S Grimshaw. Revisiting sorting for gpgpu stream architectures.
In ICPAC, 2010. 6
[28] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 2, 3
[29] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM TOG, 2022. 3, 6,
[30] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In CVPR, 2022. 6
[31] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint, 2022. 3
[32] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. In ICLR, 2023. 2,
[33] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic 123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR, 2023. 2,
[34] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. arXiv preprint, 2023. 2, 3, 6, 7, 10, 16
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML,
2021. 9, 16
[36] Lukas Radl, Michael Steiner, Mathias Parger, Alexander Weinrauch, Bernhard Kerbl, and Markus Steinberger. Stopthepop: Sorted gaussian splatting for view-consistent real-time rendering. arXiv preprint, 2024. 6
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 2, 3,
[38] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022. 2,
[39] Sara Fridovich-Keit and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022. 3
[40] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. NeurlPS, 2021. 2,
3, 6. 8, 9, 16
12

---

[41] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu. Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image consistent multi-view diffusion base model. arXiv preprint, 2023. 3
[42] Yichun Shi, Peng Wang. Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. In ICLR, 2024. 3, 10
[43] Cheng Sun, Min Sun, and Hwann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022. 3
[44] Boshi Tang. Jianan Wang, Zhiyong Wu, and Lei Zhang. Stable score distillation for high-quality 3d generation. arXiv preprint, 2023. 3
[45] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint, 2024.
2,
[46] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024. 2, 4, 8
[47] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023. 3
[48] Peihao Wang, Zhiwen Fan, Dejia Xu, Dilin Wang, Sreyas Mohan, Forrest landola, Rakesh Ranjan, Yilei Li. Qiang Liu, Zhangyang Wang, et al. Steindreamer: Variance reduction for text-to-3d score distillation via stein identity. arXiv preprint, 2023. 3
[49] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-viww reconstruction. arXiv preprint, 2021. 5.5
[50] Peng Wang and Yichun Shi. Imagedream: Imagc-prompt mult diffusion for 3d generation.
arXiv preprint, 2023. 3
[51] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.
In NeurIPS, 2024. 3, 9, 10
[52] Chao- Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3D reconstruction. In CVPR, 2023. 3
[53] Tong Wu, Jiaqi Wang, Xingang Pan, Xudong Xu, Christian Theobalt, Ziwei Liu, and Dahua Lin. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. In ICI.R, 2023.
[54] Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, and Hanwang Zhang. Consistent3d: Towards consistent fideliiylity text-to-3d generation with deterministic sampling prior. In ICLR, 2024.
3
[55] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffussin models. In CVPR, 2023. 3
[56) Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. CVPR, 2024.
[57] Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, and Xiaojuan Qi. Text-to-3d with classifier score distillation. arXiv preprint, 2023.
[58] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeie, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In NeurIPS, 2022. 3
[59] Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo. Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance. In ICLR, 2023.
A More implementation details of TeT-Splatting
In this section, we provide additional implementation details about the tetrahedron splatting process.
13

---

A.1 Barycentric coordinates
Recall that projecting a single tetrahedron, encompassing four vertices Va. vb, Ve: Vd RR' along with their SDF values fa: fo. fe, fa I. onto 2D plane results in an array of 2D vectors Vy v/y v v'g I2a representing the coordinates these vertices on the image plane.
Given pixel p with coordinate v/ R2, its barycentric coordinates u',v' v' E IR with repsect to triangle (w' t/y v'2) satisfy V/2 (1 v' v')o's + u'vl, + v'v6.. We can apply the following equations to derive the barycentric coordinates u', n/:
u
v
u'
p
(3)
1
1
v
1
u
v
v/2 1
u
(4)
If w/, r/ E [0, 1], the pixel p is considered inside the triangle.
A.2 Perspective correction
The calculated barycentric coordinates w', v' are based on 2D projections and require adjustment to reflect the original 3D spatial relationships accurately. This adjustment, known as perspective correction, necessary because 3D depth information is preserved in the 2D projection. We perform this correction using:
(5)
1
[1 1-
Za
Ze
where zu denotes the depth of each vertex. Subsequently, the SDF value and depth of the 3D position corresponding to this pixel are interpolated:
fp = (1 (1- v)fa ust + ufe,
2p
(1- 10-0'''1
8a
20
A.3 Gradient of the SDF value inside a tetrahedron
Consider arbitrary 3D point Vg with SDF value fg inside the tetrahedron (Va, vo, Vc, va). We establish the SDF value f, using the 3D barycentric coordinates u, v, wi fq = (1 (1-10 v - w)fa ufb + vfe w fa- The derivation of u, v, similar 2D case in Section A.
Va
Ub
Uc
Vd
U
Vy
1
W
1
u
Va
Ub
U2
Vd
V0g
(8)
1
W
Therefore, the formula of f, can be expressed by:
11
fq= fa fa fo fo fa
21
(9)
W
14

---

[a fa fb. fo fa
Va
Vb
Ve
Vd
(10)
1
1
Moreover, the gradient of the SDF value Vg. denoted by g. is straightforwardly derived by differentiating with respect to Vn- This gradient is constant across the tetrahedron:
Vo
fa Je
g
97b
(1I)
This results in constant vector within any tetrahedron, providing consistent gradient that aids in precise mesh extractions and surface optimizations.
B More implementation details of 3D generation
this section, we provide additional implementation details of 3D generation. Note that experi- ments are conducted one NVIDIA RTX A6000 GPU.
B.1 Geometry stage
Unlike 3DGS [13], we separate operations that are repeated across multiple images from different camera viewpoints within a single training iteration and shift them to the beginning of each iteration, including the inference of SDF values and deformations for each vertex from the hash grid, pre- filtering tetrahedra based on their 0maz and calculating the per-tetrahedron normal. These pre- processed results are then passed to the rasterizer for rendering batch of images. Moreover, we implement coarse-to-hi
approach the pre-filtering process: initially, we establish a tighter axis-aligned bounding box from the pre-filtered tetrahedral grid in the first round and then resize the tetrahedral grid based on this bounding box for a second round of pre-filtering which enhances the precision of the geometry. For the schedule of the value in Eq. 1, we set Smatio 5 and Sstart 20, which allows the curve of Φ (x) to be sufficiently steep at the final optimization. Additionally, we set both Ask and Ane to 1000.
B.2 Evaluation with vanilla RGB-based diffusion priors
This part is implemented based on the threestudio codebase [9] using the settings of Fantasia3D [3]. The tetrahedral grid resolution set to 128, and the batch size is set 1.
Text-to-3D For the text-to-3D task, for 3,000 iterations and the texture for another 1,000 iterations. Following Fantasia3D) [3], during the concatenate the rendered normal and depth maps
Stable Diffusson 2.1 base. The geometry is optimized
initial training iterations, latent code for the diffusion models, facilitating rapid convergence to basic shape. As Magic3D [18] the implementation from threestudio. To ensure a fair comparison
directly the
haven't t released their code, with Fantasia3D [3], we also adopt the threestudio implementation.
Image-to-3D In the image-to-3D task, we use Stable Zero- 1-to-1 and adapt our pipeline by incor- porating &, in the initial stage to encode the materials at the center of each tetrahedron. Specifically, we start by inferring the materials at center of each tetrahedron. Next, we compute the per- tetrahedron PBR color using the rendering equation for each tetrahedron. This per-tetrahedron PBR color e is subsequently passed to the rasterizer to perform the Hipha-blending for the final output: C ΣiEN C, To further refine the output, introduce MSE loss that aligns the rendering color map C and opacity map Or at reference view with the provided reference image &r and mask Ör: fref A CrI 1/ + Ammss Ór 12g We set the loss weights Arb and Amask to 10,000 and 1,000, respectively. Also, decrease Agk and Statio to 100 and 2, respectively.
15

---

B.3 Evaluation with rich diffusion priors
This part implemented based on the RichDreamer [34] codebase using the settings of DMTet [40]. The tetrahedral grid resolution is set to 256 and the batch size is set to 4 for two stages. We optimize the geometry for 3,000 steps. with the first 1,000 steps using latent code, followed by an additional 2,000 steps for texture optimization. While RichDreamer reports significantly increased stability at rendering resolution of 1024, we achieve stable results at lower resolution of 512. Therefore, we set. our rendering resolution to 512.
CLIP score We adopt the evaluation process in RichDreamer [34] calculate CLIP scores using the CLIP model [35] (vit-g-14). For geometry CLIP scores, we render generated meshes with uniform albedo and produce 16 different views for each object. The average CLIP scores are computed by discarding the highest and lowest scores from the provided text prompts. For texture CLIP scores, textured meshes are rendered. Since RichDreamer has not released the prompt list (113 objects) used for their metrics, utilize alternative list named "prompts_dmtee"" dxtet. (183 objects) available on their official GitHub repository.
C More results
We present extensive gallery of visual results in Figure 10-13.
D Discussions on the potential social impacts
The proposed TeT-Splatting method can make it easier for people to enter the animation and related industries. This can simplify production processes, reduce costs, and allow more people create high-quality 3D assets. However, these improvements could also lead to job losses for professionals who work in traditional roles. To address this, may be necessary gradually adjust training programs and align the workforce with future demands.
Additionally, while our method improves the efficiency of 3D generation, it also carries the biases present in the foundational models we use. These models can have built-in biases related to race, gender, and culture, which might appear in the generated content, reinforcing stereotypes. The casier creation of realistic 3D models also raises concerns about copyright infringement and misuse, highlighting the need for strong ethical guidelines and regulatory frameworks.
16

---

zoomed DSLR photo baby bunny sitting on top of a stack pancakes"
"a DSLR photo of bear dancing ballet"
DSLR photo of a bear dressed in medieval armorr
"a beautiful dress made out of garbage bags, mannequin. Studio lighting, high quality, high resolution"
"a zoomed out DSLR photo of beautifully carved wooden knight chess piece"
"a zoomed DSLR photo of a blue tulip"
brightly culored mushroum growing on a log"
DSLR photo of cake covered in colorful frosting with a slice being taken out, high resolution"
a capybara wcaring a top hat, low poly"
"a DSLR photo of a cauldron full of gold coins"
"a completely destroyed car"
crocodile playing a drum set"
Figure 10: More results of TeT- TeT-SSllatiig with rich diffusion priors.
17

---

DSLR photo of a delicious croissant"
delicious hamburger"
DSLR photo of Mont Saint-Michel, France, acrial view" "
"a ficus planted pots"
"a zoomed out DSLR photo of fresh cinnamon roll covered in glaze"
"a DSLR photo of giant worm emerging from the sand in the middle of the desert"
"a hotdog in a tutu skirt"
"a llama wearing suit"
'a DSLR photo of mug of hut chocolate with whipped cream and marshmallows"
"a DSLR photo of a pair of tan cowboy bouts, studio lighting, product photography'
"a palm tree, poly model"
"a DSLR photo of plate fried chicken waflles with maple syrup on them"
Figure 11: More results of TeT- TeT-Splatting with rich diffusion priors.
18

---

a DSLR photo of a model of the eiffel tower made out of
"a DSLR photo of a plush t-rex dinosaur toy, studio lighting, high resolution"
toothpicks"
"a DSLR photo of red cardinal bird singing"
a zoomeed out DSLR photo of a red rolary lelephune"
"a sliced loaf of fresh bread"
"a squireel dressed like Ienry VIII king of England"
a squirrel dressed like Victorian woman"
"a DSLR photo of stack of pancakes covered in maple
DSLR photo of basket with cobra coming of it"
"a tiger karate master"
"a zoomed DSL.R photo of tiger wearing sunglasses and leather jacket, riding motorcycle"
"a DSLR photo toilet made of gold"
Figure 12: More results of TeT-Splatting with rich diffusion priors.
19

---

"a DSLR photo a train engine made out of clay"
"a DSLR photo of very cool and trendy pair of sncakers,
studio lighting"
"an amigurumi bulldozer"
"a DSLRR photo astronaut standing on surface of
"a DSLR photo an old car overgrown by vines and weeds"
"an opulent couch from the palace of Versailles" "
origami motorcycle"
"a DSLR photo an ornate silver gravy boat sitting on patterned tablecloth"
'a DSLR photo of an unstable rock cair in the middle of
"banana"
"Coffee with many hules"
"a zoomed out DSLR photo of miniature schnauzer wooden sculpture, high quality studio photo"
Figure 13: More results of TeT-Splatting with rich diffuuson priors.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The motivation and contributions of this study are clearly stated in the intro- duction, with brief overview of them provided in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of this work in Appendix 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
Our main theoretical result is the gradient of SDF inside a tetrahedron, which is proved in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provided the detailed design and experimental setting in paper.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: Our results do not rely on any private data. We provide a link to our project page, which contains the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/ Soweess//ooSSumiiiiinnene
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httpps.ccppubiic//eec gomdeeeSSS ssi wwisnnolicyyiooeoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details are provided in Appendix and B. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report statistically significant following the common practice in this field.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We report the generation time in Section 5 and the used GPU resources in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the httpppbbic//hhccsuuuiieeeeeee
NeurlPS Code of Ethics https:
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code Ethics and ensure that our research strictly conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential societal impacts are discussed in Appendix D.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper of the used assets.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "8tGyP1EwSv", "reviewer": "Reviewer_39Vr", "summary": "This submission is proposing a new representation for 3D content generation combining the three following benefits:\n- easy to optimize,\n- real-time rendering,\n- allowing the extraction of precise meshes.\nRecent advances in novel view synthesis, i.e. NeRF and then 3D Gaussian Splatting (3DGS) are eventually bringing two of these benefits but the lack of quality output meshes still limits a number of applications.\n\nThe proposed approach follows the deformable tetrahedral grid approach of DMTet while using it instead for volume rendering and making it suitable for splatting as in 3DGS. Meshes can be easily extracted which facilitates chaining a first geometric optimization stage based on splat rendering with a later mesh-based texturing stage for complete 3D generation of geometry and appearance.\n\nThe paper makes the following contributions:\n- using a deformable tetrahedral grid rendered via tetrahedra splatting as a new 3D representation,\n- adapting 3DGS to work with such new representation with the implementation of a fast differentiable rasterizer for tetrahedra-based splatting,\n- leveraging this representation and optimization in a two-stage 3D generation pipeline, first to optimize tetrahedra geometry before optimizing mesh texturing,\n- An array of qualitative evaluations against other generative baselines (with some quantitative evaluations), either on image-to-3D or text-to-3D tasks using vanilla RGB diffusion priors or with rich diffusion priors.", "questions": "- To address the above weakness, do the authors have some measures of surface quality to provide some form of quantitative evaluation against baselines?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes to tackle a known shortcoming of 3D generative approaches when leveraging 3D representation from modern novel view synthesis techniques: poor meshed outputs. Identifying a suitable representation that is easy to optimize, enables real-time rendering while allowing mesh extraction is very relevant to the research community, especially to better bridge 3D and generative work.\n\n- The presented approach elegantly solves the issue of DMNet where only part of the representation (near the zero level set) can effectively be updated. This is addressed by turning the optimization into a full volumetric rendering of tetrahedra via an adaptation of 3DGS to different primitives.\n\n- The qualitative evaluations using vanilla RGB-based diffusion priors against the selected comparable techniques: Magic3D, Fantasia3D and DreamGaussian but based on NeRF, DMTet or 3DGS respectively do a rather convincing job to demonstrate the superiority and versatility of the proposed approach (Figure 4) that maintains high mesh quality with real-time rendering speed. Similarly the qualitative results with rich diffusion priors show pretty high surface quality with a reasonable training time,", "weaknesses": "- Apart from the CLIP scores of Table 2 (or the training time and rendering speed), the shown results are essentially qualitative. For a stronger comparison against other baselines, it would have been more helpful to try to integrate measures of the quality of the generated surface to demonstrate the benefits of the approach (instead of relying only on normal maps and renders). Also note that the presentation of these qualitative results could be improved by zooming on specific areas with artifacts (or lack or thereof) to better highlight them and guide the reader.\n\n- It seems there is no information on how the tetrahedral grid is initialized (besides its spatial resolution and Figure 3 hinting at a sphere).\n\n- Some suggestions for improvements:\n  - l. 107-109: the formulation make it sound like DMTet invented the established marching tetrahedra algorithm - rephrase\n  - Figure 2: Geometry Opimization -> Geometry Optimization\n  - Figures 4 and 6: it would be helpful to match the orientation for normals across objects (i.e. have consistent coloring for the whole figure)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer 39Vr**\n\nThank you for your valuable feedback and constructive suggestions. We've addressed your concerns in the revised manuscript and our response below.\n\n**Q1: Measures of surface quality.**\n\nWe thank the reviewer for raising this question. The best results from our method are achieved using the pipeline with rich diffusion priors in the text-to-3D task. Since there is no image input as guidance, we cannot evaluate the quality of the generated assets with metrics such as PSNR or Chamfer distance. Therefore, following the previous work (RichDreamer[1]), we report the geometry CLIP score and appearance CLIP score. As explained in lines 509-515, the geometry CLIP score is calculated by rendering the extracted meshes with a uniform albedo from 16 different views for each object. We believe this metric is sufficient to measure the surface quality of the extracted mesh and its alignment with the input text. Additionally, we have provided more viewpoints of the generated assets shown in the teaser in the attached PDF (Figure 1). Please refer to it for more details.\n\n**Q2: Presentation of these qualitative results**\n\nThank you for your advice! We will highlight and zoom in on specific areas in the revised manuscript to better view experience and improve the presentation of qualitative results.\n\n**Q3: Initialization of tetrahedral grid.**\n\nAs explained in line 198, we initialize the geometry as a sphere. This process involves fitting the signed distance field (SDF) to a sphere, following Fantasia3D[2]. Specifically, we employ the implementation from threestudio[3], where points are sampled in close proximity to the predefined sphere and the ground truth signed distance values are computed to optimize the SDF. Initializing the geometry as a sphere is a common practice in 3D generation methods using Score Distillation Sampling (SDS) loss, as it helps stabilize the generation process.\n\n**Q4: Some suggestions for improvements.**\n\nThank you for the suggestions! We will revise the paper to improve the writing quality. Also, we will unify the orientation of the normal maps across objects in Figures for consistent visualization.\n\n[1] Qiu, Lingteng, et al. \"Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Chen, Rui, et al. \"Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.\" In \\textit{ICCV}. 2023.\n\n[3] Liu, Ying-Tian, et al. \"Threestudio: A modular framework for diffusion-guided 3d generation.\" In \\textit{ICCV}, 2023.\n\nWe hope our responses address your queries. If you have any further concerns, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "erMEGLKtzl", "reviewer": "Reviewer_7DJZ", "summary": "The paper presents a new differentiable 3D shape representation based on DMTet. It borrows the efficient rasterization-based rendering techniques from Gaussian Splatting to improve the global optimization of DMTet. The new TeT splatting is shown to perform well in the task of text-to-3D.", "questions": "-  How to ensure the sharpness of texture obtained without extracting mesh during optimization?\n\n- Can the proposed method be used in a feedforward generative model?", "limitations": "Yes, the limitation and societal impact discussions look good to me.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an interesting new differentiable representation for 3D meshes. It is more efficient than DMTet in optimization.   \n\n- The representation works well in text-to-3D generation.", "weaknesses": "- The comparison in the experiment section is mostly qualitative.  A quantative comparison with DMTet on a 3D mesh reconstruction task would be appriciated.\n\n- The quality of underlying mesh connectivities is unclear to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer 7DJZ**\n\nThank you for your valuable feedback and constructive suggestions. We've addressed your concerns in the revised manuscript and our response below.\n\n**Q1: Sharpness of texture.**\n\nAs discussed in the limitation section of the Appendix, lines 521-523, TeT-Splatting uses tetrahedra as rendering primitives while modeling the 3D object with a structured tetrahedral grid. The use of a structured tetrahedral grid allows for the direct extraction of polygonal mesh through Marching Tetrahedra. On the other hand, it also limits TeT-Splatting's ability to model high-frequency textures. To address this, we decompose the 3D generation process into two stages in sequence: first, we optimize the rendering normal maps to obtain the geometry, and then we transition to a polygonal mesh through Marching Tetrahedra for further high-frequency texture optimization. We will further clarify and stress.\n\n**Q2: TeT-Splatting in a feedforward generative model.**\n\nAbsolutely! Integrating TeT-Splatting into a feedforward generative model as a 3D representation is a compelling concept. For instance, InstantMesh[1] uses a ViT encoder and a tri-plane decoder to generate tri-planes from multi-view images. They point out that volume rendering is memory-intensive, limiting the use of high-resolution images. To address this, they employ a differentiable isosurface extraction method (FlexiCubes[2]) to extract meshes from the tri-plane. However, this method lacks global awareness, as it can only backpropagate to the zero-level set of the signed distance field. This is where our TeT-Splatting can come into play as the 3D representation to overcome these limitations. This is because TeT-Splatting allows for precise mesh extraction, retains global awareness, while enabling real-time rendering. We reckon this integration could significantly enhance the performance and capabilities of the generative model.\n\n**Q3: Quantitative comparison with DMTet on 3D reconstruction.**\n\nAs discussed in Q1, TeT-Splatting is not designed to model high-frequency textures as required by 3D reconstruction, but focuses on the geometry stage in 3D generation, where only normal maps are needed as output. Due to the time constraints of the rebuttal period, we plan to investigate this in future work.\n\n**Q4: Quality of underlying mesh connectivities.**\n\nAs explained in lines 105-109, we establish a signed distance field (SDF) by interpolating the SDF values within each tetrahedron of a deformable tetrahedral grid, with each vertex's SDF value provided by a continuous hash network. The tetrahedral grid is structured, and its vertices can only deform within a local region, ensuring that tetrahedra do not intersect each other. After the first stage of optimization, we extract the mesh using the Marching Tetrahedra. This process assigns one or two triangles to each tetrahedron that intersects the zero-level set of the signed distance field. Consequently, as long as we maintain a smooth signed distance field, the extracted mesh will not be fragmented. As shown in Figure 7, TeT-Splatting produces a more coherent mesh compared to DMTet, due to the use of alpha-blending and normal consistency loss during optimization which leads to a more compact signed distance field. Additionally, we provide more viewpoints of the generated assets in the teaser in the attached PDF. Please refer to it for further details.\n\n[1] Xu, Jiale, et al. \"Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models.\" arXiv preprint arXiv:2404.07191 (2024).\n\n[2] Shen, Tianchang, et al. \"Flexible Isosurface Extraction for Gradient-Based Mesh Optimization.\" ACM Trans. Graph. 42.4 (2023): 37-1.\n\nWe hope our responses address your queries. If you have any further concerns, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oKB8wrZjUZ", "reviewer": "Reviewer_dtJc", "summary": "The paper presents a novel method for generating 3D models from the given inputs. The authors, for that purpose, propose the use of tetrahedra instead of 3D Gaussians in the recent proposed Gaussian Splatting framework.", "questions": "Please see weakness section.", "limitations": "The authors provide limitations and potential societal impact in the supplementary pages of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper proposes the use of TeT Splatting, that are well formulated and demonstrated  throughout the paper. The method allows for precise and fast mesh extraction, that is a well-known limitation of 3d gaussian splatting. The motivation to use TeT Splatting against the well-known DMTet is also well tackled.", "weaknesses": "1. For the task of 3D Reconstruction, 3D Consistency is highly important. However, lack of videos of the generated result is a major drawback that highly affects my decision since just two views are not enough to judge the quality of the output and created mesh. This is also a standard in the baseline papers such as DreamFusion that the authors compare with. I am happy to revise my scores otherwise.\n\n2. The paper uses hash grids as described in sec 4.1. Is it really necessary to use that, or how well the method will work without a hash grid or some other encoding such as positional encoding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer dtJc**\n\nThank you for your valuable feedback and constructive suggestions. We've addressed your concerns in the revised manuscript and our response below.\n\n**Q1: Results from multiple viewpoints.**\n\nWe apologize for the lack of videos. \nWe are not allowed to provide links to external pages.\nSo we have provided additional viewpoints of the generated assets shown in the teaser in the attached PDF. Please refer to Figure 1 in the attached PDF. The generated assets are compact, exhibiting descent 3D consistency.\n\n**Q2: Hash grid or some other encoding?**\n\nThis component is flexible in terms of choice of encoding algorithm.\nHash grid is a common choice for modeling the signed distance field (SDF) due to its efficiency, so it is selected in our work. Nevertheless, other methods such as positional encoding with MLPs or assigning a learnable SDF value to each vertex can also be considered. Per this suggestion, we have provided a comparison in Figure 2 of the attached PDF. Hash grids offer more efficient querying, and faster convergence than positional encoding. Compared to assigning learnable SDF values to vertices, hash grids enhance the 3D consistency and smoothness of the model.\n\nWe hope our responses address your queries. If you have any further concerns, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z7GKrDVjvQ", "reviewer": "Reviewer_Tjej", "summary": "This paper proposes a method for 3D mesh generation from a text. The idea is to improve gradient propagation ability of DMTet framework by splatting multiple tetrahedra instead of rendering only one surface triangle per pixel. Qualitative results on 3D mesh generation show good quality results with competitive processing time.", "questions": "-\tHow many tetrahedra are used in the experiments?\n\nThe proposed idea to splat tetrahedra and improve gradients propagation ability of original DMTet is interesting. Qualitative results show good performance. However, the explanations in the paper of the 3D generation process is lacking. Quantitative evaluation is few and only few qualitative results are maybe not enough to confirm advantage of proposed method.", "limitations": "There is no specific section or discussion about the limitations of the proposed method", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "-\tThe proposed idea is interesting. Splatting multiple tetrahedra around the surface allows to backpropagate gradients further in the model and avoid getting stuck in local minima.\n-\tFigure 3 is interesting and shows advantage of the proposed framework.\n-\tI am not so familiar with the datasets and tasks of 3D mesh generation from text but qualitative results look good and competitive with state-of-the-art", "weaknesses": "-\tSome explanations are missing to fully understand the experiments section.\n-\tResults are few and maybe cherry picked. Only qualitative cannot prove better performance of the proposed method. Again, more explanations about the generation process is required to understand advantages. For example, in figure 6 the 2D images are different, with different level of details in the 2D images. Then it is expected than the 3D models are also different and level of details also different. Then does it really make sense to compare these output 3D meshes? The difference may come from other parts in the framework than the proposed model.\n-\tThe authors motivate many of text for computational speed. But in the end advantage in computational power is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer Tjej**\n\nThank you for your valuable feedback and constructive suggestions. We've addressed your concerns in the revised manuscript and our response below.\n\n**Q1: Explanations of pipeline and experiment.**\n\nWe apologize for any confusion. Here, we further summarize our pipeline (as stated in Section 4) and experiment settings (as stated in Section 5).\n\nPipeline: The 3D generation process is decomposed into two stages. In the first stage, we use TeT-Splatting to render normal, depth, and opacity maps from the tetrahedral grid and signed distance field (SDF). We then use SDS loss to guide these maps, resulting in detailed geometry. In the second stage, we fix the geometry and extract the polygonal mesh from the tetrahedral grid and SDF using Marching Tetrahedra. Through rasterization and querying the texture network, we render the albedo and PBR maps, which are again guided by SDS loss.\n\nExperiment: We assess the efficacy of TeT-Splatting using two different diffusion priors: vanilla diffusion priors (leveraging Stable Diffusion as the diffusion model) and rich diffusion priors (leveraging the Normal-Depth diffusion model and Albedo diffusion model proposed in RichDreamer).\n\nFor vanilla diffusion priors, we compare our method with the prior art methods that also use only the Stable Diffusion model for a fair comparison. In Figure 4, we present experiments on text-to-3D and image-to-3D tasks. We also evaluate the mesh exportation process in Figure 5.\n\nFor rich diffusion priors, Figure 6 compares our method with state-of-the-art methods in text-to-3D, demonstrating that TeT-Splatting can handle high-fidelity 3D generation. In Figure 7, we visualize the normal maps during the early iterations, showing TeT-Splatting's superiority in stable and compact optimization compared to using DMTet (RichDreamer) as the 3D representation. Additionally, we conduct ablation studies in Figure 8 to demonstrate the effectiveness of our design choices in regularizations and hyperparameters. In Table 2, we report the CLIP scores to quantitatively illustrate the geometry and appearance quality of the generated assets.\n\n**Q2: In figure 6 the 2D images are different?**\n\nWe apologize for any confusion. As explained in Q1 above, Figure 6 compares those methods on the text-to-3D task, which do not depend on 2D image input. Regarding the concern that \"the difference may come from other parts in the framework than the proposed model\", we have already conducted dedicated experiments as shown in Figure 4, where all methods use the Stable Diffusion model as the diffusion prior, eliminating any differences in the diffusion prior. When comparing with RichDreamer in a truly fair manner (see Figures 6 and 7), we only substitute the representation in the first stage from DMTet to TeT-Splatting while keeping all the others the same.\n\n**Q3: Computational speed.**\n\nWe have extensively tested the computation cost and speed in the paper. In lines 249-254 and Figure 4, we did compare the training and rendering speeds of different methods. Specifically, in cases using vanilla RGB-based diffusion priors, TeT-Splatting demonstrates quick convergence, taking only 15 minutes for Text-to-3D and 5 minutes for Image-to-3D tasks, achieving a rendering speed of 133 FPS. Although TeT-Splatting operates at a lower FPS compared to DMTet (Fantasia3D) and 3DGS (DreamGaussian), this does not significantly hinder the entire generation process, as the primary speed bottleneck lies in the diffusion model. As stated in lines 275-276, employing TeT-Splatting during the geometry optimization stage significantly reduces the generation time from 70 minutes to 40 minutes.\n\n**Q4: Number of tetrahedra.**\n\nFor the experiments using vanilla diffusion priors, the resolution of the tetrahedral grid is set to 128 (please see line 486), resulting in 1.5 million tetrahedra initially, which is then reduced down (lines 150-155) to 200k after optimization. For the experiments using rich diffusion priors, the resolution of the tetrahedral grid is set to 256 (line 504), resulting in 12 million tetrahedra initially, and then dropping down to about 1.5 million after optimization. We show that a denser tetrahedral grid can lead to more detailed geometry as expected.\n\n**Q5: Few quantitative evaluation.**\n\nOverall, we follow the previous 3D generation works (e.g., RichDreamer) for quantitative evaluations. This is because in this work we focus on an alternative representation (TeT-Splatting) to be integrated into any optimization-based 3D generation pipelines.  In general, there is no ideal evaluation metrics for 3D generation tasks thus far, and it is an ongoing research problem. \nDue to no ground truth available for 3D generation evaluation, it makes sense that some proxy metrics must be used, such as the CLIP score for evaluating the similarity between the rendered images and the given prompts we are using. Specifically, to assess the geometry quality, we render the extracted meshes with uniform albedo;  For the texture quality, we render the meshes with the generated texture.\n\n**Q6: No specific section about the limitations.**\n\nDue to page limitations, we discuss the limitations of TeT-Splatting in Appendix D (lines 521-527), focusing on three main aspects:\n\n1) High-Frequency Features: TeT-Splatting struggles with modeling high-frequency features because it uses tetrahedra as rendering primitives, and the resolution of the tetrahedral grid would impose constraints on top.\n\n2) Rendering Speed: Although TeT-Splatting achieves real-time rendering, its speed is still relatively slower than 3DGS.\n\n3) Pre-Filter Operation: The current pre-filter operation may not fully exploit the potential of TeT-Splatting in terms of generation quality and speed. A densification process might improve these aspects, which we leave as future work.\n\nWe hope our responses address your queries. If you have any further concerns, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We provide two additional figures in the attached PDF:

1) **Figure 1: More viewpoints of the generated assets shown in the teaser.**

2) **Figure 2: Comparison of different encodings used for modeling signed distance field.**

Meta-review: All the reviewers liked the submission. Some queries were raised in the initial reviews, but the authors satisfactorily resolved them during the rebuttal phase. 

The paper proposes a novel approach that employs a deformable tetrahedral grid for volume rendering and splatting, enabling mesh extraction and optimization for 3D (geometry and appearance) generation. The main contributions, as highlighted in the reviews, are a new 3D representation utilizing tetrahedra splatting, its adaptation with a fast differentiable rasterizer, and its utilization in a two-stage 3D generation pipeline. The method performs effectively in text-to-3D generation tasks, offering high-quality mesh extraction. The technique is clearly explained and extensively evaluated, producing competitive results in 3D mesh generation tasks.
Decision: Accept (spotlight)