Unifi3D: A Study on 3D Representations for Generation and Reconstruction in a Common Framework
Authors: Nina Wiedemann
Sainan Liu
Quentin Leboutet
Katelyn Gao
Benjamin Ummenhofer
Michael Paulitsch
Kai Yuan
Venue: Accepted by TMLR ()

Abstract: Following rapid advancements in text and image generation, research has increasingly shifted towards 3D generation. Unlike the well-established pixel-based representation in images, 3D representations remain diverse and fragmented, encompassing a wide variety of approaches such as voxel grids, neural radiance fields, signed distance functions, point clouds, or octrees, each offering distinct advantages and limitations. 
In this work, we present a unified evaluation framework designed to assess the performance of 3D representations in reconstruction and generation. We compare these representations based on multiple criteria: quality, computational efficiency, and generalization performance. Beyond standard model benchmarking, our experiments aim to derive best practices over all steps involved in the 3D generation pipeline, including preprocessing, mesh reconstruction, compression with autoencoders, and generation. Our findings highlight that reconstruction errors significantly impact overall performance, underscoring the need to evaluate generation and reconstruction jointly. 
We provide insights that can inform the selection of suitable 3D models for various applications, facilitating the development of more robust and application-specific solutions in 3D generation. 
The code for our framework is available at https://github.com/isl-org/unifi3d.

Published in Transactions on Machine Learning Research (08/2025)
Unifi3D: A Study on 3D Representations for Generation and Reconstruction in a Common Framework
Nina Wiedemann* Intel Corporation
ain..uiedemmonn@intell.
Sainan Liu Intel Corporation
sainam. lww.iiiiininne...
Quentin Leboutet* Intel Corporation
muentin..ddoutee..innee.com
Katelyn Gao Intel Corporation
katelyng3@gmaii.com
Benjamin Ummenhofer Intel Corporation
Michael Paulitsch Intel Corporation
michaee..pruliscchiineeee...
Kai Yuan Intel Corporation
koi yuan Gintel.
Reviewed OpenReview: https: openreveeee.....ew.. net/ forum? id- GQpIWpXILA
Abstract
Following rapid advancements in text and image generation, rescarch has increasingly shifted towards 3D generation. Unlike the well-established bixel-based representation in images, 3D representations remain diverse and fragmented, encompassing a wide varicty of approaches such as voxel grids, neural radiance fields, signed distance functions, point clouds, or octrees, each offering distinct advantages and limitations. In this work, present unified evalua- tion framework designed to assoss the performance of 3D representations in reconstruction and generation. We compare these representations based on multiple criteria: quality, com- putational elficiency, and generalization performance. Beyond standard model benchmark- ing, our experiments aim to derive best practices over all steps involved in the 3D generation pipeline, including preprocessing, mesh reconstruction, compression with autoencoders, and generation. Our findings highlight that reconstruction errors significantly impact overall performance, underscoring the need evaluate generation and reconstruction jointly. We provide insights that inform the selection of suitable 3D models for various applications, facilitating the development of more robust and application-specifc solutions in 3D genera- tion. The code for our framework is available at hhttps:///st //tt://///thb..../.// thub. cowccooo///////e sis/sslloon commiss-org//miffll
1 Introduction
Recent advancements in generative image synthesis architectures, such as Generative Adversarial Networks
(GAss) and Diffusion Models, have driven significant progress in the field of 3D generation (Gezawa et al.
2020 Li al. 2024 Zhao and Larsen 2024 Liu
al.] 2024a. Jiang 2024). While image generation has
'Equal contribution

---

Published in Transactions on Machine Learning Research (08/2025)
Represenation b)
Reprosntation
PC
PC
E
Compression
Decompression
D.
Original Mesh
Vosel Geid
Denoising
Voxel Grid
Reconstructed
Diffasion
Process
SDF
SDP
Reprosentaaiinn
Laikent
Noney
(T-1)x Demolse
Denoised Lateont
Figure 1: Overview of the steps involved in a standard 3D generation pipeline: a) the mesh is transformed into suitable representation, b) an encoder Eup is pre-trained to compress into latent vector, c) a diffusion model is then trained denoise the latent, d) the latent finally decompressed into target representation using pre-trained decoder D& and turned back into mesh using dedicated algorithm such marching-cubes. (PC: point cloud, SDF: signed distance field)
reached a stage of maturity, mainly standardizing around pixel-based representations (Crowson et al. 2024 the landscape for 3D representations remains fragmented and varied. A wide range of 3D representations, including Voxel Grids (Ren ct al. 2024). Neural Radiance Fields (NeRFs) (Mildenhall et al. 2021 Signod Distance functions (SDRs) (Park et al. 2019), Point Clouds (PC) Nichol et al. 2022), and Octrees (Wang et al. 2022b), have been proposed, each suited to different applications and tasks. These methods vary not only in the way they encode geometry but also in how they handle the trade-offs between quality, computational efficiency, memory requirements, and the ability to generalize to novel objects and scenes (iu
et al.
2024a Peng et al. 2020 Wang et al. 2025).
While recent advances in 3D generative models can be attributed to various factors such as the availability of improved datasets (Deitke et al. 2023), optimized sampling techniques for diffusion models (Ren et al. 2024), and enhanced generative architectures (Zhang et al. 2024 the choice of the underlying represen- tation remains key factor. It dictates the information loss prior to encoding, inftuences the models used for compression and generation, and defines the computational resources for reconstructing mesh from a generated sample. Therefore, assessing the suitability of different 3D representations for reconstruction and generation of paramount importance.
However, objectively comparing 3D representations based on existing literature poses a significant challenge. While the representation core component, it is deeply embedded within complex 3D generative pipelines that utilize various models, loss functions, and datasets. On top of that, wide variety of pre- and post- processing techniques is applied, significantly affecting the results. For instance, handling non-watertight meshes is often not well-documented,, which tends to create inconsistencies across studies (Zhang et al. 2024). Another key issue lies in the difficulty of evaluating 3D object quality. Traditional metrics such as Chamfer Distance (CD) are commonly used to assess geometric accuracy, but they fall short in capturing perceptual quality and finer details (Mescheder et al., 2019; Wu et al., 2020 In turn, user studies can provide valuable qualitative insights, but they are labor-intensive and time-consuming. As a result, many papers resort presenting qualitative results cherty-picking favorable examples, which undermines objective evaluation and hinders progress in the field (Zhao and Larsen 2024h.
This paper introduces unified evaluation protocol designed to benchmark 3D representations. We have developed a standardized pipeline that integrates all components of the generative process - from data preprocessing to mesh encoding, generation and mesh reconstruction - into a common framework, as illustrated in Figure This design allows for the interchangeable use of 3D representations, ensuring that any observed performance differences are inherent to the representations themselves. Our standardized pipeline ensures fair comparison between different representations with respect to reconstruction and generation,
2

---

Published in Transactions on Machine Learning Research (08/2025)
instead of relying on the individual implementations of different 3D generation approaches that leverage the representations in varying manner, evaluate them with varying metries, and oftentimes solely report conditional generation performance of the complete pipeline.
In contrast to model benchmarking, we aim to generate insights into the strengths and weaknesses each representation while controlling for confounding factors like the diffusion model. Unlike traditional review papers that primarily offer qualitative overviews of existing methods (Po al. 2024 Wang al. 2025 Cao et al. 2024 Li et al. 2024 Gezawa et al., 2020 Liu et al., 2024a work emphasizes quantitative assessments grounded in empirical evidence. We make our entire pipeline open source to ensure the repeata- bility of our results and their generalization to different representations or experimental conditions. This allows rapid protolyping of novel 3D generative methods or 3D representations while ensuring adherence to best practices, such as proper preprocessing and parammer optimization. Our codebase also in- troduces previously unavailable open-source components, including a novel Dual-Octroes-based generative approach, training code for Shap-E, and transformer-bast based occupancy network generation. Summarized, the contributions of this paper are:
Standardized generation pipeline: We implement a generation pipeline with plug-and-play functionality to test 3D representations (voxel grids, SDFs, point clouds, octrees, triplanes, and NcRFs) in latent diffusion sctting as the most common approach for 3D generation.
Jointly evaluating reconstruction and generation: We analyze the relation between recon- struction and generalion capability. Reconstruction errors are as high as 20% of the generation error, proving their significance.
Best practices: We provide insights for common problems and derive best practices, such as the effects of different data preprocessing methods and the choice of meaningful sample sizes for evaluation.
Open-Source Modular Codebase: We provide an open-source codebase with structured, mod- ular architecture (see Figure 1 for rapid development and evaluation of new methods with various generation models and 3D representatious.
2 Benchmarking 3D Representations and Generative Algorithms
We put forward framework for comparing tensorial representations of 3D objects. To ensure fair and meaningful comparison, the following assumptions and requirements are established:
Meshes ground truth: The target representation in this study meshes, due to their fun- damental role in 3D computer graphics. They facilitate rapid rendering and are highly efficient in terms of space. Prominent datasets, such as ShapeNet (Chang et al. 2015) and Objaverse (Deitke et al. 2023 also provide objects as meshes.
Modularity: The pipeline should be modular, allowing for the integration of different representa- tions while controlling other components.
Coverage of related work: The framework is designed to align with state-of-the-art methods in the field, accommodating the most significant representations utilized in recent research.
Based on these features, we propose unified pipeline that captures the essential elements of contemporary 3D generation methods. As depicted in Figure I our multi-stage generation pipeline involves the following stages:
Representation Conversion: Transforming 3D mesh into a suitable representation such as a voxel grid, SDF, or point cloud. This step often employs algorithmic transformations without requiring training.
3

---

Published in Transactions on Machine Learning Research (08/2025)
2. Representation Compression: Utilizing an autoencoder to compress the high-dimeensional represen- tation into a lower-dimennionall latent space using architectures like Autoencoder (AE) (Kim et al.
2023), a Variational Autoencoder (VAE) (Luo and Hu 2021 Ren et al. 2024h, or a Vector Quantized Variational Autocncoder (VQ-VAE) CCheng ct al.
2023
3. Latent Generation: Training a generative model, typically a diffusion model (Sohl-Dicksteinn et al.
2015; Ho et al. 2020 Lyu et al.
2021 Zhou et al. 2021 to produce latent. vectors that the decoder
can reconstruct. If the representation is tokenizable, autoregressive models (Yan et al. 2022 Mittal
et al.
2022 Siddiqui et al.
2023 may be employed instead.
4. Mesh Reconstruction: Reconstructing the final 3D mesh from the decoded representation, often using algorithms like Marching Cubes (Lorensen and Cline 1987).
To support the claim that this pipeline is representative for most of the influential direct 3D generation approaches published over the past years, we list related work and their instantiations of representation, compressor and generator grid with a VQ-VAE and denoises the latent with a U-Net-basod diffusion model; XCube Ren et al. (2024) encodes voxel grid with a VAE and uses a multi-resolution U-Net diffusion; and Point-E
Table 1 For instance, SDFusion Cheng al. (2023) compresses an SDF-
Point-E[Nichol Nichol al. (2022)
compresses point cloud and denoises it via Diffusion Transformer (DIT). For an in-depth discussion of related work, we refer to Appendix A
2.1 Diffusion models
A suitable tensorial representation of 3D objects is part of any deep learning generative approach, whether it employs GANs, diffusion or autoregressive generation. Given their widespread use in the field, we opt for diffusion in our experiments; however, the main findings and identified shortcomings of certain represen- tations are generic. We implement two diffusion models: a Diffusion Transformer (DiT) (Peebles and Xie, 2023) and a U-Net diffusion model (Rombach ct al., 2022 Cheng al., 2023). DiTs are adept at handling digh-dimensional data by utilizing self-attention mechanisms and can be applied to any representation with an appropriate tokenization scheme, making them suitable for representations like point clouds and meshes. Conversely, U-Net architectures excel with grid-based representations such as voxel grids and SDFs, where the data can be organized into 2D or 3D tensors. For a detailed technical introduction to diffusion processes and for implementation details, see Appendix B
2.2 Selected representations and architectures
The experiments in this study are conducted using a carefully selected set of 3D representations, chosen for several koy reasons. First, we prioritized representations that are widely used in the field, ensuring the relevance of our analysis and aligument with existing literature. Second, we aimed to include a diverse range of representations, such as point clouds used as intermediate representation in the training process, and implicit representations like density fields and SDFF, which we use as the output representation to facilitate conversion to meshes for all methods. Third, scalability to large datasets and compatibility with diffusion models was a critical consideration, as this ensures that each representation can be effectively integrated with modern generative techniques. The encoders and decoders for each representation are trained through a reconstruction task on the same dataset used for the generative task. To ensure normally distributed latents, which is a crucial prerequisite for the diffusion process, the AE encoder always implements LayerNorm as final layer. We implemented our representations, encoders, and diffusion models within a standardized training pipeline utilizing the hydra and accelerate libraries.
Voxel and SDF Grid Encoding Voxel grids offer an intuitive and explicit encoding for 3D objects, While used prodominantly in carly rescarch (Wu et al. 2015 Maturana and Scherer 2015 Choy et al. 2016 Wu et al., 2016 Brock et al. 2016 Dai et al. 2018). they are still used in SOTA work (Ren el al. 2024 Zheng et al. 2023 Liu et al. 2023a]. Instead of binary occupancy indicators, some works fill the grid cells with sampled SDF values CCheng et al. 2023 Mittal ct al. 2022) to increase the expressiveness. We implement both approaches: a standard voxel grid where each cell holds a binary occupancy value and an SDF grid where each cell contains the signed distance to the nearest. point the mesh surface, sampled
4

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Overview of representations, autoencoders and generators used in influential contemporary research on 3D generation (PC: Point cloud). Optimization-base methods are not included since they follow a differ- ent structure. Some methods do not compress the representation and directly denoise on the representation.
Paper
Representation
Compression
Generation
gDNA (Chen et al.
2022)
Occupancy Field
GAN
BlockGAN (Nguyen- Phuoe et al.
2020
Voxel Grid
GAN
XCube (Ren et al.
2024
Voxel Grid
VAE
Sparse UNet diff.
Ne Kim al.
2023
Voxel
AE (CNN)
Hierarchical diff.
Trellis (Xiang et al.
(2224)
Voxel /
SparseFlex
Transformer VAE
Rectified Flow
LAS-Dillussie
Zheng et al.
2023
Voxel SDF
Diffusion UNet
One-2-3-45-t+ Liu et al.]
2023a
Voxel SDF
Diffusion UNet
Diffusion-SDI et al.
2023
Voxelized SDF
Patch-VAF
Diffusion UNet
LDM (Xie et al. 2024
Voxelized SDF
Transformer
SDFusion Chengg al. 2023
Voxelized SDF
VQ-VAE
Diffusion UNet
DualOctreeGNN [Wang et al.
2022a
Octree SDF
AE
(Hui et al.
202-
SDF
Wavelet features
Diffusion ViT
Cannonical mapping (Cheng et al.
2022
PC
VAE
Autoregressive
DPM (Loo and Hu 2021
PC
VAE
Diffusion
PVD Zhou et al.] 2021
Voxel / PC
Diffussoo CNN
r- r-GAN
GAN
Achlioptas et al.
2018
PC
AE
GAN
SoftFlow
Kim et al.
20020
PC
AE
Normalizing Flow
DPF-Net Klokuv et al.
2020
PC
AE
Normalizing Flow
Shape-GF Cai et al. 2020
PC Density Field
AE
GAN
PointFlow Yang et al.
2019
PC
VAE
Normalizing Flow
PointGrow Sun et al.
2020
PC
MLP
Autoregressive
3d (Zamorski et al.]
2020
PC
VAE
AAE
tree-GAN [Shu et al.
2019
PC
GAN
Point-E (Nichol et al.
2022
PC
Coordinates
DIT
3DShape2Verrie VerSet (Zhang et al.
2023
PC SDF
Transformer
DiT
CLAY [Zhang al. (2024
PC SDF
Transformer
DIT
Michelangelo (Zhao et al. 2023
PC / Occupancy
VAE
Diffusion UNet
SparseFlex (He al. 2025
PC SparssFlex
Transformer VAE
Rectified Flow
AutoSDF Mittal et al. 2202
SDF
VQ-VAE
Autoregressive
SurfGen Luo et al.
2021
SDF
GAN
3D-LDM
Nam et al.
2022
SDF
MLP
Diffusion (MLP)
TripoSG
Li et al. 2025
SDF
Transformer VAE
Rectified Flow
SDM-NET
Gao et al.)
2019
Mesh
VAE
PolyGen Nash al.
2020
Mesh
Coordinates
Autoregressive
MeshGPT Siddiqui et al.
2023
Mesh (face tokens)
GraphAE
Autoregressive
MeshXL C Chen et al. 2024b
Mesh
Coordinates
Autoregressive
GIRAFFE (Fu et al. 2022
NeRF
GAN
HoloDiffusion (Karnewar et al.
2023
NeRF
ResNet
Diffusion
Shape-F. F. (Jun and Nichal
2023
NeRF SDF
Transformer
DIT
SSDNeRF [Chen et al. 2023a
NeRF / Triplane
MLP
DiT
EG3D (Chan et al. 20022
Triplane
GAN
3DGen Cupta et al.
2023
Triplane
PointNet/ eunett
Diffusion UNet
Direct3D
Wu al.
2024aa
PC/ / Triplane
DiT
Get3D
Gao et al.
2022
DDill
StyleGAN
TriFlow
(Wizadwongsa et al.
2024
Triplane
MLP
DiT
ShapeFormer YYan et al. 2022
PC/ / VQDIF
Transformer
Autoregressive
from truncated SDF with a cutoff of 0.2. Meshes were converted to 3D grids of resolution 643 by ray casting and closest point queries. For encoding, we implemented a 3D CNN following Cheng el. al. (2023) For example, a grid of resolution 64 is transformed into a latent tensor of shape 3 x 16 x 16 x 16. This latent ropresentation is typically diffused using a 3D U-Net (Cheng et al. 2023). Here, we additionally introduce a censfoommr-bbseed approach, where the latent tensor is tokenized by dividing it into 3D patches of size 43, analogous to patch-based tokenization used in vision transformers. This results in total of 12 potential
5

---

Published in Transactions on Machine Learning Research (08/2025)
generative approaches: Voxel
SDF grid, each combined with AE /AEE / VQ-VAE, and denoised with
DiT or U-Net.
3DShape2VecSet Encoding Occupancy Networks (Mescheder et al. 2019) and their extensions (Pnng
et al. 2020 Atzmon and Lipman 2020 Zhang et al. 2023) use implicit representations to sample density or
occupancy at any spatial point, enabling detailed surface generation beyond grid limitations. This mothod
has recently been leveraged for conditional 3D generation, achieving remarkable results (Zhang et al., 2024: Yang et al. 2024 Li et al. 2025). Our implementation, based the 3DShape2VeSeel model (Zhang et al. 2023), transforms meshes into point clouds X and sub-samples them (X). The positional embeddings of the points are processed through cross-attentio
between X and X, resulting in a set of k-dimensional latent vectors. The decoder passes the vector set through multiple self-attention layers, with cross-attention applicd between the outputs and embedded query points. Following Zhang el al. (2023) latent generation employs DiT.
Dual Octree Graph Encoding To efficiently capture high-resolution geometric details while maintaining computational scalability, we implemented the dual octree graph representation proposed by Wang et al. 2022b). This method leverages the hierarchical structure of octrees combined with graph neural networks (GNNs) to effectively represent complex 3D geometries. The dual octree graph network takes a set of point clouds as input, builds a dual octree graph, and outputs an adaptive feature volume via a graph-CNN CNN-basod encoder-decoder nelwork structure. In all experiments, we use a tree depth of 6 and force the octree to be full when depth d < 2. During decoding, the encoded Octree-features are converted to signed distance field via a multilevel neural partition-of-uni extended this AE structure to VAE and VQ-VAE and trained a simple two-layer 3D U-Net diffusion model for generation. To the best of our knowledge, this is the first generative approach based on Dual Octrees.
(Ncural MPU) module to construct a surface. Our approach
Triplanes Triplane representations efficiently encode 3D scenes in a hybrid explicit-implicit manner Wang et al. 2025), utilizing three orthogonal 2D feature planes Peng et al. (2020). From these planes, a lightweight MLP occupancy network can perform volumetric rendering. We employ the autoencoder architecture pro- posed in Peng et al. (2020) to generate triplane latents from point clouds sampled the mesh. Following Wu et al. (2024b), latent generation employs a UNet with 3D-aware convolution layers to better capture the unique properties of the triplane latents.
NeRFs Neural radiance field (NeRF) another implicit representation that was first proposed for novel view synthesis Mildenhall et al. (2021), parameterizing the color and density at each point by a neural network. To render the 3D object from novel views, a differentiable volumetric ray casting approach is utilized. The vast majority of generative methods using NeRF representation are optimization-based. iteratively updating the parameters of the neural network to maximize the likelihood of rendered images
under an image generation model Poole et al. (2022); Lin et al. (2023 Liu et al. (023bb); Wang et al. (2024)
Di Sario et al. 20024.. Therefore, in this work, we adopt instead the NeRF representation used in Shap- E Jun
and Nichol (2023), which falls into the framework illustrated in Figure 1 Each mesh is converted into a point cloud and multi-view RGB images, which are encoded into a latent vector using point convolution, cross attention, and Transformer. To ensure that the latent vector is bounded, it is input into a tanh activation function, and noise is added; however, to be consistent with the other representations, we replace this step with layer normalization. During decoding, the latent vector is projected to the parameters of the NeRF's neural network. The autocncoder is trained on combination of RGB and transmittance reconstruction losses on multi-view renderings. In the original paper, the latent vector is also projected to the parameters of a signed texture field, and the autoencoder is fine-tuned with SDF and color distillation losses. However, we exclude this step to have a fair comparison of NeRF with the other representations. As in the original paper, we leverage DiT for generation. For both reconstruction and generation, meshes are obtained from the NeRF by applying Marching Cubes to its density function.
6

---

Published in Transactions on Machine Learning Research (08/2025)
2.3 Evaluation protocol: Joint evaluation of reconstruction and generation
The ability to transform mesh to and from latent space upper-bounds representation's generation ca- pabilities, as generated vectors must ultimately be converted into meshes. Thus, we advocate for jointly benchmarking reconstruction and generation to understand how reconstruction errors limit generation. Re- construction error comprises appresettationninlen sion errois due to the encoder. Additionally, an encoder trained on limited data may struggle with novel objects, nccessitating an assossment of out-of-distribution (OOD) performance. Consoquently, we propose the following evaluation protocol:
errors, like information loss in voxel grids and compres-
1. Assess reconstruction (mesh + representation - mesh)
2. Benchmark compression performance (mesh + representation + latent vector -+ representation + mesh).
Test the encoder's generalization in OOD tasks
Measure generation performance quantitatively and qualitatively.
3 Experimental setup
We benchmarked the selected methods on subset of categories of the ShapeNet datasel (Chang el. al. 2015), specifically car, airplane, and chair, following [en et al. (2024). The dataset was split into training, validation, and test sets according to the official divisior We evaluated reconstruction performance on the full lest. dataset (806 objects for airplane, 703 for car; and 1,163 for chair). Following Peng el. al. (2020) we assessed reconstruction quality using Chamfer Distance (CD), F-Score, and Normal Consistency (NC). For precise definitions, please refer to [pppendix C The selected metrics can be directly computed on the meshes and, unlike metrics like Intersection over Union (IoU), do not assume meshes being manifold, watertight, or convertible to grid representations assumptions that do not hold for many meshes in the dataset.
Quantitatively evaluating generative models for 3D shapes is challenging due to the diversity of tasks and metrics in related work, such as single-image or multi-view reconstruction, text-to-3D synthesis, or uncondi- tional generation. To ensure a fair comparison and to avoid adding complexity to our benchmarking pipeline, we opted to evaluate the models in an unconditional gencration sctting. This approach tests the ability the representations to generale objects that are both similar to the training set and diverse. Following previous work (Gao al.) 2022 Lei al. 2023 Yang et al.) 2019). we implemented three distributional metrics: Coverage (COV), Minimum Matching Distance (MMD), and 1-Nearest Neighbor Accuracy (1-NNA). All these metrics are based on the pairwise CD between a set of generated samples Sg and reference dataset Sr, which is a subset of the test data. Coverage measures the fraction of S, that is matched to Sg. reflecting the diversity of the generated samples (higher is better). MMD computes the average minimum distance from samples in S, to those in Sy, indicating the quality of the generated samples (lower better). The 1-NNA metric assesses the overfitting of the model by measuring the accuracy of classifying samples into S, and Sg based on their distances. An ideal model achieves 1-NNA of 0.5, indicating that generated samples are indistinguishable from real data. For formal definitions of these metrics, see Appendix C. In Figure 2] measure the stability of these metrics by computing them random subsets of ShapeNet, imitating perfect generative model. The meaningfulness of the metrics strongly depends on the chosen set size for S, and Sg, making it difficult to compare numbers in the literature. For instance, the MMD metric decreases with larger set sizes as the pool of meshes to find the closest neighbor increases. We also observe a large spread of values for small set sizes and recommend that the set size should be larger than 200 for the metrics to converge.
To evaluate the subjective mesh quality, we conduct a user study. In each question, two meshes generated by different approaches are presented to the user, who is asked to indicate which is preferable based on object complexity and surface quality. Overall, we collect a datasct of 575 preferences from 24 users. To obtain scores for each approach, we model the preferences using the Bradley-Terry statistical model (Bradley and Terry 1952 (see Appendix J for details).
http://shapenet..c www.:aantt....ttt.......om. sww.sord.dd.o.... www.sttnfor..eeeeeaeeeeeeeeeeeee

---

Published in Transactions on Machine Learning Research (08/2025)
COV
MMD
1-NNA
0.06
Chamier Distance
0.05
0.6
0.7
0.04
0.03
0.4
0.3
0.3-
0.2
0.02
10
100
200
400
100
200
400
10
100
200
400
15c. 15cl
|5rl. 15pl
[sy 15pl
Figure/ 2: We select 100 random subsets of different sizes of the train and split of the ShapeNet airplane category and compute the COV, MMD, and 1-NNA metrics. Small set sizes result in large spread of values. We further observe that the Coverage (COV) is not close to the optimal value 1.0 and the 1-Nearest Neighbor Accuracy (1-NNA) is above the optimal value 0.5, indicating that learning from the training set may lead to biases.
Human Prefer.
F-score
MMD
Figure 3: Rankings of 3D rep- resentations based on genera- tion and reconstruction metrics. The outer cirele indicates the top rank. This chart compares reconstruction metrics (CD, F- score, NC), reconstruction gen- eralization
CD
1-NNA
OOD task, and reconotruction elficiency (mem- ory footprint, encoding size, and inference time) as well as the metrics for unconditional gen- eration performance with user study rankings (I-NNA, MMD, COV, surface-to-volume ratio (S2VR), Human Preference).
CD on OOD-task
S2VR
Latent size
Time
Memory
DualOctree
SDF
Shape2VecSet
Triplane
Voxel
Reconstruction
Generation
4 Results and Discussion
Figure 3 summarizes the results, demonstrating that the SDF-encoder achieves the best. reconstruction performance, but our DualOctree diffusion model excels generation. In the following, the results will discussed step-by-step.
4.1 Benchmarking generation performance
The comparison the best generative approaches, one per representation, is shown in Table 2 For ablation studics on encoders, i.c. selecting the best from AE / VAE / VQVAE, and on diffusion models (DiT) UNet), see Appendix G Interestingly, we found approaches with AE encoders usually outperforming VAEs, contrary to prior findings. Apparently, adding layer normalization to AE has similar positive effect as the
8

---

Published in Transactions on Machine Learning Research (08/2025)
KL-loss in VAE. Surprisingly, the novel DualOctree-basee diffusion model achieves best performance in all metrics, followed by SDF and Shape2VecSet. This finding contrasts with the widespread use of Shape2VecSet in state-of-the-art (SOTA) research, highlighting the critical role of multi-scale input schemes and the larger transformer models employed in recent SOTA developments Zhang et al. (2024).
Table 2: Performance of representations in unconditional generation setting.
Method
COV 1
MMD
1-NNA + 0.5
DualOctrce VAE UNet
0.365
0.031
0.824
SDF AE DIT
0.357
0.032
0.860
Shape2VecSet
0.344
0.033
0.864
Triplane AE UNet
0.297
0.036
0.921
Voxel AE DIT
0.319
0.040
0.937
However, the user study (see Figure 3 paints different picture, ranking SDF (score 0.25), Shape2vecset (-0.122) and Triplane (-0.140) above DualOctrre (-0.178). This shows the necessity aiming to assess human- perceived object quality, One reason for the users' preference for SDDeggenratted objects may be the tendency of SDF-grids to generate smooth surfaces. Figure provides qualitative results for the generative approaches. On the other hand, we found that DualOctree generates more compler but potentially imperfect assets, measured in terms of the surface-to-volume ratio. [Appendix 1 expands on the analysis of complexity and provides further evidence that SDF tends to smoothen surfaces whereas Shape2VecSet and DualOctree are more prone to creating artifacts. Furthermore, Appendix E provides the corresponding results on a subset of the Objaverse dataset, demonstrating similar results but. better performance of the Triplane representation.
user st study when
4.2 Reconstruction performance
Table 3 provides the roconstruction quality in terms of CD, F-Score, and NC. The best-vectorized rep- resentations are the voxel and SDF grid encodings using AE and VAE, with an average F-score of 88%. NeRF-encoding performs worst, probably due to modifications our implementation for the sake of compa- rability (see section 2.2), and NeRF-based generation did not reach comparable generation performance and thus only analyze its reconstruction performance. Ablation studies with respect to the encoder model (AE / VAE / VQ-VAE) are provided in Appendix F
Furthermore, we hypothesized that larger latent representations reduce the information loss during com- pression but come along with longer runtimes. Figure 5 shows the trade-off bet reconstruction loss, reconstruction runtime, and the size of the latent vector (in terms of the number of elements the tensor). The results shows well the different characteristics and importance of different representations. For instance, OualOctree with the smallest latent size achieves better reconstructions than Triplane with larger latent thanks to the spatial adaptive structure but is less accurate and slower than the Voxel representation that uses much larger latent. Selecting the right representation can therefore significantly influence the runtime, accuracy and memory requirements. Generative methods targeting CAD applications may want represen- tation with minimal reconstruction error, while methods running on devices with small memory may favor a representation with a small latent.
4.3 Generalization ability of the encoder
When encoding 3D meshes into diweemnnniooall vectors, there is a trade-off between the compression capacity and the model's capability to generalize to representations. Since the generator aims to design new objects, applicability beyond the training data is crucial. We quantified the gencralization capability (sce Table 3 right) where we Lest the encoders trained on the Chair calegory when applied to the Airplane category. While most encoders show very high generalization performance, on par with category-specific training. NcRFs and DualOctrees stand out as represcntations that struggle with OOD data. In the former, the MLP-based latent may be prone overfitting, while for the latter, the low dimensionality of the latent could be problematic.
9

---

Published in Transactions on Machine Learning Research (08/2025)
SDF AE DiT
Voxel AE DiT
DualOctree VAE UNet
Shape2VecSet AE DiT
Triplane AE UNet
Figure 4: Qualitative results for mesh generation. We show results for each representation using the best encoder configuration for each.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Reconstruction quality in terms of CD, F-score and NC.
Trained tested Airplane
Car Chair
OOD (trained Chair, tested Airplane)
Method
F-score (0.0125) f
CD (*1c-4) 4
NC
F-score (0.0125) 1
CD (*1e-4)
NC 1
DualOctroe VAE
76.12 = 13:44
0.01
0.766
0.07
48.38
= 11.39
0.047
0.02
0.677
NeRF AE
58.44 + 13:22
0.034 = 0.02
0.723
+ 0.07
26.229
= 11.61
0.107 + 0.01
0.589
= 0.05
SDF AE
88.434 + 6.58
0.012 = 0.0
0.827
0.06
91.123
+ 6.02
0.01 = 0.01
0.843 = 0.05
Shape2VeeSet AE
79.37 17.04
0.023
0.776
0.07
75.338
8.87
0.022 0.01
0.717
1 0.07
Triplane AE
66.4445
16.06
0.028
0.759
0.08
41.69
+ 11.57
0.073
0.03
0.688 +
Voxel AR
85.666
10.54
0.016
# 0.01
0.787
+ 0.06
85.602
+ 9.48
0.017 + 0.01
0.8 + 0.05
Generation error (MMD)
Reconstruction error
0.035
1M
Compression error
Chamfer distance
6K
Method
Percentage MMD
80
Voxel AE
0.020
Triplane AE
60
Shape2VecSet
0.015
SDF AE
40
12K
NeRF AE
0.010
DualOctree VAE
20
0.005
DualOctree
SDF
Shape2VecSet
Triplane AE
Voxel AE UNet
0.000
VAE
AE
AE UNet
0.2
0.4
0.6
0.8
1.0
1.2
UNet
DIT
DIT
Time mesh reconstruction [s]
Method
Figure 5: Reconstruction quality by runtime for infer- ence (decoding and mesh reconstruction), and by size of the latent (see label and bubble size).
Figure Decomposition errors during genera- tion. Reconstruction errors and imperfect docom- pression play a substantial role.
4.4 Effect of preprocessing
For all methods except those encoders operating directly point clouds meshes preprocessing meshes is essential. Non- Non-watertight meshes introduce significant artifacts as they do not have well defined inside or outside, which is crucial information for many methods, adversely affecting the quality of the reconstructed shapes. Mesh manifoddiztion is an active research area, with various methods proposed like the Monifold library (Huang al. 2018), which thickens surfaces to ensure that every mesh component forms solid volume. While effective, this process can alter the original geometry by artificially expanding thin structures. ManifoldPlus (Huang et al. 2020) extends Manifold but has been reported to produce inconsistent results (Zhang et al. 20244.. Wang et al. (2022b) uses simple manifoldization (Mesh2SDF)) based on contouring the unsigned distance function, which also produces significant thickening artifacts or requires high grid resolutions, resulting in high compute and memory requirements. The thickening of the mesh helps to preserve thin structures but adds an irrevocable bias. To this end, we introduce a lcan and mean preprocessing step that transforms meshes to SDFs without thickening. Figure 7a shows visual explanation of our preprocessing step using the flood fill algorithm. Instead of altering the mesh we define the inside and outside outlined in Figure 7a
We define a target grid encompassing the mesh. We mark all voxels that touch the mosh surface (gray) and then flood fill starting from corner voxel guaranteed to be outside to define the outside region (blue). Voxels that touch the mesh but not an outside voxel defined by the 3e-voxelnnigbbbortoool get removed. This step effectively climinates internal structures not considered part of the outer shape of an object.
(4) All unlabelled voxels get labelled as inside (red).
(5) For all gray voxels we determine whether the voxel center lies inside or outside by comparing the plane defined by the closest. point to the surface and the normal approximated by the sum of
11

---

Published in Transactions on Machine Learning Research (08/2025)
the positions of the outside neighbor voxels. The surface points sampled in this step reused for computing distances for the SDF.
Finally, we directly sample points and compute distances with the determined sign.
We compare the effects of manifoldization as preprocessing step by converting meshes grid SDF representation and back to the mesh representation in Figure 7b (see Appendix H for additional results).
F-score with == 7=1 voxel size
80
60
(1)
(2)
(3)
40
20
64
128
256
384
(4)
(5)
Grid size
(a) Chamfer Distance
(b) F-score
Figure 7: We compare different methods for converting mesh to an SDF. Ours uses the flood approach
define the inside (red) and outside (blue) of mesh, as explained in a Naive uses simple raycasting
approach to determine the sign. Mesh2SDF from Wang al. (2022a) creates watertight mesh using the unsigned distance and suffers from artificial thickening of the shape. This effect is reduced with increasing grid sizes.
4.5 Error decomposition
In alignment with the goal to evaluate reconstruction and generation jointly, investigate the relation between generation, compression, and reconstruction errors. Since MMD measures generation performance as the mean CD between each sample in S, and its best match in Sg, it can be compared to the CD at the reconstruction or compression stage. While the generation of new shapes inevitably induces an MMD > 0 (see Figure 2), the MMD loosely lower bounded by the reconstruction and compression error in terms of CD. To decompose the error, for each sample p Sr, we compare its MMD, its compression error (e.g. :ncoddingdecodding). and its mesh reconstruction error (e.g. marching cubes inaccuracies). The reconstruction errors taken from the analysis in Figure 7b] while the compression errors correspond to the sample-wiss wise result from Table 3 It is worth noting that these errors not simply additive; however, it is interesting to investigate the size of the lower bounds (reconstruction and compression errors) in relation to the absolute size of the generative error. Figure shows that the reconstruction and compression errors amount to 12.9% and 39.3% of the MMD, respectively, when averaging over all representations. The MMD and the reconstruction CD corrclate on average with a Pearson R correlation of 0.30 (p < 0.01). As expected, there is also a positive correlation of 0.24 between compression and reconstruction error. This analysis underlines the important role of 3D representations and their reconstruction performance, reconstruction errors significantly reduce the quality of the generated samples.
5 Conclusion
We have presented systematic comparison of 3D representations for reconstruction and generation. Our analysis leads to recommend the following best practices:
12

---

Published in Transactions on Machine Learning Research (08/2025)
Evaluate reconstruction and generation jointly. Since generation quality is upper- hoper-bounded by the reconstruction error, the errors resulting from mesh reconstruction and representation encoding alone should be reported for transparency.
Compute the errors with respect to the original mesh. Results should not be distorted by deriva- tives obtained following the application of methods like Manifold and avoid metrics that require additional preprocessing.
Include conversions of the output representation to meshes. Comparing only generated surface points to the ground truth mesh may skew results.
A sufficient number of samples is crucial when evaluating unconditional generation with metries such as MMD, I-NNA, and Coverage. More than 200 samples are generally necessary to achieve robust outcomes.
This framework covers direct 3D generation, in contrast to optimization based methods such as Score Distillation Sampling (SDS) based methods (Poole et al., 2022 Chen et al. 2023b Lin et al. 2023;
Limitations
Sun et al. 2023 Wang et al. 2024 Shi et al. 2023b) other approaches that use multi-view images for inference Shi et al. (2023a); Liu et al. (2024c]; Long et al. 2024). Despite the impressive results generated with these techniques, recent developments have shifted the focus of the field back to direct 3D generation, due to 1) high computational costs of inference-time optimization Li et al. (2024). 2) dependence of the generation quality on the fidclity of the multi-vicw images Wu et al. (2024a), 3) preferability of explicit 3D outputs for artists Yang et al. (2024), and 4) feasibility of training for direct generation due to large-scale 3D datasets such as Objaverse Deitke al. (2023) and ObjaverseXL Deitke et al. (2024). Similarly, Gaussian Splatting has become popular in the 3D field Tang et al. (2023): Kerbl ct al. (2023); Lan et al. (2024), but is not included in our study, as it is usually trained with SDS-loss methods and focuses realistic and efficient rendering, in contrast to our assumption of meshes as the ground truth.
Furthermore, some generative mothods fit into our unified pipeline but remain unimplemented, such as autoregressive generation. Follow-up work could also extend the analysis to other datasets. While we already provide results on a subset of Objaverse in Appendix [ further experiments distinguishing different types of objects (e.g. thin structures or very complex shapes) could bring further insights on the advantages and drawbacks of each representation.
Outlook There are several open challenges in the field, such as accounting for interior structure of objects, transferring 3D object generation methods to a scene level Ren et al. 2024), or enabling model articula- tion (Leboutet et al. 2024 Lei et al.) 2023 Liu et al., 2024b). While our benchmark shows the general ability any representation to encode and generate high-resolution objects, it also shows the challenges such as dealing with thin structures while retaining computational efficiency. With the presented pipeline implemented in an open-source code base, we offer unified framework to develop and benchmark novel 3D representations for generation.
References
Panos Achlioptas, Olga Diamanti, Toannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018.
Matan Atzmon and Yaron Lipman. Sal: Sign agnostic lcarning of shapes from raw data. In Proceedings of the IEEE/CVE conference on compuler vision and pallern recognilion, pages 2565 2574, 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biomelrika, (9(3/4):324 345, 1952.
Andrew Brock, Thcodore Lim, James M Ritchic, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXio preprint arXiv:1608. 04200 2016.
200
Table 6 and Figure for the influence of preprocessing techniques
13

---

Published in Transactions on Machine Learning Research (08/2025)
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Computer Kision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pages 364 381. Springer, 2020.
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-An Ann Heng, and Stan Z Li. A survey on generative diffusion models. IEEE transactions on knowledge and data engineering, 36(7): 2814 2830, 2024.
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leomidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- 3d generative adversarial networks. In Proceedings of the IEEF/CV conference on computer vision and pattern recog- nition, pages 16123-16133, 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rxii:1512.0000 030112, 2015.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. In Proceedings of the IEFE/CVP international conference on computer vision, pages 2416 2425, 2023a.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geomelry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22246 22256, 2023b.
Rui Chen, Jianfeng Zhang, Yixun Liang, Guan Luo, Weiyu Li, Jiarui Liu, Xiu Li, Xiaoxiao Long, Jiashi Feng, and Ping Tan. Dora: Sampling and benchmarking for 3d shape variational auto-encoders. arXiv preprint Xrxiv:2412 17808, 2024a.
Sijin Chen, Xin Chen, Anqi Pang, Xianfang Zeng, Wei Chong, Yijun Fu, Fukun Yin, Billzb Wang, Jingyi Yu, Gang Yu, Bin Fu, and Tao Chen. Meshxl: Neural coordinate field for generative 3d foundation models. In NewrIPS, 2024b.
Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J Black, Andreas Geiger, and Otmar Hilliges. gdna: Towards generative detailed neural avalars. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pattern Recognátion, pages 20427-20437, 2022.
An-Chich Cheng, Xucting Li, Sifci Liu, Min Sun, and Ming-Hsuan Yang. Autorogressive 3d shape generation via canonical mapping. In Europeon Conference on Computer Vision, pages 89 104. Springer, 2022.
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Gui. SDuusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Con- [erence on Compuler Vision and Pallern Recognilion, pages 4456 1456-4465, 2023.
Christopher B Choy, Danfei Xu, Jun JunYoung Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 628 644. Springer, 2016.
Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Shippole. Scalable high-tesolution pixel-space image synthesis with hourglass diffusion transformers. Tn Forty-firs International Conference Machine Learning, 2024.
Angela Dai, Daniel Ritchic, Martin Bokcloh, Scott Reed, Jürgen Sturm, and Matthias Nicßner. Scancom- plete: Large-scale scene completion and semantic segmentation for 3d scans. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognilion, pages 4578 4587, 2018.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142
13153, 2023.
Malt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
Francesco Di Sario, Riccardo Renzulli, Enzo Tartaglione, and Marco Grangetto. Boost your nerf: A model- agnostic mixture of experts framework for high quality and efficient rendering. In European Conference on Compuler Vision, pages 176-192. Springer, 2024.
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVE conference on computer vision and pallern recognilion, pages 5501 5510, 2022.
Tianlin Fu, Renyu Deng, Yue Gao, and Fuquan Zhang. Representing scenes as compositional generative neural feature fields based on giraffe for 3d reconstruction of classroom scenes. In International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pages 227-237. Springer, 2022.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Newral Information Processing Systems, 35:31841-316 118844,,,, 2022.
Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. Sdm-net: Deep generative network for structured deformable mesh. ACM Transactions on Graphics (TOG), 38(6):1-15, 2019.
Abubakar Sulaiman Gezawa, Yan Zhang, Qicong Wang, and Lei Yunqi. review on deep learning approaches for 3d data, representations retrieval and classifications. TEFE access, (87566657593,, 2020.
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint ar 3rXi:23033053311, 2023.
Xianglong Hc, Zi-Xin Zou, Chia-Hao Chen, Yuan-Chen Guo, Ding Liang, Chun Yuan, Wanli Ouyang, Yan- Pei Cao, and Yangguang Li. Sparseflex: High-resolution and arbitrary-topology 3d shape modeling. arXiv preprint Xrv: 2025.
Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceedings of the IEEE/CVE CEE//CVF International Conference on Computer Vision, pages 2008,, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Jingwei Huang, Hao Su, and Leonidas Guibas. Robust watertight manifold surface generation method for shapenet models. arXiv preprint arXiv:1802.0100 2018
Jingwei Huang, Yichao Zhou, and Leonidas Guibas. Manifoldplus: A robust and scalable watertight manifold surface generation method for triangle soups. arXiv preprint arXiw:2005.11621 2020.
Ka- Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, and Chi-Wing Fu. Mako-a-shape: a ten-million-scale 3d shape model. In Forty-first International Confer- ence on Machine Learning, 2024.
Chenhan Jiang. A survey on text-to-3d contents generation in the wild. Xrxiv preprint orXiv:2,05.09431 2024.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint w.Xiv:2305.02400 2023.
Animesh Karnewar, Andrea Vedaldi, David Novolny, and Niloy Mitra. Holodiffusion: Training a 3d diffusion model using 2d images. In Proceedings of the LEEE/CVE conference on computer vision and pottern recognition, pages 18423-18433 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
Hycongju Kim, Hyconseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim. Softflow: Probabilistic framework for normalizing flow manifolds. Advances in Neural Informalion Processing Systems, 33: 16388-16397, 2020.
Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, and Sanja Fidler. Neuralfield-Idm:: Scene generation with hierarchical latent diffusson models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8496-8506, 2023.
Roman Klokov, Edmond Boyer, and Jakob Verbeek. Discrete point flow networks for efficient point cloud generation. In European Conference on Computer Vision, pages 694 710. Springer, 2020.
Yushi Lan, Shangchen Zhou, Zhaoyang Lyu, Fangzhou Hong, Shuai Yang, Bo Dai, Xingang Pan, and Chen Change Loy. Gaussiananything: Interactive point cloud flow matching for 3d object generation. arXiv preprint arXto:2411.08033 2024.
Quentin Leboutet, Nina Wiedemann, Zhipeng Cai, Michael Paulitsch, and Kai Yuan. Midgard: Modular interpretable diffusion over graphs for articulated designs. In Advances in Newral Information Processing Systems, 2024.
Jiahui Lei, Congyue Deng, William Shen, Leonidas J Guibas, and Kostas Danillidis. Nap: Neural 3d articulated object prior. Advances in Neural Information Processing Systems, 36:31878 31894, 2023.
Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusson. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12651, 2023.
Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, and Ying Shan. Advances in 3d generation: A survey. arXiv preprint arXiv:2401118800 2024.
Yangguang Li, Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, et al. Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models. arXiv preprinl 250: 06608, 2025.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolttin text-to-3d content creation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
Jian Liu, Xiaoshui Huang, Tianyu Huang, Lu Chen, Yuenan Hou, Shixiang Tang, Ziwei Liu, Wanli Ouyang, Wangmeng Zuo, Junjun Jiang, et. al. A comprehensive survey on 3d content, generation. arxiv preprint wwXiv22422001166, 2024a.
Jiayi Liu, Hou In Ivan Tam, Ali Mahdavi-Amiri, and Manolis Savva. Cage: Controllable articulation generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 17880-17889, 2024b.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651- 15663, 2020.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.0788 2023a.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024c.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference computer vision, pages 0298-9309, 2023b.
Xiacxiao Long, Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song- Hai Zhang, Marc Habermann, Christian Theobalt, and Wenping Wang. Wonder3d: Single image to 3d using cross-domain difflusion. In CVPR, pages 9970 9980. IEEE, 2024.
William E. Lorenson and Harvey E. Clinc. Marching cubes: A high resolution 3d surface construction algorithm. In SIGGRAPH, pages 163 169. ACM, 1987.
Andrew Luo, Tianqin Li, Wen-Hao Zhang, and Tai Sing Lec. Surfgen: Adversarial 3d shape synthesis with explicit surface discriminalors. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pages 16238 16248, 2021.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2837-2845, 2021.
Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan, and Dahua Lin. A conditional point diffusion- refinement paradigm for 3d point cloud completion. arXiv preprint arXiv:2112.055. 2021.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 922-928. IEEE, 2015.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CV conference computer vision and pattern recognition, pages 4460 1460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 306-315, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multircsolution hash encoding. ACM transactions on graphics (TOG), 1144:11115, 2022.
Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero, 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint wxXi:2212.008422 2022.
Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference on machine learning, pages 7220-7229. PMLR, 2020.
Thu H. Nguyen-Phuoe, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. Advances in neural information processing systems, 33:6767-6777 6778, 2020.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: system for generating 3d point clouds from complex prompts. ar preprint 222022222.08711, 2022.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pages 165 174, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Iision-ECCC 2020: 16th European Conference, Glasgou, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit Bermano, Eric Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State the art on diffusion models for visual computing. In Computer graphics forum, volume 43, page e15063. Wiley Online Library, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arX:::2209. 14988, 2022.
Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xeube: Large- scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4209-4219, 2024.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pages 10684-10695, 2022.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. +erol23++: a single image to consistent multi-view diffusion base model. arXiv preprint arX::22310.1111 15110, 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusson for 3d generation. arXiv preprint arXiv:2308.16511 16512, 2023b.
Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3d point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the TEEE/CVF international conference on computer vision, pages 3859-3868, 2019.
Yawar Siddiqui, Antonio Alliegro, Alexcy Artemov, Tatiana Tommasi, Danicle Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshipp: Generating triangle meshes with decoder-only transformers. arXiv preprint arXio:2311.15475 2023.
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256 2265. PMLR, 2015.
Cheng Sun, Min Sun, and Hwamn-Tzong Chen. Direct voxel grid optimization: Super-fast. convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5459-5463 2022.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xic, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2810.16816 2023.
Yongbin Sun, Yue Wang, Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Antoregressively learned point cloud generation with self-attention. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 70, 2020.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arxiv preprint Xrx:23300. 16653, 2023.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Chen Wang, Hao- Yang Peng, Ying-Tian Liu, Jiatao Gu, and Shi-Min Hu. Diffusion models for 3d generation: A survey. Compulational Visual Media, 11(1):1-28, 2025.
Hao-Yang
Peng-Shuai Wang, Yang Liu, and Xin Tong. Dual octree graph networks for learning adaptive volumetric shape representations. ACM SIGGRAPH 2022 Conference Papers, 2022a.
Peng-Shua Wang, Yang Liu, and Xin Tong. Dual octree graph networks for learning adaptive volumetric shape representations. ACM Transactions on Graphics (TOG), 41(4): 15, 2022b.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognilion, pages 4563 4563-4573, 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Newral Information Processing Systems, 36, 2024.
Suttisak Wizadwongsa, Jinfan Zhou, Edward Li, and Jeong Joon Park. Taming forward reconstruction models as latent encoders for 3d generative models. arXiv preprint ar 2tr:2501.00633 2024.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3d goncrative-adverarial modeling. Advances in neural information processing systems, 29, 2016.
Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages #29-838, 2020.
Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Torr, Xun Cao, and Yao Yao. Direct3d: Scalable image-to-3d generation via 3d latent diffusion transformer. arXiv preprint arXiv:2405. 14832, 2024a.
Zhemnan Wu, Yang Li, Han Yan, Taizhang Shang, Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, et al. Blockfusion: Expandable 3d scene generation using latent tri-plane extrapolation. ACM Transactions on Graphics (TOG), 43(4):1-17, 2024b.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on compuler vision and paldern recognilion, pages 1912 1920, 2015.
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. ar preprinl arXiv:24 12.00566 2024.
Rengan Xie, Wenting Zheng, Kai Huang, Yizheng Chen, Qi Wang, Qi Ye, Wei Chen, and Yuchi Huo. Ldm: Large tensorial sdf model for textured mesh generation. arXiv preprint arXiv:2405. 14580 2024.
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in neural information processing systems, 32, 2019.
Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Shape- former: Transformer- based shape completion via sparse representation. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 6239-6249, 2022.
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541-444 1551045500 2019.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Xianghui Yang, Huiwen Shi, Bowen Zhang, Fan Yang, Jiacheng Wang, Hongxu Zhao, Xinhai Liu, Xinzhou Wang, Qingxiang Lin, Jiaao Yu, et al. Hunyuan3d-1.00 A unified framework for text-to-3d and image-to-3d generation. arXiv preprint arXio:2411. 02299,, 2024.
Maciej Zamorski, Maciej Zikeba, Piotr Klukowski, Rafal Nowak, Karol Kurach, Wojeiech Stokowiec, and Tomasz Trzciński. Adversarial autocncoders for compact representations of 3d point clouds. Computer Vision and Image Understanding, 198:102921, 2020.
Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generalive diffusson models. ACM Transactions on Graphics (TOG), 42(4):1 16, 2023.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. arXiv preprint arX::22406. 13897, 2024.
Ke Zhao and Andreas Larsen. Challenges and opportunities in 3d content generation. arXio preprint 2XXi:22455.1395 2024.
Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-imago-text aligned latent representation. In NeurIPS, 2023.
Xin- Yang Zheng, Hao Pan, Peng-Shual Wang, Xin Tong, Yang Liu, and Heung- Yeung Shum. Locally allentional sdf diffusion for controllable 3d shape generation. arXiv preprint urXiv:2305.044 2023.
Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.0677 2023.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 5826-5835, 2021.
Appendix
A Representations and Algorithms in 3D Generative Modeling
Meshes are foundational in 3D computer graphics and vision, representing surfaces in terms of vertices, edges, and faces to form polygonal approximations of objects. While meshes are extensively used across various fields due to their broad compatibility with rendering pipelines and ability to depict complex geometries with high fidelity, processing them within neural network architectures presents significant. challenges. The variable topology and connectivity inherent in meshes make them difficult to process using standard neural architectures that expect regular input structures. Consequently, processing meshes often requires special- ized neural networks, such as graph neural networks (Li et al., 2024), autoregressive architectures (Yan) et al. 2022 Mittal et al., 2022 Siddiqui et al.) 2023 which can be complex and computationally ineffi cient. Moreover, high-resolution meshes can have substantial memory footprints, raising concerns about the scalability of the resulting learning pipclincs. These limitations have led researchers to explore alternative three-dineensionll representations derived from meshes through algorithmic conversions that typically do not require training to facilitate processing, learning, and rendering within deep neural frameworks Li et al. 2024h.
Voxel grids reprosentations in 2D images. Early rescarch predominantly relied on dense voxel grids (Wu et al. 2015
Voxel grids offer intuitive and straightforward encoding for 3D objects analogous to pixel
Maturana and Scherer 2015 Choy et al. 2016 Wu et al. 2016 Brock et al. 2016 Dai et al. 2018). Despite their convenience, voxel grids suffer from a large memory footprint, limiting the achievable resolution and computational efficiency. Recent methods have revisited voxel grids to leverage their compatibility with convolutional neural networks (CNNs). For instance, X-Cube (Ren et al. (024)) introduces a multi-resolution approach involving denoising and decoding steps, while methods like LAAS-Diffusion (Zheng al. 2023 and
20

---

Published in Transactions on Machine Learning Research (08/2025)
One-2-3-45++ (Luu et al., 2023a utilize voxel grids in the initial stages of their generation pipelines. These approaches demonstrate that voxel grids can still be effective when combined with modern techniques like diffusion models. To mitigate memory constraints, sparse or hierarchical voxel grids have been proposed {Liu ct al. 2020 Fridovich-Keil et al.] 2022. Sun et al., 2022). Notably, Instant Neural Graphics Primitives (InstanI-NGP) (Mäller et al. 2022 use a multi-level voxel grid encoded via a hash function, enabling fast
optimization and rendering while maintaining a compact model size.
Implicit neural functions Implicit representations model 3D geometry continuous functions, allowing for high-resolution detail without the memory overhead of dense grids. Occupancy Networks (Mescheder ct al. 2019) introduced an implicit representation that predicts the occupancy probability of arbitrary points in 3D space, conditioned on input data such as images or point clouds. This approach allows for smooth and detailed surface representations. Extensions of this idea. include Convolutional Occupancy Networks (Peng iet al. 2020 and Neural Implicit Surfaces (Atzmon and Lipman 2020), which improve the ability to capture fine geometric details. Recently, Zhang et al. (2023) proposed 3DShape2VecSet, employing a Transformer- based encoder and decoder where occupancy
query points is predicted via cross-attention mechanisms. This represcntation and follow-up work Chen et al. (2024a) have been used for conditional generation, achieving remarkable results (Zhang el al.] [024 Yang el al. 2024 Li et al. 2025). In contrast lo these works, Neural Radiance Fields (NeREs) (Mildenball et al. 2021 jointly model geometry and texture using implicit representations, enabling rendering from arbitrary views via volume ray casting. NeRFs are especially popular for 3D generative methods that only rely on image data for supervision, such as DreamFusion (Poole
et al. 2022).
Signed Distance Functions (SDF) Signed Distance Functions (SDFs) provide a scalar field where each point in space is assigned value representing its signed distance to the closest surface, with negative values indicating points inside the object. SDFs offer more expressive alternative to occupancy functions, capturing both the geometry and topology of 3D shapes. DeepSDF (Park et al. 2019) pioncered the use of neural networks to learn continuous SDF representations from data. Building on this, DISN (Xu et al. 2019) introduced an amortized approach that eliminates the need for test-time optimization, enabling more efficient inference. SDFs can be discretized over a voxel grid, resulting in a sampled SDF or grid SDF, which facilitates the use of convolutional architectures. SDFusion (Cheng al. 2023) demonstrated the effectiveness of this approach for 3D generation. Additionally, Mittal et al. (2022) proposed AutoSDF, which generates SDF grids for shape patches autoregressively after encoding them with a VQ-VAE. Further advancements include methods like 3D-LDM (Nam et al.) 2022), where an MLP autoencoder predicts SDF values from latent codes and query points, blending occupancy and SDF representations. Xie et al. (2024) compared different representations, concluding that tensorial SDF' outperform triplane SDFs and tensorial NeRFs terms of fidelity and efficiency.
Triplanes Triplane representations encode 3D scenes using three orthogonal feature planes (xy, æz, yz) E NNNNC significantly reducing memory requirements while retaining spatial information. This approach was popularized by EG3D (Chan et al., 2022), which employs a triplane representation within a Generative Adversarial Network (GAN) framework for high-quality 3D-aware image synthesis. Recent works have ex- tended triplane representation ations to 3D) generative modeling. Approaches such as 3DGen Gupta et al. (2023)
RODIN Wang et al.) (2023), , Blockfusion Wu et al. (2024b) and Direct3D Wu et al. (2024a) rely on latent diffusion to denoise and then up-sample triplane latents, which are then decoded via a lightweight MLP occupancy network and rendered volumetrically, facilitating the generation of high-fidelity 3D avatars or even complete 3D environments. Practically, triplanes can be obtained from occupancy data by pretraining a dedicated PointNee-UNNe--cccVee autoencoder on a reconstruction loss, as detailed in Peng et al. (2020). In this approach, the triplane latent resolution can be adjusted using specific UNet layers within the autoen- coder. Alternatively, a triplane dataset can be directly optimized from a mesh dataset, as proposed in Wu here as an optimization
ct al. (2024b). by jointly training an MLP decoder and its triplanc input-considens variable on par with the weights and biases of the MLP decoder on a reconstruction loss over the entire dataset. In this context, since the triplane equivalent of each asset in the database is readily available after pretraining, the encoder part of the generative approach consists only of projection layers. These layers encode high-dimensional triplanes into diver-dimeniional triplane latents, on which the denoising diffusion model will subsequently act.
21

---

Published in Transactions on Machine Learning Research (08/2025)
Apart from reconstruction and generation tasks, 3D representations were leveraged for CLIP-like foundation models that learn an encoding of 3D objects, which is used for downstream tasks such classification segmentation Zhou et al. (2023); Hegde et al. (2023).
B Generative pipelines
Diffusion models (Sohl-Dickstein et al. 2015) have emerged as a powerful class of generative models, demon- strating remarkable success in image synthesis (Ho et al.) 2020). They operate by progressively corrupting training data through the sequential addition of Gaussian noise (forward process) and then learning to re- cover the original data by reversing this noising process (reverse process). This framework has been extended to 3D data, allowing for the generation of complex 3D structures (Lyu al. 2021 Zhou al. 2021
Forward Process Given data sample æ5 drawn from distribution y (ara), the forward diffuusion process generates sequence of increasingly noisy samples 22, Vt [I,... -]] from æn by adding Gaussian noise ~~ N (0, I) at each timestep according to predefined variance schedule 0 < B1 <.. < Br < 1:
T q (11r)k0))) == II q x1,1), t=1
(1)
q (2 = N ( B,a,,,
(2)
A relevant property of such diffusion process that 24 can be sampled from mg using the closed-form expression:
((<,|<0)) .V V(Var (1 ā.) I),
(3)
where Or = 3g and a₂ = II Ot.- This leads to the practical sampling equation:
a4 Vojag + v1 ā,e, E e~N(o.T).
(4)
Reverse Process. Initiating from a standard Gaussian distribution, ør (0, I), a denoising model po parameterized by trainable weights 0, learns to approximate series of Gaussian transitions po (a:-1 lx.). These transitions incrementally denoise the signal such that
T
(Morr = (FT) 1=1
Po II
(5)
Po (x 1
= (1s (Lt, 1). Eg (xx,, ())-
(6)
Following the approach from Ho al. (2020), we define
B.
μe
Zt
(x,, t)
(7)
a2
Eg (ate t)
of
(8)
yiclding the following Langevin dynamics
Br VI- ai Eo (x). + t,z
xt-1
(9)
Vas
where 2 (O, and Eg (z,,t) is a learnable network approximating the per-step noise on æt. Loss Function The model is trained by minimizing the variational bound on negative log-likelihood. However, Ho et al. (2020) showed that simplified loss focusing on the noise prediction yields good empirical rosults:
L = Exo.er [||Er Ey (Varzo + vI a,e. t) [?
(10)
This loss encourages the network to predict the noise added each timestep, facilitating the denoising process during generation.
22

---

Published in Transactions on Machine Learning Research (08/2025)
We adopt open-source implementations for Di and 3D U-Net Specific parameter settings will be provided as hydra config files with our open-source codebase.
C Evaluation Metrics
We use the following metrics for evaluating our approach. measure the distance between two point clouds and Y.
The symmetric Chamfer distance is selected to
CD(X,Y)
min Σ
|x
yl12
min lyy x||2. xeX yEY
(11)
X
yEY
|
The F-score is the harmonic mean of precision and recall for generated mesh g and reference mesh R. The precision is defined as
100
min lgg r|1 <T:
P(t)
(12)
PE
with g as points sampled on the surface of the generated mesh, r points sampled from the reference mesh, I as a threshold, and []] as the Iversion bracket. The recall is defined accordingly as
100 R
R(T)
min || REG
8ll2 <T
(13)
The final F-score is then computed as
2P(T) F(T) = P(T) +R(T)
(14)
We define the normal consistency between a generated mesh G and reference mesh R as
NC(G, R)
Σ BEG
(15)
G
The function n(.) retrieves the normal the respective point, g is a point sampled on the surface of the generated mesh and N, is the closest point on the point g. (..-) the dot product which measures the similarity of the normals.
To measure the quality of the generated meshes w.r.t. set of reference meshes the metrics Coverage (COV), Minimum matching distance (MMD), and 1-nearest neighbor accuracy (1-NNA).
Karg miny es, D(X,Y)) E S
COV(S,, S,)
(16)
S,
S,)
Σ
min D(X,Y) Xes,
(17)
[S, YES.
I.NNA(S,. Exes, I[N2 S. + Eyes, HN,, E S.]
(18)
S SS
Nx is the nearest neighbor of X in the set S, U. - (X}. Sr, S are the sets of reference and generated meshes and are of equal size. We use the Chamfer distance for in all our experiments.
23

---

Published in Transactions on Machine Learning Research (08/2025)
D
Training Details
All training parameters can be found in the config files in the configs directory in the root of the repository. The main entry point is the configs/traing yaml files. For specific models and experiments, training param- eters can be found in the respective confige/model and eexfiigeexppriment subfolders. For instance, the configuration for the generation experiment with the SDF representation using an autoencoder and diffusion såf laff__a____e ae ddit_airplane yaml.
transformer on the Airplane category in maffig//xxerimmntte
We give summary of the most important training parameters here. We trained all models using Adam optimizer until convergence. The learning rate and schedule highly depends on the model used and have been found by hyperparameter search. For most experiments, we use biep-wise exponential decay learning rate schedule. For Shape2VecSet and Triplane models, we use a half-cycle cosine decay started after a warm-up phase. We have trained the models using up to 2 NVIDIA A6000 GPUs. Training times for all models are within days.
E Reconstruction and generation results on Objaverse data
To validate our results on a second widely-used dataset, we repeat the analysis on Objaverse data. We selected five categories for which most category-labeled samples were available: chair, seashell, apple, banana, mug, shield and skatcboard (together 876/113 (0//11//18) main/val/test samples). We first train the encoders, then the diffusion models on the samne data with the same split.
The reconstruction results are shown in Table 4 These results mostly align with the ShapeNet results: 1) The value ranges are similar. 2) They confirm SDF AE as the best method for roconstruction. 3) The ranking of methods mostly aligns (compare Table 3). The only change for Objaverse for DualOctree, surpassing Shape2vecset; however, these methods had similar F-score for ShapeNet (F-score of 79 for Shape2VecSet vs 76 for DualOctree).
better performance
Table 4: Reconstruction performance of best generation approaches on the Objaverse dataset.
Representation Encoder
F-score (0.0125) (7 +
CD (+le 4)
NC
DuslOctrew
VAE
83.86 +: 15.173
0.016 10.51
0.857 0.081
SDF
AE
96. 19.423 6.107
0.009 4.948
0.896 0.062
Shape2VecSet
AE
67.578 28.472
0.045 55.053
0.815 0.128
Triplane
AF
66.883 + 21 21.589
0.036 + 41.114
0.82 + 0.092
Voxel
AE
96.878 6.935
0.008 7.339
0.85 + 0.06
Furthermore, Table shows the metries for unconditional generation. Only the Triplane results improved considerably compared to the experiments ShapeNet, surpassing Shape2Vecset in the Objaverse experi- ments. The main result, putting DualOctree and SDF as the best methods, remains the same.
It is worth noting that we computed the metrics using 183 samples for the generated and reference set respectively, since this is the size of the test set. As shown in Figure 2 low sample size biases the results. Indeed, we see higher coverage and lower 1-NNA than for our evaluation on ShapeNet, where we evaluated on 400 samples.
Table Evaluation metrics for unconditional generation trained on Objaverse dataset including COV, MMD, and I-NNA scores.
Representation Generator Encoder
COV
MMD
1-NNA
DualOctree
VAE
UNet
0.47541
0.0237217
0.584699
SDF
AE
DiT
0.491803
0.0277514
0.68306
Shape2VecSet
AE
DiT
0.349727
0.0298354
0.819672
Triplane
AE
UNet
0.47541
0.0248811
0.729508
Voxel
AE
DiT
0.409836
0.0371019
0.803279
24

---

Published in Transactions on Machine Learning Research (08/2025)
F
Ablation study: Reconstruction performance
Table 6 shows the results for reconstruction for the ShapeNett Chair category across all tested encoders. In most cases, standard auloencoder (with LayerNorm) performs best. This is expected since penalizing the KL-divergence in VAËs stirs the distribution at the cost of lower reconstruction performance. However, we find that the encoder choice only plays a minor role compared to the differences between representations. Specifically, the inter-representation standard deviation (belween best CD per representations) 0.0098 whereas the intratrepresentttion standard deviation (between encoder-wise CD for each representation) is only 0.005
Representation Encoder
F-score (r 89)
CD (*1o-4)
NC
DualOctree
AE
91.629
0.012
0.821
DualOctree
VAE
83.152
12.38
0.017
0.01
0.798
t: 0.07
DualOctree
VQVAE
73.763
14.23
0.023
0.01
0.773
0.07
NeRF
AE
58.162
t 13.74
0.032
0.02
0.714
+ 0.07
NeRF
VAE
57.326
0.033
0.786
SDF
AE
94.332
6.69
0.011
0.01
0.847
+ 0.06
SDF
VAE
92.497
7.05
0.013
0.0
0.842
SDF
VQVAE
94.355
+ 6.37
0.011
0.01
0.84
= 0.06
Shape2VecSet
AE
85.127
13.81
0.019 0.01
0.812
+ 0.06
Triplane
AE
63.666
17.73
0.033
0.03
0.761
Triplane
VAE
43.952
14.02
0.052
0.04
0.733
1 0.08
Triplane
VQVAE
45.216
14.22
0.046
0.733
0.08
Voxel
AE
90.433 + 13.14
0.016
0.02
0.821
+ 0.07
Voxel
VAE
90.5
12.93
0.016
0.821
Voxel
VQVAE
90.408
13.03
0.016 0.02
0.821
+ 0.07
Table 6: Reconstruction performance by encoder (solely ShapeNet-Chair category).
Furthermore, Figure shows qualitative results for reconstruction. Thin or delicate structures lead to visible errors as missing parts of the object or as loss of details. This error mode is common for all methods and includes grid-less methods like NeRF.
G Ablation study: Generation performance
Table 7 compares all variants of the generative model. The applicability of DIT and Unet depends signifi- cantly on the structure of the latent vector. In most cases, AE outperforms VQ and VQVAE. DiT is more suitable for NeRF, Voxel grid and Shape2VecSet, whereas U-Net yields better results with DualOctree, SDF grid and Triplane. Interestingly, DiT also works well for grid-based representations that were usually just used with U-Net based diffusion in related work. Due the shorter training times and higher memory efficiency of DiT, this is promising result for future model development.
F Figure and
Figure 10 provide further qualitative results for the best generative approaches per represen-
tation.
H Mesh reconstruction under varying preprocessing methods
H.1
Preprocessing non-watertight meshes
The analysis provided in subsection 4.4 shows that mesh preprocessing methods, such Mesh2SDF substantially alter the meshes by making them thicker and thereby introduce bias. In accordance with the
The intra-reprresentation StD was only computed for representations where results available for all three encoder models
25

---

Published in Transactions on Machine Learning Research (08/2025)
GT
DualOctree AF
Triplane AF
NeRF AF
Voxel AF
SDF AF
Shape7Vec5et AF
Figure 8: Qualitative results for mesh reconstruction.
26

---

Published in Transactions on Machine Learning Research (08/2025)
SDF AE DIT
Voxel AE DiT
DualOctree VAE UNet
Shape2VecSet AE DIT
Triplane AE UNet
Figure Qualitative results for the Airplane calegory
27

---

Published in Transactions on Machine Learning Research (08/2025)
SDF AE DIT
Voxel AE DiT
DualOctree VAE UNet
Shape2VecSet AE DIT
Triplane AE UNet
Figure 10: Qualitative results for the Car category
28

---

Published in Transactions on Machine Learning Research (08/2025)
Representation
Generator
Encoder
COV
MMD
1-NNNA
DualOctree
Unet
AE
0.367
0.059
0.900
DualOctree
Unct
UNet
0.458
0.046
0.686
DualOctroe
Unet
VAE
0.440
0.047
0.662
DualOctree
Unet
VQVAE
0.393
0.049
0.708
NeRF
DiT
AE
0.240
0.085
0.965
SDF
DiT
AE
0.432
0.048
0.726
SDF
DiT
VAE
0.445
0.050
0.718
SDF
DiT
VQVAE
0.385
0.053
0.728
SDF
Unet
AE
0.372
0.054
0.787
SDF
Unet
VAE
0.378
0.058
0.743
SDF
Unet
VQVAE
0.347
0.054
0.744
Shape2VecSet
DiT
AE
0.400
0.048
0.790
Triplane
Unet
AE
0.422
0.052
0.815
Voxel
DIT
AE
0.427
0.056
0.894
Voxel
DiT
VAE
0.388
0.066
0.878
Voxel
DiT
VQVAE
0.357
0.060
0.891
Voxel
Unet
AE
0.385
0.059
0.881
Voxel
Unet
VAE
0.380
0.058
0.828
Voxel
Unet
VQVAE
0.438
0.061
0.826
Table 7: Ablation study on generation performance (only "Chair" category). Combinations that did not converge to state of proper 3D model generation are left out.
0.07
Chamfer Distance
0.06
0.05
sdf fill
0.04
sdf naive
0.03
mesh
0.02
0.01
0.00
64
128
256
384
Grid size
Figure 11: Comparison of different mesh conversion methods in terms of Chamfer distance.
F-score comparison in Figure 7b] the comparison of CD provided in Figure 11 shows that our preprocessing step yields a mesh that diverges less from the original data.
illustrates the reconstruction of an airplanc from an SDF grid with a resolution of 643 We first apply the flood-fill algorithm ensure walertightness. Then, we either directly transform the mesh into sampled SDF and back to mesh FFiuure 12b apply Manifold and then transform and reconstruct the mesh Figure 12ch.
Figure 12
Without applying Manifold, the limited grid resolution fails to capture thin struct ures such as airplane wings accurately, leading to incomplete or distorted reconstructions. Conversely, manifoldizing the mesh ensures that thin structures are represented as solid volumes, allowing grid-based methods to capture these features within the constraints of the grid resolution. However, this comes at the expense of altering the original mesh geometry, which may not be desirable in applications requiring high fidelity.
29

---

Published in Transactions on Machine Learning Research (08/2025)
(a) Ground truth
(b) Reconstructed w/o Manifold
(c) Reconstructed with Manifold
Figure 12: Reconstruction quality when converting to sampled SDF and back
H.2 Dataset quality comparison
This reconstruction analysis can not only be used to compare preprocessing methods, but also to compare the quality of datasets. We analyze ShapeNet and Objaverse with our pipeline by converting all meshes SDF grids and then convert this representation back to the mesh format. Figure 13 shows the distribution of reconstruction errors measured here F-score. We observe that a larger fraction of objects in Objaverse is of lower quality, ie., conversion to implicit representation produces larger error.
Further, we can observe on the ShapeNet dataset that quality differs significantly between categories. Fig- ure 14 shows the reconstruction error for the three categories airplane, car, chair. The F-scores for the car category are low even for a grid of size 3843, which we attribute to internal structures not captured by our preprocessing method.
17500
200000
15000
12500
150000
10000
100000
7500
5000
50000
2500
0
20
40
60
80
100
0
20
40
60
80
100
F-score with = voxel size
F-score with T= voxel size
(a) ShapeNet
(b) Objaverse
Figure 13: Distribution of F-scores between original and reconstructed mesh using SDF grid of size 3843. The threshold for the F-score is of the size of voxel.
Representation Scalability in Complex Objects
We analyze how well representations handle complex shapes. We approached this in three ways: (1) We
measure object complexity in the ShapeNet test set and relate to the reconstruction performance, (2) We use shapes with sharp edges (cube), smooth surfaces (sphere), and complex fractal surfaces (Mandelbulb), and trained the methods to encode and reconstruct them, and (3) we evaluate the complexity the generated objects study difficulties in generating complex objects. "Complexity" in (1) and (3) is measured using triangle count and nurface-tovolumee ratio ($).
30

---

Published in Transactions on Machine Learning Research (08/2025)
80
F-score with T= voxel size
75
70
8
65
60
airplane
55
chair
50
car
64
128
256
384
Grid size
Figure 14: Effect of the grid resolution the round trip conversion errors from mesh to SDF grid and back with our fold-fill method. Note that the threshold т for computing the F-score is relative to the voxel size of the used grid.
100
40
1 (12 60)
Surface volume ratio (quantlee)
1106 148)
(186 2411
20
1/6 -.3kk
Number of faces (quamile)
3 (5.8k 1910
5 (42k-82k)
6244
Triplane
NARF
DualOctree Shape2Veccet
Triglane
NeRF
Vowel
Dua Muctrre
Shape22
AE
Figure 15: Complexity analysis. Best viewed zoomed in.
For approach (1), i.c. mcasuring object complexity in ShapeNet, Figure 15 (a) shows that voxel grids and NeRF suffer from higher &, whereas other representations not allected. Figure 15 (b) shows that higher triangle count impedes the reconstruction performance for all representations. Shape2VecSeds however, is expected to better deal with high $ when more points are sampled. Furthermore, our analysis of specific complex shapes (approach (2)) shows that there are failure cases when reconstructing fractal shapes. SDF- based generation smoothes the surfaces, whereas 3DShape2 VecSet introduces small artifacts (see) Figure 16)
3DShape2VecSet- AE
DualOctree VAE
Ground truth
SDF-AE AE
Figure 16: Mandelbulb reconstruction
Approach (3), i.c. evaluating the complexity of generated objects, finds that the is generally lower for
generated objects (see Figure 17), specifically 40 + 18 for SDF-generated, 61 + 27 for DualOctree-generatel
31

---

Published in Transactions on Machine Learning Research (08/2025)
0.05
ShapeNet
DualOctree
0.04
SDF
Shape2VecSet
Voxel
Density
0.03
Triplane
0.02
0.01
0.00
10a
102
Surface-to-volume ratio
Figure 17: Surface-to-volmme ratio for generated objects and ShapeNet, using categories airplane, and chair.
vs 90 for ShapeNet objects (N = 400). This suggests a bias towards generating simple, smooth objects. Training on more complex assets, e.g. Objaverse, could alleviate this issue.
J User Study
To capture human preferences of objects generated by different methods, we asked 24 users to rate objects generated for the chair category. Each user was shown pairs of objects and asked to select the one they preferred considering the object complexity and surface quality. To rank the methods we use the Bradley- Terry model, which models the probability method being better than method B as
exp(pл)
P(A> B) exp(pa) exp(PB)
(19)
The parameters PA and py are scores for the respective methods. We estimate the scores p by minimizing the cross entropy CE
min CE((D.)))
(20)
with p as the vector of scores and y as the preference labels collected from the participants. The results are visualized in Figure 3 showing that objects generated with the SDF representations are preferred over the other representations.
32


Reviews:
[{"review_id": "9vWRMSTEBJ", "reviewer": "Reviewer_fZtW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Thorough benchmarking of a wide range of 3D representations, including SDFs, meshes, voxel grids, vecsets etc, which would be useful for the 3D generation community.\n- Proposed a unified framework that integrates all 3D representations for fair comparison and evaluation.\n- Sufficient experiments of different encoders and evaluation of generation and reconstruction performance, including geometric metrics and statistic analysis and user study.\n- Utilize Dual Octree graph encoding, which achieves a good performance over other representations.\n\nWeaknesses:\n- The experiments were done on simple ShapeNet datasets, which is less used with Objaverse becoming the main training datasets. Therefore, the benchmark might not be general enough.\n- Some of the conclusions are a bit surprising. For example, Shape2VecSet doesn't achieve not good results, however, most of the recent SoTA models are based on vecsets. For some representations (Voxel), using AE achieves better generation performance than VAE, which also not aligns with common experience.  There are no analysis towards these points.\n- Lack of training details (iterations, training efficiency, model size etc) of the encoding and generation. Lack of visualization results.\n- Missing citation: [1] State of the art on diffusion models for visual computing, [2] Diffusion models for 3D generation: A survey, [3] A survey on generative diffusion models", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a unified evaluation framework across different 3D representations (voxel grids, SDFs, point clouds etc.) to assess their performance in 3D reconstruction and generation. The authors transformed ground truth meshes into given 3D representations and trained different encoding networks to compress them into vectors. The generation network is trained on the latent space. The authors evaluated the performance of both reconstruction and generation using standard metrics and user studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address the weaknesses part.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mB0BYWGnqx", "reviewer": "Reviewer_bY2D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This work offers a \"universal\" framework to compare 3D representations.\n- It also highlights that generation and reconstruction capability should be considered jointly during evaluation.\n- The paper is well-written and organized.\n\n### Weaknesses\n- The authors should better clarify why the proposed framework is better to benchmark 3D representations instead of reconstructing/generating with the standard original recipe of the different models.\n- Gaussian splatting methods are missing in the benchmark. I think the authors should incorporate 3DGS into the framework, as it's now the state-of-the-art (SOTA).\n- Some sentences need to be clarified (e.g., \"To ensure that the latent vector behaves well [...]\", what does it mean by \"behaves well?\").", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a systematic comparison of 3D representations for reconstruction and generation in a common framework.\nThe steps involved in this framework are: transforming the mesh into a suitable representation, using an encoder to compress it into a latent vector, applying a denoising model to denoise it, and finally, using a decoder to decompress and transform the representation back into a mesh. 3D representations are compared on quality, computational efficiency, and generalization performance metrics. The proposed framework also provides insights into best practices for 3D representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- See weaknesses.\n- Additional references can be added (\"3D Gaussian Splatting for Real-Time Radiance Field Rendering\" by Kerbl et al., \"Uni3D: Exploring Unified 3D Representation at Scale\" by Zhou et al.,  \"Boost Your NeRF: A Model-Agnostic Mixture of Experts Framework for High Quality and Efficient Rendering\" by Di Sario et al.).", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PRxf78rf7u", "reviewer": "Reviewer_RxWz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is well-written, with clear structure and illustration.\n2. The representations covered are comprehensive and clearly introduced. The proposed pipeline is clear, and easy to adapt to new representations.\n\nWeakness:\n1. The proposed pipeline is limited to the latent diffusion setting. Although a new representation could be easily adapted, the latent-diffusion setting limits the scope of analysis.\n2. The evaluation is only conducted on ShapeNet. It is possible the evaluation results are largely affected by the dataset bias. The authors should provide the similar evaluation on at least one more dataset and compare the results.\n3. In the conclusion, the authors state \"the errors resulting from mesh reconstruction and representation encoding alone should be reported\". However, it is not clear how separated reported errors could benefit the generation task.\n4. Some claims are not clearly explained or supported with evidence. For example, the authors state in the limitation section that \"Consistent results across ShapeNet suggest similar outcomes would apply to Objaverse\", which is not supported by experiments on Objaverse. Also, in the conclusion part, the authors suggest that \"Results should not be distorted by derivatives obtained following the application of methods like Manifold, and avoid metrics that require additional preprocessing,\" which the reviewer do not find strong support other than Figure 10.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors implement a latent diffusion based generation pipeline to analyze different 3D representations. The authors analyze the reconstruction as well as generation capability. In addition, the authors provide some insights for common problems and derive best practices. The authors also provide an open-source codebase for the future research.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. For the Coverage (COV) metric in section 3, the equation (in the appendix) is not aligned with the text description. In Eqa. (16) there is no distance threshold while in the text description the authors mention \"within a certain distance\".\n2. For the user study result in section 4.1, it is interesting that only DualOctree-based method ranks significantly different.\nIt seems the user study and these metrics are evaluating different aspects. If I understand it correctly, user study is comparing among generated approaches while the metrics are evaluating each approach towards ground truth. A more fair comparison (or to say, closer setting) would be also providing the ground truth to the users.\n3. In Figure 4 qualitative results it is not clear why different representations are not using the same setting. For example, DualOctree is using VAE while others are using AE.\n4. For Figure 5 and its corresponding analysis in section 4.2, the authors could explain a little bit more and provide some clear conclusion.\n5. Section 4.4 provides some results (Figure 7) but the detailed approach is in Appendix, which would confuse the readers. The reviewer suggest detailing some approaches here, or moving the results to the Appendix with the approach.", "broader_impact_concerns": "No obvious concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)