MeshDiffusion: Score-based Generative 3D Mesh Modeling
Authors: Zhen Liu
Yao Feng
Michael J. Black
Derek Nowrouzezahrai
Liam Paull
Weiyang Liu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: generative model
diffusion model
3D mesh
shape generation

Abstract: We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and physical simulation. Compared to other 3D representations like voxels and point clouds, meshes are more desirable in practice, because (1) they enable easy and arbitrary manipulation of shapes for relighting and simulation, and (2) they can fully leverage the power of modern graphics pipelines which are mostly optimized for meshes. Previous scalable methods for generating meshes typically rely on sub-optimal post-processing, and they tend to produce overly-smooth or noisy surfaces without fine-grained geometric details. To overcome these shortcomings, we take advantage of the graph structure of meshes and use a simple yet very effective generative modeling method to generate 3D meshes. Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parameterization. We demonstrate the effectiveness of our model on multiple generative tasks.

Published as a conference paper at ICLR 2023
MESHDIFFUSION:
SCORE-BARE GENERATIVE 3D MESH MODELING
Zhen Liu"?, Yao Feng" Michael J. Black,, Derek Nowrouzezahrai, Liam Paull', Weiyang Liu Mila, Université de Montréal
Max Planck Institute for Intelligent Systems Tübingen
ETT Zürich *McGill University UUniversity of Cambridge
Project Page: meshdi f fusion, .otasio..ggiiii...... wee.nnggthub...... github.io
snaycher
cadeed comonbie
Ddiagi
splaac paxtnng
Unconditionally generated 3D mesh sumples from MeshDiffusion
(b) Our meshes with text-conditioned
Figure I: (a) Unconditionally generated 3D mesh samples randomly selected from proposed MeshDiffusion, a simple diffusion model trained on a direct parametrization of 3D meshes without bells and whistles. (b) 3D mesh samples generated by MeshDiffusion with text-conditioned textures from [39]. Mesh produces highly realistic and fine- ine-grained geometric details while being easy and stable train.
ABSTRACT
We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such automatic scene generation and physical simulation. Com- pared to other 3D representations like voxels and point clouds, meshes are more desirable in practice, because (1) they enable easy and arbitrary manipulation of shapes for relighting and simulation, and (2) they can fully leverage the power of modern graphics pipelines which mostly optimized for meshes. Previous scal- able methods for generating meshes typically rely on sub-optimal post-pree post-processing, and they tend to produce overly-smood geometric details. To overcome these shortcomings, we take advantage of the graph structure meshes and use a simple yet very effective generative modeling method generate 3D meshes. Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parametrization. We demonstrate the effectiveness of our model multiple generative tasks.
or noisy surfaces without fine-gr -gaiindd
INTRODUCTION
As of the most challenging tasks in computer vision and graphics, generative modeling of high-quality 3D shapes is of great significance in many applications such as virtual reality and metaverse [11]. Traditional methods for generative 3D shape modeling usually built upon repre- sentations of voxels [51] or point clouds [1], mostly because ground truth data of these representations are relatively easy obtain and also convenient to process. Both representations, however, do not produce fine-level surface geometry and therefore cannot be used for photorealistic rendering of shapes of different materials in different lighting conditions. And despite being convenient to process
Work done partially during an internship Max Planck Institute for Intelligent Systems.

---

Published as a conference paper at ICLR 2023
for computers, both voxels and point clouds are relatively hard for artists to edit, especially when the generated 3D shapes are complex and of low quality. Moreover, modern graphics pipelines are built and optimized for explicit geometry representations like meshes, making meshes one of the most desirable final 3D shape representations. While it is still possible to use methods like Poisson reconstruction to obtain surfaces from voxels and points clouds, the resulted surfaces are generally noisy and contain many topological artifacts, even with carefully tuned hyperparameters.
To improve the representation flexibility, sign distance fields (SDFs) have been adopted to model shape surfaces, which enables us to use marching cubes [29] to extract the zero-surfaces and thus 3D meshes. However, SDFs are typically harder learn it requires a carefully designed sampling strategy and regularization. Because SDFs are usually parameterized with multi- perceptrons (MLPs) in which a smoothness prior implicitly embedded, the generated shapes tend to be so smooth that sharp edges and important (and potentially semantic) details are lost. Moreover, SDFs are costly to render and therefore less suitable for downstream tasks like conditional generation with RGB images, which require efficient differentiable renderer during inference.
We instead aim to generate 3D shapes by directly producing 3D meshes, where surfaces are repre- sented as a graph of triangular polygon faces. With 3D meshes, all local surface information is completely included in the mesh vertices (along with the vertex connectivity), because the surface normal of any point on the shape surface is simply nearest neighbor some local linear combination of vertex normals. Such a regular structure with rich geometric details enables us to better model the data distribution and learn generative models that are more geometry-aware In light of recent advances in score-based generative modeling [16, 47J where powerful generative performance and effortless training are demonstrated, we propose train diffusion models on these vertices to generate meshes. However, it is by no means a trivial task and poses two critical problems: (1) the numbers of vertices and faces are indefinite for general object categories, and (2) the underlying topology varies wildly and edges have to be generated at the same time.
natural solution to enclose meshes with another structure such that the space of mesh topology and spatial configuration is constrained. One common approach is to discretize the 3D space and encapsulate each mesh a tiny cell, and it is proven useful in simulation [37] and human surface modeling [35]. Observing that this sort of mesh modeling is viable in recent differentiable geometry modeling literature [32, 38, 43], we propose to train diffusion models on a discretized and uniform tetrahedral grid structure which parameterizes a small yet representative family of meshes. With such a grid representation, topological change is subsumed into the SDF values and the inputs to the diffusion model now assume fixed and identical size. More importantly, since SDF values now independent scalars instead of the outputs of an MIP,, the parameterized shapes are no longer biased towards smooth surfaces. Indeed, by such explicit probabilistic modeling we introduce explicit geometric prior into shape generation, because score matching loss of diffusion models on the grid vertices has direct and simple correspondence the vertex positions of triangular mesh.
We demonstrate that our method, dubbed MeshDiffisson, is able to produce high-quality meshes and enables conditional generation with a differentiable renderer. MeshDiffusion is also very stable to train without bells and whistles. We validate the superiority of the visual quality of our generated samples qualitatively with different rendered views and quantitatively by proxy metries. We further conduct ablation studies to show that our design choices are necessary and well suited for the task of 3D mesh generation. Our contributions are summarized below:
To our knowledge, we are the first to apply diffusion model for unconditionally generating 3D
high-quality meshes and to show that diffusion models are well suited for 3D geometry.
Taking advantage of the deformable tetrahedral grid parametrization of 3D mesh shapes, we propose a simple and effortless way to train a diffusion model to generate 3D meshes.
We qualitatively and quantitatively demonstrate the superiority of MeshDiffusion on different tasks, including (1) unconditional generation, (2) conditional generation and (3) interpolation.
2 RELATED WORK
3D Shape Generation. 3D shape generation commonly done by using generative models on voxels [51] and point clouds [53]. due to their simplicity and accessibility. These resulted (often noisy) voxels or point clouds, however, do not explicitly encode surface information and therefore has to be processed with surface reconstruction methods in order to be used in applications like relighting and
2

---

Published as a conference paper at ICLR 2023
simulation. Recent advances in implicit representations [31] lead series of generative models on neural fields [5, 6, 23, 42]. Based on these representations, one can learn 3D shapes directly from 2D images [5, 6, 42] with differentiable rendering in the generator. Most of these implicit representations, despite representing shapes in a more photorealistic way, require additional post-processing steps to extract explicit meshes, and are often more time-consuming to render images. If the underlying topology is known, one may directly generate the vertices of mesh [30, 483. [33] extends this approach the topology-varying case with autoregressive models, sequentially generating vertices and edges, but it is hardly scalable and yields unsatisfactory results on complex geometry. A batch of concurrent work propose similar solutions to mesh generation, including: GET3D [13] which uses StyleGAN [19] with a differentiable renderer on tetrahedral grid representations and learns to generate 3D meshes from 2D RGB images, TetGAN [14] which trains generative adversarial networks (GANs) on tetrahedral grids and LION [54] which uses trained Shape-As-Points [36] latent point clouds.
network to build meshes from diffusion-
Mesh Reconstruction. Reconstructing 3D meshes of generic objects is challenging and often ill- posed due to its highly complex structures [18, 20, 28, 50, 52]. One often resorts to some non-mesh intermediate representations that are easy to process, and then transforms them back to meshes with mesh reconstruction methods. One of the most popular methods is marching cubes [29] which assumes that surface is represented by the zero level set of some continuous field, and this continuous field can be well approximated by linear interpolation of discrete grid points. It is possible to avoid implicit representations and directly construct meshes. For instance, by assuming the points lie on a surface, one can build triangular meshes by connecting these points, a process known as Delaunay triangulation [3. 10, 15, 22]. It is relatively rare because of its strong assumption on data. With known mesh topology, one can deform a mesh template to fit given representation. This approach can be easily used fit multiview images of shapes with known topology (e.g.. human faces) using a differentiable renderer [25]. It is recently shown that we are able to parametrize meshes of varying topology [38, 43] and optimize them using differentiable renderers [32], which our work leverages.
Score-based Generative Models. Recent years have witnessed a surge in modeling data distributions with score-based models [16, 46, 47J, which parameterizes the logarithm of the gradient of the probability, known as the score function, rather than the probability directly. Different from energy- based models, the often intractable normalization constant can be avoided and therefore training can be performed by simply matching the score function. It has been shown that, by using multi-scale models [16]. U-Net architectures [41] and denoising score matching [49], score-based models can perform high-fidelity image generation and inpainting [16, 40, 47].
3 PRELIMINARIES
Deep marching tetrahedra (DMTet) [43] is a method to parametrize and optimize meshes of arbitrary topology differentiable way. The 3D space is discretized with a deformable tetrahedral grid, in which each vertex possesses SDF value. The SDF of each 3D position in the space is computed by marching tetrahedra [12], which assumes SDF values to be barycentric interpolation of the SDF values of the vertices of the enclosing tetrahedra. More specifically, for point xy inside a tetrahedron of vertices T1, 22, 33, 24 with SDF values being $1, $2, $3, $1, we obtain its unique barycentric coordinate (a1,02, 03; 04) such that N. = E, ag3; (a; [0, 1)). The SDF value 8, of 2ig is then computed as 8, = E, 048; As a result, if there exists a triangular mesh in the tetrahedron (i.e., $1, 82, $3, 84 are not of the same sign), then we can know that the triangular mesh is exactly the zero surface in the tetrahedron. The triangular mesh vertex Up on a tetrahedron edge (Va, vb) is therefore computed by - (v.58 - ubss.) (s6 sa). The advantage of using DMTet, compared to variants of deep marching cubes [26, 38] which rely on the marching cubes algorithm [29], is that the grids are deformable and thus more capable of capturing some finer geometric details like very thin handles of chairs. While the cubic grid used for deep marching cubes can be deformable as well, the deformed cubes are much worse objects to deal with, compared to tetrahedra which will remain tetrahedra after deformation. Notably, DMTet can be fitted with differentiable renderer by jointly optimizing for the geometry, the texture and the lighting of a 3D shape given multi-view RGB images within a reasonable amount of time and memory budget [32].
Diffusion models are a family of score- generative models which learn and infer with the ground-trute -ruth -ependeenddnt score function defined by a forward diffusion process [16]. Under the stochastic differential equation (SDE) framework proposed in [47]. diffusion models construct the probability of each point in space by diffusing all data points 35 D to target distribution at time T
3

---

Published as a conference paper at ICLR 2023
Reprovstation
Corniing
Cfftur conidrane 5OD valaa
Dervor EDC
Dedomable Tenueoka
Image Space
Tetrabekaa ard
Cubie
Signad Liuaree Field
Signol Distanne Ficid
Figure Overview of the proposed MeshDiffusion model.
(normally Pr (xT) = Nx(r: where pe (2 (zo) is modeled by the following Itô process, with w(t) being standard Wiener process: dr = F(x, t)dt - G(t)dw, which F(x, L) and G(I) are some predefined functions. The training objective is approximate the score function V, log (raa) with neural network denoted by
[ 0, I)) in a time-varian manner: log Pa (D) = ExpeD llog Pe (2) [20]].
sa (Te, t). where A(t) is a scalar weight subject to the model design:
L(O; D) -E
E
X(t) (=.. t) V. log (2r)
(1)
With the trained score function predictor, samples can be generated by solving the reverse SDE [47] with the initial distribution (xT)) using solvers like Euler-Maruyama method [21]:
dx (F(z,t) t) (z) log Po (2o)) G(t)dw.
(2)
popular diffusion model DDPM [16], which utilizes a discrete diffusion process p(ara (xx-1)
N Man - B,)ara-1, B),, be formulated within this SDE framework [47]: dr - -Barl
B.dw. The parameters 3r are chosen such that pr(rT) ) is close to a standard Gaussian distribution.
MeshDiffusion DIFFUSION MODEL ON MESHES
With DMTet, 3D meshes can be approximated and parameterized with tetrahedral grids. Moreover, by imitializing the tetrahedral grid using body-centered cubic (BCC) tiling [24] and treating the deformation each vertex only as an attribute, we obtain uniform and almost-regular tetrahedral grid (with fewer vertex degrees the boundary) in which 3D translational symmetry is preserved. Such nice properties enable us to easily train diffusion model for generating meshes. From now on, we always assume that the undeformed tetrahedral grid is uniformly initialized and predict the deformation of tetrahedral grid vertices from their initial positions. As result, our model takes as input uniform tetrahedral grid with 4-dimensional attributes (specifically, 3 dimensions are for deformation and dimension is for SDF). These structural priors inspire us to use 3D convolutions.
4.1 3D CONVOLUTIONAL NETWORKS FOR DEFORMABLE TETRAHEDRAL GRIDS
While it is natural to use graph neural networks (GNNs) for diffusion models on tetrahedral grids, we argue that is better to use convolutional neural networks (CNNs) which generally have better model capacity and contextual information than GNNs due to the embedded spatial priors in the convolution operators. Specifically, the set of vertex positions a uniform tetrahedral grid, created in the way described in [24], is subset of the vertex position set of uniform cubic lattice. A standard voxel-like representation can then be easily created by infilling the "missing" sites of a tetrahedral grid (illustrated in Figure 2). We follow the common practices diffusion models [47] and a 3D U-Net for the score function predictor. Detailed architectures and settings are in the appendix.
4.2 TRAINING OBJECTIVE
Since the tetrahedral grid representation of each object in the datasets is given, our training objective can be formulated by the following constrained optimization:
min
E SECai( [1 I..... T..... ae p (b6))
X(t) [[so(E,, t) V.
0² arg min Trended (yo (go)))
which is the 3D shape dataset, yo ~ D is the set of 2D views of an object sampled from D, t is uniformly sampled from (1, ., 7), s00 1) is the score function approximator parameterized by
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 Training and Inference
Algorithm 2 Conditional Generation
Training:
1: Fitting z' for cach E DD and normalize the SDF 1: Randomly initialize and fit deformable tetrahedral
values to =1, resulting the dataset (x' 2)) E DD'
grid using the given RGBD view.
2: Fitting for each (x', E D'' by conditioning the
2: Using rasterization, find all occluded tetrahedra of which mask all tetrahedral vertices. Re-initialize the masked tetrahedral vertices and run the completion algorithm using the trained diffusion model.
SDF values on those x'.
3: Train a diffusion model so (I, 1) on D' by treating the normalized SDF values as float numbers.
Inference:
1: Obtain zo by solving the reverse SDE with initial distribution ((T)) and normalize the SDF values of no to +1.
4: (Optionally) Finetune the unmasked tetrahedral ver- tices as well near the end of the completion process.
2: (Optionally) Regenerate by conditioning on the previously generated (and normalized) SDF values.
neural network 0, (yo) is the mapping from 2D views of yo to its tetrahedral grid representation and LRender is the rendering (and relevant regularization) loss used in [32].
While in theory it is possible to train Is such that sy can be better learned, we instead take the simplest two-stage and non-annnnonne con-amottzeed approach as detailed optimization and encoder architecture design to use is beyond our scope. Specifically, we first solve for the constraint Leaader and create a dataset Dr of tetrahedral grids by fitting tetrahedral grid zo for each yo ~ 'D in a non-amortized way. The second stage is simply to optimize for the main objective, i.e,, L-2 denoising loss of the diffusion model, on this generated dataset D,. Despite simplicity, we find that this simple procedure is very stable and yields reasonably good performance.
In principle, the first data fitting stage can be trained with multiview RGB images by following the procedure in [32]. However, we notice that with RGB images only, fails to learn some complex geometries, especially when surface materials are highly specular. To demonstrate our idea while not delving too deep into the rendering details, we instead assume that we have access to 3D mesh datasets (in our experiments, ShapeNet datasets [7]) so that RGBD images can be rendered with random lighting and simple surface materials. More specifically, we use a single default material with diffuse components only for all ground truth meshes, and render multiview RGBD images with some known but randomly rotated environment light (represented as a cubemap [44]). With the additional depth information, our reconstruction objective is LRender
Timage-ineee + d'eept ddeptt Ochamter Lhhanter + Openalty Lenattty in which Limage and Luepth are the image loss (RGB and silhouette) and the depth loss between the rendered views and the ground truth views, respectively; Lehamfer is the Chamfer distance [1] between sampled point cloud from the predicted mesh and one from the ground truth mesh; Tppenalty is the regularization terms to move reconstructed meshes out from bad local minima. Details of these losses are included in the appendix.
In order to remove absolute scales from SDFs (Section 4.3), we perform the optimization twice, where during the second pass we fix the SDFs to be the signs (i.e., 11) of the resulted SDF values from the first pass. This can effectively improve the results.
4.3 REDUCING NOISE EFFECT OF MARCHING TETRAHEDRA
It is tempting to train diffusion models on arbitrary Dz. the dataset of fitted tetrahedral grids of objects, with low LRender- However, the naively trained models generate highly non-smooth surfaces (Figure 3) due to a mismatch between the Ly denoising objective on the 4-dimensional inputs (deformation and SDF) and the ideal L2 reconstruction objective on triangular mesh vertex positions. Recall that the triangular mesh vertex position Up on single tetrahedron edge e = (a, b) is computed by linear interpolation up = (Na*8 - Ubba)/ (s6 8a). in which Va. 1. are the positions of tetrahedron vertices and b, and Sa, S6 are the corresponding SDF values. With perfectly fitted ca and U/ but a small noise e on both S. and sb. the triangular mesh vertex is subject to a perturbation inversely proportional to 83 Sa. which may incur unexpected consequences: Up,noisy 2,, é(v, vb)/(s, 5.). An arbitrarily fitted dataset Dr does not provide any guarantee on the scale of [sb - sal across different locations in different data points. Such behavior under the simplified assumption implies that the error in predicted mesh vertex positions can vary a lot even with the same Lg denoising loss in SDFs and deformation, indicating unevenly weighted training and inference in the view of mesh vertices.
5

---

Published as a conference paper at ICLR 2023
IM-GAN
SDF StyloGAN
MeshDiflusion (ours)
Chair
Car
Airplane
Table
Riilee
Figure 4: Qualitative comparison among different generative models. All the samples are randomly selected.
A similar issue arises when we consider the topological changes due to small noise in the SDF values. For illustration, (C < is a tiny positive scalar), and all its neighboring vertices possess SDF value of 1. As result, with small noise on $a. the probability of local topological change (e.g., an undesirable hole) in the resulted 3D meshes can differ a lot even when the Le denoising loss is the same.
consider the case where some tetrahedral vertex 8a with -C 8a
Due to these undesired behaviors, we propose to normalize the SDF values and adopt the two-pass optimization scheme described in Section 4.2, such that we can ensure Sa, Sb E {+1. 1} and |s6 sal 2 for all mesh-generating tetrahedron edges.
With normalized SDFs values and continuous deformation vectors, it natural train a hybrid diffusion model in which the SDFs are mod- eled like in D3PM [2] since normalized SDFs take discrete values of #1. However, we empirically find that suffices to simply treat the normalized SDFs as float numbers and train a standard Gaussian diffusion model. The inference thus requires a final normalization operation round SDFs to +1 according to their signs.
Figure Undesirable artifacts produced by naive DDPM due to the sensitivity of marching tetrahedra to SDF noises. Better viewed by zooming in.
Because the generative model is trained under the assumption that SDF values are un-normalized, further refinement step can be applied to improve visual quality. Specifically, we simply set the generated and normalized SDFs as the conditional inputs and then run conditional generation step the deformation vectors only.
4.4 CONDITIONAL GENERATION WITH SINGLE- VIEW IMAGES
In many cases, we would like to generate 3D objects given a (possibly partial) single view RGBD image. Such a task is simple with MeshDiffusion by two-stage process: first to fit tetrahedral grid with the given single RGBD view and second use our MeshDiffusion model (stochastically) correct the wrong parts. We follow the procedure shown in Algorithm 2. For the conditional inference process, we consider the simplest conditional generation method [16] (also called replacement method in [17]). Specifically, sample -.-1 from and replace # by Pt- (+ 12)) which is defined by the forward diffusion. More details are included the appendix.
EXPERIMENTS AND RESULTS
5.1 GENERAL SETTINGS
Data Fitting. We fit tetrahedral grids on five ShapeNet subcategories: Chair, Airplane, Car, Rifle and Table, the same set of categories used in [9, 55]. Fitting each object takes roughly 20-30 minutes single Quadro RTX 6000 GPU. We use the same train/test split in [55]. The detailed architecture, training and hyperparameter settings explained in the appendix.

---

Published as a conference paper at ICLR 2023
MMD (4)
COV (%;,
I-NNA (%; 4)
Method
JSD (10 - 9.4)
CD
EMD
LFD
CD
EMD
LFD
CD
EMD
LFD
IM-GAN
13:928
1816
3615
49.64
41.%6
47.79
58.59
69.05
68.58
6.298
SDF-SIyLGGAN
15.763
3730
45.60
45.50
43.95
63.25
67.66
6.846
Chair
GET3D
15.972
1.843
3801
43.36
42.77
44.48
75.26
72.49
82.82
4.732
MeshDiffusion
13.212
1.731
3472
46.00
46.71
42.11
53.69
$7.63
63.02
5.038
IM-GAN
5.209
1.197
2645
28.26
24.92
30.73
95.69
94.79
89.30
42.586
SDF StyleGAN
5.064
29.93
32.06
41.93
88.34
15,960
Car
GET3D
6.243
1.252
2657
15.04
18.38
31.13
75.26
72.49
89.07
69.107
MeshDiffusion
4.972
1.196
2477
34.07
25.85
37.53
81.43
87.84
70.83
12.384
IM-GAN
3.736
1.110
4939
44.25
37.08
45.86
79.48
82.94
79.11
21.151
Airplanc
SDF StyleGAN
4.558
40.67
32.63
38.20
48
26.304
MeshDiffosion
3.612
1,022
4538
47.34
42.15
45.36
66.44
76.26
67.24
11,366
IM GAN
3.550
1.058
6240
46.53
37.89
42.32
70.00
72.74
69.26
25,704
Ride
SDF StyleGAN
4,100
1.069
6475
46.53
40.21
41.47
73.68
73.16
76.84
33.624
MeshDiffusion
3.124
1.018
5951
52.63
42.11
48.84
57.68
67.79
55.58
19,353
IM GAN
11.378
1.567
3400
51.04
49.20
51.04
65.96
63.17
62.49
4.865
Table
SDF StyleGAN
13.896
1.615
3423
42.21
41.80
42.98
68.35
68.21
66.19
4.603
MeshDiffusion
11,005
1.548
3427
49.56
50.33
51.92
59.35
59.47
58.97
4.310
Table 1: Shape metrics of our model and baseline models.
Generative Model. To show the effectiveness and universality of our approach of training diffusion models on 3D meshes, for the U-Net architecture take extremely approach by simply replacing the 2D convolution and other 2D modules in the U-Net architecture used by [47] with their 3D counterparts. To improve model capacity, indicating which lattice sites are artificially introduced. We slightly increase the depth of the network due to the larger input size. We do not tune the hyperparameters of diffusion SDEs but instead use the same set of hyperparameters as described in [47], which also validates our MeshDiffusion is easy to train. We train the discrete-time category-speciff diffusion models for all datasets for total 90k iterations with batch size 48 on 8 A100-80GB GPUs. The training process typically takes 2 to 3 days.
provide the model as conditional input mask
For ablation on alternative architectures, we train GANs on our datasets with an architecture similar to the one used in [51]. Our SDF-based baselines include IM-GAN [9] and SDF-StyleGAN [55]. We also compare MishDuffusion against GET3D [13] which also uses DMTet for mesh parametrization.
5.2 UNCONDITIONAL GENERATION
For qualitative comparison, we randomly sample 3D shapes from each of the trained generative models for each dataset. We remove isolated meshes of tiny sizes and then apply both remeshing and the standard Laplace smoothing [34] on all the generated meshes (smoothed with À = 0.25 and 5 optimization steps). We visualize samples produced by MeshDiffusion and the existing state-of- the-art 3D mesh generative models in Figure 4. We note that MeshDiffusion produces the sharpest samples and preserve the finest geometric details, while pure based methods tend to be too smooth. Part of the reason is that these bDF-b methods assume a very smooth interpolation between points, while MeshDiffusion explicitly models the interpolation in piecewise linear way.
5.2.1 QUANTITATIVE EVALUATION
In Table 1, we show various point cloud distances [1, 55] and light field distance (LFD) [8] between the test set and the generated samples by sampling point clouds of size 2048 from both ground truth and gen- erated meshes. The results consistently show that MeshDiffusion can better capture the geometric de- tails, and therefore achieves better point cloud metries in most of the cases. For a more detailed description on the metrics and experimental settings, please refer to the appendix.
Model
Chair
Airplane
Car
Rifle
Table
IM-GAN [9]
64.19
74.57
141.2
103.3 51.70
SDF StyleGAN [55]
36.48
65.77
128.70
65.50 42.29
MeshDiffusion
39.62
64.30
130.20
54.73 48.55
Table 2: FID scores averaged across 24 views.
We also follow [55] and measure Frechet inception distances (FIDs) [4] on rendered views as a proxy of goodness of surfaces. Same as in [55]. the views are rendered with 24 camera poses uniformly distributed on a sphere; 3 fixed point light sources are used to light the meshes; a gray diffuse material is used for all meshes. The FID score of each view is computed with Imaeeenet-rrerraieed model
7

---

Published as a conference paper at ICLR 2023
Method
(
(%., 1)
I-NNA (%,
JSD (10->_
CD
EMD
LFD
CD
EMD
LFD
CD
EMD
LFD
on Tots
16.116
1,537
4173
45.13
46.39
41.74
72.97
72.57
85.91
5.353
MeshDiffusion
13.212
1.731
3472
46.00
46.71
42.11
53.69
57.63
63.02
5.038
Ours Smoothing
13.885
1.772
3446
43.36
45.50
43.36
60.88
62.54
62.32
4.716
Ours w/o Normalization
14.324
1.816
3690
44.76
46.61
44.69
63,94
65.01
65.34
5.178
Table Ablation study of MeshDiffusion the Chair category in ShapeNet.
and averaged to obtain the FID scores for 3D shapes. With suitable hyperparameters for resolution and Laplacian smoothing, our MeshDiffusion is able to achieve competitive scores compared to SDF-StyleGAN. We note, however, the computed FID is not genuine score for 3D shapes since the distribution of rendered views of these meshes very different from the distribution of natural RGB images. In addition, the multiview FID score used by SDF-StyleGAN [55] assumes flat shading, while it is more natural and common to use other shading methods like Phone shading for smoother images. Results in Table 2 show that even with such over-simplified shading. MeshDiffusion still yields very competitive FID scores compared to recent state-of-the sof-the-arr art methods.
Additionally, we perform ablation study on the choices of models in Table 3. can be observed that the our SDF normalization strategy described in Section 4.3 is indeed beneficial for the diffusion model, and our customized diffusion model is better suited for our mesh generation setting.
5.3 CONDITIONAL GENERATION
Singlo-viow Fiting
Completion Results
Our model generate meshes by condi- tioning on a single-view RGBD image, as shown in Figure 5. Because the geometry estimated from the single-view fitting is perfect even in the given single view, we allow the originally-fixed tetrahedral vertices to be slightly updated by the dif- fusion model near the end of the diffusion completion process (in our experiments, T 50: the inference process starts at 1000 and ends at T 0). Results demonstrate that MeshDiffusion can gen- erate plausible and reasonable completion results conditioned on the given view.
Ground Truth
Given View
View
View
View
View 2
View
View
View
View
Figure 5: Conditional generation on a single RGBD view.
5.4 INTERPOLATION RESULTS
We use DDIM [45] to convert the stochastic sampling process into a deterministic one. With DDIM, the initial random Gaussian noise øт is treated as the "latent code" of the generated image, and we run the following sampling process: Zt-1 or- [ar (1 a: ( t)] + (1 (4-1)so(az, (z,, t). Following the settings in DDIM paper [45], we use spherical interpolation for the latent codes. We visualize some of the interpolation sequences in Figure We set the number of inference steps to 100 for faster inference and use quadratic time spacing as described in [45].
5.5 TEXT-CONDITIONED TEXTURE GENERATION
We show Figure and Figure 6 that the 3D meshes generated by MeshDiffusion can be easily painted with some texture generation methods. In our experiment, Ture [39] to generate rext-conditioned textures on generated raw 3D meshes. As can be observed from the results, MeshDiffusion along with TEXTure produce fairly realistic and reasonable textured 3D mesh models. With more advanced texture generation methods, we believe that the advantage of MeshDiffusion's high-quality 3D geometry be better demonstrated.
recent work TEX-
6 DISCUSSIONS
Optimization issues with DMTet. While DMTet is capable of fitting geometries, it fails in cases where the underlying topology is complex. Besides, it is not encouraged to learn the true topology of
8

---

Published as a conference paper at ICLR 2023
sola with anime chaneter
blue and pupple leather swivel chair
Acyberpunk atyle van
A medical vehiele
SturWars jet
WWT style Brilish plane
car tank style
photo of car lego bricks
Figure 6: More examples of text-conditioned textures synthesized by [39] on our generated moshes.
shapes and may produce invisible topological holes by contracting the neighboring triangular mesh vertices close enough. Furthermore, we observe that the optimization process with differentiable ras- terization 3D meshes may produce floating and isolated meshes, especially when depth supervision introduced. It therefore worth designing better optimization techniques, regularization methods and possibly better parametrization of meshes for the purpose of training mesh diffusion models.
Diffusion model design. Our experiments demonstrate the effectiveness of the sim- ple design with few hyperparameter and architecture changes: a 3D-CNN-based U- Net augmented cubic grids with DDPM. We note that it is straightforward to utilize advanced diffusion models and switch to more memory-efiie -fffiietefficient architectures. Es- pecially, since it is possible to regularize SDFs with methods such Lipschitz con- straints [27], is a promising approach to train diffusion models on the latent space produced by regularized autoencoder, the same strategy adopted in [40, 54].
Limitations. Diffusion model typically as- sumes known dataset in the input modal- ity (augmented tetrahedral grids in our case), but efficiently train diffusion mod- els on 2D images, we need a better way to amortize the costs and fully leverage the power of differentiable rendering. Our paper avoids this important aspect but instead adopts the two- stage approach of "reconstruction- -hen--teeeeee generneeenrr.n.. Moreover, our formulation, the differentiable renderer is useful only during the tetrahedral grid creation process, while in principle we believe there can be ways to incorporate the differentiable render in the training and inference process of diffusion models. Finally, diffusion model is built with very naïve architecture, thus limiting the resolution of input tetrahedral grids, while we notice that some of the fine details cannot be fully captured with the current resolution of 64 during the dataset creation stage. With better architecture designs or adaptive resolution techniques (as [43]), we may greatly increase the resolution and generate more diverse set of fine-level geometric details.
interpolation
Figure 7: Some examples of our interpolation results.
7 CONCLUDING REMARKS
We demonstrate that our MeshDiffusion, diffusion model on 3D meshes parameterized by tetrahedral grids, able to generate fine-details of 3D shapes with arbitrary topology. Such model minimally designed but still outperforms the baselines. We demonstrate that, despite being trained and perform- ing inference in the 3D space, can be easily used when only 2.5D information is available. We believe that model potentially shed some light future studies of score- models for learning and generating high-fidelity shapes. Future work may include but not limited to text-conditioned mesh generation with diffusion models, joint synthesis of texture and geometry with more realistic materials, and motion synthesis and physical simulation on generated meshes.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank Yuliang Xiu, Jinlong Yang, Tim Xiao, Haiwen Feng, Yandong Wen for constructive suggestions. We would like to thank Samsung Electronics Co., Ldt. for funding this research.
Disclosure. MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon. Datagen Technologies, and Meshcapade GmbH. While MJB is part-time employee of Meshcapade, his research was performed solely at, and funded solely by, the Max Planck Society. DN is supported by NSERC Discovery Grant (RGPIN-5011360) and LP is supported by NSERC Discovery Grant (RGPIN-04653).
Ethics Statement. Our model is developed for general 3D mesh generation and is in very preliminary stage in terms of automatically generating product-quality meshes. Still, our model may potentially be used to generate inappropriate contents if trained on specific datasets.
REFERENCES
[i] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In ICML, 2018
[2] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete space-spaces. In NeurlPS, 2021.
[3] Jean-Daniel Boissonnat. Geometric structures for three-dimensional shape representation. ACM Transac- tions Graphics, 3(4):266 286, 1984.
[4] Andrew Brock, Jeff Donahuc, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2018.
[5] Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry 3d generative adversarial networks. In CVPR, 2022.
[6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
[7] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Haurahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rXiv:1512.03012. 2015.
[8] Ding-You Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. In Computer graphics forum, volume 22, pages 223-232. Wiley Online Library, 2003.
[9] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling, In CVPR, 2019.
[10] Tamal K Dey, Joachim Giesen, and James Hudson. Delaunay based shape reconstruction from large data. In PVG, 2001.
[11] John David N Dionisio, William G Burns III, and Richard Gilbert. 3d virtual worlds and the metaverse: Current status and future possibilities. ACM Computing Surveys, 45(3):1-38. 2013.
[12] Akio Doi and Akio Koide. An efficient method of triangulating vaui-valued surfaces by using tetrahedral cells. IEICE Transactions on Information and Systems, 74(1):2142224, 1991.
[13] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin. Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In NewrIPS, 2022.
[14] William Gao, April Wang, Gal Metzer, Raymond A Ych, and Rana Hanocka. Tetgan: A convolutional neural network for tetrahedral mesh generation. arXiv preprint arXiv:22 2022.
[15] Meenakshisundaram Gopi, Shankar Krishnan, and Cláudio T Silva. Surface reconstruction based on lower dinensional localized delaunay triangulation. In Computer Graphics Forum, volume 19, pages 467-478 2000.
[16] Jonathan Ho, Ajay Jain, and Picter Abheel. Denoising diffusion prohabilistic models. In NeurlPS, 2020.
10

---

Published as a conference paper at ICLR 2023
[17] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models, arXiv preprint orXiv:2204.33456 2022.
[18] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning eategory-specific mesh reconstruction from image collections. In ECCV, 2018.
[19] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NeurlPS, 2020.
[20] Hiroharu Kato, Yoshitaka Ushiku, and Taisuya Harada. Neural 3d mesh renderer. In CVPR. 2018
[21] P.E. Kloeden and E. Platen. Numerical Solutoon of Stochastic Differential Equations. Springer Berlin Heidelberg, 2011.
[22] Ravikrishna Kolluri, Jonathan Richard Shewchuk, and James F FO'Brien. Spectral surface reconstruction from noisy point clouds. In SGP, 2004.
[23] Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In ICML, 2021.
[24] François Labelle and Jonathan Richard Shewchuk. Isosurface stuffing: fast tetrahedral meshes with good dihedral angles. In SIGGRAPH, 2007.
[25] Tianye Li, Shichen Liu, Timo Bolkart, Jiayi Liu, Hao Li, and Yajie Zhao. Topologically consistent multi-view view face inference using volumetric sampling. In ICCV, 2021.
[26] Yiyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representa- tions. CVPR, 2018.
[27] Hsuch-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. In SIGGRAPH. 2022.
[28] Weiyang Liu, Zhen Liu, Liam Paull, Adrian Weller, and Bernhard Schölkopf. Structural causal 3d reconstruction. ECCV, 2022.
[29] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In SIGGRAPH, 1987.
[30] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black. Learning dress 3d people in generative clothing. In CVPR, 2020.
[31] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[32] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans. Thomas Mueller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. arXiv:2111.12503. 2021.
[33] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In ICMI., 2020.
[34] Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Mare Alexa. Laplacian mesh optimization. In Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia, 2006.
[35] Hayato Onizuka, Zehra Hayirei, Diego Thomas, Akihiro Sugimoto, Hideaki Uchiyama, and Rin-ichiro Taniguchi. Tetratsdf: 3d human reconstruction from single image with a tetrahedral outer shell. In CVPR, 2020.
[36] Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape as points: differentiable poisson solver. In NeurlPS, 2021.
[37] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning mesh-based simulation with graph networks. ICLR, 2020.
[38] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoît Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. NeurlPS, 2020.
[39] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. arXiv preprint a0Xiv:2302.00721, 2023.

---

Published as a conference paper at ICLR 2023
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[42] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for aware image synthesis. NeurIPS, 2020.
[43] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. NeurlPS, 2021.
[44] Peter Shirley, Michael Ashikhmin, and Steve Marschner. Fundamentals of computer graphics. AK Peters/CRC Press, 2009.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[46] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In UAI, 2020.
[47] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[48] Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. Variational autoencoders for deforming 3d mesh models. In CVPR, 2018.
[49] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7)11661-167, 2011.
[50] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018
[51] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-adversarial modeling. In NIPS, 2016.
[52] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In CVPR, 2020.
[53] Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongic, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. ICCV, 2019.
[54] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In NeurlPS, 2022.
[55] Xin- Yang Zheng, Yang Liu, Peng-Shuai Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In SGP, 2022.
12

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
A Details on Datasct Preparation
14
A.1 Architecture and Losses
14
A.2 Implmentation Details
15
B Details on Mesh Diffusion Models
16
C Quantitative Metrics for 3D Meshes
17
D Generation Sequence
18
E Nearest Neighbor Visualization
19
F Qualitative Comparison to GET3D
20
G Inner Structure Generation
21
H More Unconditional Generation Samples
22
13

---

Published as a conference paper at ICLR 2023
A
DETAILS ON DATASET PREPARATION
A.1 ARCHITECTURE AND LOSSES
RGB and silhouette loss. By interpolating between a black background image and the binary mask (produced by rasterization to indicate the existence of rasterized meshes each pixel), obtain smoothed binary mask M and Mgt for the fitted meshes and predicted meshes respectively, We then compute the silhouette loss:
Lilhonette ||
Mgell2.
We compute the averaged pixel-wise log-Lı loss between rendered images and ground truth images:
LRGB
1g Mg Render(x, Q) @ Mll)
=ED,QEDpM
in which Dpose is the set of random camera poses which always look at the center of the object, and xc the sampled tetrahedral grid from the dataset D.
Depth loss. We use a mixed loss for depth: when the depth is greater than a threshold (set to 1.0), we La loss; otherwise, we switch to Lı loss find it producing much better surfaces.
deptt
dd Renderasput (Ir, Q)
DED,QED
The depth images are computed by barycentric interpolation following rasterization, in which the background is assumed have large default depth of 20, which is much greater than the object depth (constrained in [-1, 1J).
To better remove inner artifacts and generate inner structures, we also include the depth loss on the second layer from rasterization, i.e., the second triangular mesh (if any) intersected by each view ray. We weight this second layer depth loss term by 0.1 as we observe that it can interfere with the first layer depth loss.
The complete depth loss term is thus
T.depth, complete
E LED,QED,
[gg. Render (2; mixed +0.1 +0.1+|Idg 2nd Renderaephh (2,, Q) mixxdd
Chamfer loss. every iteration, we randomly sample 50, 000 points from both ground truth meshes and predicted meshes and compute the standard Chamfer distance:
LChamfer
min | PEP
2112
Σ VEP
min |
y1
arE Pined
SDF regularization loss. Following [32], we use the same L2 regularization loss and penalize the difference of SDFs two neighboring tetrahedral vertices (i.e., Ere: the set all edges in the tetrahedral grid) as in [32] so that the occluded regions do not produce complex inner geometry. It is also helpful for optimizing tetrahedral grids.
LSDF =
Σ Et.t
SSDF(u) - SDF(v))12.
Hyperparameter setting. We set Olimage = O'Chamder = 1.0 and C'eeph = 100.0. We set Qspr to 0.2 and use linear decay of the scale towards 0.01. We Adam optimizer for all the parameters with learning rate 5e and (8,, 32) = (0.9, 0.999). We train both reconstruction passes with 5000 iterations.
14

---

Published as a conference paper at ICLR 2023
A.2 IMPLMENTATION DETAILS
The initial tetrahedral grid is initialized in a cube 1 by a dense body-centered cubic (BCC) tiling of tetrahedra [24] (see https: https://git /tg/ttttbbb.///. com/ crawfo crawforddore quartet for code examples). As some tetrahedral generation packages produce additional tetrahedral vertices on the boundary of the cube which breaks translational symmetry (hence detrimental to the performance of 3D CNN), we remove such brreakty-breaking boundaries if any.
We find that using the clipped deformation vectors better than using the tanh-ed deformation vectors as they remove some nonlinearity. And to ensure that the deformation vectors are always differentiable, we clip the deformation vectors only after each gradient update, but not during the forward computational pass.
We set the range of deformation to be three times of that used in [43] so that the meshes can better capture details in the absence of SDF scales (Section 4.3), especially when the grid resolution is low. As coff-intersecting meshes may appear, is necessary to perform standard mesh processing operations including remeshing and Laplacian smoothing. For grids of higher resolution (e.g., 128), we instead set the deformation to be 1.5x of hat used in [43].
Because normalize SDFs after the first reconstruction pass, it is more desirable for the underlying shape to be captured mostly by the topology implied by SDF values, not the vertex deformation. To encourage convergence to such solutions, during the first 2000 iterations, we periodically scale the learned SDF values by 0.4 for every 300 iterations.
We observe that depth supervision is not enough to remove some isolated floater meshes in the space. Therefore, we cull these floaters by setting the SDF values of all tetrahedral vertices lying outside the visual hull to positive values (with the assumption that positive values represent ouside-ness). Due constraints on computational resources, we do not perform culling in full 3D way, but use rendered shape silhouette to determine floaters visible in the rendered views only.
We notice that some of the meshes in ShapeNet are not watertight while some others have parts which are very thin. Depth supervision on these parts can lead to more topological holes. Therefore, during rasterization we extract the depths (stored in Z-buffers) of the two closet meshes for each pixel and decide if we compute the depth loss on these pixels according to the difference in depth of these two meshes. Specially, depth supervision is not included if the difference is too small or there is only one rasterized mesh (meaning that the ray only intersects with a single mesh).
Motivated by the fact that only SDFs of the mesh geeeragene tetrahedra (ie., tetrahedra with vertices of different SDF signs) matter, we set the SDFs of all the non-mesh- non-mesh--ennrrainn tetrahedral vertices to +1 (signs depending on if vertices inside/outside the shape) reduce the complexity of the dataset.
For the single-view reconstruction stage in the conditional generation experiments, we observe that the lack of other views leads to many floating artifacts when the target shape consists of a bulk volume. Therefore, during the first 300 iterations, we periodically (every 10 iterations) query the nearest neighbor ground truth mesh vertices for every tetrahedral vertex, and determine if the vertex is in front of the camera and not behind the current view (by computing the nearest surface normal, the displacement from the nearest face center and the view direction from camera). If the tetrahedral vertex satisfies the condition, we set its SDF value to +1.
15

---

Published as a conference paper at ICLR 2023
B
DETAILS ON MESH DIFFUSION MODELS
Input
Encoder
Input
X ResBlocks. 5 X kemel, width 128
Pooling, Stride
ResBlocks, kernel, widhh
2 ResBlocks, 3 kernel width 128
Paoling, Stride 2
Pooling Stride 2
x ResBlocks, kernel, width
ResBlocks, kernel width 256
Pooling, Stride 2
Pooling, Stride 2
3 ResBlooks (with one atlention layer in between),
ResBlocks (with one attention layer in between),
kemel, width 128
x
kemel,
256
Pooling, Stride 2
Pooling, Stride
3 x ResBlocks, 3 x kernel, width 256
2. ResBloocks, 3 x kemel, width 512
Puoling, Stride 2
Pooling, Siride 2
ResBlocks, kornel, width
RosBlocks, 3 komall width 512
FC
FC
x ResBlocks, 3 x kernel, width 256
x RosBlocks, 3 x kemel, width 312
Attention
Attention
X ResBlocks, 3 x kernel, width 256
ResBlocks, 3 kernel width 512
Decoder
Decoder
3 X ResBlocks, 3 x kernel, width
ResBlocks, 3 x kernel width 512
Upsampling, Stride 2
Upsampling, Stride
3 X ResBlocks, 3 X kernel, width 256
2. Resfllocks, 3 x 3 kemel, width 512
Upsampling, Stride
ResBlocks Upsampling, Stride
ResBlocks one
layer between).
layer in between),
3 kemel, width 128
3 3 kemel, width 256
Upsampling, Stride 2
Upsampling, Siride
ResBlacks, kernel, widlh
Resfflocks,
width
Upsampling, Stride
Upsamplingg Stride
x ResBlocks, x kernel, width
X ResBlocks, x 3 kemnll width 128
Upsampling,
x ResRlocks, 5 5 kemel, width 128
Table 4: Architecture of the 3D U-Net (Left:: resolution 64, Right: resolution 128). The shortcuts from the encoder to the decoder are not shown.
We adapt the network of DDPM in [16] and use base width of 64. The encoder in the U-Net is shown in Table 4, and the decoder follows the same but reverse pattern. For higher resolution grids, slightly reduce the number of layers in each resolution stage but double the base width.
All the values at the artificilly-intrrouueed sites cubic lattice are set to zero. We append to the cubic grid in the first and last few layers binary mask that indicates which vertices are from the tetrahedral grid and which are fake. The score matching loss is also masked accordingly so that the augmented vertices in the predicted cubic grids do not contribute.
To prevent overfitting, we augment the dataset by randomly translate all tetrahedron vertices by the same but tiny amount.
16

---

Published as a conference paper at ICLR 2023
C QUANTITATIVE METRICS FOR 3D MESHES
We briefty explain the metrics we used for evaluating the quality 3D Meshes.
Minimum Matching Distance (MMD). MMD measures the average distance to the nearest neighbor of individual points in one point cloud to another point cloud.
Coverage. Suppose every point cloud in set A is approximated by its nearest neighbor in another set B. Coverage measures the fraction elements B which are used to cover in the nearest neighbor matching sense. Higher the coverage, Better that a representative set of B.
Leave-one-out Accuracy (1-NNA). As its name indicates, 1-NNA is the average leave-one-out accuracy of the I-NN classifier fitted A classify B. measures if each element in the A is important in representing another set B. A low 1-NNA score means that A well covers B.
Jensen-Shannon Divergence (JSD). With JSD, we compute the distance between distribution of ground truth point clouds and that of the generated point clouds: JSD(PA, (L -
(KL(PBll MM), in which M (PA + PB).
Light Field Distance (LFD). LFD leverages so-called light field descriptor [8], which combines local region-base and contour-based descriptors to measure the silhouettes and thus object shapes.
17

---

Published as a conference paper at ICLR 2023
D GENERATION SEQUENCE
For each time step t E (1....,7) 7) during the DDPM inference process, we can compute the predicted zo by the current value of z, [16]:
60 (1s VI (1r 1)),
in which (I, t) is the learned denoising network, a, = II (1 3a) and St is the noise scale hyperparameter DDPM.
We show example generation trajectories in Figure by visualizing the predicted zo through time. All fo's 's are clipped to [0. 1]. We visualize the sequence from T 250 50, as the predicted tetrahedral grids before T = 250 produce nothing other than noise.
Figure 8: Generation sequences of predicted uio-
18

---

Published as a conference paper at ICLR 2023
E NEAREST NEIGHBOR VISUALIZATION
We show the nearest neighbors of some generated samples in the validation set in Figure 9.
Generated
Nearest Neighbor
Generated
Nearest Neiglibor
Genorated
Nearest Neighbor
Generated
Nearest Neighbor
Figure Nearest neighhor (from the ShapeNet dataset) the generated samples.
19

---

Published as a conference paper at ICLR 2023
F QUALITATIVE COMPARISON TO GET3D
We visualize some generated meshes from MeshDiffusion and GET3D in Figure 10. Both models are able to generate finer details of shapes due to the use of DMTet. On the car category of ShapeNet, generated meshes by GET3D tend to have fewer or less noticeable holes, possibly because GET3D directly trained on adversarial losses on rendered images. In comparison, MeshDiffusion has to learn the complex and often invisible 3D structures, which is also one of our advantages for modeling complex 3D geometry with inner structures. However, potentially due to the limited capacity of the decoder (compared to a recurrent U-Net in diffusion models) and the hardness of GAN training (v.s. supervised learning with denoising loss), we observe that the failure cases appear more often in GET3D compared to MeshDiffusion. And the novel examples in MeshDiffusion seem to make more sense compared to those from GET3D.
GET3D
MeshDiffusion
Figure Qualitative Comparison between MeshDiffusion and GET3D.
20

---

Published as a conference paper at ICLR 2023
G INNER STRUCTURE GENERATION
Since our MeshDiffusion explicitly trained with 3D information, it is capable of generating inner structures invisible from the outside. We give comparison between samples generated by MeshDiffusion (trained with a higher resolution of 128) and GET3D in Figure 11.
Figure 11: Inner structures of generated car meshes. Left: GET3D, Right: MeshDiffusion.
To fully evaluate the capability of surface generation (as GET3D is only trained to optimize surface appearance), we follow the same procedure described https: /////gtttuu/ com/nv hatnn/vvllbbbs GET3D/tres /masttr//////////aaa /ereee evv I luatiiin iination_stte seensssrrippi scripts and compare the point cloud metrics in Ta- 5, with generated sample set 5x larger than the validation set.
MMD (L)
COV (%, 1)
Model
CD
EMD
CD
EMD
GRT3D
9. 8.8477
0.3668
42.93
54.20
MeshDiffuuson
6.7965
0.3997
64.13
68.80
Table Shape metrics of MeshDiffusion and GET3D on surface point clouds of generated car meshes.
21

---

Published as a conference paper at ICLR 2023
H MORE UNCONDITIONAL GENERATION SAMPLES
Figure 12: Generated 3D chair meshes.
22

---

Published as a conference paper at ICLR 2023
Figure 13: Generated 3D car meshes.
23

---

Published as a conference paper at ICLR 2023
Figure 14: Generated 3D airplane meshes.
24

---

Published as a conference paper at ICLR 2023
Figure 15: Generated 3D rifle meshes.
25

---

Published as a conference paper at ICLR 2023
Figure 16: Generated 3D table meshes.
26


Reviews:
[{"review_id": "fvIjxn44S58", "reviewer": "Reviewer_jFa3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a diffusion model MeshDiffusion for 3D meshes generation.  The meshes are embedded in a deformable\ntetrahedral grid, and then employ a score based method to train a diffusion model on this direct parameterization. Besides using in the 3D space, the network can be used when only 2.5D information is available as well. The model is tested  on multiple generative tasks.", "summary_of_the_review": "Overall it is an interesting application of the score based method in the 3D mesh generation domain which is very important. The paper does show some good results over the existing method. The paper would benefit from more time spent on the presentation, illustration, proof reading etc as it contains some typos and errors.", "strength_and_weaknesses": "The paper is for 3D mesh generation using the score based diffusion model. Score based diffusion model for generative modeling such as image synthesis has shown significant improvement over existing methods such as GAN, VAE, etc. This paper tries to extend the score based diffusion method into 3D mesh generation domain, which is a very important area to explore.\n\nThe main framework is based on Deep Marching Tetrahedra, plus the score based diffusion model. These are the main strength of the proposed method which essentially combines these two and applies to mesh generation.\n\nThe results seem good in comparison with existing methods.\n\nThe work on the other hand, seems still at a very earlier stage, and the paper seems generated in a little rush, e.g. there are two citations of 38,38] in page 3, second line, second paragraph. There are probably still some issues that need to solve before it can be fully utilized, as some of the limitations mentioned by the authors.\n\nNonetheless, this paper does point to an interesting direction in 3D mesh generation which the 3D community might not be fully aware of before (including myself), in this point the paper has merit and is to be accoladed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UwGn1ht8TrF", "reviewer": "Reviewer_Mo7X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents MeshDiffusion, a method for generating 3D meshes. The authors use the deformable tetrahedral grid as the 3D representation, and train a diffusion model on this parameterization. The authors demonstrated the effectiveness of the proposed model on multiple generative tasks, including unconditional and conditional generation.", "summary_of_the_review": "The paper proposes a mesh generation model based on tetrahedral grid representation and diffusion model. The results show that the proposed method outperforms the baseline methods. However, the main issue is that one main state-of-the-art method \"GET3D\" was not mentioned or compared in this paper. Thus I would suggest the authors add this comparison and re-submit the paper.", "strength_and_weaknesses": "+ The tetrahedral grid representation is able to generate fine details of 3D shapes with arbitrary topology.\n\n+ The mesh output can be easily used in graphics pipelines.\n\n- The main problem is: There's a recently published paper  \"GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images\" in NeurIPS 2022. This paper is released on Sep. 22, which is before the paper deadline of ICLR 2023 (Sep. 28). So it should be considered as \"previous work\". The GET3D paper uses a similar idea (tetrahedral representation) to perform a similar task. However, in the submission the authors didn't mention, review, or compare with GET3D.\n\n- Multiple loss functions are used for training the network. However, the ablation study does not show the necessity of each loss function.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3OqWai7yGUB", "reviewer": "Reviewer_NSt1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a diffusion model-based mesh generator. \nIt adopts the representation from the previous `deep marching tetrahedral`.\nThe method works in two steps by first learning to reconstruct 3D meshes and then training a diffusion model on the grid points using a 3D U-Net. \n", "summary_of_the_review": "The paper proposes the first diffusion model on 3D meshes. \nHowever, due to its limited evaluation, I have doubts on its generation capacity and its claims on stable training and reducing noise. \nIn addition, quantitative comparisons in FID metrics with baseline methods can not show its superiority.\nThe paper also misses a comparison with recent GET3D. \nSo I am slightly leaning toward rejection. ", "strength_and_weaknesses": "Strength:\n- To my knowledge, it is the first diffusion model on 3D **mesh**. \n- On unconditional generation evaluation, it truly beats one of the latest SOTA(SDF-StyleGAN).\n\nWeakness:\n- Not particularly novel. The paper is more like a combination of previous tetrahedral generation work and the diffusion model, especially when I see that the method has two training steps. \n\n- Misses evaluation on SVR. Obviously, the first step can reconstruct shapes from 2D images. However, I could not find any evaluation on it neither in the paper nor in the appendix. \n\n- Miss closest shape results when showing generated shapes. Can you show the nearest shapes in training set to validate the generation capability of your generative model? \n\n- I saw claims like `extremely easy and stable to train.` and `MeshDiffusion is also very stable to train without bells and whistles.`. Is there any evidence to prove this?\n\n- Holes seem still exist on the generated meshes. Are there any quantitative results that can support your point `REDUCING NOISE-SENSITIVITY OF MARCHING TETRAHEDRA`? \n\n- Why FID metric of the car category is so high(even worse than IM-GAN)? \n\n- Misses comparison with GET3D. \n\nMisses references:\npoint cloud diffusion model:\nLION: Latent Point Diffusion Models for 3D Shape Generation\n\nrecent tetrahedral mesh generation method:\nGET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images\n\nTypos:\n`including unconditional generation, conditional generation and interpolation` -> \n`including unconditional generation, conditional generation**,** and interpolation`", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%