Ghost on the Shell: An Expressive Representation of General 3D Shapes
Authors: Zhen Liu
Yao Feng
Yuliang Xiu
Weiyang Liu
Liam Paull
Michael J. Black
Bernhard Schölkopf
Venue: ICLR 2024 oral (2024)
Keywords: Non-watertight mesh; generative model; 3D geometry; differentiable rendering
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a general 3D mesh representation to include non-watertight meshes, which enables efficient mesh reconstruction and generative models.

Abstract: The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they enable 1) fast physics-based rendering with realistic material and lighting, 2) physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range of object shapes, any 3D representation must be able to model solid, watertight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight surfaces, we parametrize open surfaces by defining a manifold signed distance field on watertight templates. With this parametrization, we further develop a grid-based and differentiable representation that parametrizes both watertight and non-watertight meshes of arbitrary topology. Our new representation, called Ghost-on-the-Shell (G-Shell), enables two important applications:  differentiable rasterization-based reconstruction from multiview images and generative modelling of non-watertight meshes. We empirically demonstrate that G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks, while also performing effectively for watertight meshes.

Published as a conference paper at ICLR 2024
GHOST ON THE
SHELL
AN EXPRESSIVE REPRESENTATION OF GENERAL 3D SHAPES
Zhen Liu Yao Fengla Yuliang Xiu
Weiyang Liu
Liam Paulf
Michael J. Black Bernhard Schölkopf
Max Planck Institute for Intelligent Systems- Tübingen
Miila Quebec Al Institute, Université de Montréal
ETH Zürich *University of Cambridge
"Equal contribution
Shhared last author
Project page: gshell3d github.
(b) Application I: Reconstruction from multi-view images
Shell watertight surface
Ghost: open surficee defined on the Shell
(a) Ghost-on-the- Shell: A general 3D representation
(c) Application II: Unconditional mesh generation
Figure I: Left: Illustration of mesh extraction with G-SHELL through manifold signed distance surface;
Right: Applications G-SHELL, multiview mesh reconstruction (top) and mesh generation (bottom).
ABSTRACT
The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they 1) enable fast physics-based rendering with realistic material and lighting, 2) support physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range object shapes, any 3D representation must able to model solid, water- tight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight surfaces, parameterize open surfaces by defining manifold signed distance field on watertight templates. With this parameterization, we further develop a grid-based and differentiable representation that parameterizes both watertight and non-watertight meshes of arbitrary topology. Our new representation, called Choost-nnthe-Sheee (G-SHELL), enables two important applications: differentiable rasterization-batt reconstruction from multiview images and generative modelling of non-we wateerittg meshes. We empirically demonstrate that G-SHELL achieves srate-ofthhe-e art performance on waterrtttt mesh reconstruction and generation tasks, while also performing effectively for watertight meshes.
INTRODUCTION
The creation of high-fidelity 3D virtual worlds requires representation of 3D shape that can be rendered and simulated efficiently and realistically. Most commonly, 3D shapes are represented as meshes for which modern graphics pipelines are highly optimized. Because the manual creation of 3D mesh assets is time-consuming. research has focused on the automatic creation from images or generative models. While much of the recent work has focused watertight meshes [55, 56, 60, 67).

---

Published as a conference paper at ICLR 2024
many 3D objects, such as clothing, paper leaves, are non -attertttt,gh, open' and thin. The capture and generative modeling of such surfaces is relatively underexplored.
Existing modeling methods for non- watertight meshes typically build unsigned distance field (UDF) [35, 38], a scalar field of the absolute distances of 3D coordinates to the nearest surface. With UDF, one may obtain oon-watertight meshes by extracting and discretizing the zero UDF levelset. However, isosurface extraction from a UDF, compared to that of signed distance fields (SDF), is a non-trivial task: the bisection search strategy in classical algorithms such as Marching Cubes [39] does not simply apply to UDFs. Common workarounds include 1) post-processing double-layered watertight mesh [48, 70]. introducing local pseudo-signs for bisection search [15]. and using implicit- free point-to point-to-meet to-mesh reconstruction methods [10]. These methods inevitably introduce modeling errors, posing challenge for non- oon-watertight mesh reconstruction and generation.
We take a different approach to modeling con-watertight meshes with the following key observation most open surfaces can be viewed as entities floating on watertight surfaces, analogous to continents floating on the Earth's 's surface. In other words, it suffices to model the open surface boundary on some watertight surface template. To formalize this idea, we define a manifold signed distance field (mSDF) on the watertight template, in which the sign indicates whether point lies in the open surface or not, and the absolute scale indicates the geodesic distance to the boundary. An open surface can now be extracted via isoline extraction with mSDF.
We follow this intuition and design a general representation, Ghost-on-the-Shell (dubbed G-SHELL), which jointly parameterizes the watertight template and the non-watertight mesh living on it. Specifi- cally, we discretize the 3D space into a grid of cells, of which the vertices store both SDF and mSDF values, and then apply Marching-CCues--iig isoline. Our implementation exploits an efficient mesh extraction algorithm instead of following the naïve two-stage upproach of "isosurface to isoline". In nutshell, we adapt the look-up table in Marching- Saangg-Cubeslllle like algorithms, which enumerate all possible configurations of isosurfaces in each cell according to both SDF and mSDF signs. Such an implementation effectively reduces the number of mesh faces created and thus the computational cost of the grid-to-ne grid-to-mesh mapping.
extraction obtain the SDF isosurface and the mSDF
Since G-SHELL uses only simple, deterministic and parallelizable operations for mesh extraction, mesh-based inverse rendering can now be applied to non-watertight meshes due to optimization landscapes (with simple Marching- Cubes-like extraction) and memory-friendly compu- tation (with efficient mesh rasterizers). This efficient mesh rasterization means that we are now able to optimize both material and lighting from pixel information by exploiting physics-based rendering of meshes. Furthermore, the regular grid structure of G-SHELL allows the extension of recent generative methods, such as diffusion models, to non-watertight meshes for the first time.
In summary, major contributions are listed below:
Mesh representation. G-SHELL is a differentiable representation that effectively parameterizes both watertight and non- watertight meshes of different shape topologies.
Efficiency. With the designed mesh extraction algorithm for G-SHELL, we achieve fast reconstruc- tion of non- watertightt meshes with differentiable rasterizers. Specifically, we design efficient mesh extraction algorithm for G-SHELL.
baysics-based inverse rendering. G-SHELL enables joint optimization of topology, material, and lighting of both watertight and not-witert meshes.
Mesh generation. The grid parameterization and efficient mesh extraction of G-SHELL enables effective generative modeling of both watertight and non-watertight meshes with diffusion models.
We qualitatively and quantitatively compare G-SHELL with current popular 3D representations on reconstruction from realistic images and unconditional generation of both watertight and non- watertight meshes, demonstrating the superiority of G-SHELL in representing general 3D shapes.
2 RELATED WORK
Mesh parameterization and extraction. In contemporary computer graphics pipelines and software, 3D meshes serve as a crucial and foundational representation. The means of mesh reconstruction can mostly be classified into three categories: from mesh templates, from point clouds and from implicit
Le., bordered. Not be confused with the mathematical term of openness.
2

---

Published as a conference paper at ICLR 2024
fields. Mesh templates enable the optimization of vertex positions to align with object surfaces but can be inflexible for diverse topologies [16, 64]. Some methods [45, 59, 61] utilize remeshing [5. 20] to accommodate topology changes, but careful initialization remains essential to avoid bad local minima during optimization. Point clouds, compared to mesh templates, can be directly obtained from Lidar seans and at the same time offer flexibility in capturing diverse shape topologies. Such flexibility comes at the cost of the challenge in inferring point connectivity whether two points are adjacent each other on the target surface. Common methods for building surfaces from point clouds, such as Ball-Pivoting [3] and Delaunay Triangulation [28], are not only slow due to non-parallelizable operations but susceptible to noises in source point clouds.
It is therefore more common to extract meshes from implicit fields, which may be built from point clouds with Poisson reconstruction [24, 46, 54], or from multiview images [41, 67]. Subsequently, meshes can be extracted by identifying and triangulating the zero levelsets with methods like Marching Cubes [39], Marching Tetrahedra [58] and Dual Contouring [23]. Many of these algorithms are made differentiable, such as in Deep Marching Cubes [30], Neural Dual Contouring [9], MeshSDF [49], DMTet [55] and FlexiCubes [56]. However, most of them only apply to watertight meshes due to the of SDF. To handle non- watertight meshes, some papers propose differentiable methods using UDF. For instance, MeshUDF [15] computes pseudo-signs on grid vertices - the signs of inner products between UDF gradients on grid vertices and reuses Marching Cubes to extract meshes from the resulted pseudo SDFs. These methods are sensitive to input noises, due to the challenge in locating zero levelsets from non-negative fields. While implicit representations other than UDF exist, such as variants of generalized winding number [2, 11, 21] and DeepCurrents [44], there is no efficient differentiable mesh parameterization for them. In comparison, our method robustly and efficiently models non- won-watertight meshes by extracting zero levelsets in a Marching -aagg--Cbbsiiiie like manner.
Differentiable inverse rendering. It is popular in recent years to perform differentiable inverse rendering through implicit representations, such as NeRF [41] and SDF [60, 67], with which one may utilize differentiable volumetric rendering [37, 41] or surface rendering [22, 32, 67] methods. These methods are adapted to reconstruct non-watertight surfaces with UDF (e.g.. NeuralUDF [38], NeUDF [35]) and SDF (e.g., NeAT [40]). However, rendering with implicit representations typically requires multiple and likely expensive queries for each pixel in the rendered image. And since the geometry and color are encoded in an implicit way, it is relatively hard to disentangle geometry, ma- terial and lighting during inverse rendering. In contrast, explicit representations, such as point clouds and meshes, can be efficiently rendered with rasterization [25, 33], and allow easy disentanglement of physical properties [17, 34, 42]. Compared to the implicit-based methods for non-watertight mesh modeling, our method takes advantage of mesh-based rasterization to enable fast joint optimization of shapes, materials and lighting from multi-view images.
Generative modeling of geometry. 3D generation is widely studied for various representations, including explicit representations like mesh [12. 13, 36, 43], point clouds [65, 68], voxels [62], and implicit ones like signed distance functions (SDFs) [6, 69] and neural radiance fields (NeRF) [6, 7, 26, 53]. However, apart from few mesh-based based methods, all the other ones do not directly generate meshes with arbitrary topology. As result, one typically has to perform additional post-processing steps to extract meshes, which can be time-consuming and may introduce additional errors.
More recently, methods have been proposed to generate meshes using intermediate grid represen- tations either through direct 3D modeling [12, 13, 36] or through lifting information from 2D generative models [8. 31]. While these methods achieve success in generating watertight 3D meshes, none of them can generate non-watertight meshes. is also possible to directly generate meshes in an autoregressive way: for instance, PolyGen [43] builds transformer-bantt based autoregressive model to alternately produce vertices and edges. However, autoregressive models can be so flexible that they hardly scale to complex meshes (especially those not created by designers) with a very dense set of vertices and often produce selffintersecting meshes. Our method, instead, is capable of generating both watertight and non wot-tttett meshes with fine geometric details and without self-intersections.
3 PRELIMINARIES: SDF-BASED MESH EXTRACTION
We briefly summarize Marching Cubes, a classical SDF-based mesh extraction method, and introduce some of its variants. In nutshell, Marching Cubes discretizes the 3D space with 3D cubic grid and extracts faces from each cubic cell with a simple linear assumption: the SDF value of any point in each cell is barycentric interpolation of those cell corners. Specifically, given any point x in
3

---

Published as a conference paper at ICLR 2024
cell with 8 corners (P1, Po........P Pb), first obtain the barycentric coordinate (c1, C2 C2,,.., Cg) such that æ = " cpP- Under the linear assumption, the extracted mesh must be polygons with their vertices on the cubic cell edges. Therefore, it suffices to compute the mesh vertex position p' on each edge (pt, Py) with SDF values 8; < 0 < 8j, respectively. The vertex position simply a linear interpolation between p; and Pj: (scP) (sp:)/(s; 8z).
The connectivity of these extracted mesh vertices can be ef- ficiently inferred through look-up table. One may instead use tetrahedral grids, leading to the variant called Marching Tetrahedra, for which we visualize the look-up table Figure 2. Since the vertices are computed through simple differentiable operations, one may parameterize watertight meshes with a po- tentially deformable grid of SDF values egg., deep marching cubes (with some relaxation) [30] and DMTet [55].
Figure 2: Look-up table for Marching Tetrahedra (up to rotation symmetry). Grid vertices with and without red dot possess SDF values opposite signs.
G-SHELL: AN EXPRESSIVE REPRESENTATION OF GENERAL 3D SHAPES
4.1
OPEN SURFACE LIVING ON WATERTIGHT SURFACE
We start with the following simple observation on a category of open surfaces, which guides our insight to parameterize general 3D shapes with open surfaces that live watertight surface.
Any smooth and simply-connected open surface be smoothly deformed be subset of sphere.
This is direct consequence of classical topological theories on surfaces, of which more mathematical details are given in Appendix A. Indeed, large number of surfaces (e.g., plain T-shirts) can be completed to a watertight surface by first contracting the holes and later deforming it into a sphere. Inspired by this observation, we define a continuous and differentiable mapping v M - IR on the template sphere M characterize if a point belongs to the open surface Ma:
((x) 0, Vz E Interior
v(x)= = 0, Vx E a.M.
v(x) < 0, Otherwise,
Case inside the open surface
Case 2: on the surface boundary
Case 3: outside the open surface
where y can be instantiated as the signed geodesic distance to the open surface boundary living on the watertight template. While the number of choices of given some M and M, be infinite, without loss of generality we call the field of y manifold signed distance field (mSDF), since it is defined on manifold surface and characterizes the boundary in way that similar SDF.
The problem now effectively reduces to learn- ing a "2D mesh" defined by the zero isoline of v. Just as 3D meshes on the zero isosurface can be parameterized by 3D cubic grids deep marching cubes, "2D meshes" (polygo- nal curve) can also be parameterized by a "2D grid", i.e., mesh for the (deformed) sphere: simply to learn / value on each of the sphere mesh vertices from which watertight meshes. This process is illustrated in Figure 3. Intuitively speaking, an open sur- face can be viewed as the remaining essence after cutting out the hollow vacuum on a 3D shell, and hence we name the proposed 3D representation Chostt-n-the-Shell
(p).
(pz. P2)
extract
Figure 3: Illustration of non-watertight mesh extraction from some watertight triangular mesh. p1, are the po- sitions (watertight) mesh vertices. Ap= -|| - Pall and vi > are the corresponding mSDF values. The or- ange triangle is extracted and the blue polygon is discarded.
Such a naïve construction, however, poses modeling challenges when applied to general objects. First, cannot capture watertight surfaces that are not homeomorphic to spheres (e.g., donuts). Furthermore, some naïve deformation of surface in 3D may result in self-intersection and therefore addressing this requires additional regularization and/or modeling techniques.
Inspiration drawn from the manga series Ghost in the Shell
4

---

Published as a conference paper at ICLR 2024
Figure 4: G-SHELL. look-up table (up to rotational symmetry) for tetrahedral grids. Grid vertices with and without a red dot possess SDF values of opposite signs, and green dots on watertight mesh vertices indicates negative mSDF values. The pink regions represent the final extracted faces while the blue regions on the watertight template mesh. Colored polygons other than triangles are cut along dashed lines.
the discarded
Instead, we propose to jointly learn general watertight mesh template, parameterized by a 3D grid of SDF values, in order to capture a larger set of meshes². As we are not able to define v with mesh topology changing over time, we instead define in the 3D space. Specifically, we store the discretized values of y in a 3D grid. The mSDF value of any point in a grid cell can therefore be computed by barycentric interpolation of the values the grid cell vertices. We note that G-SHELL reduces to a typical watertight surface representation if all mSDF values on the grid are to positive values (i.e., no valid topological hole defined on the manifold).
4.2 EFFICIENT MESH EXTRACTION WITH G-SHELL
With SDF and mSDF values stored in the same 3D grid, we obtain for G-SHELL an efficient Marching- Cungg-Cube-liiie like algorithm which reuses the interpolation coefficient (rf. Eqn. 3) for the mSDF sign computation. Specifically, with an edge (pr. P,). ). the corresponding SDF values s; < 0 and mSDF values Vj, we can compute the mSDF value on the extracted mesh vertex as 1/ (8;4z - (,x:)/(s; - 8;). We give an example of the look-up table for tetrahedral grids in Figure which enumerates all possible cases of SDF signs (on grid vertices) and mSDF signs (on watertight mesh vertices). Despite using tetrahedral grids as an example, we note that G-SHELL generally applicable to other grid structures and not limited to tetrahedral grids.
5
APPLICATIONS OF G-SHELL
5.1 MESH RECONSTRUCTION FROM MULTIVIEW IMAGES
With G-SHELL, existing differentiable rasterization- based rendering methods (e.g., [17, 42]) can be seamlessly applied for end-to-end reconstruction of both 3D watertight and non-watee wattertttt meshes from multiview RGB and binary mask images. Reconstruction with rasterization not only allows the final geometry be explicitly optimized without pose-procsssing. but also compared to volumetric rendering with UDFs [35, 38] there is no need to evaluate densities on number of sample points per ray anymore. Moreover, with physics-basce mesh rendering, one can jointly optimize geometry, material and lighting in a single stage.
memory and time
We note, however, some particular difficulties in non-ward wateeriighhh mesh reconstruction from images. Unlike watertight meshes where we never get to see the inside surface, non- wwtertightt surfaces have two sides and one side may be more visible than the other. For example, the inside of long dress may not be fully observed and, when actually seen, is also likely to be poorly illuminated. Similarly, indirect illumination has to be considered along with realistic materials (especially highly specular ones) and potentially complex geometry. To simplify the problem while still being able to demonstrate the effectiveness of the G-SHELL representation, we use Nydiffrecme [17]. an occlusion-awan differentiable renderer that ignores indirect illumination but considers shadow rays.
Indeed, any orientable open surface without self-interseetion he modeled therehy.
5

---

Published as a conference paper at ICLR 2024
Another technical challenge is how identify the existence and location of topological holes with only 2D images, particularly when only a limited number of views are available. We therefore propose to regularize the mSDF values of the reconstructed mesh by introducing "holc-opening" loss (the mSDF is parameterized by a function with some parameter set 0MSDF):
LmSDF-reg (OMSDF)
Σ
Unuber(V0 (u)) +
Σ
Limber (VBier(VWasor (e) (1)
w.Vii (u)20
"esice 148 ( (u) ==0)=0
u' visable from some E Q
Encourage bole opening
Regularize holes from being too large
in which Q is the of training camera poses, T and are some positive scalars, Luuber is the Huber loss function. We introduce the second regularization term to discourage topological holes from being too large, especially during the early stage of the optimization process. We provide all the details regarding the remaining regularization losses and other training settings in Appendix B.
5.2 G-MESHDIFFUSION: GENERATIVE MODELING OF GEOMETRY
With the regular grid structure of the G-SHELL parameterization, it is straightforward to train generative models to produce the grid attributes (SDF, mSDF and potentially grid deformation) to enable non-watertigh mesh generation. Indeed, G-SHELL enables generative modeling with diffusion models [19] in which a regular input structure is necessary.
To demonstrate the generative modeling of G-SHELL, we consider MeshDiffusion [36], which gen- erates watertight meshes by sampling SDF and grid deformation in 3D tetrahedral grid, generate non-watertigh meshes. Although it is possible to simply introduce the additional dimension of mSDF on the grid vertex attributes, the generated shapes can be noisy as pointed out in [36]. Specifically, the boundary vertices of a generated non- non-watertight mesh are computed via an interpolation with mSDF:
11
1u2
VI <<<<< 122
V2
in which (w1, (2) is an edge on the extracted watertight template and V1, 1/ are the corresponding mSDF values. Because v, can be any real number, naive diffusion loss results in unevenly weighted prediction on u'. One could normalize y to +1, similar to what MeshDiffusion does for the SDF values, but it comes at the cost of expressiveness as the grid deformation is already used compensate the eтror resulting from the normalization of SDF values.
We therefore propose a modified version of the MeshDiffusion architecture to generate continuous mSDF values. Specifically, by setting a = |pal/ |r 1//. we rewrite Eqn. 2 into linear mapping of (u, u2): u' aug (1 a)u,. As a result, we may alternatively generate the linear interpolation coefficient which is bounded in [0. 1]. In the case of tetrahedral grids as in DM'ett, there are 12 candidate edges (u,, uzy) in each single cell since u, 's always lie on tetrahedral grid edges. We simply set the diffusion model to generate all a's I's for these candidates edges in each single cell. The a's to be used are eventually chosen based on the configuration of each generated tetrahedral cell (as in Figure 4). Similar to MeshDiffusion, we collect dataset of non-watertight meshes by running inverse rendering on multiview datasets of 3D objects. We term our final diffusion model on the G-SHELL representation as G-MeshDiffusion. More details can be found in Appendix C.
6 EXPERIMENTS AND RESULTS
6.1 RECONSTRUCTION FROM MULTIVIEW IMAGES
Baselines. We compare our method to current sfate-of-the-ard art methods for waterrittt mesh reconstruction: NDU [38], NeUDF [35] and NeAT [40]. We also evaluate watertight recon- struction methods including NeuS [60] and Nydiffrecme [17] (with DMTet [55]). We follow the original settings of these baselines, but train them for 400,000 iterations with mask loss weighted by 0.1. During testing, the novel views are rendered and subsequently evaluated at a resolution of 512. For UDF-based methods, the watertight explicit meshes are extracted using MeshUDF [15], while for SDF-based methods (NeuS/NeAT), the meshes are extracted with Marching Cubes [39] with the same resolution. For Nvdiffrecme with DMTet, use grid resolution of 128.
Dataset. We use DeepFashion3D [18] to quantitatively evaluate the performance of reconstruction with G-SHELL on non-watertight meshes. Specifically, ground truth meshes of 9 instances

---

Published as a conference paper at ICLR 2024
Ground
Nydiffremme TeD
NeuS
NeuralUDF
NeUDF
NeAT
Nediffremme wi G-Shell
Figure 5: Comparison between reconstruction w/ G-SHELL and other baseline methods on multiview recon-
struction on DeepFashion3D dataset. Top row: reconstructed texture. Bottom rows: reconstructed meshes.
Avg
NeuS [60]
31.876
26.850
29.234
29.692
30.405
33.827
31.591
32,846
26.282
30,289
Nydittreeme w/ DMT [55]
32.570
33.542
28.143
30.814
28.781
32.336
33.917
32.144
32.872
31.677
NeuralUDF [38]
30.264
25.887
28.908
31.115
28.463
30.739
32.185
29.965
32.574
30.011
NoUDF [35]
30.312
31.957
27.448
30,275
28.324
32,568
32.511
31.371
34.898
31,074
NeAT [-0]
27.407
28.228
24.129
26.944
24.887
29.630
30.846
27.149
29.841
27.674
Nydiffrecme G-SHELL
33.165
33,959
30.204
33,164
31.139
33,429
34.997
32,724
34.579
33.039
Nydiffrecms w/ G-SHELL (FC)
33.169
34.615
30.223
33,368
31.738
33.611
34.897
32,499
35.427
33,277
Table I: PSNR (1) DeepFashion3D garment instances. Marker: 1st rank and 2nd rank.
from DeepFashion3D-v2 that overlap with the instances of DeepFashion3D-vv used in [35, 38]. Different from previous work, such as NeuralUDF, that uses albedo images for training and testing, we instead use images rendered with realistic lighting and shading. Specifically, for each instance we use Blender with Cycles engine and realistic environment lightmap to render 72 views (RGB images and binary segmentation masks) for training and 200 views for testing.
Settings. For the multi- view reconstruction with G-SHELL, we set the grid resolution to 128 for tetrahedral grids and for FlexiCubes, and train the models with 5000 iterations using batch size of 2 views. For all baseline methods, use the default settings in their papers. Our experiments are carried out with tetrahedral grid implementation G-SHELL by default, if not otherwise specified. To demonstrate that the idea of G-SHELL be quite general, also evaluate simple variant of G-SHELL implemented with FlexiCubes [56], denoted by G-SHELL (FC) in Table and Table 2.
Reconstruction quality. Table shows the PSNR averaged over all test views. For all shapes fitted with iaplicit-based baseline methods (i.e., except for Nydiffrecme with DMTet), we render images directly from the learned implicit fields instead of the extracted meshes. We also evaluate all meth- ods on the quality of the reconstructed geometry by computing the bi-directional pointcloud-to-mee Chamfer distance. The results are given Table 2, from which we see that G-SHELLL achieves ex- cellent reconstruction quality and outperforms a number of fate-offthe-art methods. Figure 5 gives a qualitative comparison, where the front and back mesh faces (re-oriented to be consistent to their neighbors) are painted in different colors. The qualitative results in Figure 6 demonstrates that the mesh topology reconstructed by G-SHELL with FlexiCubes more regular than G-SHELL with tetrahedral grids, making better suited for physical simulation. More results are in Appendix F.
G-Shell Tetrahedral
G-Shell w/ FlexiCubus
Figure 6: Reconstructed mesh topology G-SHELL with tetrahedral grids and G-SHELL with Flexicubes. Mesh vertex count: Left 10,658, Right 6813.
7

---

Published as a conference paper at ICLR 2024
Method Instance ID
30
92
117
133
164
320
448
522
591
Avg
NeuS [60]
0.450
1.244
0.505
0.709
0.528
0.426
0.734
0.598
1.737
0.770
Nvdiffroome w/ DMT& [55]
0.629
0.466
0.724
0.856
0.722
0.444
0.444
0.649
1.026
0.662
NeuralUDF [38]
0.457
0.867
0
0.215
0.198
0,554
0.197
0.561
0.206
0.393
NeUDF [35]
0.315
0.458
0.204
0.107
0.184
0.432
0.159
0.585
0.128
0.286
NeAT [40]
0.605
0.325
1,010
0.873
0.538
0.305
0.323
0.591
0.237
0.534
Nvdiffrecme w/ G-SHELL
0.212
0.207
0.133
0.144
0.160
0.173
0.173
0.208
0.178
0.177
Nvdiffreeme w/ G-SHELL (FC)
0.235
0.227
0.146
0.154
0.165
0.229
0.195
0.261
0.234
0.203
Table Chamfer distance (cm 4) on DeepFashion3D garment instances, Marker:
rank and 2nd rank.
Method \ Metalnes
0.2
0.4
0.6
0.8
Avg
Method
Metalness
0.2
0.4
0.6
0.8
Neu5 (60)
33.59
32.03
31.62
31.21
30.58
30.45
31.5 18
Neu8 [60]
0:47
0.59
0.50
0.57
0.68
0.72
0.59
NeuralUDF [18]
31:42
29.54
29:45
29.39
29.25
28.98
29.67
NeuraUDF [38]
0.51
1.06
1.05
0.98
0.53
0.85
0.83
NeUDF [35]
32.45
29.32
29.04
29.19
29.13
29.18
29.72
NeUDF [55]
0.26
0.52
0.40
0.45
0.70
0.44
0.46
NeAT [10]
28.41
27.37
28.07
26.93
26.58
26.85
27:37
NeAT [80)
0.59
0.65
0.61
0.68
0.70
0.73
0.66
G-SBELL
36,01
34.23
32,86
33.08
32,93
32.31
33.57
G-SHELL
0.20
0.22
0.28
0.24
0.24
0.27
0.24
Table 3: Ablation study on metallic materials. We use Nydiffrecmm along with the proposed G-SHELL for our
reconstruction in the experiment. Left: PSNR (1). Right: Chamfer distance (em 1).
From Figure 5. we can observe that our method achieves much better prediction on novel views than the other baselines due to the better modeling of lighting, occlusion and material. Due to the highly concave structure in the clothing interior, we can also see that watertight reconstruction baselines often fail to reconstruct the inner side of clothing in the DeepFashion3D dataset.
Efficiency training and inference. addition, we compare the runtime of all watertihtt methods (tested on the same machine with a single NVIDIA RTX 6000 GPU). Along with Nvd- iffreeme, G-SHELL takes only 3 hours to fit a ground truth shape while NeuralUDE, NeUDF and NeAT take 17.3, 16.4, and 4.3 hours, respectively. For novel-view synthesis with all images rendered with a resolution of 512x512, our method runs at 2.7 sec/img (inferring from a learned tetrahedral grid with the Nvdiffrast rasterizer (27)), while NeuralUDF, NeUDF and NeAT run at 1.8 min/img, 1.4 min/ing, and 9.7 min/img, respectively. Compared to the other methods, ours is significantly faster in both training and inference due to highly efficient rasterization.
6.2 HYBRID WATERTIGHT AND NON-WATERTIGHA MESH RECONSTRUCTION
To demonstrate that G-SHELL may be used for recon- structing both watertight and non-watertight shapes at the same time, we test our method on an instance from the synthetic NeRF dataset [41] in which all the images are taken from the objects above and never below. We adopt a minimalist assumption that unseen regions should always be empty and reconstruct the target shapes with G-SHELL. On the left of Figure 7, we show that the upper surface of the target shape is well reconstructed without adding any unnecessary lower surface faces. On the right hand side of Figure 7, we also visualize the generalized winding number field [2. 21] by randomly sampling points near the surface to quantify local manifoldness. It shows that in- verse rendering with G-SHELL can reconstruct a hybrid shape of both watertight and non-watertight parts where the regions with only visible upper surface are reconstructed as single-layered meshes.
Figure 7: Left: the reconstructed mesh with back faces painted orange. Right: general- ized winding number field; points near the mesh boundary are colored white.
6.3 RECONSTRUCTION UNDER SPECULAR LIGHTING
One key advantage of mesh inverse rendering that geometries may be reconstructed in cases of complex lighting and materials. To demonstrate such an advantage, we conduct ablation experiments and compare the performance of non-watertight mesh reconstruction methods on shapes with specular surfaces. Specifically, we modify the material of the sweater mesh presented in Figure 5 by setting the roughness parameter to 0.4 and create a set of meshes with the metalness parameter ranging from to 1. We use the same set of hyperparameters for the experiment. Results in Table and Table 3 verify that G- G-SHELL can well reconstruct both images and shapes under complex lighting and materials.
8

---

Published as a conference paper at ICLR 2024
Upper
Lower
Figure 8: Samples of unconditionally generated upper and lower outfits by G-MeshDiffusion.
MMD (10" 3.
COV (St, 1)
I-NNA (%. J)
Method
MV-FID (L)
CD
EMD
EMD
CD
EMD
MeshDifflusion [36]
68.72
$8.99
Lower
GET3D [12]
1.57
$7.19
46.00
48.13
79.66
72,82
95.69
Garments
MesshDifuusion
55,30
MeshDiffusion [36]
1.24
55.37
$1.51
46.71
77.44
78.77
167.69
Upper
GET3D [12]
1.37
56.86
43.87
84.37
Garments
G-MeshDiffusion
LO5
$1.15
51.33
49.20
61.37
60.66
88.98
Table 4: Quantitative results the proposed G-SHELL diffusion model (G-MeshDiffusion))
6.4 GENERATIVE 3D MESH MODELING
Baselines. As there are no mesh generative models for open surfaces with manifold structures, we compare method with watertight mesh generative models: MeshDiffusion [36] and GET3D [12], both using the SDF-based DMTet [55] as the representation. Detailed training settings are deferred Appendix C. We implement G-SHELL with tetrahedral grids in this experiment.
Dataset. We collect meshes in 4 categories (T-shirt, Top, Skirts, Trousers) from the Cloth3D dataset [4] and regroup them into new categories: upper garments (including T-shirt and Top) and lower garments (including Skirts and Trousers). For both MeshDiffusion (using the DMTet representation) G-MfssDiifusion (using the G-SHELL representation), we run inverse rendering on meshes with known environment lightmaps and known materials using RGB, binary mask, and depth supervision. We generally follow same settings of [36] for G-MeshDiffusion. For GET3D, we follow the same training setting as [12] and render multiview RGB images for training.
Evaluation metrics. For each model, we sample set of meshes, with the size of the test sets, using 100 steps of DDIM [57] and apply standard Laplacian smoothing to these meshes. Similar to [12, 36], we evaluate point cloud metrics between the point clouds sampled from generated meshes and those from ground truth meshes. To compensate the lack of perceptual measure in the point cloud metrics, we also evaluate the generated results with multiview FID (MV-FID) [36, 69], which is computed by average of FID (Fréchet Inception Distance) scores of 20 views (rendered with fixed light sources

---

Published as a conference paper at ICLR 2024
Upper
Lower
Interpolation
Figure 9: Interpolation results of our generated samples.
Geverated
Nearest neighher
Figure 10: Nearest neighbor of our generated sample in the training set.
and diffuse-only mesh material). During rendering, we do not re-orient the face normals towards the camera so that the difference between watertight and open surfaces can be taken into account.
Qualitative and quantitative results. The quantitative results are given in Table 4. We observe that G-MeshDiffusson generally achieves better performance than the watertight mesh generation methods (MeshDiffusion and GET3D), but more importantly, G-MeshDifflssio the single-sided nature of non- watertiht meshes as it achieves a significantly better MV-FID score. In addition, qualitatively show some unconditionally sampled meshes from G-MeshDiffusion in Figure 8. Following [36], we also provide some interpolation sequences (obtained by spherical linear interpolation with the initial Gaussian noises using 100-step DDIM inference) in Figure 9. The interpolation results demonstrate a smooth transition across different clothing styles. Finally, we show in Figure 10 the nearest neighbor of our generated meshes in the training set. The results show that G-MfssDiffusion does not memorize the training samples and can generate novel shapes.
better capture
DISCUSSION ON LIMITATIONS
Representation. G-SHELL is not able to model shapes with self-intersections. Nor does model non-orientable surfaces such as Möbius strips, since the of SDF implies orientability. Besides, compared to implicit-basee methods of which the resolution can be viewed as "infinite", G-SHELL generally may require higher resolution in order model tiny topological holes.
Mesh reconstruction. the target geometry is complex, since the entanglement between geometry and indirect illumination comes into play. In this paper, we only take shadow rays into consideration, but in more complex scenarios one might need to model indirect illumination.
The discontinuity in mesh rendering poses a severe optimization problem when
Mesh generation. 3D U-Nets are not memory-efficient, and it is hard to scale them to high resolutions. Future work for better generative modeling with G-SHELL may include more efficient architectures such as those using triplane features [6], and alternative 3D generative methods similar to GET3D [12].
8 CONCLUDING REMARKS
Our proposed G-SHELL is a general 3D mesh representation which models both watertight and non-watertight meshes. By introducing the new quantity of manifold signed distance field (mSDF) on a learnable watertight mesh template with varying topology, we enable both rasterization based reconstruction and diffusion- morisiioo-moll--eee model-based unconditional generation non-watertigh meshes. Such design leads to better performance and greater flexibility in non-wa watertight mesh modeling. Still, G- SHELL is only able to parameterize limited subset of meshes, especially when limited computational resources are available. Furthermore, it calls for better inverse rendering and generative modeling techniques to fully leverage the flexibility of G-SHELL.
10

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We sincerely thank Peter Kulits for paper proofreading and Zhouyingcheng Liao for creating physical simulation demos. Authors listed as equal contribution (resp., shared last author) are allowed to switch their orders in the author list in their resumes and websites. The paper title was proposed during an after-dinner coffee chat among Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu and Tim Xiao, especially due to Yuliang Xiu (for proposing "Shell") and Weiyang Liu (for the connection to the manga series Ghost in the Shell ").
Disclosure. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: OIIS O 18039B, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 PProecct number 390727645. MIB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook,, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a part-time employee of Meshcapade, his research was performed solely at, and funded solely by, the Max Planck Society. LP supported by the Canada CIFAR AI Chairs Program and NSERC Discovery Grant. WT. was supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP XX. project number: 276693517. YF is partially supported by the Max Planck ETH Center for Learning Systems. YX is funded by the European Union's 's Horizon 2020 research and innovation programme under the Marie Aklodowska-Curds grant agreement No.860768 (CLIPE).
REFERENCES
[1] L.V. Ahlfors and L. Sario. Riemann Surfaces: (PMS-26). Princeton Mathematical Series. Princeton University Press, 2015.
[2] Gavin Barill, Neil G Dickson, Ryan Schmidt, David IW Levin, and Alec Jacobson. Fast winding numbers for soups and clouds. ACM Transactions on Graphicx, 3- 2018.
[3] Fausto Bernardini, Joshua Mittleman, Holly Rushmeier, Cláudio Silva, and Gabriel Taubin. The ball- pivoting algorithm for surface reconstruction. IEEE transactions on visualization and computer graphics. 5(4):349-359, 1999.
[4] Hugo Bertiche, Meysam Madadi, and Sergio Escalera. Cloth3d: clothed 3d humans. In ECCV, 2020.
[5] Mario Botsch and Leif Kobbelt. A remeshing approach to multiresolution modeling. In Eurographics, 2004.
[6] Eric R Chan, Connor 7 Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR, 2022.
[7] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
[8] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023.
[9] Zhiqin Chen, Andrea Tagliasacchi, Thomas Funkhouser, and Hao Zhang. Neural dual contouring. ACM Transactions on Graphics, (1)),, 2022.
[10] Julian Chibane, Gerard Pons-Moll, et al. Neural unsigned distance fields for implicit function learning. In NewrlPS, 2020.
[11] Thor Christiansen, Jakob Andreas Bærentzen, Rasmus Paulsen, and Morten R Hannemose. Neural representation of open surfaces. In Computer Graphics Forum, volume 42, page e14916. Wiley Online Library, 2023.
[12] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In NewrIPS, 2022.
[13] William Gao, April Wang. Gal Metzer, Raymond A Yeh, and Rana Hanocka. Tetgan: A convolational neural network for tetrahedral mesh generation. In BMVC, 2022.
hhtp:://wwwwiiinppiiiioo/iiiiihh shelS

---

Published as a conference paper at ICLR 2024
[14] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In ICML, 2020.
[15] Benoit Guillard, Federico Stella, and Pascal Fua. Meshudf: Fast and differentiable meshing of unsigned distance field networks. In ECCV, 2022.
[16] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: A self-prior for deformable meshes, ACM Trans. Graph., 39(4), 2020.
[17] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkherg. Shape, Light, and Material Decompasition from Images using Monte Carlo Rendering and Denoising. In NeurlPS, 2022.
[18] Zhu Heming, Cao Yu, Jin Hang, Chen Weikai, Du Dong, Wang Zhangye, Cui Shuguang, and Han Xiaoguang. Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images. In ECCV, 2020.
[19] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion prohahilistic models. In NewrlPS, 2020.
[20] Jingwei Huang. Hao Su, and Leonidas Guibas. Robust watertight manifold surface generation method for shapenet models. arXiv preprint 180.01088, 2018.
[21] Alec Jacobson, Ladislav Kavan, and Olga Sorkine-Hornung Robust inside-outside segmentation using generalized winding numbers. ACM Transactions on Graphics, 32(4):1-12 -2, 2013.
[22] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In CVPR, 2020.
[23] Tao Ju, Frank Losasso, Scott Schaefer, and Joe Warren. Dual contouring of hermite data. In SIGGRAPH, 2002.
[24] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. SGP, 2006.
[25] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4): 14, 2023.
[26] Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In ICMI,, 2021.
(27] Samuli Laine, Janne Hellsten, Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics, 1914 2020.
[28] Der-Tsai Lee and Bruce Schachter. Two algorithms for constructing a delaunay triangulation. Interna- tional Journal of Computer Information Sciences, 9(3):219-222. 1980.
[29] Lee, Introduction to Topological Manifolds. Graduate Texts Mathematics. Springer New York, 2010.
[30] Viyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representa- tions. In CVPR, 2018
[31] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
[32] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In CVPR, 2020.
[33] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Leamnnng to infer implicit surfaces without 3d supervision. In NeurlPS, 2019.
[34] Weiyang Liu, Zhen Liu, Liam Paull, Adrian Weller, and Bernhard Schölkopf. Structural causal 3d reconstruction. In ECCV, 2022.
[35] Yu-Tao Liu, Li Wang, Jie Yang. Weikai Chen, Xiaoxu Meng, Bo Yang, and Lin Gao. Neudf: Leaning neural unsigned distance fields with volume rendering. In CVPR, 2023.
[36] Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffu- sion: score-hased generative 3d mesh modeling. In ICI.R, 2023.
12

---

Published as a conference paper at ICLR 2024
[37] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics, 38(4):1-14, 2019.
[38] Xiaoxiao Long, Cheng Lin, Lingji Liu, Yuan Liu, Peng Wang, Christian Thcobalt, Taku Komura, and Wenping Wang. Neuraludf: Learning unsigned distance fields for multi-view reconstruction of surfaces with arbitrary topologies. In CVPR, 2023.
[39] William Lorensen and Harvey Cline. Marching cubes: A high resolution 3d surface construction algorithm. In SIGGRAPH, 1987.
[40] Xiaoxu Meng, Weikai Chen, and Bo Yang. Neat: Learning neural implicit surfaces with arbitrary topologies from multi-view images. In CVPR, 2023.
[41] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. In ECCV, 2020.
[42] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Mueller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In CVPR, 2022.
[43] Charlic Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of meshes. In ICML, 2020.
[44] David Palmer, Dmitriy Smirnov, Stephanic Wang, Albert Cher, and Justin Solomon. Deepeurrents: Learning implicit representations of shapes with boundaries. In CVPR, 2022.
[45] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In ICCV, 2019.
[46] Songyou Peng. Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape points: diffcrentiable poisson solver. In NeurlPS, 2021.
[47] Xucbin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool. Highly accurate dichotomous image segmentation. In ECCV, 2022.
[48] Lingteng Qiu, Guanying Chen, Jiapeng Zhou, Mutian Xu, Junle Wang, and Xiaoguang Han. Rec-mv: Reconstructing 3d dynamic cloth from monocular videos. In CVPR, 2023.
[49] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. In NeurlPS, 2020.
[50] Ian Richards. On the classification of noncompact surfaces. Transactions of the American Mathematical Society, 106(2):259-25) 269, 1963.
[51] Johannes Lutz Schönberger and Jan MunMMichael Frahm. Structure- motion revisited. In CVPR, 2016.
[52] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefcys, and Jan-Michacl Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
[53] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In NeurlPS, 2020.
[54] Silvia Sellán and Alec Jacobson. Stochastic poisson surface reconstruction. ACM Transactions Graphics. 2022.
[55] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In NeurIPS, 2021.
[56] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojeie, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Graphics, 42(4):1-16, 2023.
[57] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
[58] Graham M Treece, Richard W Prager, and Andrew H Gee. Regularised marching tetrahedra: improved iso-surface extraction. Computers & Graphies, 23(4):583-598, 1999.
[59] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei I Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018.
13

---

Published as a conference paper at ICLR 2024
[60] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
[61] Xingkui Wei, Zhengging Chen, Yanwei Fu, Zhaopeng Cui, and Yinda Zhang. Deep hybrid self- self-prior for full 3d mesh generation. In ICCV, 2021.
[62] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic seeerative-aveesariid modeling. NIPS, 2016.
latent space object shapes
[63] Yuliang Xiu, Jinlong Yang. Xu Cao, Dimitrios Tzionas, and Michael J. Black. ECON: Explicit Clothed humans Optimized Normal integration. In CVPR, June 2023.
[64] Yuxuan Xue, Bharat Lal Bhatnagar, Riccardo Marin, Nikolaos Sarafianos, Yuanlu Xu, Gerard Pons-Moll, and Tony Tung. Nsf: Neural surface fields for human modeling from monocular depth. In ICCV, 2023.
[65]
Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: point cloud generation with continuous normalizing flows. ICCV, 2019.
[66] Guandao Yang, Abhijit Kundu, Leonidas Guibas, Jonathan Barron, and Ben Poole. Learning diffusion prior for nerfs. arXiv preprint MrxXiv:3304.14777 2023.
[67] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurlPS, 2020.
[68] Xiaohui Zeng, Arash Vahdat, Francis Williams. Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffrusion models for shape generation. NeurlPS, 2022.
[69] Xin- Xin-Yang Zheng, Yang Liu, Peng-Shuai Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for shape generation. In SGP, 2022.
[70] Heming Zhu, Lingteng Qiu, Yuda Qiu, and Xiaoguang Han. Registering explicit implicit: Towards high-fidelity garment mesh reconstruction from single images. CVPR, 2022.
14

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Some Mathematics Surfaces
16
B Detailed Settings on Mesh Reconstruction
17
B.1 Losses
17
B.2 Differences in implementations for lighting regularization
19
B.3 Rendering, optimization and other settings
19
C
Detailed Settings on Diffusion Models with G-SHELL
20
C.1 Grid storage of interpolation coefficients from mSDF ................................
20
C.2 Architecture training settings
................................
20
C.3 Data preparation
20
Brief Introduction on MeshDiffusion
E
Implementation of G-SHELL
22
F More Qualitative Examples of Multi-view Mesh Reconstruction
23
G Reconstructed Meshes for Surfaces with Complex Patterns
26
H Reconstruction of Shapes with Metallic Materials
27
On mSDF Regularization
28
Experiment on Real Data
29
K Visualization of Learned Watertight Mesh Templates
30
15

---

Published as a conference paper at ICLR 2024
A SOME MATHEMATICS ON SURFACES
We first state the condition for homeomorphism between compact triangulable bordered surfaces?:
Theorem ([50)) Two compact triangulable bordered surfaces are homeomorphic if and only if they both have the same number of boundary curves, the same Euler characteristic, and are either both orientable or else both non-orientable.
It is apparent that any simply-connected surface is homeomorphic to compact disk sphere, since simply-connected bordered surfaces are triangulable.
Indeed, one may establish a more general relation between compact bordered surfaces and closed surfaces with the following classification theorem:
Theorem (Classification Theorem of Surfaces [29]) Every compact surface (without boundary) is homeomorphic to one of the following:
The sphere S.
connected sum of tori: TT...T.
connected sum of projective planes: TP #P.
Together with Theorem 1, one may conclude that any connected orientable bordered surface (which triangulable) is homeomorphic to compact bordered surface either sphere connected sum of tori.
We conclude this section by noting the following theorem:
Theorem ([1)) Every bordered surface F can be regularly embedded in surface. F is compact, can be regularly embedded in a closed surface
Lie. 2-manifolds, opposed the casual daily-life definition of surfaces.
16

---

Published as a conference paper at ICLR 2024
B DETAILED SETTINGS ON MESH RECONSTRUCTION
B.1
LOSSES
RGB and mask supervision. Given multiview RGB images IGT and the corresponding binary masks MGT (to segment foreground from background), we optimize the following losses:
LRGB = ||RGBB Rendered
? MGT RGB. GT @ Marll
(3)
Lmask
Rendered
MGT l12
(4)
Since the direct output of the renderer is high-dynamic range (HDR) image /RGB, Rendered, we follow [17] and use the same tonemapper I' to map the HDR image into sRGB image with pixels in the range [0, 255]:
IRGB, Rendered = T(RRBB, Remdered) (log (RRGB, Rendered + 1));
(5)
where I' is the sRGB transfer function [17].
mSDF supervision. Since all pixels with Mor should have a non-positive projected mSDF value we introduce the following loss on mSDF values:
LMSDF |1 Mor)+ sgn(1 Mor)ll
(6)
in which M, is the rendered (i.e., projected) mSDF image.
Depth supervision. In cases when ground truth depth images provided, we include depth supervision loss:
Depph Depth. rendored Depth. GT
Eikonal regularization. To reconstruct smoother surfaces, we use the standard Eikonal regularization [14] for MLP-paramettrrized SDF values on the vertices of the reconsiructed mesh:
L Tiikonal
E (II 1)² eEV
(8)
in which V is the set of vertices of the extracted mesh.
mSDF regularization. We rewrite the mSDF regularization, introduced in the main paper, into the following (with 0.001): LmSDDF-regooeu (0 (OMSDE)
Σ
Lauber (vo (u)).
(9)
uite (u)20
I TeSSDE-re reg. close (OMSDF)
Σ
Thuberr (vo (w') e).
(10)
(1 =
al visible from q
To determine if mesh vertex u' is visible for the set of all training camera poses is computationally costly. Therefore, we instead use an approximated loss for LmSDF.reg-cosee
LISDE reg close (0mSDr)
Σ
Liuaerr (vo(e)) -e).
(11)
28 '2 (w )=0 n' visible from some E Qe
in which Qe is the batch of camera poses at training iteration t.
Note that stop the gradient flow from these two losses to both SDF and grid node offsets. While it is possible to allow gradients from mSDF regularization to update SDF and grid node offsets as well, choose not do stabilize the optimization process.
17

---

Published as a conference paper at ICLR 2024
We follow [17] and use the following regularization losses:
Albedo smoothness regularization. To effectively decouple light from material, smoothness regular- ization is imposed on albedo hy:
Lka
Σ
(Fa ka(FFuut Cka)l,
(12)
Isurt
in which Cke ol) (we obe 0.01).
SDF regularization. The following SDF regularization is to reduce floaters and inner geometry:
LSDD reg =
Σ
H(((s:), sgn(s;)) + H(((s;), sgn(s;))
(13)
JJES
where S is the set of unique edges on the 3D grid, H is the binary cross-entropy loss, 8,, 8j are the SDF values of the corresponding grid vertices and sgn is the sign function. This regularization loss aims to smooth the SDF in the whole space; as a result, isolated tiny meshes are less likely. This regularization term is complimentary to the Eikonal loss, Eikonal loss is mostly concerned with the surface smoothness as it is computed on the surface only.
Monochrome regularization. The following loss helps decoupling lighting, geometry and material:
(14) where ca and c, are the diffuse and specular light components, Lref is the target reference image, T is the tonemapping function introduced earlier, Y (xx is the simple luminance operator by averaging RGB channels (I, + 2g + /33, and V max(E, Er, gg Z&).. With this loss term, the demodulated lighting assumed be mostly monochrome, i.e., (x)) x ((r).
Lightt = [ + <x))) (Felll
In addition, we penalize the specular component with the following loss:
Tspecclar Aspecntar (cx))
(15)
min ( (ca)llaa
Mesh topology regularization. Finally, for the G-SHELL variant with FlexiCubes, we follow [56] and use the proposed mesh topology regularization Laev which penalizes mean absolute deviation between the primal and dual vertices in FlexiCubes. The precise loss and parameter definitions are beyond the scope of the paper and we do not elaborate them here.
Total loss. In summary, the total loss is
Trotal
Trendering
+ T Tegeumetry reg Tematerial light
(16)
with
Lyendering
LRGB + Mask Mask + "Depth Depth
+ YMSDD IISDFF
(17)
I yeeometry reg
Tikoonll Taloonal + Feg TSDD
TISSDF LmSDD..eg.ppe + TnsDD
Tide Lavvs
(18)
and
Lmaterial-lighh-reg Light + Tsspecular P + Thk Lig-
(19)
Choice of loss scales. For DeepFashion31) instances, we set "Mask = 1, YMSDDE = 0.5, TmSDD
10-6/ (0-6/p, 7mSDDF-eg-open (grid-resolution/64]] We use linear schedule for Eikonal regularization: Yeikonal
3 10 10-"/p, Nigh 0.15 and Tispecular = 0.0025. where
Tigh
0.3 for the
first 500 iterations, YEikonal 0.1 for iteration 500 to 2000 and YElkonal 0.01 for the remaining iterations.
For the synthetic chair experiment, we set Tiskonal -pee to 2e 5/p for the first 1500 iterations and 2e 6/p for the remaining ones.
5e 3, TmsDD--regclss 2e 4/p. We set
We set Yavv not mentioned are set to default values as in [17].
0.25 for the G-SHELL variant with FlexiCubes but otherwise 0. All other loss scales
'There slight differences between the Nydiffreeme paper, the official repository of Nydiffremme and our implementation. Please Sec. B.2 for more details.
18

---

Published as a conference paper at ICLR 2024
B.2 DIFFERENCES IN IMPLEMENTATIONS FOR LIGHTING REGULARIZATION
We note the difference between the Nvdiffrecme paper [17] and the official implementation'. In the original paper, the lighting regularization is written such that T is the first be applied to the rendered and reference image:
(T(ca c.))) ((
(20)
The official implementation uses a modified regularization loss, in which the first term tends to encourage specular components (e = 0.001 in the repo):
|| (ca)ll Adiffuse min (ca + co)ll [Ca + c.)) - + Aspecalar (c,)ll
(21)
B.3 RENDERING, OPTIMIZATION AND OTHER SETTINGS
For G-SHELL with tetrahedral grids, we set the number of rays samples of the Nydiffrecme renderer (for Monte-Carlo approximation of the rendering equation) Nample 24 for better performance on the interior of mehses (e.g., clothing in DeepFashion dataset). In practice, setting Nample = 12 leads to faster convergence (approx. 2 hours) without too much sacrifice on reconstruction quality (PSNR values lowered by less than 1). For G-SHELL with FlexiCubes, we instead use Nasmple- = 16.
As occlusion heavily depends on shape topology, it will be hard to optimize both lighting and shape topology at the same time with some initialized shape. To alleviate this issue, we follow Nydiffrecme [17] and use soft occlusion during the first few hundred iterations. Specifically, the occluded light rays will be scaled by 7) which gradually decreases from 1 at iteration O to O at iteration 1000. is fed to the denoiser Nydiffrecme as well. Please refer to [17] for implementation details.
The learnable PBR materials are parameterized in the same way as in [17]. We do not learn a normal map (and therefore the normal regularization in the Nydiffrecme paper does not apply here).
We use Adam optimizer with (B), _B2) = (0.9, 0.99). The total number of iterations for each shape is 5000. The initial learning rate the MLP for parameterizing SDF values is set to 3e while those for mSDF and grid node offsets are set to 0.15. The initial learning rate for the rest of parameters used in G-SHELL with FlexiCubes are set to 3e 4. The learning rates follow a decay schedule of 10 10-0.0000 0.0000- where t is the iteration number.
We use an MLP with hidden layers and a hidden channel size of 128 (with a concatenation shortcut from input to the output of the 4th layer) parameterize SDFs. We use the standard positional encoding (for instance in [41]): [1, cos(2 2). sin(2' z), cis(2)))sssn((²r
cos(2 x). sin(2 x)] with K = 6. We use Softplus activation with 3 100 for all layers except for the final output layer (in which no activation function is used). The MLP is initialized by fitting the SDF of a sphere with the diameter being roughly half of the grid size.
The learnable grid node offsets, after multiplied by the scale of deformation, are directly added to the canonical (i.e., undeformed) grid vertex positions. After each gradient update, we clip the learned (and unscaled) grid node offsets to [-1, 1]. These learnable offsets are initialized to O.
The images for the DeepFashion3D ground truth shapes are rendered using Cycles engine in Blender with 5,000 samples, number of max bounces of 24 and fixed environment lightmap.
We sample 100,000 points per mesh to compute Chamfer distances. Here we use the implementation in ECON [63] 8 which computes the bi-directional metric with cointlooud-to-urriee distances.
19

---

Published as a conference paper at ICLR 2024
C DETAILED SETTINGS ON DIFFUSION MODELS WITH G-SHELL
C.1 GRID STOR AGE OF INTERPOLATION COFFFICIENTS FROM MSDF
As described in the main paper, in each tetrahedral cell there are 12 candidate (watertight) mesh edges if we count all possible SDF configurations. Since these 12 candidate edges are in general spatially separated, we take the simplest approach and store them on larger cubic grid.
Suppose (normalized) SDF values and grid node offsets are stored in a cubic grid of resolution R. the same as in MeshDiffusion. Due to the spatial regularity of these 12 candidate edges (as long as we use regular enough tetrahedral grid), we design positional embedding for each edge. Specifically, the "location" of each candidate edge (u,, Uz), of which uy lies on the tetrahedral grid edge (PI. P2) and uj (p2, 12).P.). is
P1 + P3 + 2D2
(22)
where p's are the undeformed grid vertices respectively. With these positional embeddings, we may store them to the nearest coordinate in larger cubic grid of resolution 2R. As result, we are able to use 3D U-Net for a diffusion model to jointly generate SDF values, deformation vectors and mSDF-induced interpolation coefficients.
For each candidate edge (u,, uy), we may either generate a or aj (note that aj = =1 oz). We pick the simplest convention: first sort two nodes (ua, Mý) of each candidate edge in lexicographical order of the three dimensions of a and b, and then for each sorted edge (a, b), we always pick Og as the interpolation coefficient to generate.
C.2 ARCHITECTURE AND TRAINING SETTINGS
We use the same architecture of MeshDiffusion but add two input layers (one for the mSDF values and one for the mSDF grid mask which indicates where the grid stores mSDF values; each implemented with 3D convolution layer of kernel size 3) and output layer (implemented with transposed 3D convolution layer with kernel size 4, stride 2 and padding 1) to accommodate the introduction of an additional grid for MuSDF-induced interpolation coefficients. The output of the additional input layer is directly added to the output of the original input layer in the MeshDiffusion U-Net architecture. Similar to MeshDiffusion, the predicted mSDF noise is multiplied by the mSDF grid mask so that only the used mSDF grid locations are counted.
We follow the training settings in MeshDiffusion with reduced learning rate of le 5. We use batch size of 8 with 4 gradient accumulation steps (with 8 80GB-mem A100 GPUs). We use mixed precision training to speed up the training process.
C.3 DATA PREPARATION
We use the official GET3D implementation" to prepare datasets of rendered images for ground truth meshes (with a simple diffuse-only material). For MeshDiffusion (with DMTet), we mostly follow [36] but fit each object with 1000 iterations for both coarse-fitting and finetuning stages (instead of 5000) with the deformation scales set to and 2.0, respectively. For G-MeshDiffusion we use the depth supervision loss with Ydepth 100 but disable the mSDF regularization losses the depth information is enough for identifying topological holes on ground truth shapes. As in the reconstruction experiments, SDFs are parameterized by an MLP instead of being directly stored as learnable scalars in the tetrahedral grid. We recenter the meshes to the origin (in the world coordinate), and then rescale all meshes in an isotropic way so that the minimum and maximum of the bounding box coordinates are -1 and 1, respectively. After rescaling, we compute the minimum and maximum of xr y, 2 coordinates in each dataset (upper and lower garments) and scale the tetrahedral grid with H (0.45( ), 0..5
Vmin) ). 0..5( Zmmx >min)). The SDF values are initialized to fit a ellipsoid obtained by scaling a unit sphere with H. The mSDF values stored on tetrahedral grids are initialized in the same way in reconstruction experiments: we randomly initialize them by sampling from a uniform distribution 24( (-0.01, 0.99).
20

---

Published as a conference paper at ICLR 2024
D A BRIEF INTRODUCTION ON MESHDIFFUSION
MeshDiffusion [36] a diffusion model [19] that generates a tetrahedral grid representation [55] for watertight meshes. It assumes that every mesh in the dataset is parameterized by a deformable tetrahedral grid (with fixed grid topology) which stores SDF values. By setting the canonical tetrahedral grid uniform grid, one may measure the node offsets from the deformed position to the canonical position. The deformable grid of SDF values (1 dimension) can therefore be turned into uniform grid of offsets and SDF values (in total 4 dimensions). As a uniform tetrahedral grid can be seen as a subset of a cubic grid, a dataset of cubic grids can be created by introducing some artificial sites to augment the tetrahedral grid. By using cubic grids, standard 3D U-Nets (with standard 3D convolution layers) can be used for the diffusion model, and MeshDiffusion is trained on and produces such augmented grids.
While directly producing a 4-dimensional vector of node offsets and SDF values is feasible often it produces many artifacts in generated meshes. The reason is mostly due to that the mesh vertex positions are computed in a non-linear way:
$2 U281
/2/ + |s1|u2
122
(23)
82 81
[8]
|82|
with u1, U2 be two nodes of an edge on a tetrahedral grid and $1 > 0 > 82 the corresponding SDF values. Notice that u' stays the same if one scales both 81 and 82 by same positive scalar. As the SDF values are not directly computed from the ground truth shapes but instead optimized [36, 42, 55], they cannot be uniquely determined. As a result, the scale of them can vary in different positions and vary across the dataset, which makes the diffusion model harder to learn. To see why it could be an issue, let's suppose that there is a node in the tetrahedral grid which always has a negative SDF value with tiny scale in the dataset, and its surrounding nodes have positive SDF values with large scales. A small perturbation on the SDF values of the node with tiny SDF scale may easily lead to a topological change in the extracted mesh while one on the neighboring nodes is much less likely to change the mesh topology.
Here is another perspective to demonstrate the effect of the unconstrained scale: suppose the signs of SDFs are known (therefore, known mesh topology) and the tetrahedral grid is not deformable. The model now only needs to learn the scale of SDF values. Let's further assume small and identical noise é' > 0 on all SDF values. We have
m12,noisy
M12
(u2 u1)
(24)
[81
[82]
one trains a DDPM model [19] on the extracted mesh vertex positions instead (as the mesh topology is assumed to be known and fixed), the loss becomes
E
we 1ilc-ollatte toe, !)1?
(25)
where D is the dataset of meshes, U is the vector of all mesh vertex positions and w, is a weighting coefficient and fe is the denoising U-Net. With the mesh extraction formula with noisy SDF values, one may see that the standard Gaussian noise in mesh vertex positions in each local region is roughly the standard Gaussian noise in SDF values multiplied by the inverse of the average local SDF scale. Uneven SDF scales in the dataset lead to a unevenly weighted loss (in the sense of mesh vertex positions) on different data points and on different mesh vertex positions.
To alleviate this issue, MeshDiffusion employs SDF normalization strategy which rounds SDF values in the dataset to 11, depending on the sign of SDF values. Such an operation leads to additional errors, and therefore in the data collection process one needs to finetune the optimized grid node offsets after the normalization step. As the model is trained on normalized datasets, SDF values of the generated grids need to be normalized well. Here we note that similar phenomenon has been observed in [66] in which learning diffusion models to generated NeRF grid fields from an arbitrary NeRF dataset leads to artifacts and constraints on NeRF datasets have to be imposed for better quality.
21

---

Published as a conference paper at ICLR 2024
E IMPLEMENTATION OF G-SHELL
Algorithm Mesh Extraction with G-SHELL
1: For any grid edge (21, with opposite SDF signs, compute mesh vertex positions by (p182- - p281 -1))
2: Project all mSDF values stored on the 3D grid to the extracted mesh vertices. The resulted mSDF
value v' for any u extracted from (p1,P2) is v' (21 //51)/($2 $1).
3: With the mSDF on watertight mesh vertices and SDF signs on grid nodes known, check the look-up table and determine the non-watertight mesh topology for each single cell. Compute
the position of any boundary vertex with (vbua vaul)//( Va) for any watertight mesh edge (ua, Ub) to be cut.
22

---

Published as a conference paper at ICLR 2024
F
MORE QUALITATIVE EXAMPLES OF MULTI-VIEW MESH RECONSTRUCTION
Ground Truth
Nydiffreome DMTet
NeuS
NeuralUDF
NeUDF
NeAT
Nydiffreeme G-Sbell
Reconstructed fexture
Roconstructed mash
Figure 11: Comparison between reconstruction w/ G-SHELL and baseline methods on instance in DeepFash- ion3D datasct.
23

---

Published as a conference paper at ICLR 2024
Ground Truth
Nvdiffrceme
NeuS
NeuralUDF
NeUDP
NeAT
Nvdiffrecme
Reconsrrcceed toxture
Roconstructsd mesh
Figure 12: Comparison between reconstruction w/ G-SHELL and baseline methods multiview reconstruction on instance 117 DeepFashion3D dataset.
24

---

Published as a conference paper at ICLR 2024
Ground Truth
Nvdiflieeme w/ DMTet
Neu8
Neu
NeUDF
NeAT
Nvdiflreeme w/ G-Shell
Roconstructed texture
Reconstrccted mesh
Figure 13: Comparison between reconstruction w/ G-SHELL and baseline methods on multiview reconstruction on instance 320 in DeepFashion3D dataset.
Nydiffreeme w/ DMTet
Nvdiffiremee G-Shell
Ground Truth
NeuS
NeuralUDF
NeUDF
NeAT
Reconstrucied texture
Roconstriessd mesh
Figure 14: Comparison between reconstruction w/ G-SHELL and baseline methods on multiview reconstruction on instance 522 in DeepFashion3D dataset.
25

---

Published as a conference paper at ICLR 2024
G RECONSTRUCTED MESHES FOR SURFACES WITH COMPLEX PATTERNS
We also show the reconstruction result of a non-watertigh surface with complex surface patterns in Figure 15. The results show that G-SHELL can almost perfectly reconstruct the geometry.
Ground truth multi-view images
Reconstruted non-watertight mesh
Figure 15: Reconstruction results of watewrittt surface with complex surface patterns from multi-view imgaes.
26

---

Published as a conference paper at ICLR 2024
H RECONSTRUCTION OF SHAPES WITH METALLIC MATERIALS
We visualize the ground truth and reconstructed shape in the specular lighting setting. With the increase of metalness in the material, the reconstructed shapes start contain artifacts. Yet the reconstructed 3D meshes still replicate most of the target geometry and texture. The missing specular lighting effect in the rendered image is mostly due to the limitation of the renderer no indirect illumination is considered. More advanced differentiable rendering methods may solve this issue.
Ground Truth
Reconstrution
n
m=0.2
m=0.4
m=0.6
m=
m=
Figure Qualitative results ablation study on the change specular parameter the ground truth mesh material, m represents metalness parameter.
27

---

Published as a conference paper at ICLR 2024
I ON MSDF REGULARIZATION
might be tempting to ignore the second term the mSDF regularization loss with some well-tuned of loss coefficients. However, we empirically observe that it is hard find good choice for "neoDe-eeeopnn to achieve conservative yet effective hole opening regularization. A too large JmSDF-reg-pen typically results in large topological holes when the shape to optimize is still far from the target shape (and often leads to completely empty meshes); a too small value fails to create topological holes when needed and fails to remove meshes unobserved/occluded regions. possible to find a schedule of "MSDFF-ry rey open without /mSDF close. However, we note that the schedule synchronized with the schedule of shadow ray contribution: once the
of "omSDe-reg-open has occluded light rays hardly contribute in rendered colors, it is in general hard for the shape topology to change dramatically, A time-invariant loss coefficient of LinDDF-rgeeloss is general easier to control and more robust.
On the other hand, a too small scale for LmSDF-rg-sloe often leads to incorrect shape topology while too large scale for LesDF-eg-close may produce some artifacts. These losses correlated with different initialization strategies and influence the optimization process in coupled way.
In Fig. 17, show extreme cases when inappropriate mSDF regularization scale leads to failures. We do not include the case for very large scales of LMSDF open as nearly no meshes will be produced.
(4) Small mSDF hole-closing loss
(b) Large mSDF hole-closing loss
(c) Tiny//cco mSDF hole-opening loss
Figure 17: Extreme with inappropriate mSDF regularization scales.
28

---

Published as a conference paper at ICLR 2024
J EXPERIMENT ON REAL DATA
To demonstrate that our method works on real data, we collected a set of multiview images by shooting video with hand-held smartphone. We extract 94 frames out of the video and obtain camera poses via COLMAP [51, 52]. The binary segmentation masks are obtained by running an off-the-shelf foreground segmentation model [47] on all the images. We fit the G-SHELL representation with the same set of parameters as in the chair example and show the results in Figure 18. Even the images include inconsistent lighting (due to the occlusion by the video shooter), motion blur and some specular lighting from the piece of the paper, Nvdiffrecme inverse rendering with G-SHELL is still able reconstruct relatively reasonable shape with texture.
Input image
Input mask
Reconstructed image
Reconstructed mesh
Figure 18: Empirical results on real data collected by a hand-held smartphone.
29

---

Published as a conference paper at ICLR 2024
K VISUALIZATION OF LEARNED WATERTIGHT MESH TEMPLATES
We show the resulted watertight mesh templates for some of the DeepFashion3D instances. We note that in theory these watertight mesh templates can be arbitrary, though the inductive bias of MLP and the Eikonal loss largely regularize the watertight template to be smooth enough.
Figure 19: Visualization of the watertight mesh templates for cach instance in DeepFashion3D datasct the end
of optimization.
30


Reviews:
[{"review_id": "ehEXBawMov", "reviewer": "Reviewer_RYwJ", "summary": "This paper proposes a new representation, G-Shell, for 3D data. Existing papers mostly focus on modeling solid, watertight shapes, while the modeling of thin, open surfaces has not been widely studied. To fill this gap, this paper proposes a parameterization and develops a gruid-based method for both watertight and non-watertight meshes of arbitrary topology. Experiments show that G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks, while achieving competitive results for watertight meshes.", "questions": "Please see questions above", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The topic studied in this paper is interesting and important.\n2. Being able to model thin, open surfaces will be useful for a couple of applications.\n3. The proposed method is technically sound.\n4. Experiments show that the proposed method is effective in modeling non-watertight meshes.", "weaknesses": "1. While the proposed method is faster than other methods as shown in Table 3, 3 hours is still too long.\n2. The thin shape examples in Figure 4 and Figure 7 don't have complicated geometry. If there is a more complicated geometry, how well would the proposed method perform in reconstructing / modeling? For instance, when dropping a cloth onto an object, the cloth will have a lot of folds, wrinkles and even a lot of self-contacts. Can the proposed method deal with this case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKlh00WrIn", "reviewer": "Reviewer_hJqu", "summary": "The paper proposes a new representation for surfaces with boundary (open surfaces) geared toward inverse rendering and surface reconstruction. The representation views an open surface as a sub-level set of a function on a closed surface, which is in turn represented as a level set of a function in the ambient space. The two functions are discretized together on a background grid, and an extended marching-tetrahedra lookup table enables extraction of a mesh for the open surface. The advantages of the representation are demonstrated for reconstruction from images as well as generative modeling by a diffusion method.", "questions": "The paper shows a lookup table for a marching-tetrahedra approach for G-shells. Is there also an extended version of marching-cubes?\n\nThe \"hole-opening\" loss in eq. (1) sounds like it really promotes hole-closing. In any case, it requires more explanation. What does the parameter $\\epsilon$ do? How should one set it? The paper refers to \"topological holes\" but I assume what is meant here are boundaries, not handles.\n\nThe appendix lists many more loss terms with free parameters, and it is unclear whether all of these terms are necessary. As usual, it would be helpful to add an ablation study. Some of the descriptions of the loss terms are cryptic and could do with elaboration. E.g., how does (12) reduce \"floaters\" and \"inner geometry\"? Why do you need a second SDF regularization after imposing the eikonal regularization? Would it be helpful to include some form of eikonal regularization on $\\nu$ as well—ideally taking the gradient along the surface?\n\n## Minor quibbles:\n- p. 2: \"generation modeling\" -> \"generative modeling\"\n- p. 3: What does \"validity value\" mean?\n- p. 4: referring to a curve as a \"2D mesh\" and a surface as a \"3D mesh\" is confusing. It would be better to say \"polygonal curve\" and \"surface mesh,\" respectively, or something similar.\n- p. 5: \"pose-processing\" -> \"post-processing\"\n- 6.2: \"winding number\" should really be \"generalized winding number.\" Also, I am not sure what it means to look at the generalized winding number on the surface when it is a function in the ambient volume that has a sharp jump at the surface.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The basic idea is elegant, and it avoids the problems of unsigned distance fields. The comparisons to previous work are also compelling. The generative modeling results look cool, though it would be nice to see some examples other than clothing if other datasets are available.", "weaknesses": "The authors claim this is the first work to propose a differentiable representation suitable for both open and closed surfaces. Though they mention representations for open surfaces based on unsigned distance fields, they should also include citations to the following two works, which offer alternative approaches:\n- D. Palmer, D. Smirnov, S. Wang, A. Chern, and J. Solomon, “DeepCurrents: learning implicit representations of shapes with boundaries,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18665–18675.\n- T. V. Christiansen, J. A. Bærentzen, R. R. Paulsen, and M. R. Hannemose, “Neural Representation of Open Surfaces,” 2023, doi: 10.1111/cgf.14916.\n\nThe exposition could use some polishing. In addition to general copy-editing, some details of the method require further elaboration. Most prominently, the mesh extraction algorithm, which is a main contribution of the paper, is described only very briefly in section 4.2, and the lookup table for G-shells is only explained pictorially in figure 3. It would be helpful to readers to include at least a little more explanation of what is going on in that figure and why (even if it has to go in an appendix or supplemental material).\n\nThe description of the generative modeling approach in the paragraphs following eq. (2) is unclear. What does \"unevenly weighted prediction\" mean? What is the naïve diffusion loss, and what are you replacing it with? Why does predicting the linear interpolation coefficient instead of the value of $\\nu$ help? Is there any tradeoff in doing this? If the normalization of SDF values is an issue, would there be an advantage to using more general implicit functions instead? If you are going to extend MeshDiffusion, it would be helpful to include at least a brief summary of how that method works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAJ1BVOueK", "reviewer": "Reviewer_dsfR", "summary": "This paper extends the 3D grid representation by introducing an additional mSDF field defined on grid vertices for the modeling of open surfaces.  This new representation can be used in the reconstruction and generative modeling of open 3D surface meshes.", "questions": "1. The ability to handle specular surfaces is emphasized in the experimental results. However, the experimental setting to handle the specular surfaces is not clear enough.  Do you simultaneously reconstruct lighting, geometry and material properties from the captured images?  or you assume lighting condition is known. \n\n2. How is the performance of the proposed method when reconstructing open surfaces that are homeomorphic to donuts or other complex surfaces with many topology handles?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. A nice extension of 3D grid representation that can handle open surface meshes. \n2. Impressive experimental results on open surface reconstruction and generation.", "weaknesses": "While the experimental results are impressive, it is not clear whether the proposed modified marching cube or tetrahedra with mSDF value can guarantee the correct topology of the 3D mesh. For example, an isolated edge with no incident triangles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XK7BhTCVel", "reviewer": "Reviewer_9iXa", "summary": "This paper presents a novel mesh representation called Ghost-on-the-Shell (G-SHELL). This representation parametrizes non-watertight surfaces by defining a manifold signed distance field on watertight templates. It enables reconstruction from multiview images and generative modeling of both watertight and non-watertight meshes of arbitrary topology. The paper demonstrates that G-SHELL performs well in tasks related to non-watertight mesh reconstruction and generation while also being effective for watertight meshes.", "questions": "Regarding the experiments in mesh generation, G-SHELL is compared against other implicit  SDF based methods that can only represent watertight surfaces. Since the experiments are only conducted with a dataset of non-watertight surfaces (clothes), it is thus expected that G-SHELL is going to be better at the job than the rest of methods. How well MD w/ G-SHELL performs with watertight surfaces compared with MeshDiffusion?   \n\n\nPolyGen [40] is regarded on page 3 as ineffective for high vertex counts. Although this is probably accurate, it is worth clarifying in the paper that methods like PolyGen generate non-uniform meshes and use the vertices more efficiently than the regular grid used in G-SHELL.  Comparing vertex counts between the two methods does not seem fair. \n\n\nThe paper lacks experiments on real data, especially in reconstruction. It would be interesting to compare G-SHELL + differentiable rasterizers against the baselines based on volume rendering or even vanilla NERF + marching cubes. Do the authors plan to include such experiments in the camera-ready version?\n\n\nIt would be illustrative to visualize the watertight template being estimated jointly with the open surface in some of the experiments.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper introduces an original approach to implicitly modeling non-watertight 3D meshes. Treating open surfaces as entities floating on watertight surfaces is a novel idea with significant advantages over other methods that require unsigned distance fields (UDF). It leads to the development of a manifold signed distance field (mSDF) on the watertight template, which is a sound contribution.\n\nIn terms of quality, the methodology is well-described, and the authors offer a clear rationale for their Ghost-on-the-Shell (G-SHELL) implicit modelling approach. The paper also provides empirical evidence of the advantages of G-SHELL in tasks such as mesh reconstruction and generation, which enhances the overall quality of the work.\n\nThe paper is well-written and the contributions are cleary presented. The authors effectively communicate complex concepts.\n\nIn terms of significance, G-SHELL is a differentiable and efficient implicit representation for both watertight and non-watertight meshes which broadens its impact. Besides it has the potential to be easily adopted in computer graphics pipelines.  G-SHELL applications in reconstruction and mesh generation problems are notable. In particular, G-SHELL’s enables reconstruction methods where the topology, material and lighting are jointly optimized which is crucial for scaning 3D assets from images. \n\nOverall, this paper presents an original, high-quality, clear, and significant contribution to deal with non-watertight surfaces and demonstrating practical applicability through empirical validation.", "weaknesses": "G-SHELL inherits common disadvantages of implicit surface representation methods. Mainly, since it employs a regular grid, surfaces with high and unbalanced entropy will require many grid elements, which can be inefficient and limiting for real applications.  \n\nG-SHELL uses a marching cubes-like algorithm to extract the surface. Even though this method is highly parallelizable, it also represents a burden in computation compared to explicit methods.\n\nG-SHELL does not model self-intersecting and non-orientable surfaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)