DMesh: A Differentiable Mesh Representation

Authors: sanghyun  son
matheus  gadelha
yang  zhou
zexiang  xu
ming c lin
yi  zhou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 06:55:07

Abstract: We present a differentiable representation, DMesh, for general 3D triangular meshes. DMesh considers both the geometry and connectivity information of a mesh. In our design, we first get a set of convex tetrahedra that compactly tessellates the domain based on Weighted Delaunay Triangulation (WDT), and select triangular faces on the tetrahedra to define the final mesh. We formulate probability of faces to exist on the actual surface in a differentiable manner based on the WDT. This enables DMesh to represent meshes of various topology in a differentiable way, and allows us to reconstruct the mesh under various observations, such as point clouds and multi-view images using gradient-based optimization. We publicize the source code and supplementary material at our project page (https://sonsang.github.io/dmesh-project).

Main paper: 
 introduction: 
 Polygonal meshes are widely used in modeling and animation due to their diverse, compact and explicit configuration. Recent AI progress has spurred efforts to integrate mesh generation into machine learning, but challenges like varying topology hinder suitable differentiable mesh representations. This limitation leads to reliance on differentiable intermediates like implicit functions, and subsequent iso-surface extraction for mesh creation (Liao u. a., 2018; Guillard u. a., 2021; Munkberg u. a., 2022; Shen u. a., 2023 Shen u. a., , 2021;; Liu u. a., 2023b) . However, meshes generated by such approaches can be misaligned at sharp regions and unnecessarily dense (Shen u. a., 2023) , not suitable for down-stream applications that require light-weight meshes. This limitation necessitates us to develop a truly differentiable mesh representation, not the intermediate forms.The fundamental challenge in creating a differentiable mesh representation lies in formulating both the vertices' geometric features and their connectivity, defined as edges and faces, in a differentiable way. Given a vertex set, predicting their connectivity in a free-form way using existing machine learning data-structures can cost significant amount of computation and be difficult to avoid irregular and intersecting faces. Consequently, most studies on differentiable meshes simplify the task by using a mesh with a pre-determined topology and modifying it through various operations (Zhou u. a., 2020; Hanocka u. a., 2019; Palfinger, 2022; Nicolet u. a., 2021) . This work, on the contrary, ambitiously aims to establish a general 3D mesh representation, named as DMesh, where both mesh topology and geometric features (e.g. encoded in vertex location) can be simultaneously optimized through gradient-based techniques.Our core insight is to use differentiable Weighted Delaunay Triangulation (WDT) to divide a convex domain, akin to amber encapsulating a surface mesh, into tetrahedra to form a mesh. To create a mesh with arbitrary topology, we select only a subset of triangular faces from the tetrahedra, termed the "real part", as our final mesh. The other faces, the "imaginary part", support the real part but are not part of the final mesh (Figure 4 ). We introduce a method to assess the probability of a face being part of the mesh based on weighted points that carry positional and inclusiveness information. Optimization is then focused on the points' features to generate the triangular mesh. The probability determination allows us to compute geometric losses and rendering losses during gradient-based optimization that optimizes connectivity and positioning.The key contributions of our work can be summarized as follows.• We present a novel differentiable mesh representation, DMesh, which is versatile to accommodate various types of mesh (Figure 2 ). The generated meshes can represent shapes more effectively, with much less number of vertices and faces (Table 2 ).• We propose a computationally efficient approach to differentiable WDT, which produces robust probability estimations. While exhaustive approach (Rakotosaona u. a., 2021) requires quadratic computational cost, our method runs in approximately linear time. After optimization, one can recover meshes using iso-surface extraction techniques (Lorensen und Cline, 1998; Ju u. a., 2002) .Differing from neural representations, another class of methods directly produce meshes and optimize them. However, they assume that the overall mesh topology is fixed (Chen u. a., 2019; Nicolet u. a., 2021; Liu u. a., 2019; Laine u. a., 2020) , only allowing local connectivity changes through remeshing (Palfinger, 2022) . Learning-based approaches like BSP-Net (Chen u. a., 2020) allow topological variation, but their meshing process is not differentiable. Recently, differentiable isosurface extraction techniques have been developed, resulting in high-quality geometry reconstruction of various topologies (Liao u. a., 2018; Shen u. a., 2021 Shen u. a., , 2023;; Wei u. a., 2023; Munkberg u. a., 2022; Liu u. a., 2023b; Mehta u. a., 2022) . Unfortunately, meshes relying on iso-surface extraction algorithms (Lorensen und Cline, 1998; Ju u. a., 2002) often result in unnecessarily dense meshes that could contain geometric errors. In contrast, our approach addresses these issues: we explicitly define faces and their existence probabilities, and devise regularizations that yield simplified, but accurate meshes based on them (Table 2 ). See Table 3 for more detailed comparisons to these other methods. 
delaunay triangulation for geometry processing: 
 Delaunay Triangulation (DT) (Aurenhammer u. a., 2013) has been proven to be useful for reconstructing shapes from unorganized point sets. It's been shown that DT of dense samples on a smooth 2D curve includes the curve within its edges (Brandt und Algazi, 1992; Amenta u. a., 1998a) . This idea of using DT to approximate shape has been successfully extended to 3D, to reconstruct threedimensional shapes (Amenta u. a., 1998b) for point sets that satisfy certain constraints. However, these approaches are deterministic. Our method can be considered as a differentiable version of these approaches, which admits gradient-based optimization.More recently, Rakotosaona u. a. (2021) focused on this DT's property to connect points and tessellate the domain, and proposed a differentiable WDT algorithm to compute smooth inclusion, namely existence score of 2-simplexes (triangles) in 2 dimensional space. However, it is not suitable to apply this approach to our 3D case, as there are computational challenges (Section 3.2). Other related work, VoroMesh (Maruani u. a., 2023) , also used Voronoi diagrams in point cloud reconstruction, but their formulation cannot represent open surfaces and is only confined to handle point clouds.3 PreliminaryQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed the two limitations of our current approach in Section 6, which are about computational efficiency and manifoldness of the generated mesh.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
probabilistic approach to mesh connectivity: 
 To define a traditional, non-differentiable mesh, we specify the vertices and their connectivity. This connectivity is discrete, meaning for any given triplet of vertices, we check if they form a face in the mesh, returning 1 if they do and 0 otherwise. To overcome this discreteness, we propose a probabilistic approach to create a fully differentiable mesh -given a triplet of vertices, we evaluate the probability of a face existing. This formulation enables differentiability not only of vertex positions but also of their connectivity.Note that we need a procedure that tells us the existence probability of any given face to realize this probabilistic approach. This procedure must be 1) differentiable, 2) computationally efficient, and 3) maintain desirable mesh properties, such as avoiding non-manifoldness and self-intersections, when determining the face probabilities.Among many possible options, we use Weighted Delaunay Triangulation (WDT) (Figure 6 (a)) and a point-wise feature called the "real" value (ψ) to define our procedure. Each vertex in our framework is represented as a 5-dimensional2 vector including position (3), WDT weight (1), and real value (1) (Figure 3 (a)). Given the precomputed WDT based on vertices' positions and weights, we check the face existence probability of each possible triplets. Specifically, 1) a face F must exist in the WDT, and then 2) satisfy a condition on the real values of its vertices to exist on the actual surface. We describe the probability functions for these conditions as Λ wdt and Λ real :EQUATIONThen we get the final existence probability function, which can be used in downstream applications (Figure 3 ), as follows:Λ(F ) = P (F ∈ Mesh) = Λ wdt (F ) • Λ real (F ).(2)This formulation attains one nice property in determining the final mesh -that is, it prohibits self-intersections between faces. When it comes to the other two criteria about this procedure, Λ wdt function's differentiability and efficiency is crucial, as we design Λ real to be a very efficient differentiable function based on real values (Section 4.2). Thus, we first introduce how we can evaluate Λ wdt in a differentiable and efficient manner, which is one of our main contributions. 4 ) and unreliable estimation when (k < d).We will discuss later how our formulation conquer these computational challenges. Our setting for 3D meshes are similar to 2D meshes, the (d = 2, k = 1) case in Figure 5 (d), where a triangular face reduces to a line. Therefore, we will mainly use this setting to describe and visualize basic concepts for simplicity. However, note that it can be generalized to any (d, k) case without much problem. For given ∆ 1 , reduced power cell of p 1 for the ∆ 1 (R p1|∆ 1 ) is rendered in blue, with the original power cell (grey). We can evaluate the existence of ∆ 1 in WDT by computing the signed distance from D ∆ 1 to R p1|∆ 1 .We generalize the basic principles suggested by Rakotosaona u. a. ( 2021) to address our cases. For formal definitions of the concepts in this section, please refer to Appendix B.Let S be a finite set of points in R d=2 with weights. For a given point p ∈ S, we denote its weight as w p . We call those weighted points in S as "vertices", to distinguish them from general unweighted points in R 2 . Then, we adopt power distance π(p, q) = d(p, q) 2 -w p -w q as the distance measure between two vertices in S. As depicted in Figure 6 , ∆ k=1 is a 1-simplex, which is a line connecting two vertices p i and p j in S. Its dual form D ∆ 1 (red line) is the set of unweighted points3 in R 2 that are located at the same power distance to p i and p j . In the power diagram, the power cell C p (gray cell) of a vertex p is the set of unweighted points that are closer to p than to any other vertices in S. We can use C p and D ∆ 1 to measure the existence of ∆ 1 . From Figure 6 , we can see that when ∆ 1 = {p i , p j } exists in WDT, its dual line D ∆ 1 aligns exactly with C pi 's boundary, while when∆ 1 = {p i , p j } doesn't exist in WDT, D ∆ 1 is outside C pi .To make this measurement less binary when ∆ 1 exists, we use the expanded version of power cell called "reduced" power cell (R p|∆ ), introduced by Rakotosaona u. a. (2021). The reduced power cell of p i ∈ S for ∆ 1 = {p i , p j } is computed by excluding p j from S when constructing the power cell4 . For example, when ∆ 1 exists, R pi|∆ 1 will expand towards p j 's direction (Figure 6 (c)), and D ∆ 1 will "go through" R pi|∆ 1 . In contrast, when ∆ doesn't exists, even though we have removed p j , R pi|∆ 1 will not expand (Figure 6 (d)), and thus D ∆ 1 stays outside of R pi|∆ 1 . Now we can define a signed distance field τ pt (x, R) for a given reduced power cell R, where the signed distance is measured as the distance from the point x ∈ R d to the boundary of the reduced power cell (sign is positive when inside). Then, we can induce the signed distance between a dual form D and a reduced power cell R:EQUATIONAs illustrate in Figure 6 (c) and (d), τ (D ∆ 1 , R p1|∆ 1 ) is positive when ∆ 1 exists, while negative when it does not exist in WDT. This observation can be generalized as follows:Remark 3.1. ∆ k exists in WDT if and only if ∀p i ∈ ∆ k , τ (D ∆ k , R pi|∆ k ) > 0.In fact, the sign of every τ (D ∆ k , R pi|∆ k ) is same for every p i ∈ ∆ k . Therefore, we can use its average to measure the existence probability of ∆ k , along with sigmoid function σ:EQUATIONwhere α wdt is a constant value used for the sigmoid function. Λ wdt (∆ k ) is greater than 0.5 when ∆ k exists, aligning with our probabilistic viewpoint and being differentiable.Computational Challenges. As mentioned before, Rakotosaona et al. (2021) solved the problem for the case where (d = k = 2) where the dual D ∆ k is a single point. Naïvely applying their approach for computing Eq. 3 to our cases poses two computational challenges:• Precision:When (d = 3, k = 2) or (d = 2, k = 1), D ∆ k becomes a line, not a single point.Finding a point on the line that maximizes Eq. 3 is not straightforward. • Efficiency: When naïvely estimating the reduced power cell in exhaustive manner, the computational cost increases with the number of points at a rate of O(N 2 ), where N is the number of points.See Appendix B.2 for a detailed discussion of these limitations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [No] Justification: Even though we tried our best to describe every theoretical detail in main paper, especially in Section 3.2, and Appendix B, our method assumes that the reader has some background knowledge about the geoemtrical concepts. However, we specified the sources that the readers can refer to learn details about the theoretical claims we made in the paper.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
formulation: 
 4.1 Practical approach to compute Λ wdtWe introduce a practical approach to resolve the two aforementioned challenges. Specifically, we propose constructing the PD first and use it for computing lower bound of Eq. 3 in an efficient way.We also propose to handle the two cases separately: whether ∆ k exists in the WDT or not.First, when the simplex ∆ k does not exist, we choose to use the negative distance between the dual form and the normal power cell, -d(D ∆ k , C p ). This is the lower bound of Eq. 3: 6 (d) for this case in (d = 2, k = 1) case. We can observe that computing this distance is computationally efficient, because the normal PD only needs to be computed once for all in advance. Moreover, C p is a convex polyhedron, and D ∆ k is a (convex) line, which allows us to find the distance between line segments on the boundary of C p5 and D ∆ k , and choose the minimum.-d(D ∆ k , C p ) ≤ τ (D ∆ k , R p|∆ k ) < 0, (5) as C p ⊂ R p|∆ k . See FigureSecond, we analyze the case when the simplex ∆ k exists in WDT. In this case, we have to construct the reduced power cell R p|∆ k for given ∆ k , which requires much additional cost. Instead of doing it, we leverage pre-computed PD to approximate the reduced power cell. Then, we pick a point v ∈ D ∆ k ∩ R p|∆ k , where p ∈ {∆ k }, and the following holds:EQUATIONbecause v ∈ R p|∆ k and by the definition at Eq. 3. In our case, since D ∆ k ∩ R p|∆ k is a line segment, we choose its middle point as v to tighten this lower bound. See Figure 6 (c) for the line segment in (d = 2, k = 1) case. We use this bound when ∆ k exists. Note that computing this lower bound is also computationally efficient, because we can simply project v to the reduced power cell.To sum up, we can rewrite Eq. 3 as follows.EQUATIONBy using this relaxation, we can get lower bound of Eq. 3, which is reliable because it always has the same sign. Also, we can reduce the computational cost from O(N 2 ) to nearly O(N ), which is prerequisite for representing meshes that have more than 1K vertices in general. See Appendix B.2 for the computational speed and accuracy of our method, compared to the previous one. Finally, we implemented our algorithm for computing Eq. 7 in CUDA for further acceleration.Λ real evaluates the existence probability of a k-simplex ∆ k in our mesh when it exists in WDT.To define it, we leverage per-point value ψ ∈ [0, 1]. To be specific, we compute the minimum ψ of the points in ∆ k in differentiable way: q) , and Ψ is function that maps a point p to its ψ value. We set β = 100.Λ real (∆ k ) = p∈∆ k κ(p) • Ψ(p), where κ(p) = e -β•Ψ(p) / q ∈∆ k e -β•Ψ(Along with Λ wdt that we discussed before, now we can evaluate the final existence probability of faces in Eq. 2. We also note here, that when we extract the final mesh, we only select the faces of which Λ wdt and Λ real are larger than 0.5.DMesh can be reconstructed from various inputs, such as normal meshes, point clouds, and multiview images. With its per-vertex features and per-face existence probabilities Λ(F ), we can optimize it with various reconstruction losses and regularization terms. Please see details in Appendix C.First, if we have a normal mesh with vertices P and faces F, and we want to represent it with DMesh, we should compute the additional two per-vertex attributes, WDT weights and real values. We optimize them by maximizing Λ(F) since these faces lies on the reference mesh. Conversely, for the remaining set of faces F that can be defined on P, we should minimize Λ( F). Together, they define the reconstruction loss for mesh input (Appendix C.1).For reconstruction from point clouds or multi-view images, we need to optimize for all features including positions. For point clouds, we define our loss using Chamfer Distance (CD) and compute the expected CD using our face probabilities (Appendix C.2). For multi-view images, we define the loss as the L 1 loss between the given images and the rendering of DMesh, interpreting face probabilities as face opacities. We implemented efficient differentiable renderers to allow gradients to flow across face opacities (Appendix C.3). Being fully differentiable for both vertices and faces, DMesh allows us to develop various regularizations to improve the optimization process and enhance the final mesh quality. The first is weight regularization (L weight ), applied to the dual Power Diagram of the WDT (Appendix C.4). This regularization reduces the structural complexity of the WDT and discards unnecessary points, controlling the final mesh complexity (Figure 7 ). The next is real regularization (L real ), which enforces nearby points to have similar real values and increases the real values of points adjacent to high real value points (Appendix C.5). This helps remove holes or inner structures and makes faces near the current surface more likely to be considered (Appendix D). The final regularization, quality regularization (L qual ), aims to improve the quality of triangle faces by minimizing the average expected aspect ratio of the faces, thus removing thin triangles (Appendix C.6).To sum up, our final loss function can be written as follows:L = L recon + λ weight • L weight + λ real • L real + λ qual • L qual ,where λ values are hyperparameters. In Appendix E, we provide values for these hyperparameters for every experiment. Also, in Appendix E.3, we present ablation studies for these regularizations. In this section, we provide experimental results to demonstrate the efficacy of our approach. First, we optimize vertex attributes to restore a given ground truth mesh, directly proving the differentiability of our design. Next, we conduct experiments on 3D reconstruction from point clouds and multi-view images, showcasing how our differentiable formulation can be used in downstream applications.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: In Appendix D and E, we provided the pseudocode of our optimization process, and detailed hyperparameters to reproduce the experimental results. Also, we included code in the supplementary material. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Justification: For the nature of our optimization based experiments, random initialization has little affect in the results. In addition, our experiments exam the effect of resolution and topology varieties, it does not use large amount of testing data. Thus, error bars for showing statistical significance of the experiments are not suitable for our paper. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments and applications: 
 For the mesh reconstruction problem, we used three models from the Stanford 3D Scanning Repository (Curless und Levoy, 1996) . For point cloud and multi-view reconstruction tasks, we used four closed-surface models from the Thingi32 dataset, four open-surface models from the DeepFashion3D dataset, and three additional models with both closed and open surfaces from the Objaverse dataset and Adobe Stock. These models are categorized as "closed," "open," and "mixed" in this section. Additionally, we use nonconvex polyhedra of various Euler characteristics and non-orientable geometries to prove our method's versatility.We implemented our main algorithm for computing face existence probabilites and differentiable renderer used for multi-view image reconstruction in CUDA (Nickolls u. a., 2008) . Since we need to compute WDT before running the CUDA algorithm, we used WDT implementation of CGAL (Jamin u. a., 2023) . We implemented the rest of logic with Pytorch (Paszke u. a., 2017) . All of the experiments were run on a system with AMD EPYC 7R32 CPU and Nvidia A10 GPU.In this experiment, we demonstrate that we can preserve most of the faces in the original normal triangular mesh after converting it to DMesh using the mesh reconstruction loss introduced in 4.3.In Table 1 , we show the recovery ratio (RE) and false positive ratio (FP) of faces in our reconstructed mesh. Note that we could recover over 99% of faces in the original mesh, while only having under 1% of false faces. Please see Appendix E.1 for more details. This result successfully validates our differentiable formulation, but also reveals its limitation in reconstructing some abnormal triangles in the original mesh, such as long, thin triangles.In this experiment, we aim to reconstruct a mesh from partial geometric data, such as (oriented) point clouds or multi-view images. For point cloud reconstruction, we sampled 100K points from the ground truth mesh. We can additionally use point orientations, if they are available. For multi-view reconstruction, we rendered diffuse and depth images of the ground truth mesh from 64 view points. In Appendix E, we illustrated the example inputs for these experiments. Also, please see Appendix D to see the initialization and densification strategy we took in these experiments.To validate our approach, we compare our results with various approaches. When it comes to point cloud reconstruction, we first compare our result with classical Screened Poisson Surface Reconstruction (PSR) method (Kazhdan und Hoppe, 2013)6 . Then, to compare our method with optimization based approach, we use recent VoroMesh (Maruani u. a., 2023) method. Note that these two methods are not tailored for open surfaces. To compare our method also for the open surfaces, we use Neural Dual Contouring (NDC) (Chen u. a., 2022b) , even though it is learning-based approach.Finally, for multi-view reconstruction task, we compare our results with Flexicube (Shen u. a., 2023) and Neural Implicit Evolution (NIE) (Mehta u. a., 2022) , which correspond to volumetric approaches that can directly produce meshes of varying geometric topology for given visual inputs.In Figure 8 , we visualize the reconstruction results along with the ground truth mesh for qualitative evaluation. For closed meshes, in general, volumetric approaches like PSR, VoroMesh, and Flexicube, capture fine details better than our methods. This is mainly because we currently have limitation in the mesh resolution that we can produce with our method. NIE, which is also based on volumetric principles, generates overly smoothed reconstruction results. However, when it comes to open or mixed mesh models, which are more ubiquitous in real applications, we can observe that these methods fail, usually with false internal structures or self-intersecting faces (Appendix E.2). Since NDC leverages unsigned information, it can handle these cases without much problem as ours. However, we can observe step-like visual artifacts coming from its usage of grid in the final output, which requires post-processing. Additionally, to show the versatility of our representation, we also visualize various shapes reconstructed from oriented point clouds in Figure 2 .Table 2 presents quantitative comparisons with other methods. We used following metrics from Chen u. a. (2022b) to measure reconstruction accuracy: Chamfer Distance (CD), F-Score (F1), Normal Consistency (NC), Edge Chamfer Distance (ECD), and Edge F-Score (EF1) to the ground truth mesh. Also, we report number of vertices and faces of the reconstructed mesh to compare mesh complexity, along with computational time. All values are average over 11 models that we used. In general, our method generates mesh of comparable, or better accuracy than the other methods. However, when it comes to ECD and EF1, which evaluate the edge quality of the reconstructed mesh, our results showed some weaknesses, because our method cannot prevent non-manifold edges yet. However, our method showed superior results in terms of mesh complexity -this is partially due to the use of weight regularization. Please see Appendix E.3 to see how the regularization works through ablation studies. Likewise, our method shows promising results in producing compact and accurate mesh. However, we also note that our method requires more computational cost than the other methods in the current implementation.Before moving on, we present an experimental result about shape interpolation using DMesh in Figure 9 . We used multi-view images to reconstruct a torus first, and then optimized the DMesh again The computational cost rises sharply beyond 20K points, with most of the time spent on WDT construction ("WDT"), while the probability computation ("Prob") requires significantly less time.to fit a double torus. The results show that DMesh effectively reconstructs the double torus, even when initialized from a converged single torus, highlighting the method's robustness to local minima. However, this also indicates that our representation lacks meaningful shape interpolation, as it does not assume any specific shape topology. 
limitations: 
 As shown above, our method achieves a more effective and complete forms of differentiable meshes of various topology than existing methods, but still has several limitations to overcome.Computational Cost. Currently, the resolution of DMesh is limited by computational cost. Although our theoretical relaxation and CUDA implementation reduce this burden, processing meshes with over 100K vertices remains challenging due to the computational bottleneck of constructing the WDT at each optimization step. In Figure 10 , we analyze computational costs relative to the number of points. As shown, costs rise sharply beyond 20K points, with WDT construction consuming most of the time. This limits our method's ability to handle high resolution mesh.Non-Manifoldness. As we have claimed so far, DMesh shows much better generalization than the other methods as it does not have any constraints on the shape topology and mesh connectivity. However, due to this relaxation of constraint, we can observe spurious non-manifold errors in the mesh, even though we adopted measures to minimize them (Appendix D.2.7).Specifically, an edge must have at most two adjacent faces to be a "manifold" edge. Similarly, a "manifold" vertex should be adjacent to a set of faces that form a closed or open fan. We refer to edges or vertices that do not satisfy these definitions as "non-manifold." In our results, we found that 5.50% of edges and 0.38% of vertices were non-manifold for point cloud reconstruction. For multi-view reconstruction, 6.62% of edges and 0.25% of vertices were non-manifold. Therefore, we conclude that non-manifold edges are more prevalent than non-manifold vertices in our approach.To address the computational cost issue, we can explore methods that reduce reliance on the WDT algorithm, as its cost increases significantly with the number of points. This is crucial since representing complex shapes with fine details often requires over 100K vertices. To tackle the non-manifold issue, we could integrate approaches based on (un)signed distance fields (Shen u. a., 2023; Liu u. a., 2023b) into our method, ensuring manifold mesh generation. Finally, future research could extend this work to solve other challenging problems, such as 3D reconstruction from real-world images, or applications like generative models for 3D shapes. This could involve encoding color or texture information within our framework, opening up exciting new directions for exploration.Table 3 : Traits of different optimization-based shape reconstruction methods. We compare methods based on template mesh (Palfinger, 2022; Nicolet u. a., 2021) , neural SDF (Wang u. a., 2021 (Wang u. a., , 2023)) , neural UDF (Long u. a., 2023; Liu u. a., 2023a) , differentiable isosurface extraction techniques (Shen u. a., 2021; Munkberg u. a., 2022; Shen u. a., 2023; Liu u. a., 2023b) O O O O X X O Neural SDF O X X O O X O Neural UDF O O X O O X △ Diff. Isosurface O △ O O O X O DMesh (Ours) O O O O O O XHere we provide conceptual comparisons between our approach and the other optimization-based 3D reconstruction algorithms, which use different shape representations. To be specific, we compared our method with mesh optimization methods starting from template mesh (Palfinger, 2022; Nicolet u. a., 2021) , methods based on neural signed distance fields (SDF) (Wang u. a., 2021 (Wang u. a., , 2023)) , methods based on neural unsigned distance fields (UDF) (Liu u. a., 2023a; Long u. a., 2023) , and methods based on differentiable isosurface extraction (Shen u. a., 2021; Munkberg u. a., 2022; Shen u. a., 2023; Liu u. a., 2023b) . We used following criteria to compare these methods.• Closed surface: Whether or not the given method can reconstruct, or represent closed surfaces.• Open surface: Whether or not the given method can reconstruct, or represent open surfaces.• Differentiable Meshing: Whether or not the given method can produce gradients from the loss computed on the final mesh.• Differentiable Rendering: Whether or not the given method can produce gradients from the loss computed on the rendering results.• Geometric topology: Whether or not the given method can change geometric topology of the shape. Here, geometric topology defines the continuous deformation of Euclidean subspaces (Lee, 2010). For instance, genus of the shape is one of the traits that describe geometric topology.• Mesh topology: Whether or note the given method can produce gradients from the loss computed on the mesh topology, which denotes the structural configuration, or edge connectivity of a mesh.• Manifoldness: Whether or not the given method guarantees manifold mesh.In Table 3 , we present a comparative analysis of different methods. Note that our method meets all criteria, only except manifoldness. It is partially because our method does not assume volume, which is the same for methods based on neural UDF. However, because our method does not leverage smoothness prior of neural network like those methods, it could exhibit high frequency noises in the final mesh. Because of this reason, we gave △ to the neural UDF methods, while giving X to our approach. When it comes to methods based on differentiable isosurface extraction algorithms, we gave △ to its ability to handle open surfaces, because of (Liu u. a., 2023b) . They can represent open surfaces as subset of closed ones, but cannot handle non-orientable open surfaces. Finally, note that our method is currently the only method that can handle mesh topology.Likewise, DMesh shows promise in addressing the shortcomings found in previous research. Nonetheless, it has its own set of limitations (Section 6). Identifying and addressing these limitations is crucial for unlocking the full potential of our method.B Details about Section 3.2Here, we provide formal mathematical definitions of the terms used in Section 3.2. We mainly use notations from Aurenhammer u. a. ( 2013) and Cheng u. a. ( 2013).Generalizing the notations in Section 3.2, let S[w] be a finite set of weighted points in R d , where w is a weight assignment that maps each point p ∈ S to its weight w p . We denote a weighted point as p[w p ] and define power distance to measure distance between two weighted points. Definition B.1 (Power distance). Power distance between two weighted points p[w p ] and q[w q ] is measured as: EQUATIONEQUATION)Note that some points may have empty power cells if their weights are relatively smaller than neighboring points. We call them "submerged" points. As we will see later, weight regularization aims at submerging unnecessary points in mesh definition, which leads to mesh simplification.To construct the power cell, we can use the concept of half space. A half space H p<q is the set of unweighted points in R d that are closer to p[w p ] than q[w q ]. Definition B.3 (Half space). Half space H p<q is defined as:EQUATIONNote that we can construct a power cell by intersecting half spaces, which proves the convexity of the power cell. Now we call H(p, q) as a half plane that divides R d into two half spaces, H p<q and H q<p . Definition B.4 (Half plane). Half plane H p,q is defined as:EQUATIONThen, for a given k-simplex ∆ k comprised of weighted pointsEQUATIONwhich is equivalent to:EQUATION)Note than when k = d, D ∆ k becomes a point, while it becomes a line when k = d -1. As discussed in Section 3.2, we leverage the distance between this dual form D ∆ k and reduced power cell (Rakotosaona u. a., 2021) of the points in D ∆ k to query the existence of ∆ k . The reduced power cell R p|∆ k is a power cell of p that does not concern the other points in ∆ k in its construction. Definition B.6 (Reduced power cell). Reduced power cell of a weighted point p[w] ∈ S[w] for given ∆ k is defined as:EQUATION)Using these concepts, we can measure the existence probability of a k-simplex, as provided in Section 3.2.As mentioned in Section 3.2, the previous approach of Rakotosaona u. a. ( 2021) has two computational challenges in computational efficiency and precision. These limitations are mainly rooted in not knowing the power diagram structure before evaluating Eq. 3. We summarize the overall procedure of the previous approach, and point out how our method is different from it.Collecting simplexes to evaluate First, we need to collect simplexes that we want to compute probabilities for. Here we assume the number of simplexes increases linearly (O(N )) as the number of points (N ) increases. This is a plausible assumption, because we often search for k-nearest neighbors for each point, and combine them to generate the query simplexes.Sampling a point on the dual forms of the simplexes The previous method relies on point projections to evaluate Eq. 3. This did not incur any problem for their case (d = k = 2), because the dual form was a single point. However, when (k < d) as in our cases, the dual form contains infinite number of points, which makes unclear how to apply this point-based approach. One possible solution is sampling the most "representative" point on the dual form, and leveraging the point to estimate Eq. 3. By definition, this estimation is lower bound of Eq. 3. However, the problem arises when this sample point does not give reliable result. For instance, we can consider the case shown in Figure 6(c ).In the illustration, we can observe that ∆ 1 exists in WDT, and thusD ∆ 1 goes thorugh R p1|D ∆ 1 . If we sample a point on D ∆ 1 that is included in R p1|D ∆ 1, the signed distance from the sample point would have same sign as Eq. 3. However, if the sample point is selected outside R p1|D ∆ 1 , the sign would be different from the real value. In this case, even if ∆ k exists, we can recognize it as not existing. Note that this false estimation produces false gradients, which could undermine optimization process.In contrast, we do not have to concern about this precision issue, because we construct PD, which tells us good sample points that give reliable lower bounds of Eq. 3, when the given simplex exists in WDT. Otherwise, we explicitly compute minimum distance between the dual form and the reduced power cell, as discussed in Section 4.1.Projecting sample points to reduced power cells The final step is point projection, where we project the sample points from dual forms to the reduced power cells to estimate Eq. 3. Based on the definitions in Appendix B, we can observe that a (reduced) power cell's boundaries are comprised of half planes. That is, the boundaries of C p is comprised of multiple half planes between p and the other weighted points. However, when we do not know which half planes comprise the boundaries, we have to do exhaustive search to find the signed distance from the sample point to the boundaries of the reduced power cell. As the number of half planes that are associated with a point p is N , the computational cost to precisely compute the signed distance is O(N ).Note that it does not hold for our case, because by constructing WDT and PD, we know which half planes form the boundaries of each power cell. Also, note that even when the number of points increases, the average number of half planes that comprise the boundaries of power cells remains constant. Therefore, in our case, this step requires only O(1) computational cost.Summary To sum up, the computational cost of the previous approach amounts to O(N 2 ), as the number of simplexes to evaluate increases linearly, and the cost for the projection step also increases linearly as the number of points increase. However, the cost for ours remains at O(N ). Moreover, the previous approach does not guarantee satisfactory estimations of Eq. 3.Before moving on, we point out that the original implementation limited the number of half planes to consider in evaluating Eq. 3 to reduce the computational cost to O(N ). This relaxation is permissible to the case where the precision is not very important. However, the precision is important in our case, because we aim at representing mesh accurately.To prove the aforementioned theoretical claim, we conducted experiments to measure the computational speed and accuracy of the probability estimation for (d = 2, k = 2) and (d = 2, k = 1) cases, for varying number of points. We randomly sampled points in a unit cube uniformly, and set the Table 4 : Comparison of computational speed and accuracy between the previous method (Rakotosaona u. a., 2021) and ours. For each number of points in 2D, we report the triplet of (computational speed (ms) / false positive ratio (%) / false negative ratio (%)), along with the number of query simplexes that we give to the algorithm. weights by random sampling from a normal distribution: N (0, 10 -3 ). For fair comparison, we did not use CUDA implementation that we mainly use in the paper. We implemented both algorithms in PyTorch, and ran 10 times for each setting to get fair values. In each experiment, we fed the query simplexes into the algorithm, and computed their existence probabilities. If the computed probability for a simplex was over 0.5, but did not exist, it is counted to false positive ratio. In contrast, if the computed probability for a simplex was below 0.5, but did exist, it is counted to false negative ratio.We measured the computational accuracy with these metrics.In Table 4 , we can see that as the number of points increases, the computational cost increases exponentially for the previous approach, while ours remain fairly stable up to 30K points. When the number of points reached 100K, the previous method failed because of excessive memory consumption. However, when the number of points is smaller than 1K, the previous method ran faster than ours, because we need to construct PD even for the small number of points, which consumes most of the time for all these cases.In terms of accuracy, we can observe that the previous method and ours both give accurate estimations when (d = 2, k = 1). However, when (d = 2, k = 1), we can see that the false negative ratio is almost 50% -as we set the number of existing simplexes and non-existing simplexes in the query set as the same, it means that the previous method chose wrong sample points for most of the query simplexes, so that it predicted most of them as not existing. To select the most representative point, we found the intersection point between the dual form and the affine space created by the weighted points in the simplex. This could be another interesting direction to explore, but this approach found out to be not very accurate in this experiment. In contrast, our approach gave stable estimations for all the cases. This result demonstrates that our method is more scalable, and gives reliable probability estimations than the previous approach, which is necessary for accurate 3D mesh representation.Here we provide formal definitions for the loss functions that we use in the paper.In this section, we explore the loss function used to transform the ground truth mesh into our DMesh representation. As previously mentioned in Section 4.3, the explicit definition of ground truth connectivity in the provided mesh allows us to establish a loss function based on it.Building on the explanation in Section 4.3, if the ground truth mesh consists of vertices P and faces F, we can construct an additional set of faces F. These faces are formed from vertices in P but do not intersect with faces in F. F = F * -F, where F * = every possible face combination on P.Then, we notice that we should maximize the existence probabilities of faces in F, but minimize those of faces in F. Therefore, we can define our reconstruction loss function asEQUATIONIf the first term of the loss function mentioned above is not fully optimized, it could lead to the omission of ground truth faces, resulting in a poorer recovery ratio (Section 5.1). Conversely, if the second term is not fully optimized, the resulting DMesh might include faces absent in the ground truth mesh, leading to a higher false positive ratio (Section 5.1). Refer to Appendix D.1 for details on how this reconstruction loss is integrated into the overall optimization process.In the task of point cloud reconstruction, we reconstruct the mesh by minimizing the (L 1 -norm based) expected Chamfer Distance (CD) between the given point cloud (P gt ) and the sample points (P ours ) from our reconstructed mesh. We denote the CD from P gt to P ours as CD gt , and the CD from P ours to P gt as CD ours . The final reconstruction loss is obtained by combining these two distances.EQUATIONTo compute these terms, we start by sampling P ours from our current mesh. First, we sample a set of faces that we will sample points from. We consider the areas of the triangular faces and their existence probabilities. To be specific, we define η(F ) for a face F asη(F ) = Λ(F ), η(F ) = F area • η(F ),and define a probability to sample F from the entires faces F asP sample (F ) = η(F ) F ′ ∈F η(F ′ ).We sample N faces from F with replacement and then uniformly sample a single point from each selected face to define P ours . In our experiments, we set N to 100K.In this formulation, we sample more points from faces with a larger area and higher existence probability to improve sampling efficiency. However, we observed that despite these measures, the sampling efficiency remains low, leading to slow convergence. This issue arises because, during optimization, there is an excessive number of faces with very low existence probability.To overcome this limitation, we decided to do stratified sampling based on point-wise real values and cull out faces with very low existence probabilities. To be specific, we define two different η functions:η1 (F ) = Λ wdt (F ) • min(ψ i , ψ j , ψ k ), η 1 (F ) = F area • η1 (F ) η2 (F ) = Λ wdt (F ) • max(ψ i , ψ j , ψ k ), η 2 (F ) = F area • η2 (F )where (ψ i , ψ j , ψ k ) are the real values of the points that comprise F . Note that η 1 is the same as η7 .For the faces in F, we first calculate the η1 and η2 values and eliminate faces with values lower than a predefined threshold ϵ η . We denote the set of remaining faces as F 1 and F 2 . Subsequently, we sample N 2 faces from F 1 and the other N 2 faces from F 2 , using the following two sampling probabilities:P sample,1 (F ) = η 1 (F ) F ′ ∈F1 η 1 (F ′ ) , P sample,2 (F ) = η 2 (F ) F ′ ∈F2 η 2 (F ′ ).The rationale behind this sampling strategy is to prioritize (non-existing) faces closer to the current mesh over those further away. In the original η = η 1 function, we focus solely on the minimum real value, leading to a higher sampling rate for existing faces. However, to remove holes in the current mesh, it's beneficial to sample more points from potential faces-those not yet existing but connected to existing ones. This approach, using η 2 , enhances reconstruction results by removing holes more effectively. Yet, there's substantial potential to refine this importance sampling technique, as we haven't conducted a theoretical analysis in this study.Moreover, when sampling a point from a face, we record the face's existence probability alongside the point. Additionally, if necessary, we obtain and store the face's normal. For a point p ∈ P ours , we introduce functions Λ pt (•) and N ormal(•) to retrieve the face existence probability and normal, respectively:EQUATIONp) = the face where p was sampled from.Now we introduce how we compute the CD gt , which is CD from P gt to P ours . For each point p ∈ P gt , we first find k-nearest neighbors of p in P ours , which we denote as (p 1 , p 2 , ..., p k ).Then, we define a distance function between the point p and the k-nearest neighbors as follows, to accommodate the orientation information:D(p, p i ) = ||p -p i || 2 + λ normal • Dn (p, p i ),where Dn (p,EQUATIONwhere λ normal is a parameter than determines the importance of point orientation in reconstruction.If λ normal = 0, we only consider the positional information of the sampled points.After we evaluate the above distance function values for the k-nearest points, we reorder them in ascending order. Then, we compute the following expected minimum distance from p to P ours ,D(p, P ours ) = i=1,...,k D(p, p i ) • P (p i ) • P (p i ), P (p i ) = Λ pt (p i ) • I prev (F (p i ), P (p i ) = Π i=1,...,k-1 (1 -P (p i )),where I prev is an indicator function that returns 1 only when the given face has not appeared before in computing the above expected distance. For instance, if the face ids for the reordered points were (1, 2, 3, 2, 3, 4), the I prev function evaluates to (1, 1, 1, 0, 0, 1). This indicator function is needed, because if we select p i as the nearest point to p with the probability Λ pt (p), it means that we interpret that the face corresponding to p i already exists, and then we would select p i on the face as the nearest point to p rather than the other points that were sampled from the same face, but have larger distance than p i and thus come after p i in the ordered points.Note that we dynamically change k during runtime to get a reliable estimation of D(p, P ours ). That is, for current k, if most of P (p k )s for the points in P gt are still large, it means that there is a chance that the estimation could change a lot if we find and consider more neighboring points. Therefore, in our experiments, if any point in P gt has P (p k ) larger than 10 -4 , we increase k by 1 for the next iteration. However, if there is no such point, we decrease k by 1 to accelerate the optimization process.Finally, we can compute CD gt by summing up the point-wise expected minimum distances.CD gt = p∈Pgt D(p, P ours ).In computing CD ours , which is CD from P ours to P gt , we also find k-nearest neighbors for each point p ∈ P ours , which we denote as (p 1 , p 2 , ..., p k ). Then, for a point p, we use the same distance function D in Eq. 17 to find the distance between p and (p 1 , p 2 , ..., p k ). After that, we select the minimum one for each point, multiply the existence probability of each point, and then sum them up to compute CD ours . Finally, we can compute the final reconstruction loss for point clouds as shown in Eq. 16.When we are given multi-view images, we reconstruct the mesh by minimizing the L 1 difference between our rendered images and the given images. In this work, we mainly use both diffuse and depth renderings to reconstruct the mesh.If we denote the (N img ) ground truth images of N pixel number of pixels as I gt i (i = 1, ..., N img ), and our rendered images as I ours i , we can write the reconstruction loss function asL recon = 1 N img • N pixel i=1,...,Nimg ||I gt i -I ours i ||.Then, we can define our rendered image as follows:I ours i = F(P, F, Λ(F), MV i , P i ).where F is a differentiable renderer that renders the scene for the given points P, faces F, face existence probabilities Λ(F), i-th modelview matrix MV i ∈ R 4×4 , and i-th projection matrix P i ∈ R 4×4 . The differentiable renderer F has to backpropagate gradients along P, F, and Λ(F) to update our point attributes. Specifically, here we interpret Λ(F) as opacity for faces to use in the rendering process. This is because opacity means the probability that a ray stops when it hits the face, which aligns with our face existence probability well. For this reason, we ignore faces with very low existence probability under some threshold to accelerate the reconstruction, as they are almost transparent and do not contribute to the rendering a lot.To implement F, we looked through previous works dedicated for differentiable rendering (Laine u. a., 2020; Liu u. a., 2019). However, we discovered that these methods incur substantial computational costs when rendering a large number of (potentially) semi-transparent triangles, as is the case in our scenario. Consequently, we developed two efficient, partially differentiable renderers that meet our specific requirements. These renderers fulfill distinct roles within our pipeline-as detailed in Appendix D, our optimization process encompasses two phases within a single epoch. The first renderer is employed during the initial phase, while the second renderer is utilized in the subsequent phase. If there are multiple semi-transparent faces in the scene, we have to sort the faces that covers a target pixel with their (view-space) depth values, and iterate through them until the accumulated transmittance is saturated to determine the color for the pixel. Conducting this process for each individual pixel is not only costly, but also requires a lot of memory to store information for backward pass.Recently, 3D Gaussian Splatting (Kerbl u. a., 2023) overcame this issue with tile-based rasterizer. We adopted this approach, and modified their implementation to render triangular faces, instead of gaussian splats. To briefly introduce its pipeline, it first assigns face-wise depth value by computing the view-space depth of its center point. Then, after subdividing the entire screen into 16 × 16 tiles, we assign faces to each tiles if they overlap. After that, by using the combination of tile ID and the face-wise depth as a key, we get the face list sorted by depth value in each tile. Finally, for each tile, we iterate through the sorted faces and determine color and depth for each pixel as follows.C = i=1,...,k T i • α i • C i , (T i = Π j=1,...,i-1 (1 -α j )),where T i is the accumulated transmittance, α i is the opacity of the i-th face, and C i is the color (or depth) of the i-th face. Note that α i = Λ(F i ), as mentioned above. Even though this renderer admits an efficient rendering of large number of semi-transparent faces, there are still two large limitations in the current implementation. First, the current implementation does not produce visibility-related gradients (near face edges) to update point attributes. Therefore, we argue that this renderer is partially differentiable, rather than fully differentiable. Next, since it does not compute precise view-point depth for each pixel, its rendering result can be misleading for some cases, as pointed out in (Kerbl u. a., 2023).To amend the first issue, we opt to use another differentiable renderer of Laine u. a. ( 2020), which produces the visibility-related gradients that we lack. Since this renderer cannot render (large number of) transparent faces as ours does, we only render the faces with opacity larger than 0.5. Also, we set the faces to be fully opaque. If we call this renderer as F A ′ , our final rendered image can be written as follows.I ours i = 1 2 (F A (P, F, Λ(F), MV i , P i ) + F A ′ (P, F, Λ(F), MV i , P i )).In Figure 12 , we illustrate rendered images from F A and F A ′ .Acknowledging that this formulation is not theoretically correct, we believe that it is an intriguing future work to implement a fully differentiable renderer that works for our case. However, we empirically found out that we can reconstruct a wide variety of meshes with current formulation without much difficulty.As mentioned before, this renderer is used at the first phase of the optimization process, where all of the point attributes are updated. However, in the second phase, we fix the point positions and weights, and only update point-wise real values (Appendix D.2). In this case, we can leverage the tessellation structure to implement an efficient differentiable renderer. As the second renderer does a precise depth testing unlike the first one, it can be used to modify the errors incurred by the second limitation of the first renderer (Figure 13 ).The second renderer performs precise depth ordering in an efficient way, based on the fixed tessellation structure that we have. In Figure 11 , we illustrate a 2D diagram that explains our approach. When the green ray, which corresponds to a single ray to determine the color of a single pixel, goes through the tessellation, we can observe that it goes through a sequence of triangles (tetrahedron in 3D), which are denoted as T 1 , T 2 , and T 3 . When the ray enters a triangle T i through one of its three edges, we can see that it moves onto the other adjacent triangle T i+1 only through one of the other edges of T i , because of compact tessellation. Therefore, when the ray hits one edge of T i , it can only examine the other two edges of T i to find the next edge it hits. Note that we do not have to do depth testing explicitly in this approach. Also, unlike the first approach, this renderer does not have to store all the possible faces that a ray collides for the backward pass, because it can iterate the same process in the opposite way in the backward pass to find the edge that it hit before the last edge. If we only store the last edge that each hits at the forward pass, we can start from the last edge and find the previous edges that it hit to compute gradients. Therefore, this second renderer requires much less memory than the first one, and also performs precise depth testing naturally. However, note that this renderer is also partilly differentiable, because it cannot update point positions and weights.To sum up, we implemented two partially differentiable renderers to solve multi-view reconstruction problem with DMesh. They serve different objectives in our reconstruction process, and we empirically found out that they are powerful enough to reconstruct target meshes in our experiments. However, we expect that we can simplify the process and improve its stability, if we can implement a fully differentiable renderer that satisfy our needs. We leave it as a future work.Weight regularization aims at reducing the complexity of WDT, which supports our mesh. By using this regularization, we can discard unnecessary points that do not contribute to representing our mesh. Moreover, we can reduce the number of points on the mesh, if they are redundant, which ends up in the mesh simplification effect (Appendix E.3).We formulate the complexity of WDT as the sum of edge lengths in its dual power diagram. Formally, we can write the regularization as follows,L weight = i=1,...,N Length(E i ),where E i are the edges in the dual power diagram, and N is the number of edges.Real regularization is a regularization that is used for maintaining the real values of the connected points in WDT as similar as possible. Also, we leverage this regularization to make real values of points that are connected to the points with high real values to become higher, so that they can be considered in reconstruction more often than the points that are not connected to those points. To be specific, note that we ignore faces with very low existence probability in the reconstruction process. By using this regularization, it can remove holes more effectively.This real regularzation can be described asL real = 1 i=1,...,N Λ(F i ) i=1,...,N Λ(F i ) • (σ 1 (F i ) + σ 2 (F i )), σ 1 (F i ) = 1 3 j=1,2,3 |ψ j - (ψ 1 + ψ 2 + ψ 3 ) 3 |, σ 1 (F i ) = 1 3 j=1,2,3 |1 -ψ j | • I( max j=1,2,3 (ψ j ) > δ high ).Here ψ 1,2,3 represent the real values of points that comprise F i , and δ high is a threshold to determine "high" real value, which is set as 0.8 in our experiments. Note that the faces with higher existence probabilities are prioritized over the others.After reconstruction, we usually want to have a mesh that is comprised of triangles of good quality, rather than ill-formed triangles. We adopt the aspect ratio as a quality measure for the triangular faces, and minimize the sum of aspect ratios for all faces during optimization to get a mesh of good quality. Therefore, we can write the regularization as follows.L qual = 1 i=1,...,N Λ(F i ) i=1,...,N AR(F i ) • E max (F i ) • Λ(F i ), AR(F i ) = E max (F i ) H min (F i ) • √ 3 2 , E max (F i ) = Maximum edge length of F i , H min (F i ) = Minimum height of F i .Note that we prioritize faces with larger maximum edge length and higher existence probability than the others in this formulation. In Appendix E.3, we provide ablation studies for this regularization.In this section, we explain the optimization processes, or exact reconstruction algorithms, in detail. First, we discuss the optimization process for the experiment in Section 5.1, where we represent the ground truth mesh with DMesh. Then, we discuss the overall optimization process for point cloud or multi-view reconstruction tasks in Section 5.2, from initialization to post processing.Our overall algorithm to convert the ground truth mesh into DMesh is outlined in Algorithm 1. We explain each step in detail below.At the start of optimization, we initialize the point positions (P), weights (W), and real values (ψ) using the given ground truth information (P gt , F gt ). To be specific, we initialize the point attributes as follows.P = P gt , W = [1, ..., 1], ψ = [1, ..., 1].The length of vector W and ψ is equal to the number of points. In Figure 14 , we illustrate the initialized DMesh using these point attributes, which becomes the convex hull of the ground truth mesh.Algorithm 1 Mesh to DMesh Note that during optimization, we allow only small perturbations to the positions of initial points, and fix weights and real values of them to 1. This is because we already know that these points correspond to the ground truth mesh vertices, and thus should be included in the final mesh without much positional difference. In our experiments, we set the perturbation bound as 1% of the model size.However, we notice that we cannot restore the mesh connectivity with only small perturbations to the initial point positions, if there are no additional points that can aid the process. Therefore, we periodically perform point insertion to add additional points, which is described below.The point insertion is a subroutine to add additional points to the current point configurations. It is performed periodically, at every fixed step. The additional points are placed at the random place on the faces in F, which correspond to the faces that should not exist in the final mesh. Therefore, these additional points can aid removing these undesirable faces.However, we found out that inserting a point for every face in F can be quite expensive. Therefore, we use k-means clustering algorithm to aggregate them into 0.1 • N F clusters, where N F is the number of faces in F, to add the centroids of the clusters to our running point set. On top of that, we select 1000 random faces in F to put additional points directly on them. This is because there are cases where centroids are not placed on the good positions where they can remove the undesirable faces.In Figure 14 , we render DMesh after point insertion to the initialized mesh. Note that some of the undesirable faces disappear because of the added points. In this problem, we minimize the reconstruction loss specified in Eq. 15 to restore the connectivity in the ground truth mesh, and remove faces that do not exist in it. In the formulation, we denoted the faces that are comprised of mesh vertices P, but are not included in the original mesh as F. Even though we can enumerate all of them, the total number of faces in F mounts to O(N 3 ), where N is the number of mesh vertices. Therefore, rather than evaluating all of those cases, we maintain a set of faces F that we should exclude in our mesh during optimization.To be specific, at each iteration, we find faces in the current WDT that are comprised of points in P, but do not exist in F, and add them to the running set of faces F. On top of that, at every pre-defined number of iterations, in our case 10 steps, we compute k-nearest neighboring points for each point in P. Then, we find faces that can be generated by combining each point with 2 of its k-nearest points, following Rakotosaona u. a. ( 2021). Then, we add the face combinations that do not belong to F to F.In our experiments, we set k = 8.In Algorithm 2, we describe the overall algorithm that is used for point cloud and multi-view reconstruction tasks. We explain each step in detail below.We divide each optimization epoch in two phases. In the first phase (phase 1), we optimize all of the point attributes -positions, weights, and real values. However, in the second phase (phase 2), we fix the point positions and weights, and only optimize the real values. This design aims at removing ambiguity in our differentiable formulation. That is, even though we desire face existence probabilities to converge to either 0 and 1, those probabilities can converge to the values in between. To alleviate this ambiguity, after the first phase ends, we fix the tessellation to make Λ wdt for each face in F to either 0 or 1. Therefore, in the second phase, we only care about the faces that exist in current W DT , which have Λ wdt value of 1. Then, we can only care about real values.Note that the two differentiable renderers that we introduced in Appendix C.3 are designed to serve for these two phases, respectively.In this work, we propose two point initialization methods. The first initialization method can be used when we have sample points near the target geometry in hand.This initialization method is based on an observation that the vertices of Voronoi diagram of a point set tend to lie on the medial axis of the target geometry (Amenta u. a., 1998a,b) . Therefore, for the given sample point set P sample , we first build Voronoi diagram of it, and find Voronoi vertices P voronoi . Then, we merge them to initialize our point set P:P = P sample ∪ P voronoi ,all of which weights are initialized to 1. Then, we set the real values (ψ) of points in P sample as 1, while setting those of points in P voronoi as 0.In Figure 15 , we render the mesh that we can get from this initialization method, when we use 10K sample points. Note that the initial mesh has a lot of holes, because there could be Voronoi vertices that are located near the mesh surface, as pointed out by (Amenta u. a., 1998b) . However, we can converge to the target mesh faster than the initialization method that we discuss below, because most of the points that we need are already located near the target geometry.If there is no sample point that we can use to initialize our points, we initialize our points with N 3 points regularly distributed on a grid structure that encompasses the domain, all of which has weight 1 and ψ value of 1. We set N = 20 for every experiment (Figure 16a ). Then, we optimize the mesh to retrieve a coarse form of the target geometry (Figure 16b ). Note that we need to refine this mesh in the subsequent epochs, as explained below. Until now, we introduced two point initialization techniques. When the input is a point cloud, we sample subset of the point cloud to initialize our mesh (Figure 15 ). However, when the input is multi-view images, we start from initialization without sample points (Figure 16 ), because there is no sample point cloud that we can make use of.We maintain the running set of faces to evaluate probability existence for in F. At each iteration, after we get W DT , we insert every face in W DT to F, as it has a high possibility to persist in the subsequent optimization steps. Also, as we did int mesh to DMesh conversion (Appendix D.1), at every 10 optimization step, we find k-nearest neighbors for each point, and form face combinations based on them. Then, we add them to F.At start of each epoch, if it is not the first epoch, we refine our mesh by increasing the number of points. To elaborate, we refine our mesh by sampling N number of points on the current DMesh, and then initialize point attributes using those sample points as we explained above. We increase N as number of epoch increases. For instance, in our multi-view reconstruction experiments, we set the number of epochs as 4, and set N = (1K, 3K, 10K) for the epochs excluding the first one.In Figure 16 , we render the initial and the last state of DMesh of each epoch. Note that the mesh complexity increases and becomes more accurate as epoch proceeds, because we use more points. Therefore, this approach can be regarded as a coarse-to-fine approach.When it comes to multi-view reconstruction, we found out that it is helpful to add one more constraint in defining the face existence. In our formulation, in general, a face F has two tetrahedra (T 1 , T 2 ) that are adjacent to each other over the face. Then, we call the remaining point of T 1 and T 2 that is not included in F as P 1 and P 2 . Our new constraint requires at least one of P 1 and P 2 to have ψ value of 0 to let F exist.This additional constraint was inspired by the fact that F is not visible from outside if F exists in our original formulation, and both of P 1 and P 2 have ψ value of 1. That is, if it is not visible from outside, we do not recognize its existence. This constraint was also adopted to accommodate our real regularization, which increases the real value of points near surface. If this regularization makes the real value of points inside the closed surface, they would end up in internal faces that are invisible from outside. Because of this invisibility, our loss function cannot generate a signal to remove them.In the end, we can expect all of the faces inside a closed surface will exist, because of the absence of signal to remove them. Therefore, we choose to remove those internal faces by applying this new constraint in the post-processing step.Note that this discussion is based on the assumption that our renderer does a precise depth testing. If it does not do the accurate depth testing, internal faces can be regarded as visible from outside, and thus get false gradient signal. In Figure 13a , the final mesh after phase 1 is rendered, and we can see therer are lots of internal faces as the renderer used in phase 1 does not support precise depth testing. However, we can remove them with the other renderer in phase 2, as shown in Figure 13b , which justifies our implementation of two different renderers.Finally, we note that this constraint is not necessary for point cloud reconstruction, because if we minimize CD ours in Appendix C.2, the internal faces will be removed automatically.In this section, we provide experimental details for the results in Section 5, and visual renderings of the our reconstructed mesh. Additionally, we provide the results of ablation studies about regularizations that we suggested in Section 4.3.As shown in Table 1 , we reconstruct the ground truth connectivity of Bunny, Dragon, and Buddha model from Stanford dataset (Curless und Levoy, 1996) . For all these experiments, we optimized for 20K steps, and used an ADAM optimizer (Kingma und Ba, 2014) with learning rate of 10 -4 . For Bunny model, we inserted additional points at every 5000 step. For the other models, we inserted them at every 2000 step.In Figure 17 , we provide the ground truth mesh and our reconstructed mesh. We can observe that most of the connectivity is preserved in our reconstruction, as suggested numerically in Table 1 . However, note that the appearance of the reconstructed mesh can be slightly different from the ground truth mesh, because we allow 1% of positional perturbations to the mesh vertices. • Regularization: λ weight = 10 -8 , λ real = 10 -3 , λ qual = 10 -3 for every mesh.• Number of epochs: Single epoch for every mesh.• Number of steps per epoch: 1000 steps for phase 1, 500 steps for phase 2 for every mesh.• Optimizer: ADAM Optimizer, Learning rate = 10 -3 in the first epoch, and 3 • 10 -4 in the other epochs for every mesh.• Weight Regularization: λ weight = 10 -8 for every mesh.• Real Regularization: λ real = 10 -3 for the first 100 steps in every epoch for open surface meshes and one mixed surface mesh (Plant) / 10 -2 for the first 100 steps in every epoch for closed surface meshes and two mixed surface meshes (Bigvegas, Raspberry).• Quality Regularization: λ qual = 10 -3 for every mesh.• Normal Coefficient: λ normal = 0 for every mesh (Eq. 17).• Number of epochs: 4 epochs for every mesh. In the first epoch, use 20 -3 regularly distributed points for initialization. In the subsequent epochs, sample 1K, 3K, and 10K points from the current mesh for initialization. • Number of steps per epoch: 500 steps for phase 1, 500 steps for phase 2 for every mesh.• Batch size: 64 for open surface meshes, 16 for the other meshes.In Figure 22 , 23, and 24, we provide visual renderings of our point cloud and multi-view reconstruction results with ground truth mesh. We also provide illustration of input point cloud and diffuse map. Note that we also used depth renderings for multi-view reconstruction experiments.Generally, we can observe that reconstruction results from both point cloud and multi-view images capture the overall topology well. However, we noticed that the multi-view reconstruction results are not as good as point cloud reconstruction results. In particular, we can observe small holes in the multi-view reconstruction results. We assume that these artifacts are coming from relatively weaker supervision of multi-view images than dense point clouds. Also, we believe that we can improve these multi-view reconstruction results with more advanced differentiable renderer, and better mesh refinement strategy. In the current implementation, we lose connectivity information at the start of each epoch, which is undesirable. We believe that we can improve this approach by inserting points near the regions of interest, rather than resampling over entire mesh. Also, regarding comparison to Flexicube (Shen u. a., 2023) in Table 2 , we tried to found out the reason why ours give better results than Flexicube in terms of CD to the ground truth mesh for closed surfaces in thingi32 dataset. We could observe that Flexicube's reconstruction results capture fine geometric details on the surface mesh, but also observed that they have lots of false internal structure (Figure 18 ). Note that this observation not only applies to closed surfaces, but also to open surfaces, where it generates lots of false, self-intersecting faces (Figure 19 ). Our results do not suffer from these problems, as we do post-processing (Appendix D.2) to remove inner structure, and also our method can represent open surfaces better than the volumetric approaches without self-intersecting faces.In this section, we provide ablation studies for the regularizations that we proposed in Section 4.3. We tested the effect of the regularizations on the point cloud reconstruction task.We tested the influence of weight regularzation in the final mesh, by choosing λ weight in (10 -6 , 10 -5 , 10 -4 ). Note that we set the other experimental settings as same as described in Section E.2, except λ quality , which is set as 0, to exclude it from optimization.In Table 5 , we provide the quantitative results for the experiments. For different λ weight , we reconstructed mesh from point clouds, and computed average Chamfer Distance (CD) and average number of faces across every test data. We can observe that there exists a clear tradeoff between CD and mesh complexity. To be specific, when λ weight = 10 -6 , the CD is not very different from the results in Table 2 , where we use λ weight = 10 -8 . However, when it increases to 10 -5 and 10 -4 , we can observe that the mesh complexity (in terms of number of faces) decreases, but CD increases quickly. The renderings in Figure 20 support these quantitative results. When λ weight = 10 -6 , we can observe good reconstruction quality. When λ weight = 10 -5 , there are small artifacts in the reconstruction, but we can get meshes of generally good quality with fewer number of faces. However, when it becomes 10 -4 , the reconstruction results deteriorate, making holes and bumpy faces on the smooth surface. Therefore, we can conclude that weight regularization contributes to reducing the mesh complexity. However, we need to choose λ weight carefully, so that it does not harm the reconstruction quality. The experimental results tell us setting λ weight to 10 -6 could be a good choice to balance between these two contradictory objectives.As we did in the previous section, we test the influence of quality regularization in the final mesh by selecting λ real among (10 -4 , 10 -3 , 10 -2 ). We also set the other experimental settings as same as before, except λ weight = 0. In Table 6 and Figure 21 , we present quantitative and qualitative comparisons between the reconstruction results.We provide statistics about average CD, average number of faces, and average aspect ratio of faces. Interestingly, unlike weight regularization, we could not observe tradeoff between CD and aspect ratio. Rather than that, we could find that CD decreases as aspect ratio gets smaller, and thus the triangle quality gets better.We find the reason for this phenomenon in the increase of smaller, good quality triangle faces. Note that there is no significant difference between the number of faces between λ qual = 10 -4 and 10 -3 . Also, we cannot find big difference between visual renderings between them, even though the aspect ratio was clearly improved. However, when λ qual becomes 10 -2 , the number of faces increase fast, which can be observed in the renderings, too. We believe that this increase stems from our quality constraint, because it has to generate more triangles to represent the same area, if there is less degree of freedom to change the triangle shape. Since it has more triangle faces, we assume that they contribute to capturing fine details better, leading to the improved CD.However, at the same time, note that the number of holes increase as we increase λ qual , which lead to visual artifacts. We assume that there are not enough points to remove these holes, by generating quality triangle faces that meet our needs. Therefore, as discussed before, if we can find a systematic way to prevent holes, or come up with a better optimization scheme to remove them, we expect that we would be able to get accurate mesh comprised of better quality triangles. Justification: In abstract and introduction, we described that we are presenting a differentiable mesh, and we are going to explore various aspects of it (e.g. computational cost, reconstruction task, regularization). They are accurately discussed across the entire paper, including Appendix.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provided the machine that we used for running experiments in Section 5, and reported execution time in the experimental results. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: Our research conforms to the every guideline in the Code of Ethics. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our research is mainly about geometry, especially about triangle mesh formulation. So we believe it does not possess societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our research does not deliver any trained model, but suggest a method for formulating differentiable mesh.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [No] https://sonsang.github.io/dmesh-project 38th Conference on Neural Information Processing Systems (NeurIPS2024).In 2D case, a vertex is a 4-dimensional vector including position (2), WDT weight (1), and real value (1).We treat unweighted points' weight as 0 when computing the power distance.In 3D case where k = 2 and ∆ 2 = {pi, pj, p k }, pj and p k would be ignored for R p i |∆ 2 .This holds when d = 3. When d = 2, we can use vertices of of Cp.We provide point orientations for PSR, which is optional for our method.We do not use differentiable min operator, as we do not require differentiability in the sampling process. 
 
 Appendix:  


Decision: This paper introduces a differentiable representation for meshes and efficient algorithms supporting it, allowing surface reconstruction. The paper has received favorable reviews from five expert reviewers, who generally appreciated the idea, efficiency (including a CUDA implementation), good presentation and writing, extensive evaluations (including appendix the paper is 42 pages long ...), extensive comparisons, discussed limitations, etc.

Some weaknesses were raised (manifoldness, remaining computational cost), which did not impact the suitability for publication.
The AC concurs and judges the paper to be strong and ready for publication.

Reviews: review: paper_summary: paper_summary: The authors first propose a differentiable mesh representation, DMesh, which can represent a wider variety of mesh types. They then introduce a computationally efficient approach to differentiable weighted Delaunay triangulation that can run in approximately linear time. Following this, an efficient algorithm is provided for reconstructing surfaces from both point clouds and multi-view images, with DMesh used as an intermediate representation. Finally, an effective regularization term is proposed for mesh simplification and enhancing triangle quality.
strength_weakness: strength_weakness: In this work, the inner structure is removed via a depth test, meaning that any reconstruction method producing an inner structure can be improved by using the depth test to remove the unseen parts of the mesh. While this post-processing step does help to improve the quality of the result, it is not a feature of the proposed method itself.
questions: questions: 1. During the post-processing step, the inner structures are removed via depth testing. How does the method handle the boundaries caused by the previous operation?

2. In line 91, "... and it is not." should be completed for clarity. Maybe the authors missed some words at the end.

3. The DMesh is constructed from the normal mesh. However, in real applications we often do not have access to the mesh. How does the DMesh help in reconstructing the result?
limitations: limitations: Yes.
paper_summary: paper_summary: The work introduces a differentiable mesh representation where both the topology of the surface and the connectivity are differentiable. The paper builds on previous work titled 'Differentiable Surface Triangulation,' which suggests using a soft relaxation of Weighted Delaunay Triangulation (WDT) to differentiate through meshing and vertex positions. DMesh highlights that the original work cannot be used as a differentiable mesh representation in a straightforward way and would be very inefficient due to the quadratic scaling of the time complexity with the number of vertices. In light of this, the paper suggests approximating the weights in Delaunay Triangulation using a tight lower bound, which significantly accelerates the algorithm, making it practically linear. Additionally, the work introduces another per-vertex parameter that defines the probability of existence of each face that includes this vertex (given that it exists in WDT). Extensive validation shows that DMesh is capable of reconstructing complicated meshes, including open surfaces and non-orientable surfaces. Furthermore, DMesh outperforms the baselines on the tasks of surface reconstruction from a point cloud and multi-view rendering, while also being a more flexible and general representation.
strength_weakness: strength_weakness: I am a bit confused why the paper states that the running time of “Differentiable Surface Triangulation” growth exponentially with the number of vertices, while in fact it is quadratic? I would like to see either an explanation or a corrected version by the rebuttal.

While the method is very well evaluated and ablated, I think a few additional results will be interesting to see. Here they are in the order of decreasing significance:

1.	The biggest question to me so far is if the interpolations between two DMeshes are possible, especially if there is a topology/connectivity change. All the presented experiments showcase an optimization process that can start in a good initialization of the DMesh. My concern is that, after convergence the DMesh should get somehow more binary in $\Lambda_{\text{wdt}}$ and $\Lambda_{\text{real}}$, what will make it hard to take it out of this local minima in the further optimization process. By the rebuttal I would like to see a simple demonstration of a mesh-to-mesh optimization with a surface topology change, or a discussion of this in the limitation section if this is not trivial.

2.	In the limitations it is mentioned that sometimes the resulting mesh is non-manifold. It would be interesting to see some examples of failures and an estimation how often it happens and why? (at least I did not find this in the paper).

3.	(Minor) The point cloud reconstruction results are demonstrated for 100k sampled points, which is very dense. It would be interesting to see how the quality of mesh reconstruction depends on the number of sampled points.

4.	(Minor) Table 2 is evaluated only on 11 meshes, which is not much. Would be great to see a bigger test set by the camera-ready deadline.

*I would be ready to even further increase my rating if these questions are addressed.*

Minor Notes and Typos:

- Line 91: “it is not ..”  seems to miss something
- Would be great to increase the resolution of illustrations. In particular, the pixelation is very visible on Figure 6., at least in my PDF viewer
- Footnote 3 on page 5 – seems to be a typo around “and p_j and p_k”
questions: questions: Here I have a few questions that don't affect the rating, but are rather out of curiosity. Including them in the final manuscript is up to the authors:
1. It looks like the representation allows the user to freeze the connectivity of the mesh if they want to. Is it also possible to "freeze" surface topology ?
2. Did you have to prevent the optimization process from converging to binary values of the probability ? (which I believe will make the mesh effectively non-differentiable ?)
3. is there anything we can say about the existence and uniqueness of the additional two parameters ($\Lambda_{\text{wdt}}$ and $\Lambda_{\text{real}}$) when reconstructing a mesh with the reconstruction loss from section 4.3.1. ?

After the authors addressed the questions in the rebuttal, increasing my rating to strong accept.
limitations: limitations: The limitations are discussed in great detail, apart from the lack of examples of non-manifold meshes as mentioned in Weaknesses. The broader impact discussion does not seem to be necessary here.
paper_summary: paper_summary: The paper presents a differentiable mesh representation for mesh reconstruction from input meshes, point clouds, and for multi-view reconstruction. It builds off ideas of Rakotosaona 21, and uses a weighted Delaunay triangulation framework to predict simplex probabilities. They borrow the notion of distance to reduced power cell, but consider lower bound estimations and precomputation to speed it up. Their method is the first to give a differentiable mesh representation that is flexible enough to handle open/closed meshes, and accommodate changes in mesh topology and connectivity. The method is tested against several competing SOTA works and performs better or comparably in all scenarios (save for producing exactly manifold output).
strength_weakness: strength_weakness: 1. The method does not guarantee manifoldness of the end mesh result, which is acknowledged by the authors.
2. The matter of computational cost is swept under the rug a bit. It would be nice to present detailed statistics of some sort (over meshes/point clouds of various sizes, etc.) in the main paper. 
3. It is unclear to me how much technical novelty is contained in the estimation step of 4.1 over the prior framework. The steps seem relatively natural and easy to consider once you realize that the dual simplex at hand is no longer a simple point.
4. The earlier parts of the article could be compacted to include at least an overview of the technical pipeline used in the experiments. There are a lot of details that are omitted entirely in each use scenario, especially for the multi-view reconstruction. 
5. There is a small, confusing technical mistake above equation (3). The signed distance function should be positive inside, and negative when outside for this to make sense. This is as done in Rakotosaona.
questions: questions: 1. As noted I had some technical questions about S4.1:
    * For Eq. (5), can you give an instance when this lower bound is not sharp? I thought that if the simplex does not exist in the WDT, then the power cell would remained unchanged under deletion of the other two points?
    * How is it that one approximates the reduced power cell? This is not explained at all, unless I missed something? 
2. Many of the mins,maxes, absolute values, lengths, etc. that appear in the regularization terms appear to be non-differentiable. Are these simply smoothed, or am I missing something? 
3. What does the optimization do when there is not a shot showing a particular side of the model? E.g. from the bottom of a model? 
4. Why were "reality" weights vertex-based? Was there any consideration for edge-based weights? 
5. How often does non-manifoldness occur?
limitations: limitations: The authors appropriately acknowledged their limitations, some of which I listed in Weaknesses above.
paper_summary: paper_summary: This paper proposes a novel differentiable mesh representation, which focuses on both the vertex position and connectivity. To achieve this, authors come from the probability of face existence, and decompose the probability into two parts: 1) the WDT probability; 2) a probability of vertices' existence on surface. Further, to accelerate the WDT process, authors adopt a novel relaxation operation, making the whole design applicable.
strength_weakness: strength_weakness: -: Although the method is efficient compared to traditional approaches, the computational cost is still significant, especially when dealing with large datasets. This prevents the application beyond small object level task.

-: While the method is tested on synthetic datasets and controlled environments, there is a lack of demonstration on real-world datasets, or texture-rich dataset. Applying DMesh to more complex scenarios would strengthen the paper's contributions.

-: No ablation on the influence of \lambda_WDT and \lambda_real.
questions: questions: I think this is a good work and I believe further improvements as mentioned in the weakness are not easy to complete during the rebuttal. I suggest the author to add experiment on ablation of the influence of \lambda_WDT and \lambda_real to make the core contribution more convincing.
limitations: limitations: Yes, authors discussed limitations.
paper_summary: paper_summary: This paper proposed a novel differentiable representation of a mesh. It takes advantage of two probabilities, $\Lambda_{wdt}$ and $\Lambda_{real}$, to generate the mesh, such that the mesh is differentiable.
strength_weakness: strength_weakness: The efficiency of the method seems some what slow due to the WDT construction.

In the experiments, it is better to show the deformation process of the a mesh with different topologies.
questions: questions: As there are spurious non-manifold errors in the mesh, what are the ratio of the non-manifold errors?

Is the method robust against noises and holes? Current experiments did not test point clouds with noises or holes.
limitations: limitations: The limitations should discuss the potential limits to process indoor/outdoor scenes. These data are more challenge due to noises, holes and complicated topology.

Comments: title: Thank you
About global response
comment: Thank you again for your acknowledgement!
Thank you for your responses. I choose to keep my rating.
We're happy to hear that our response solved the questions, thank you for the raise!
Thanks for the authors' response with details, it solved my questions and I'd like to change my rating to weak accept.
Thank you for your acknowledgement again, we really appreciate it.

Authors
Thank you for your acknowledgement again, we really appreciate it.

Authors
Thank you for the extensive responses. I will be keeping my score as is entering the discussion phase.

Reviewer
Thank you for your responses. I choose to keep my rating.
Thank you very much, we agree that we can consider such balancing techniques!

Also, we will definitely add the additional experimental results in the paper, thank you for the suggestion.

It would be really big help for us if you could give a raise to our rating, we will really appreciate it :)
Thank you for the fast reply. This is very interesting. It will be interesting to look if stopgradient or just rescaling the gradients will help to define the balance between $\Lambda_\text{wdt}$ and $\Lambda_\text{real}$.

I think the mesh-to-mesh example will be a good addition to support the claims of the paper, so please consider including this in the final version.

At this point, I don't have any more questions and will gladly adjust my recommendation.
Thank you for the great questions, it inspires us a lot. To answer your additional questions,

---

1. We have not embedded any specific behavior to prefer $\Lambda_{real}$ over $\Lambda_{wdt}$, or manually optimize $\Lambda_{real}$ instead of $\Lambda_{wdt}$ in the optimization process. Technically, $\Lambda_{real} = 1$ and $\Lambda_{wdt} = 1$ would get the same amount of gradient, because they are just multiplied together to get the final face existence probability (Eq. 2). However, note that $\Lambda_{wdt}$ includes sigmoid function in its formulation (Eq. 4). Therefore, the parameters that are fed into the sigmoid function could get very small gradient when $\Lambda_{wdt}$ is almost binary. In contrast, $\Lambda_{real}$ does not have sigmoid function in its formulation, which means that we do not have to worry about vanishing gradient problem for $\Lambda_{real}$. In this case, the optimization process would "naturally" optimize $\Lambda_{real}$ instead of $\Lambda_{wdt}$, because $\Lambda_{wdt}$ would not be very optimizable. So we intended to say that even when $\Lambda_{wdt}$ is binary, optimizer would be able to adjust $\Lambda_{real}$ to escape the binary state.

---

2. As described above, if $\Lambda_{wdt}$ is almost binary, optimizer would automatically adjust $\Lambda_{real}$ instead of $\Lambda_{wdt}$. And as far as we understand, your concern is about only manipulating $\Lambda_{real}$ instead of $\Lambda_{wdt}$, even when we could get better optimization result when we change $\Lambda_{wdt}$ instead to get better tessellation. Correct us if we are wrong, but if our understanding is correct, our current approach does not care about this possibility yet. Our current algorithm minimizes the loss function in blind manner right now, regardless of specific $\Lambda_{real}$ and $\Lambda_{wdt}$. Therefore, we would have to say it finds the local minimum near current tessellation, instead of finding the global minimum near the optimal tessellation. However, we believe that the coarse-to-fine strategy in Appendix D.2 is a very good method to re-initialize $\Lambda_{wdt}$, thus tessellation, near the global optimal tessellation. Therefore, we'd like to suggest to use that method to find the global minimum. 

---

3. Yes, as you mentioned, the optimization loss was defined with multi-view inverse rendering. The reason about the triangle size is related to the answers above. Because $\Lambda_{real}$ is usually easier to optimize than $\Lambda_{wdt}$, it is optimized faster. That is, while the entire tessellation does not change a lot, only the faces that we select change fast. 

To elaborate, at the end of the first optimization to fit DMesh to a single torus, most of the points are gathered around the torus surface. Since there are not so many points located at the places far away from the torus surface (as we remove unnecessary points using regularization), the faces in that region become larger than those located near the torus surface. Then, we start from that DMesh to fit different shapes like double torus, and in that process, rather than moving (single) torus surface toward the target shape, the optimizer just removes the undesirable parts on the torus surface by adjusting $\Lambda_{real}$. And then, it makes the probability of desirable faces larger, and they usually have larger faces, because they were located far away from the original single torus. After that, $\Lambda_{wdt}$ is optimized slowly to fine-tune the shape. 

After this process, if we re-initialize DMesh using the coarse-to-fine strategy, then it would get much better initial tessellation to start from. Then, it would end up in much smaller, fine-grained faces in the end.

We believe it would be very interesting to explore if we can impose any kind of guidance to select (or prefer) between $\Lambda_{wdt}$ and $\Lambda_{real}$ during the optimization. Thank you again for the inspiration.
Thank you for the detailed response.

For Q8:
1. when you say "we can simply adjust the real values stored in the 3 vertices", do you mean that the optimization process naturally has gradients that minimize $\Lambda_\text{real}$ or that you need to embed this behavior separately? 
2. In the case of the probability of face existing = 1, are there any cases when we would prefer the optimization process to decrease $\Lambda_\text{wdt}$ instead? I am curious to understand if the over-parametrization caused by having two parameters ($\Lambda_\text{real}$ and $\Lambda_\text{wdt}$) might cause any kind of dead-locks in the optimization process. As you mentioned in Q9, for a fixed tesselation there is a global minimizer in terms of $\Lambda_\text{real}$, but I believe that to efficiently optimize from one surface to another we will need to change tessellation first and only then optimize $\Lambda_\text{wdt}$.
3. Thank you for providing the mesh-to-mesh optimization experiment. This is very interesting. Do I understand correctly that the optimization loss was defined through multi-view inverse rendering? I noticed that the tessellation has changed and that in both cases the meshes seem to get bigger triangle size. Do you have any intuition why this happens?
Oh, it is visible now. Thank you very much!
It seems to me that your global answer is already visible to reviewers.

AC
Sorry for your inconvenience, but we just found out that the global response is not visible to the reviewers.

We will make it visible to reviewers as soon as possible.

--------- update

It is visible now, thank you.