Shape of my heart: Cardiac models through learned signed distance functions

Authors: jan  verhülsdonk
thomas  grandits
francisco  sahli costabal
thomas  pinetz
rolf  krause
angelo  auricchio
gundolf  haase
simone  pezzuto
alexander  effland
Venue: MIDL_io_2024_Conference
Published: 2024-01-17 16:50:42

Abstract: The efficient construction of anatomical models is one of the major challenges of patientspecific in-silico models of the human heart. Current methods frequently rely on linear statistical models, allowing no advanced topological changes, or requiring medical image segmentation followed by a meshing pipeline, which strongly depends on image resolution, quality, and modality. These approaches are therefore limited in their transferability to other imaging domains. In this work, the cardiac shape is reconstructed by means of threedimensional deep signed distance functions with Lipschitz regularity. For this purpose, the shapes of cardiac MRI reconstructions are learned to model the spatial relation of multiple chambers. We demonstrate that this approach is also capable of reconstructing anatomical models from partial data, such as point clouds from a single ventricle, or modalities different from the trained MRI, such as the electroanatomical mapping (EAM).

Main paper: 
 introduction: 
 Modern personalized precision medicine frequently targets patient-specific therapies with improved therapy outcomes, reduced intervention times, and thus lower costs. In the case of cardiac personalized treatment, this necessitates complex simulation models from patient 
related work: 
 Surface reconstruction from point clouds is an intensively researched topic and provides a variety of possible approaches (Ma et al., 2022) . Classical approaches assume no prior knowledge of the given point clouds, but in turn, require dense point clouds and are often sensitive to noise (Carr et al., 2001; Kazhdan et al., 2006; Kazhdan and Hoppe, 2013; Ummenhofer and Brox, 2015; Fuhrmann and Goesele, 2014) . Modern deep-learning approaches try to overcome these limitations by learning prior information about shapes using learned SDFs (Park et al., 2019; Hanocka et al., 2020) , occupancy fields (Mescheder et al., 2019) , level sets (Michalkiewicz et al., 2019) or implicit fields (Chen and Zhang, 2019) .In this context, the DeepSDF method is one of the most active fields of research. Such methods usually try to infer a multitude of shapes from at least partially dense point clouds. However, this method is prone to overfitting the data, thus often requiring regularization both in the latent space, as well as in the neural network by imposing dropout rules (Park et al., 2019) . In Liu et al. (2022) , Lipschitz-regularized linear layers are proposed, which are able to overcome some of these limitations. Alternative forms of regularization are, for instance, based on the eikonal equation (Gropp et al., 2020) , other methods rely on local SDFs (Chabra et al., 2020; Jiang et al., 2020; Tretschk et al., 2020; Erler et al., 2020) , neural-pulls (Baorui et al., 2021) , or learned unsigned distance functions (Chibane et al., 2020) . For further methods and comparisons, we refer the reader to Ma et al. (2022) .Statistical Shape Models (SSMs) are classical methods for (cardiac) shape reconstruction, which are tailored to the whole heart (Ecabert et al., 2008; Hoogendoorn et al., 2013; Lötjönen et al., 2004; Ordas et al., 2007; Unberath et al., 2015; Zhuang et al., 2010) , the atria (Nagel et al., 2021) , or the ventricles (Bai et al., 2015; Petersen et al., 2017) . Only a few works have shown that deep learning methods can be applied to the cardiac domain to reconstruct cardiac shapes from point clouds (Beetz et al., 2023; Kong et al., 2023; Xiong et al., 2022; Sander et al., 2023; Alblas et al., 2023; Wang et al., 2021) , whereas most of the learning based methods heavily rely on image input (Beetz et al., 2022; Kong et al., 2021) . 
method: 
 In this section, we comment on the representation of cardiac shapes by learned SDFs and provide details about training and shape completion.We assume that each shape describing the heart is volumetrically represented as an SDF given by f S : R 3 → R mapping from spatial points x ∈ R 3 to their respective signed distances s ∈ R. These signed distances encode the signed projection distance at a point x to the surface (negative values inside, positive values outside). In such a way, the cardiac surface is represented as the zero-level set of this SDF, i.e., {x ∈ R 3 : f S (x) = 0}. A straightforward approach to learning such an SDF for arbitrary shapes involves representing the SDF by a neural network S θ : R 3 → R, in which case the SDF is fully defined by the network's architecture and weights θ. However, modeling surfaces solely through their neural network weights would necessitate a separate SDF and subsequent neural network for each individual surface, while also neglecting any similarities between the available training shapes. Following (Park et al., 2019) , we instead model all shapes using a single neural network and additionally provide this network with a d-dimensional latent representation z ∈ R d of the shape as an additional input, together with the aforementioned spatial coordinates, i.e. S θ : R 3+d → R. Hence, we encode multiple heart geometries with the same network, but different latent codes. We propose to model each of the left and right endocardial and epicardial surfaces with a shared latent code and neural network by letting the DeepSDF simultaneously estimate a signed distance for each of the closed bi-ventricular surfaces s = (s 1 , s 2 , s 3 , s 4 ) ∈ R 4 . In summary, the learned DeepSDF is given by the vector-valued function S θ : R 3+d → R 4 .The anatomical samples X i := (x k , s k ) K i k=1 used for learning are K i pairs of spatial coordinates x k with their sampled signed distance vector s k for the i-th biventricular shape. The set of all N available anatomical bi-ventricular samples is denoted as X := (X i ) N i=1 . To each anatomical shape i, we associate a coupled latent code representation z i ∈ R d also learned from data. We denote the set of learned latent codes as Z := (z i ) N i=1 . Our goal during training is to minimize the mismatch between the sampled signed distances and the ones estimated by the network through a simple quadratic loss term. Additionally, to restrict the latent codes Z, we follow (Park et al., 2019) and assume a zero mean Gaussian prior distribution with covariance σ 2 I on the latent codes, which gives rise to the loss termL(θ, Z) = 1 N N i=1 (x k ,s k )∈X i 1 4K i S θ (x k , z i ) -s k 2 + 1 σ 2 z i 2 2 .The parameter σ is used to balance between reconstruction accuracy for the training shapes and regularity in the latent space, which is essential for shape completion as described in Section 3.3. See Section 4.1 for the choice of σ in our model. In Liu et al. (2022) , it was shown that learning L directly may result in overfitting and might provide poor interpolation properties for the latent space. To overcome these issues, a Lipschitz penalization on the network was proposed in order to better control the smoothness. For this purpose, the Lipschitz bound L of the network is estimated by L = M i=1 W i p for an M -layer deep network, where W i are the network weights of the i-th layer. We closely follow the implementation of (Liu et al., 2022) where the linear layers of our network are replaced with Lipschitz-normalized layers. For each layer an additional weight c i is introduced such that softplus(c i ) = ln(1 + e c i ) serves as an upper bound softplus(c i ) ≥ W i p for the Lipschitz constant. Integrating this Lipschitz-regularization into our previous loss-functional L leads to our finally used cost-functionalEQUATIONwhere C(θ) = (c i ) M i=1 denotes the network parameter dependent per-layer Lipschitz bounds. As mentioned, for samplesX i := (x k , s k ) K ik=1 the network inputs are the latent code z i and the spatial coordinates x i of the sample point. Following Liu et al. (2022) the spatial coordinates are multiplied by the factor C s = 100 to balance the Lipschitz regularity of the latent space and the spatial coordinates. We use 5 hidden layers with 256 neurons and tanh activation functions. In the third hidden layer, the latent code and the spatial coordinate are concatenated to the output of the previous layer. The last layer is a linear layer with four signed distance functions as output, one for each of the left, right, epi-and endocardial surfaces. The latent code size was chosen as d = 64. A schematic representation of the network is shown in Figure 1 (left) .For the training of our network, we built a public shape library of watertight 3D shapes for endo-/epicardium of the left/right ventricles (Grandits et al., 2024) After learning the SDF network, a new anatomical bi-ventricular shape can be inferred from sparse point clouds of any combination of surfaces or given signed distance. For this, we consider K given samples consisting of triplets Ỹ of spatial coordinates x k ∈ R 3 , a single signed distance s k ∈ R, and an index of the surface j k ∈ {1, 2, 3, 4}. If a point x k lies on the surface j k , then s k = 0. Finding the bi-ventricular reconstruction of a point cloud thus reduces to finding its latent code representation z by minimizing the following problemEQUATIONwhere the subscript refers to the vector component and β is an additional weight that is increased depending on the noise of the input point cloud. For noise-free point clouds, we set β = 1 to obtain the maximum-a-posterior (MAP) estimation (Park et al., 2019) .For noisy input point clouds, the expected data loss of a perfect reconstruction can be approximated with the variance of the noise ξ. We set β = C b ξ, where C b = 100 is a chosen scaling factor. In the case of real-world data, the underlying noise structure is unknown and estimated as follows: We start with a reconstruction S 0 θ for an arbitrary initial noise estimation ξ0 . We then iteratively estimate the empirical variance ξ2 n+1 based on S n θ viaEQUATION)This estimation exhibits a high experimental convergence rate (see appendix Section F). 
experiments: 
 In this section, we state details of the training procedure and the network architecture. Moreover, we introduce distinct metrics for the evaluation of the surface reconstruction quality, which are exploited in all benchmarks. Finally, we evaluate the network on in-vivo measured catheter data.In this section, we present the numerical results obtained with the proposed method. During the training process, we optimize both the network parameters θ and the latent code representations z i of the training data. The network is fit to 4 surfaces, namely the epi-and endocardium of the right and left ventricle, and the associated hyperparameters σ and α are optimized for the best performance of the reconstruction in the shape completion process.For the loss term, we thus obtained 1/σ 2 ≈ 1.8 × 10 -7 and α = 1.9 × 10 -6 . The network is trained with the Adam optimizer (Kingma and Ba, 2015) for 3000 epochs with a learning rate of 0.005 which is decreased twice with the factor of 0.2 after 2700 and 2900 epochs. For the mesh generation, we compute the signed distance function on a 128 3 grid on the bounding box of all training points and reconstruct the zero-level set using the contour filter of PyVista (Sullivan and Kaszynski, 2019) , based on marching cubes. The final reconstruction quality of the training models is depicted in Figure 1 (right).We test the regularity of our network with respect to the latent space input with interand extrapolation between two latent codes of the training dataset. The shapes of the reconstructed hearts change uniformly between every inter-and extrapolation point and generate meaningful results (see appendix Figure 4 ). Points that are drawn from the prior distribution with covariance σ 2 I produce meaningful heart geometries (see appendix Figure 5 ). With our model, we are able to learn multiple implicit surfaces at the same time and encode them in a joint latent space representation. Therefore, it is easily possible to calculate the latent code during inference time based on an arbitrary subset of surface points. We test this by sampling sparse point clouds on the endocardium of the left ventricles from the test set and optimizing the latent code according to (2), still providing us with a full 4-chamber biventricular shape. Not only is our method capable of a close reconstruction of the endocardium of the left ventricle, but it can also predict possible shapes of the other three surfaces. We provide visual results for a reconstruction from 50 lv endocardial points in Figure 2 and Figure 6 . Note however that the accuracy of the unmeasured right ventricular shape significantly decreases. Numerical results of the reconstruction quality on the endocardium of the left ventricle can be found in Section 4.2.Our method is compared with Points2Surf (Erler et al., 2020) , Point2Mesh (Hanocka et al., 2020) , and a variant of our network without Lipschitz regularization. In all experiments, only the endocardium of the left ventricle is considered. All methods are compared on point clouds with different cardinalities n, ranging from very sparse point clouds with n = 50 to relatively dense ones with n = 2000. Additionally, the coordinates of the points are perturbed with noise drawn from a Gaussian normal distribution with zero mean and fixed covariance ξ 2 I. For each number of points n and level of noise ξ we test the methods on the same 44-point clouds sampled from the meshes of the data set (for Points2Surf we apply a pre-trained model (Erler et al., 2020) ). In the appendix we additionally provide a comparison to a SSM (see Appendix C). To test our method on all 44 meshes of the dataset we selected 11 disjoint test sets and train a different network on each remaining training sets. For both our unregularized (α = 0) and regularized DeepSDF we take β = max(1, C s ξ 2 ) and minimize the objective in (2) using the Adam optimizer (Kingma and Ba, 2015) a learning rate of 10 -2 for 50 000 epochs. In our unregularized network we used standard linear layers without any regularization and changed the activation functions to ReLu. For the unregularized layers this seems to improve the performance drastically, whereas for the regularized case we found that the choice of activation function did not influence the results too much. The results for n = 50 on four meshes computed with our method are depicted in Figure 7 in the appendix. For the comparison, we evaluated the performance in terms of L2-Chamfer-distance (CD), which can be seen in Figure 2 . For two point clouds X and Y the CD is given asd CD (X, Y) = 1 |X| x∈X min y∈Y x -y 2 + 1 |Y| y∈Y min x∈X x -y 2 .In appendix B and Table 1 further comparisons in terms of Hausdorff-distance (HD), and Large deformation diffeomorphic metric mapping (LDDMM) with their respective definitions are provided. For every test case, we state the mean and standard deviation across the four meshes of the test set. Note that Points2Surf did not converge for sparse point clouds (n = 50 and n = 200). We thus omitted the associated results in the table. Our method performs particularly well on sparse point clouds compared to the other methods. The obtained meshes are qualitatively similar to the ground truth meshes and geometric features like the curvature can be recovered properly (see appendix Figure 8 ).Electro-anatomical mapping (EAM) is a common routine for patients undergoing catheter ablation. As the catheter is inserted into the heart, it continuously builds a three-dimensional point cloud that is triangulated into a surface model. Our EAM data consists of a set of points located on the endocardial surface of the left ventricle. The point cloud is usually sparse and unevenly distributed, but commonly allows for a reasonable estimate of endocardial geometry. The exact data acquisition process is described in detail in appendix E. The EAM data SSM ours, ξ = 3.368EAM data SSM ou SSM ours, ξ = 3.368Figure 3 : Reconstruction of the LV endocardium from the EAM data. We compare the SSM reconstruction (left panel) to our approach for an optimal noise level estimation of σ = 3.368 mm (right panel). We also report the absolute distance from the geometry provided by the EAM system.overall LV shape is well-approximated in most regions except for the apex, as can be seen in Figure 3 and Figure 9 , and includes the outflow aortic tract due to the learned modality.Compared to the LV surface obtained from the EAM system, our shape is much smoother and is even defined in regions where data is missing. Since the underlying noise structure is unknown, we estimate the noise variance iteratively as described in (3). For the EAM data, we obtain an estimate of 3.4 mm, consistent with the size of the electrode's tip (2 -4 mm).A table containing the iterates of the estimator can be found in Table 3 in the appendix. 
discussion and outlook: 
 We have presented a novel method to represent cardiac anatomy based on signed-distance functions. The quantitative comparison with other methods shows that our approach can reconstruct the shape of hearts on a state-of-the-art level, especially for sparse and noisy data. In contrast to the methods we compared against, our approach does not require knowledge of the normals at each point of the point cloud. This knowledge however could be incorporated at the shape completion stage by fitting the normalized gradient of the SDF against the given normals. The requirement of point clouds is a mixed blessing, as it requires the construction of a point cloud from images through a learned method, or by segmenting the image stack and using its surface. On the other hand, enables the algorithm to operate on different modalities. Additionally, the algorithms expects the point clouds in a canonical pose (CT-based), which however could be automated using rigid point registration methods Jian and Vemuri (2010) . Learning multiple SDFs with one network allows the generation of shapes based on input data from different chambers of the heart or a combination of multiple surfaces.While the present work only encodes bi-ventricular surfaces, the method itself can be extended to encode additional chambers and shapes, such as the atria or aorta. This method has important implications in cardiac modeling, digital twinning for precision medicine, and the creation of virtual cohort of patients. We plan to further extend it to time-dependent shape models and shape uncertainty quantification (Gander et al., 2021) .To quantify the quality of reconstructed meshes we use the L2-Chamfer-distance (CD), the Hausdorff-distance (HD), and a Large deformation diffeomorphic metric mapping (LDDMM) loss terms between the reconstructed mesh and the ground truth mesh. We calculate the L2-Chamfer distance and the Hausdorff distance by sampling 50 000 points on each mesh. For two point clouds X and Y the CD is given asd CD (X, Y) = 1 |X| x∈X min y∈Y x -y 2 + 1 |Y| y∈Y min x∈X x -y 2 .The Hausdorff distance (HD) is defined asd HD (X, Y) = max{max x∈X min y∈Y x -y 2 , max y∈Y min x∈X x -y 2 }.With the LDDMM loss, we measure how well the reconstructed mesh M r can be registered to the original ground truth mesh M gt . To obtain numerically comparable results across the different methods we first remesh the results to obtain meshes with the same resolution (for this task, we use the Blender software package (Blender Online Community, 2018) with a voxel resolution of 0.9 mm). To obtain the LDDMM loss, we calculate the center points c F , the normals n F and the area A F of every face F from the set of faces F r and F gt , respectively. For γ = 1 letC(F 1 , F 2 ) = F 1 ∈F 1 F 2 ∈F 2 e -γ c F 1 -c F 2 2 2 n F 1 , n F 2 A F 1 A F 2 .Then, the LDDMM loss is defined asd L (M gt , M r ) = C(F gt , F gt ) + C(F r , F r ) -2C(F gt , F r ).The statistical shape model describes a heart shape as a variation of a mean shape µ in different directions (modes). We used the publicly available SSM from the cardiac atlas project that is based on 630 healthy Biobank reference patients, further described in Petersen et al. (2017) . A mean point cloud X ∈ R N ×3 of N points together with 200 eigenmodes V i ∈ R N ×3 for i = 1, . . . ,J Y (α, b) = d aCD Y, X + 1 N b + 200 i=1 α i λ i V i + β α 2 .This loss function is optimized for 5 000 epochs with the Adam optimizer (Kingma and Ba, 2015) , with a learning rate of 0.005. 1 : Mean and standard deviation of the Chamfer distance (CD), the Hausdorff distance, and the LDDMM-loss for different numbers of input points n and different levels of noise ξ. For every test case, the lowest value is printed in bold and the second lowest value is printed in italic. The asterisk denotes the non Lipschitz regularized version of our network.In this section, we investigate the advantages of jointly encoding of left and right endocardial shapes in comparison to independently reconstructing the two shapes. We then compare the average chamfer distances across both endocardia for the case of a separate reconstruction and a joint reconstruction for four test meshes. Additionally, we perform the same experiment on partially observable regions, where the points on the left ventricle are only sampled from its top half (ventricular base) and the points of the right ventricle are sampled from its lower half (ventricular apex). The results of both experiments can be found in Table 2 . For the case of full data availability, the joint reconstruction seems to be advantageous only for the case of very sparse data (n = 10), but for the cases where only partial data is available, the joint reconstruction improves the resulting reconstruction in all but two cases.  
 
 Appendix:  


Decision: This paper raised quite some questions, and I appreciate that the authors and reviewers have actively engaged in the discussion during the rebuttal period. Reviewer 7Bgm raised concerns about the novelty of the work with respect to existing publications. Although the work does show similarities with the paper by Sander et al, I believe the authors have made their contributions - in particular the inclusion of Lipschitz constraints - sufficiently clear in the revised manuscript. This is an exciting work that will be of interest to the MIDL community and thus I recommend acceptance.

Reviews: review: paper_summary: paper_summary: Signed distance functions of ventricular surfaces are jointly represented using a neural field. The neural field is conditioned on both 3D coordinates as well as a latent vector, and can be trained using mesh models of the heart (points + SDF values). Additional Lipschitz regularization of the network during training ensures a smooth latent space, which allows interpolation between shapes in latent space.
main_review: main_review: The manuscript is clearly structured, innovative and shows potential for a lot of new work regarding organ shape analysis and/or disease progression. The experiments prove the effectiveness of the method.
strength_weakness: strength_weakness: The one thing I missed is the potential clinical impact of this work. The results look very promising and show potential in e.g. modeling time series and the deformation of the heart in cardiac gated CTs, as the shapes can be conditioned on space + time as well.
Additionally, more background information on the shapes included in the training data would be beneficial. Are these all healthy hearts, or are some of them diseased? What is the distribution of the included shapes and how does this influence the model's capability to represent shapes that were outside of the distribution? Can a model trained only on healthy hearts still represent diseased hearts?
questions: questions: - Are there clusters in latent space based on healthy / diseased heart shapes?
- Does your method violate topological priors?
paper_summary: paper_summary: The authors present a novel alternative to traditional linear statistical shape models of the biventricular anatomy. In essence, the proposed method comprises an auto-decoder architecture that maps spatial coordinates and a shape encoding to a four-dimensional vector, which represents the signed distances to the left and right endocardium and epicardium, respectively. Despite training a shallow Lipschitz-regularized neural network for the auto-decoder architecture on only 36 patient geometries, the method outperforms the comparators (Points2Surf, Points2Mesh, and a linear SSM) in reconstructing the left endocardium from sparse and noisy point clouds as well as when reconstructing the surface of sparse electroanatomic maps.
main_review: main_review: I truly appreciate that the authors have responded in such detail to all of my questions and concerns (thank you!!). 

From my point of view, the new experiments as well as the changes made to the manuscript are adequate and strengthen the paper. While there are obviously many different ways to further test the proposed method, I do understand that it is impossible to do everything in the short time frame of the rebuttal phase as well as to fit everything within the page limit. In my opinion, the proposed method has significant merits and the manuscript, in its revised form, would be of great interest to the MIDL community. I am happy to change my recommendation to acceptance.
strength_weakness: strength_weakness: While I strongly believe that this manuscript is of high quality and that the proposed method has significant merits over traditional SSMs, I have several questions and concerns that I hope the authors could answer:

- From my point of view, one key limitation of the method was not properly addressed in the manuscript: the requirement that the biventricular shapes have to be in a canonical pose for training and inference. Could the authors please comment on this limitation and how such a system could be applied, e.g., when the target is in a rotated position?

- It is questionable to me whether the traditional SSM is a valid comparator. First of all, the SSM was trained on a different patient population, which is only comprising healthy patients. The test data, however, was extracted from a database of healthy and diseased patients. Could the authors please provide information on whether the data split held any diseased patients and please comment on how this could impact the results? In addition, to fit the SSM, only the shape coefficients and a translational offset are optimized. Could there be a rotational bias due to the construction of the SSM on a different dataset that is offsetting the errors?

- Moreover, it seems to me like the proposed method is either compared against a too simple approach (linear SSM) or against too sophisticated deep-learning-based approaches (Points2Surf and Point2Mesh). Similarly, no ablation study, e.g., on the impact of the Lipschitz-regularization, was run, which makes it difficult to assess whether the improved results are a result of the non-linear shape representation, the SDF representation, or something else. One possible additional comparator may be a non-linear SSM constructed from the training meshes via an autoencoder. In theory, this may also allow to compare the inter- and extrapolation quality of the learned shape encoding.

- One particular advantage of the proposed method is the adjustable sampling resolution. To reconstruct a mesh, a grid sampling pattern of 128^3 was used. It is not clear how finer or coarser sampling would impact the reconstructed mesh and the errors. In addition, it is not clear why a point-based approach would be preferred over, e.g., projecting only the (learned) shape encodings to a fixed four-dimensional grid of size 128^3 holding the signed distance maps.

- Even though the paper was overall easy to read and provided in most cases complete descriptions, I often found myself constantly jumping between the main manuscript and the appendix and I do not believe that the manuscript would be self-contained without the appendix. In addition, the discussion of the results is in comparison to the extent of this paper too short and shallow.
questions: questions: Please see the comments under the weaknesses section and in the detailed comments. In addition, could the authors please comment on the following questions:

- Could the authors please explain whether and how this method could be extended to represent non-closed surfaces? 

- In traditional SSMs one can retrieve the mean shape, do the authors see any way to retrieve something similar with the proposed approach?

- Another benefit of SSMs are point correspondences, which can be easily used to transfer data between geometries. How could the proposed method be used for such tasks?

- A shape encoding of 64 dimensions was chosen. Could the authors please comment on the impact of the latent dimension size onto the results?

- Appendix, A2: Could the authors please clarify why two point clouds were generated? In addition, could the authors please comment on the impact of only sampling within a 30mm distance to the shape? What happens if one would sample further away? Could this be an issue for very large or very small geometries?
paper_summary: paper_summary: The authors present a method for cardiac shape generation using implicit neural representations. The method is very similar to the one presented in Sander et al. 2023, but rather than completing shapes in sparsely sampled CMRI, the authors propose to complete shapes in sparsely sampled electro-anatomical maps. They extend the method by utilizing multiple output heads for four classes (i.e. epi- and endocardium of both ventricles), rather than a single ventricle per MLP. Additionally, they add a Lipschitz regularization term to improve the plausibility of reconstructions from randomly sampled latent codes, as proposed in Liu et al. 2022.
main_review: main_review: The new multi-chamber comparison unfortunately has the appearance of cherry-picked and misrepresented results. In all of the original experiments, the authors show the results for 50, 200, 500 and 2000 samples. In appendix D, the table conspicuously stops at 40. The authors admit that the joint reconstruction seems to be advantageous only for the case of very sparse data (n = 10), which does not seem to be a relevant use-case (as the lower bound in the rest of the manuscript is 50). These results show that for the use cases considered in the rest of the manuscript, the multi-class approach is actually worse than the single-class version proposed in Sander et al. 2023.

The updated manuscript still includes the sentence "We demonstrate that this approach is also capable of reconstructing anatomical models from partial data, such as point clouds from a single ventricle" in the abstract, which is misleading. The authors have not added the numerical results for the other classes in this experimental setting, and the result they show in Figure 6 implies this is _not_ possible with any reasonable expectation of accuracy. Additionally, the manuscript still includes the same second and third paragraph of the introduction, which imply that the current work is the first to propose cardiac shape generation with implicit SDFs. This is only later toned down by the sentence "The proposed method shares some similarities with the very recent work by Sander et al. (2023)", which is not an adequate characterization of the similarity of the methods. Especially as the results still label the version without Lipschitz regularization as __Ours (L)__, even though the authors claim this is their best effort at a reimplementation of Sander et al. in the OpenReview comments (_"In the view of the authors, the non-Lipschitz regularized DeepSDF is the closest achievable approximation to the paper by Sanders et al., ..."_). Furthermore, it seems like the authors still use the multi-class setting for this baseline, which from the results shown in Appendix D would perform worse than the "true" baseline as presented in the original work.
strength_weakness: strength_weakness: The manuscript overstates its novelty. The authors compare their work with outdated methods, which misrepresents the benefit of the presented method. Additionally, while Sander et al. 2023 is cited in the related work section, the rest of the manuscript is written as if this (very similar) work does not exist.

The additions compared to Sander et al. 2023 are minor, and these additions are not properly compared with the baseline. The primary innovation over the Sander et al. is the Lipschitz regularization term, but the authors provide no comparison or evaluation regarding the effects and benefits of this regularization term. 

While the authors show that the method can generate plausible inter- and extrapolations from the latent vectors of two shapes, the motivation for why this is relevant is not clear from the manuscript.

The method is only evaluated on four patients.

Not all claims are substantiated with experimental results. While the authors state "[our method] can also predict the shape of the other three surfaces", they do not provide numerical results for the errors in any of these reconstructions.

The discussion is not very well written. It does not discuss the implications or effects of the proposed additions compared to previously published methods.
questions: questions: The authors should compare their work with modern baselines for this task, such as J Wang et al. 2021 (MICCAI), M Beetz et al. 2022 (PMLR) and certainly with J Sander et al. 2023 (CBM). Does the proposed method outperform these methods? If not, what is the benefit of the proposed method compared to these methods?

The paper should be rewritten such that it no longer overstates its own novelty compared to related prior work.

What is the added benefit of the proposed Lipschitz scaling regularization term? Does it actually help? This should be investigated.

The authors should evaluate on more patients using cross-validation, rather than only reporting metrics for four patients.

The authors claim the method can predict the shape of the other three surfaces if only one of the surfaces is available, but it seems like such a system could only reconstruct a hallucination that does not necessarily resemble the true structures. What are the errors for these reconstructions, numerically?

What are the implications of the proposed additions compared to previously published methods?

Comments: title: Answer to the review
Answer to the review 3/3
Answer to the review 2/3
Answer to the review 1/3
Answer to the review
General Statement
comment: Thank you for carefully reading our manuscript and providing valuable feedback. We are confident to address your critique convincingly and highlighted our novelty and differences to the related methods.

Point-by-point response to the questions to address in the rebuttal:
* The authors should compare their work with modern baselines for this task, such as J Wang et al. 2021 (MICCAI), M Beetz et al. 2022 (PMLR) and certainly with J Sander et al. 2023 (CBM). Does the proposed method outperform these methods? If not, what is the benefit of the proposed method compared to these methods?
  + Thank you for your comment. Indeed, we were aware of the highly interesting paper by J Sander et al. 2023 (CBM), which we carefully read. However, originally we decided to compare our method with the established and widespread methods for the task of point cloud reconstruction, namely Points2Surf and Point2Mesh.

    In contrast to the paper by Sanders et al., we further extend existing DeepSDF methodologies by incorporating Lipschitz regularization as proposed by Liu et al., together with a multi-SDF estimation approach. In our work, we particularly focus on the case of low density and high noise point cloud recordings, as often found in electroanatomical mapping, in which these architectural changes show major improvements. In the revised version of the manuscript, we have now added several new numerical experiments to exactly examine the benefit of the Lipschitz constraint, which outperforms non-Lipschitz regularized DeepSDFs in most cases considered. In the view of the authors, the non-Lipschitz regularized DeepSDF is the closest achievable approximation to the paper by Sanders et al., given that the training data they used is not publicly available, thus showing the superiority of our approach against the method by Sanders in the considered scenarios.

    Second, we advocate a multi-region approach, in which simultaneously four signed distance functions are estimated. For instance, the simultaneous estimation of the endocardia of the left and right ventricle can improve the overall reconstruction quality for sparse and/or partial data, as we now show in Appendix D. We have systematically analyzed this in a novel numerical experiment. All of the aforementioned differences and benefits of our method are now further outlined in the introduction.

    We also wanted to thank you for bringing the works of Beetz et al. and Wang et al. to our attention. Due to time and space restrictions of this work, we decided to compare our method in the revised version only against an approximate version of Sanders et al.. Note that in this very same work, they already showed how their method is superior to the works by Beetz et al. and Wang et al., which by extension also applies to our method since we outperform the work of Sanders et al. in most cases.

* The paper should be rewritten such that it no longer overstates its own novelty compared to related prior work.
  + Thank you for this comment. As stated in the previous reply, there exists a clear novelty to current methods. We however completely agree that we did not accurately describe these novel points, which resulted in the impression that we overstated our novelty (this was by no means our intention). In the revised version, we focused on a more careful description of our real novelty in the introduction and related work section.

* What is the added benefit of the proposed Lipschitz scaling regularization term? Does it actually help? This should be investigated.
  + Please see our response to the first item.

* The authors should evaluate on more patients using cross-validation, rather than only reporting metrics for four patients.
  + You are completely right. In the revised version, we have now included the cross-validation on all 44 available patient data as requested.

* The authors claim the method can predict the shape of the other three surfaces if only one of the surfaces is available, but it seems like such a system could only reconstruct a hallucination that does not necessarily resemble the true structures. What are the errors for these reconstructions, numerically?
  + Thank you for bringing this to our attention. We see potential in the possibility to reconstruct (unobserved) RV shapes from purely LV measurements. Since it is likely that the RV shape is not uniquely defined by the LV, the observed errors are comparably high, but significantly decrease once measurement points at the RV are added (see Appendix D in the revised version). Nevertheless, we can reconstruct a possible/likely candidate for the other structures based on the joint prior distribution. We adapted our statement in the paper accordingly.

* What are the implications of the proposed additions compared to previously published methods?
  + We clarified the implication that our method has compared to previously published methods, especially Sanders et al..
Point-by-point response to the additional questions:

* Could the authors please explain whether and how this method could be extended to represent non-closed surfaces?
  + Thank you for your question. The generalization to non-closed surfaces is beyond the scope of this paper. However, there exists some work addressing learned unsigned distance functions (e.g. “Neural Unsigned Distance Fields for Implicit Function Learning” by Julian Chibane, Aymen Mir, Gerard Pons-Moll, NeurIPS 2020).

* In traditional SSMs one can retrieve the mean shape, do the authors see any way to retrieve something similar with the proposed approach?
  + We assume a zero mean Gaussian prior, therefore the reconstruction of the zero-latent vector would provide the prior mean shape and would be easily computable. However, the computation of the mean shape in Cartesian space on the level of shapes is not as straightforward in our case, as it is for traditional SSMs.

* Another benefit of SSMs are point correspondences, which can be easily used to transfer data between geometries. How could the proposed method be used for such tasks?
  + To the best of our knowledge, there is no canonical approach for point correspondences for signed distance functions. 

* A shape encoding of 64 dimensions was chosen. Could the authors please comment on the impact of the latent dimension size onto the results?
  + Thanks for the question. We experimentally discovered that a latent space dimension of 64 provides the best tradeoff between training accuracy and prediction quality. However, we found that the results are not very sensitive to changes in the latent space dimension. Our experiments reveal that the performance mainly depends on the choice of the Lipschitz regularization parameter (alpha) and the prior weighting (sigma).

* Appendix, A2: Could the authors please clarify why two point clouds were generated? In addition, could the authors please comment on the impact of only sampling within a 30mm distance to the shape? What happens if one would sample further away? Could this be an issue for very large or very small geometries?
  + Thank you for this comment. The two point clouds previously referred to endocardial and epicardial points respectively, but are later merged into a single point cloud. To avoid any further confusion, we only mention a single generated point cloud (left, right, epi- and endocardial).
   The main aim of the proposed framework is the reconstruction of shapes by means of the signed distance function, which exhibits its highest accuracy in the neighborhood of the zero levelset. The Lipschitz regularity of the network bounds the change of the SDF in a rough sense, but the signed distance may still deviate from the true signed distance beyond the trained distance. Note however that in no experiment, we observed a second disconnected level set at another point in the discretized mesh grid, even though some portions in the bounding box exceed the distance of 30mm to at least one of the four shapes.
A point-by-point response to the detailed comments:

* Figure 1: The authors mention in the text that the output of the 2nd hidden layer is concatenated with the spatial coordinate and the shape encoding. Could the authors please adapt the network illustration to reflect this?
  + Fixed

* Figure 2: Even though one can clearly see that the shapes are well extra- and intrapolated, it may be even easier to comprehend in a small animation. Would it be possible for the authors to also publish a video of the deformation?
  + Thank you for this idea. Unfortunately, we were not able to produce such a video until the first rebuttal deadline, but we are happy to provide it during the next week in the discussion period.

* Figure 3, table: It may be easier to read if not font types but colors are used to highlight the lowest and second lowest values. In addition, how do the errors behave in relation to the heart size?
  + Following your advice, we now included a color scheme in the table. We tested the correlation of heart sizes (LV volume) to reconstruction error (CD) for our methods in all experiments, but found no consistent correlation.

* Figure 3, graphic: How many samples on the LV endocardium were used for the reconstruction?
  + We used 50 points for the reconstruction, the information can now be found in the paper.

* Methods, Lipschitz-regularized network explanation: Could the authors please clarify how the additional weight c_i is incorporated into the network forward computation (if at all)?
  + Thank you for your question. We exploited an absolute row sum as the upper boundary for the Lipschitz constant. In every forward application of the network, we multiply every row of the weight matrix by a factor to ensure that its absolute sum is below c_i. For further details, we refer to (Liu et al., 2022).

* What algorithm is used to reconstruct the surfaces from the SDFs?
  + The meshes are generated using the contour filter of pyvista, which itself is based on marching cubes. This remark was now added to Sec. 4.1.

* Inference on EAM data: What is the error when reconstructing the data with the SSM?
  + We highlight that there is no ground truth available for the heart shapes of the EAM data. Thus, a quantitative analysis is impossible in this context, which is why we can only present visual results.

* Discussion: The authors state "The requirement of point clouds is a mixed blessing, as it requires an additional step to work on image input", but the inference in the first experiment was run over positions sampled on a grid, which, to my understanding, would allow direct application to image data. Could the authors please clarify what they mean?
  + Thank you for this question. This additional step refers to creating point clouds from images, which could be achieved directly through e.g. a learned approach, or by using image segmentations. We have slightly adapted the text.

* Could the authors please clarify whether the spatial coordinates were normalized?
  + We translated the coordinates such that the left ventricular apex is in the Cartesian origin. Other than that, there were no further steps for the normalization of the coordinates, which were given in millimeters in the original space.

* Idea for future research: It would be very interesting to see whether, when trained on large amount of healthy and diseased shapes, the learned shape encodings are also discriminative of the underlying diseases. In addition, it would be very valuable to not only study the error under sparse data, but also partial shape data, e.g., providing only data on the septum or free wall
  + Thank you for these suggestions. This looks like a very promising future research direction and we are grateful for bringing this to our attention.
Thank you for your valuable feedback on our paper. We used the allocated rebuttal time to extend our manuscript both by experiments and further elaboration of our method. We hope that this revision will be able to address your concerns both in the revised paper and in the following answers.

* One key limitation of the method was not properly addressed in the manuscript: the requirement that the biventricular shapes have to be in a canonical pose for training and inference. Could the authors please comment on this limitation and how such a system could be applied [...]?
  + Thank you for this question. This is indeed a limitation of our method, as we require the left ventricular apex to be aligned and the heart to have the same pose as in a CT scan. Note however that rigid alignment methods offer reasonable, well-established methods to solve this problem. This is now shortly discussed in the Discussion.

* It is questionable to me whether the traditional SSM is a valid comparator [...]. Could the authors please provide information on the data split and the impact on the results? [...] Could there be a rotational bias due to the construction of the SSM on a different dataset that is offsetting the errors?
  + We decided to use SSM as a competitor because they are still popular in the cardiac community, see:
     - https://doi.org/10.1109/EMBC48229.2022.9871327
     - https://doi.org/10.1117/12.2550650
     - https://doi.org/10.1371/journal.pcbi.1008851

    The applied SSM should have the same spatial orientation as the meshes from the test set, therefore optimizing the shape coefficients and the translational offset should be sufficient.

    Regarding the difference between healthy and diseased patients, we further investigated how well our approach performs when only training healthy patients (Rodero et al., 2021) while later inferring the shapes of diseased patients (Strocchi et al., 2020) and compared the results to a model that was trained on a subset of the diseased hearts. Both models performed worse than our original model but we found no significant performance difference to one another. The deterioration in terms of performance of the methods when only using healthy or diseased data to infer diseased hearts could be a consequence of the reduced number of training meshes.
    Since, the training data of the SSM purely comes from cardiovascular magnetic resonance (CMR) studies, the reference orientation is the same as for the training and test data used in this study, making a rotational bias highly unlikely.

* Moreover, it seems to me that the proposed method is either compared against a too simple approach (linear SSM) or against too sophisticated deep-learning-based approaches [...]. Similarly, no ablation study, e.g., on the impact of the Lipschitz-regularization, was run, which makes it difficult to assess whether the improved results are a result of the non-linear shape representation, the SDF representation, or something else. One possible additional comparator may be a non-linear SSM constructed from the training meshes via an autoencoder.[...]
  + Thank you for your comment. We have selected Points2Surf and Point2Mesh due to their widespread use in recent baseline comparisons for shape reconstructions with point cloud data as single input (note that both methods were published in 2020). Moreover, we have now conducted the requested ablation study to validate the benefit of the Lipschitz constraint. A comparison against a non-linear SSM as you suggested would indeed be interesting but is unfortunately not realizable within this rebuttal period.

* One particular advantage of the proposed method is the adjustable sampling resolution. [...] It is not clear how finer or coarser sampling would impact the reconstructed mesh and the errors. In addition, it is not clear why a point-based approach would be preferred [...].
  + Thank you for this question. In several numerical experiments, we concluded that the 128^3 grid is sufficiently fine to fully represent the encoded shape. Experimentally, no benefit of finer meshes could be discovered, which, however, results in additional computation time. Note however that further usage of the reconstructed mesh could require a higher resolution model (e.g electro-mechanical, or bidomain simulations), which can be easily realized using our methodology.
* Even though the paper was overall easy to read and provided in most cases complete descriptions, [...] I do not believe that the manuscript would be self-contained without the appendix. In addition, the discussion of the results [...] of this paper too short and shallow.
  + Thank you for your comment. Indeed, we target a self-contained manuscript, which is challenging due to the page limit. We placed our primary emphasis on authoring a self-contained method section while striving to achieve maximum self-containedness in all subsequent sections. In the revised version, we slightly revised the text.
Thank you for your valuable feedback and the very positive assessment of our work. We are aware of the missing clinical validation of this work for the aforementioned examples. However, this is beyond the scope of this work, which, however, will be addressed in future research.

A point-by-point response to the detailed comments:

* The one thing I missed is the potential clinical impact of this work.
   +  We are grateful for this suggestion. In general we think that the presented method has the potential to offer an easier method to reconstruct anatomical models, where imaging data is scarce, not present, or limited to specific imaging modalities ( e.g. electroanatomical mapping, echocardiography). We tried to highlight possible clinical applications in the last part of the discussion in the revised version.

* Are these all healthy hearts, or are some of them diseased? What is the distribution of the included shapes and how does this influence the model's capability to represent shapes that were outside of the distribution? Can a model trained only on healthy hearts still represent diseased hearts?
  + Thank you for this question. We have now clarified in the text that 20 geometries are from healthy subjects and 24 geometries are from patients with heart failure candidate to resynchronization therapy (see Appendix A). We have examined the patients without any prior assumption regarding the underlying probabilistic model. Already in this cohort, a large inter-patient variability can be observed. Out-of-distribution shapes with a completely different anatomy such as patients suffering from single ventricle defects will likely not be properly represented by the current model unless the training set is enlarged by these diseases.

* I don't agree with the notation for the zero-levelset in Section 3.1, this implies that a signed distance function is invertible, which is generally not the case. I would write the zero level set as $\lbrace \mathbf{x} \in \mathbb{R}^3 \colon f_S(\mathbf{x})=0\rbrace$ .
  + Thank you for your comment. We now denote the zero level set as suggested.

* The endocardium and epicardium seem to be nested in one another, did you observe violations of the anatomical requirement in some representations?
  +  During training and in all inferences we performed, no topological or anatomical violations were observed using our method.

* What is shown in the right part of Figure 3? Are those multiple reconstructions or only the surface reconstruction resulting from the proposed method?
  + In the right part of Figure 3, we show the left and right endocardial reconstruction of our method based on points on the left endocardial reconstruction. Because of the joint latent code for all surfaces, our method can provide a reasonable estimation of the other surfaces.

* It might be nice to visualize the different parts of the heart (e.g. in Figure 1) using different colors so it is clear which parts are separate for reader that are not very familiar with heart anatomy.
  + Thank you for this suggestion. In the revised version, we added colors to highlight the left and right ventricles in Figure 1.  

* If you have shapes for healthy and diseased hearts available, are they clustered in the latent space? Is this also influenced by the shapes that were present in the training data?
  + Thank you for this comment. We performed a simple k-means on the final latent vectors, which did not result in clusters visually separating healthy and diseased patients.

* The last sentence of the discussion is unclear. What are the concrete implications of optimal transport in latent space? I think there are much more straightforward applications of the proposed work, e.g. modeling of time series.
  + Thank you for bringing this to our attention. We agree that optimal transport in latent space is not the most straight-forward extension of our work and have thus removed it from the last paragraph of the discussion and added a sentence on time dependent models.
We wanted to take this opportunity to thank all the reviewers for their time and effort to evaluate our presented work. We did our best to address all raised points sufficiently. The revised version contains many changes that are highlighted in a different color in the document. Additionally, the document was restructured to more efficiently emphasize our novelty and contributions. Most notably, Figure 2, the comparison with SSM and the lower half of Figure 4 were moved to the Appendix. The results of the SSM in Figure 3 were now replaced with the original DeepSDF methodology results, more closely connected to other related works. Detailed comments and answers to the reviewers’ points are provided in the comment section of the reviews.