Deep Active Latent Surfaces for Medical Geometries

Venue: NLDL_org_2025_Conference
Published: 2024-09-06 16:39:40

Abstract: Shape priors have long been known to be effective when reconstructing 3D shapes from noisy or incomplete data.  When using a deep-learning based shape representation, this often involves learning a latent representation, which can be either in the form of a single global vector or of multiple local ones. The latter allows more flexibility but is prone to overfitting. 
In this paper, we advocate a hybrid approach representing shapes in terms of 3D meshes with a separate latent vector at each vertex. During training the latent vectors are constrained to have the same value, which avoids overfitting. For inference, the latent vectors are updated independently while imposing spatial regularization constraints. We show that this gives us both flexibility and generalization capabilities, which we demonstrate on several medical image processing tasks.

Main paper: 
 introduction: 
 3D shape reconstruction from noisy or incomplete data often benefits from shape priors. With deeplearning, this usually means searching for an appropriate latent representation under regularization losses. This representation can be a single global latent vector per shape or a grid of latent vectors.In both cases, balancing the quality of fit against regularization strength is challenging. Too much regularization results in overly smooth shapes. Too little damages robustness and generalization. Our insight is to use multiple latent vectors but impose the regularity constraints on the latent vectors instead of the surface. This is effective because, if a latent vector models a sharp feature, requiring that this vector be similar to its neighbors will not detract from that. To this end, we represent surfaces as triangulated meshes and advocate using separate latent vectors at each vertex. We then impose smoothness on the latent vectors by using regularization ideas from early 3D shape modelling [1] [2] [3] .More specifically, during training, we use an auto-decoding approach [4] to jointly learn network weights and a single latent vector for all vertices of each training sample. At inference time, we then allow the latent vectors to be different at each vertex while enforcing consistency of the vectors via a regularization loss during fitting. Our Deep Active Latent Surfaces (DALS) approach, illustrated in fig. 1 , is both easy to train from relatively small 048 datasets and very expressive. By learning a single 049 latent vector per shape, we do not require huge train-050 ing sets. At model fitting time, we can smoothly 051 blend latent vectors to model complex shapes with 052 sharp features. Finally, using a triangulated mesh 053 instead of a latent vector grid makes our model more 054 compact and simpler to train because it does not 055 have to waste capacity on modeling empty regions. 056 2 Related Work 057 Active Surface Models Active contour models 058 refine contours according to local image properties 059 while remaining smooth. They were first introduced 060 in [1] for interactive delineation and then extended 061 for many different purposes [5] . Active surface mod-062 els [2, 3] replace contours by triangulated meshes to 063 instead model 3D surfaces. They have proved very 064 successful for, e.g., medical [6, 7] and cartographic 065 applications [8], and are still being improved [9-066 11]. Recently, Deep Neural Networks (DNNs) have 067 been used to evaluate the energy that active con-068 tours minimize [12, 13] and to directly predict vertex 069 offsets [14] [15] [16] . In [17] , active surface models are 070 embedded in special purpose network layers that reg-071 ularize surface meshes using the same semi-implicit 072 scheme as the original active contours [1] . Balancing data and regularization terms to avoid over-smoothing while being robust to noise remains a challenge. In [15] smoothing is added as a training loss but not during inference. In [17] , smoothing is made adaptive to allow sharp edges. Finally, [18] replaces smoothing with preconditioned gradient descent of the external energy. In all these approaches, the regularization tends to flatten sharp geometric features, i.e., over-smooth. In our work, we side step these issues by only smoothing in the latent space.Neural Shape Modeling Deep-learning is now routinely used to model 3D shapes, via models that transform latent vectors into a target shapes parameterized in terms of, e.g., triangulated meshes [19] [20] [21] [22] , tetrahedral meshes [23, 24] , surface patches [25] ,point clouds [26, 27] , voxel grids [28, 29] , occupancy functions [30] [31] [32] , signed and unsigned distance fields [33, 34] , or neural splines [35] .Methods may represent a shape with one or several latent vectors. Those that use one obtain the latent vector for a shape with an encoder [30, 36] or by directly optimizing a latent vector [4, 19, 34, 36, 37] .While effective, accurately representing fine details is difficult. Instead, the methods of [32, 33, 38, 39] use a grid of latent vectors and a shared decoder, which greatly increases the model flexibility. However, this wastes latent vectors on representing empty spaceincreasing memory use and training time. Sparse grids [40, 41] or multiscale tree structures [42, 43] mitigate this but complicate training. Spatial hash encoding [44] implicitly allocates more capacity to surface regions, but is not designed to model multiple surfaces. Our work also relies on multiple latent vectors. However, we store the latent vectors at the vertices of a triangle mesh instead of a spatial grid. This grants us high flexibility but avoids unnecessary latent vectors and a complex training pipeline. 
deep active latent surfaces: 
 We now describe our Deep Active Latent Surface (DALS) approach, illustrated in fig. 1 . We represent watertight 3D shapes by triangulated spheres with a latent vector at each vertex. Each, latent vector along with the vertex coordinate, is fed to a decoder D θ that generates an offset vector that translates the vertex to its final position. After translating all vertices, we have the final shape (see top right of fig. 1 ). During training, all vertices use the same latent vector. During model fitting, we allow the vectors to be different but impose spatial consistency. This still allows modeling sharp features because these can be predicted by individual latent vectors.Formally, let D Θ be a neural network with weights 126 Θ that takes as input alatent vector z ∈ R d and 127 a spatial location x ∈ R 3 and returns an offset 128 D Θ (z, x) ∈ R 3 . Given a triangulated sphere with 129 V vertices and F facets, we denote by M θ (z) the 130 deformed mesh we obtain by translating each vertex 131 x v by D Θ (z, x v ) for all v between 1 and V .Assume we are given a set of N training shapes 133 S = {S 1 , . . . , S N }. As in [4] , we can simultaneously 134 learn Θ and a z i for each S i by looking for135 Θ * , z * 1 , . . . , z * N = arg min Θ,z1,...,z N N i=1 L dat (Θ, z i , S i ) , (1) 136 L dat (Θ, z, S i ) = L cf (M θ (z), S i ) + (2) 137 λ reg L reg (M θ (z)) + λ n ∥z∥ 2 , 138where L cf is the Chamfer distance [45] , L reg is a 139 shape regularization term, and λ reg and λ n are 140 weighting constants.141In practice, we take D Θ to be an MLP with three 142 hidden layers of size 724, 724, and 362, with ReLU 143 activations for the hidden layers and none for the 144 last layer. Before each ReLU activation we use layer 145 normalization [46] . The input is a concatenation 146 of x and z. Our initial spherical triangulation is a 147 subdivided icosahedron. We use a pointwise MLP 148 because we want to learn a general mapping from 149 the surface of a sphere conditioned on a latent vector 150 as this allows us to increase the mesh resolution at 151 inference time. To this end, we also randomly rotate 152 the template during training.The L reg term in Eq. 2 is intended to encourage 154 the generation of high quality meshes. Experimen-155 tally, we found that using the Laplacian did not allow 156 us to generate high quality meshes without losing 157 too many details in the reconstructions. Instead, as 158 in [47] , we useEQUATIONwhere F stands for the mesh facets, a f , b f , c f for 161 the lengths of the three edges of facet f , and A f 162 for its area. L reg promotes regular triangles without 163 directly penalizing high frequency features [48] , as 164 illustrated by fig. 2 . translation from the initial sphere to where it should shape M(Z) solves a given downstream task, e.g., 176 fitting a point cloud. Formally, we look for 177EQUATIONlet energy [18, 49] to define L dir . We writeEQUATION191where L is the uniform Laplacian matrix and p is an Z i = (z * 1 + z * 2 + 208 • • • + z * N )/Nwhere z * i are the latent vectors found 209 in eq. ( 1). We then iteratively update the latent 210 vectors using the Adam optimizer. 
shape reconstruction from 3d point clouds: 
 Experimental Setup. We test the ability of the latent vector models to reconstruct unknown shapes 8.8 ± 7.53 0.19 ± 0.07 94.6 ± 2.57 96.8 ± 1.97 0.98 ± 0.00 0.00 ± 0.00 DUAL-MLP (global) [39] 13.6 ± 5.67 0.16 ± 0.05 71.7 ± 6.39 91.4 ± 3.48 0.98 ± 0.00 0.00 ± 0.00 DUAL-MLP (local) [39] 161.7 ± 39.6 0.43 ± 0.04 44.5 ± 4.05 62.1 ± 4.08 0.98 ± 0.00 0.00 ± 0.00 DALS (Ours) 2.4 ± 1.04 0.11 ± 0.04 95.4 ± 2.06 99.0 ± 0.76 0.98 ± 0.00 0.20 ± 0.40 For all methods, we use the ADAM optimizer to minimize their fitting losses. We set λ reg = 0.001 and λ dir = 0.2. Finally, for DASM+R and DALS we post process the results using five iterations of Botsch-Kobbelt remeshing [55] .To evaluate the reconstructions, we use the Cham-279 fer distance, the Hausdorff distance, and the F-score 280 [56, 57] at 1% and 2% of the surface's bounding 281 sphere diameter. We also evaluate the mesh quality 282 of the reconstructions using the quality measure of 283 Eq. 3 and the percentage of self-intersecting faces. 284Results. We report comparative results in ta-285 ble 1. DALS consistently outperforms the other 286 3.82 ± 2.24 0.21 ± 0.05 47.0 ± 7.67 74.4 ± 8.25 0.98 ± 0.00 0.00 ± 0.00 DASM+R [17] 27.41 ± 7.91 0.47 ± 0.09 14.8 ± 5.99 29.5 ± 9.99 0.98 ± 0.00 0.02 ± 0.07 DUAL-MLP (global) [39] 4.05 ± 1.58 0.23 ± 0.05 49.9 ± 6.17 74.4 ± 6.08 0.98 ± 0.00 0.00 ± 0.00 DUAL-MLP (local) [39] 15 We repeated the experiment on much simpler and smoother spleen shapes. The data are again from the Medical Segmentation Decathlon [50] prepared the same way as the liver data. We focused on DALS and DASM+R since they delivered the best results on the liver. As can be seen in table 2 and fig. 4 DASM+R delivers very slightly better metrics but a qualitatively worse reconstruction because it overfits to the staircase artifacts on the ground-truth shape.In contrast, DALS yields an organic shape that still fits the data well which can be viewed as a more realistic result. This effect also exists in the liver dataset but did not significantly affect the metrics as the original images were of higher resolution.Experimental Setup. When annotating medical images, a common time-saving practice is to only annotate three orthogonal 2D slices instead of the entire 3D image. We test the ability of our model to 313 reconstruct shapes from such weak annotations by 314 fitting to the 2D planar boundary curves extracted 315 from the 2D slice annotations. We randomly sample 316 5 000 points from the intersection curves between 317 each held out liver and three orthogonal axis-aligned 318 planes. As shown in fig. 5 (e), this results in a very 319 sparse point set, making the quality of the embedded 320 shape priors key to obtaining good results.For DALS, DASM, and DASM+R we take L task 322 to be a modified Chamfer distance that relies on dis-323 tances within the annotation planes (see appendix A 324 for details). For the other methods, we again use 325 the mean absolute SDF value to compute L task . 326 We again use the ADAM optimizer and Botsch-327 Kobbelt remeshing as in the previous section. We 328 set λ reg = 0.01 and λ dir = 100 as we want to rely 329 heavily on the shape prior in this task.Results. We report comparative results in table 3 331 and qualitative results in fig. 5 . To generate these 332 results, we only used annotations in the three orthog-333 onal axis-aligned planes. DALS results are consis-334 tently better while retaining excellent mesh quality. 335 This is also the case as one annotates additional 336 planes, which we show in appendix A.1. with the more recent nn-U-Net [60] and UNETR [61] .For this experiment, we use the 40  
4 experiments: 
 We demonstrate the benefits of DALS on several medical image processing tasks. We train all models to learn a latent representation of livers and spleens using data from the Medical Segmentation Decathlon [50] . To create ground-truth meshes, we resampled the annotated images so that their voxel size is 1×1×1 mm and used marching cubes [51] to extract isosurfaces. We standardize the surfaces to have zero mean and be contained in the unit sphere.The datasets contains 111 livers and 41 spleens. We use the first 71 livers for training and hold out the last 40 for evaluation. We also train another model on the first 31 spleens with the last 10 held out for evaluation. We augment the training data using the PointWOLF algorithm [52] to create 100 new shapes for each training shape. PointWOLF applies a smoothly varying non-rigid transformation to mesh vertices yielding diverse and realistic augmentations.We use 128 dimensional latent vectors and an icosahedron subdivided 3 times for training and 4 times for fitting as a template for the decoder. To learn these vectors and the decoder weights we solve the minimization problem of Eq. 1 with λ reg = 10 -4 and λ n = 10 -3 . We use the ADAM optimizer [53] , with learning rate 0.002, momentum terms to β 1 = 0.9, β 2 = 0.999 and train for 24 hours on a single NVIDIA Tesla V100 GPU (ca. 7,500 epochs). If the loss does not improve for 100 epochs we half the learning rate, down to a minimum of 10 -5 .Baselines We compare our model against the following baselines: DeepSDF [4] , SIREN+DeepSDF where training and inference are as in DeepSDF but with a SIREN [54] based decoder, and the DUAL-MLP approach of [39] . These are all auto-decoder based which foregoes the need to train separate encoders for each experiment. We also compare to DASM, the active surfaces of [17] , along with an improved version that we dub DASM+R which adds a re-meshing step during fitting to avoid selfintersections. DASM and DASM+R do not rely on a learned shape prior and simply promote smoothness.As the DUAL-MLP authors did not release code, we implemented two separate versions of it, one that uses one single latent vector per shape (global) and one that uses several (local). All these methods were trained as recommended in the relevant papers.Beyond the experiments listed below, we also perform ablation experiments and analyze parameter sensitivity in appendix C. 
5 conclusion: 
 We have shown that DALS is an efficient approach for learning a shape model for organic shapes represented as watertight surfaces. We use an autodecoder approach to learn a single latent vector per training shape. However, at model-fitting time, we allow the latent vectors to be different at each vertex The key ingredient is that we enforce consistency of the latent vectors across the triangulation but not on the vertex 3D locations. This enables us to learn the model from a relatively small training set while giving the necessary flexibility to model complex 3D shapes without over-smoothing. and Consistent Site Modeling". In: Congress  
 
 Appendix: We investigate whether DALS actually uses multiple different latent vectors during fitting. We use DALS to reconstruct a liver from unoriented points as in the smooth transition between latent vectors over 805 the surface. For the results in the paper, the number of fitting 818 iterations was set high enough to ensure convergence. 819 Specifically, for section 4.1, DASM+R used 2.9 sec-820 onds, DUAL-MLP used 1.3 seconds, and DALS used 821 2.4 seconds.We investigate the effect of the template mesh resolu-824 tion. Again, we use the same setup as in section 4.1 825 and vary the resolution of the template. Specifi-826 cally, we use a subdivided icosahedron subdivided 827 n = 2, . . . , 6 times. Note that we use the same model 828 for all n, which was trained using an icosahedron 829 Regarding L dir , the reconstruction metrics follow a convex shape while the mesh quality (more or less) monotonically improve as the regularization strength is increased-although only slightly. I.e., if the strength is too small, the fitting suffers as the latent vectors have too much freedom and the fitting is less stable. If the strength is too large, the latent vectors have too little freedom to fit the target, but all latent vectors will have more similar values. As the network is trained to generate high quality meshes in the single latent vector case, this then translates to slight higher mesh quality.Regarding L reg , it has a limited but measurable effect on the metrics for smaller values. As it becomes to large, however, it seems to become more unstable and significantly worsens the reconstruction quality as well as causing more self intersections. The mesh quality, on the other hand, goes up since we are putting more weight on it.Based on this, our recommendation is to leave L reg at 0.001 or similar and focus hyperparameter tuning on L dir .  


Decision: The paper proposed a novel method for 3D shapes reconstruction by learning a mesh deformation. Reviewers found the proposed approach novel and conceptually interesting with solid experimental results and reproducibility.  They also commend great paper presentation and high quality of the manuscript. 
Overall, this paper makes a significant contribution and is recommended for acceptance.

Reviews: review: title: title: Interesting approach, method details are not clear enough, evaluated only on one dataset
paper_summary: paper_summary: This paper introduces an active latent shape representation model for shape reconstruction by deforming a triangulated sphere to match the target shape. The proposed approach consists of two stages: training and fitting. While the methodology is intriguing, certain aspects lack clarity and require further elaboration.
main_review: main_review: Thank the authors for the rebuttal, which clarified a couple of things in the paper. I am now leaning toward accepting the paper.
strength_weakness: strength_weakness: 1. Key details of the proposed method are insufficiently explained. For instance, in lines 153-154, the choice of regularization is attributed to the Laplacian regularization being "too smooth," but this might be influenced by the regularization term's weighting. The reasoning behind this choice remains unclear. Additionally, it is ambiguous whether the fitting process is performed in a single step or involves multiple iterations to optimize Eq. 4.

2. The weighting terms in Eq. 4 appear to significantly influence the final results, particularly in terms of smoothness, yet the sensitivity of these terms is not adequately explored. The approach may be sensitive to changes in such hyperparameters. Given the noticeable differences in hyperparameter values between Sections 4.1 and 4.2, the authors should provide a more detailed discussion of how these hyperparameters affect different tasks.

3. The organization of the paper could be improved. For example, the task-specific losses discussed in Section 3.2 should be consolidated into a dedicated section rather than being spread across the experimental sections.

4. It is unclear how the proposed method distinguishes itself from existing literature that uses multiple latent vectors for shape representation. Including an introductory section explaining how shape representation pipelines are typically structured could provide helpful context. Additionally, experimental details (e.g., GPU setup, learning rate) could be moved to an appendix to save some space.

5. To enhance reproducibility, the authors should demonstrate their method on at least one additional dataset, as the current evaluations are all based on a single dataset.
title: title: Powerful and solid contributions to 3D reconstruction
paper_summary: paper_summary: The 3D reconstruction problem is one of the important application cases in computer vision and medical bio-information processing. This paper makes new contributions to the field of 3D reconstruction, while also following the latest trends in the field. One important strategy for conventional 3D reconstruction is the active contour/surface model. However, since these methods directly model the observed 3D shape, it is not easy to deal with noise and regularization during training. Therefore, the proposed method first proposes a new method that applies regularization to the latent variables of the auto-decoder model. Then, in 3D reconstruction, it shows a method for expressing more flexible 3D shapes by utilizing multiple vectors (i.e., a mechanism that can explicitly combine multiple clues that are effective for multiple observed 3D shapes) while satisfying spatial consistency from the auto-decoder obtained in the learning process. As a result, learning can be done efficiently with a small number of samples, and it is well suited to real-world problems in medical bioinformatics processing, such as the 3D shape of the liver used in experiments.
main_review: main_review: My slight concerns have all been addressed in the authors' response. In particular, the authors have provided supplementary material to explain the technical details so that this paper can be a self-contained explanation.
strength_weakness: strength_weakness: I have not been able to find any fatal flaws in this paper. Thank you for sharing your great contribution. The following are some of the simple questions I had while reading the paper. If there are any items that are important for a deeper understanding of the significance of the proposed method, I would be very grateful if you could let me know.

- I certainly think that the method of expanding the training data 100 times using PointWolf [ICCV2021] is appropriate. On the other hand, is such data augmentation really necessary even for relatively static organs like the liver? For example, for organs like the cardiac heart, which constantly undergo major changes in shape, I have a hunch that such data augmentation would work effectively. However, in the case of the liver, I couldn't easily imagine whether data augmentation would work effectively. I wonder if this was an effective process that was tested experimentally. If the authors had investigated this area in their preliminary study, it would be very useful for readers if they could report it briefly in the supplementary materials, etc. (This is not a critical point, so additional experiments, etc. are not necessary.)

- I found it somewhat difficult to follow the explanation of $L_{reg}$ (this seems to be a classic and well-known method, so it may be because of my lack of prior knowledge). As a result, I was only able to understand it after referring to the original paper [Bhatia&Lawrence, 1990]. However, I also understand that it is difficult to explain it further due to the page limit. If the authors could explain this point using simple illustrations in supplementary materials, etc., it may be possible for people like me with a lack of prior knowledge to follow it immediately.
title: title: Review "Deep Active Latent Surfaces for Medical Geometries"
paper_summary: paper_summary: The paper proposes a novel auto-decoder method to represent 3D shapes by training a neural net that receives a latent vector and a point on a standard 3D sphere as input and predicts the offset from the 3D sphere vertex location to the actual vertex location for the shape to be modeled. The neural net is a simple 3-layer MLP.

The latent vector is supposed to represent the entire 3D shape and is, therefore, forced to be the same across vertices during training.

At inference time, the optimization is to find latent vectors, separate for each vertex, such that the resulting predicted shape conforms to some loss, such as the agreement with 3D point clouds or human-generated segmentations.
main_review: main_review: The reviewers addressed my questions fully and provided additional insight. I stand by the evaluation.
strength_weakness: strength_weakness: If the paper has any weakness, then that it contains so much content that it is, sometimes, hard to follow for non-experts in the specific subfield the paper adresses. As such, I would rather like to pose some questions which were not entirely clear to me when reading:
* in line 125, how is x represented? Is it in 3D coordinate space or in angular space?
* is the output of D (i.e. the offset mentioned in line 125) just a scalar, regulating how far the corresponding vertex is from the center of the sphere or does it need to be a 3D vector?
* How were the layer sizes in line 140 found?
* The Fitting Scheme section (3.2) could, I think, profit from an introductory sentence introducing the problem: namely finding a matrix of latent variables Z, such that the predicted shape via D solves some downstream task, such as fitting a point cloud.
* In the Fitting Scheme in 3.2, how is Z initialized?