CorticalFlow: A Diffeomorphic Mesh Transformer Network for Cortical Surface Reconstruction
Authors: Leo Lebrat
Rodrigo Santa Cruz
Frederic de Gournay
Darren Fu
Pierrick Bourgeat
Jurgen Fripp
Clinton Fookes
Olivier Salvado
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Deep Learning
Geometric Deep Learning
Regular Surface Recosntruction
Cortical Surface Reconstruction

Abstract: In this paper, we introduce CorticalFlow, a new geometric deep-learning model that, given a 3-dimensional image, learns to deform a reference template towards a targeted object. To conserve the template mesh’s topological properties, we train our model over a set of diffeomorphic transformations. This new implementation of a flow Ordinary Differential Equation (ODE) framework benefits from a small GPU memory footprint, allowing the generation of surfaces with several hundred thousand vertices. To reduce topological errors introduced by its discrete resolution, we derive numeric conditions which improve the manifoldness of the predicted triangle mesh. To exhibit the utility of CorticalFlow, we demonstrate its performance for the challenging task of brain cortical surface reconstruction. In contrast to the current state-of-the-art, CorticalFlow produces superior surfaces while reducing the computation time from nine and a half minutes to one second. More significantly, CorticalFlow enforces the generation of anatomically plausible surfaces; the absence of which has been a major impediment restricting the clinical relevance of such surface reconstruction methods.

CorticalFlow: A Diffeomorphic Mesh Deformation Module for Cortical Surface Reconstruction
Léo Lebrat CSIRO, QUT leb0260csiro.. au
Rodrigo Santa Cruz CSIRO, QUT
Frédéric de Gournay IMT UMRS219 degourna@insa-tool..oo
Darren Fu UQ
fon0220csiro.
Pierrick Bourgeat CSIRO
Jurgen Fripp CSIRO
Clinton Fookes
Olivier Salvado
QUT
CSIRO, Data61.
Abstract
In this paper introduce CorticalFlow, geometric deep-learning model that, given a 3-dimensional image, learns to deform a reference template towards a targeted object. To conserve the template mesh's topological properties, train our model over a set of diffeomorphic transformations. This new implementa- tion of flow Ordinary Differential Equation (ODE) framework benefits from a small GPU memory footprint, allowing the generation of surfaces with several hundred thousand vertices. To reduce topological errors introduced by its discrete resolution, we derive numeric conditions which improve the manifoldness of the predicted triangle mesh. To exhibit the utility of CorticalFlow, we demonstrate its performance for the challenging task of brain cortical surface reconstruction. In contrast current state-of-the-arr, CorticalFlow produces superior surfaces while reducing the computation time from nine and half minutes to one second. More significantly, CorticalFlow enforces the generation of anatomically plausible surfaces; the absence of which has been major impediment restricting the clinical relevance such surface reconstruction methods.
Introduction
The field of 3D shape reconstruction using deep learning techniques has attracted much attention. Recently, a plethora of methods have been developed for problems such as single-view object
reconstruction 22 50 77]. surface generation (27 74. and meshing noisy point clouds 332 42 At
first, these methods solely aimed to retrieve surface meshes geometrically close as possible to the target shape. However, recent applications require generating regular meshes with a known genus, such physics simulation, 3D-printing, and clinical analysis of anatomical surfaces [24] 58 62).
In this direction, three approaches in the literature stand DeepCSR 61. Voxel2Mesh [[]] and Neural Mesh Flow (NMF) [30]. DeepCSR first predicts implicit surface functions and then employs an iso-surface extraction method along with topology correction algorithm obtain genus- zero surfaces without handles or holes. Voxel2Mesh extends the vertex-wise template deformation approach of Wang et al. (7 by optimizing several mesh-smoothing penalty functions. contrast, NMF builds an invertible mapping that enforces topology conservation upon the resolution of an Ordinary Differential Equation (ODE) through sequence of residual blocks called Neural Ordinary Differential Equation (NODE) (9] However, these methods come with several limitations. The topology correction algorithm employed by DeepCSR computationally expensive and blind
Equal contribution
Check our project web-page http::///ebrreggggttoo.ooooorrre
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

towards the anatomical validity of its reconstructions, which can result in implausible corrections and mesh artifacts. On the other hand, Voxel2Mesh and NMF rely on deme-demnnding and vertex- dependent building blocks such as graph convolution that do not scale up well as the number of vertices in the template mesh increases to accommodate complex shapes. In addition, the dynamics learned by the NODE model in NMF can be very complex and may lead to a non-diffeomorphic mapping resulting in self-intersections in the reconstructed mesh.
This paper introduces CorticalFlow (CF), a new geometric deep learning model that smoothly deforms a template mesh towards complex shapes producing high-resolution regular meshes. First, simple 3D convolution neural network predicts a dense 3D flow field from volumetric image with a modest GPU memory footprint. Second, we formulate a tractable mathematical framework to compute diffeomorphic mapping for each vertex by solving a flow ODE. We derive sufficient and comprehensible conditions for meeting the diffeomorphic properties of these transformations. Finally, sequence of these diffeomorphic mappings is composed to produce accurate high-resolution genus-7ero regular meshes.
To evaluate our approach and compare it to existing techniques for regular surface reconstruction, we consider the problem of brain cortical surface reconstruction, which is an essential step for the analysis of brain morphometry in neurodegenerative diseases [18] and psychological disorders [57]. Given a
3D MRI of the brain, the goal to describe the inner and outer surfaces of the brain cortex, which are both homeomorphic to a sphere. Cortical surface reconstruction is challenging given the complexity, high resolution, and regularity required for the predicted meshes. In our experiments, CorticalFlow more accurate than ofthh-of-theart methods, providing an average reduction of 17.38% in Chamfer distance across all cortical surfaces compared to DeepCSR (the second-best performing method in this criteria). In terms of surface regularity, it surpasses NMF or Voxel2Mesh with an average reduction of at least 32.58% of self-intersecting faces while handling template meshes with many more vertices. It is also faster and more memory-efficied than all of these competitors.
2
Related Works
2.1 Geometric deep learning for surface reconstruction
Supervised surface reconstruction can be broadly categorized according to the 3D shape representation used to encode the prediction as either volumetric, implicit surfaces representation, novel geometric primitives, or geometric [22].
Volumetric methods predict shapes encoded as 3D grid of voxels containing discretized surface representations such as occupancy [I] and level-sets (46). From this representation, surfaces are obtained using iso-surface extraction methods, such as marching cubes [41] While 3D volumetric processing amenable to a convolutional neural network, the memory requirements are often a limitation to attain high high-resolution reconstructions (it grows cubically with the voxel-grid resolution). To overcome this issue, approaches based on octrees [35 68 72 output resolution from voxel-grid of 323 to 2563. Unfortunately, these approaches sacrifice speed and necessitate the redefinition of standard network operations such as convolution, pooling, and unpooling for this hierarchical data structure. Furthermore, as presented in [61], even this level of resolution, the precision is too coarse to capture the highly curved regions of the cortical surfaces.
have been proposed to increase the
Implicit surface methods alleviate resolution limitations of the volumetric methods by directly pre- dicting surface representations like occupancy (45), signed distance [54 755,, and 3D Gaussians (26) for points with continuous coordinates. This formulation allows synthesizing grids at an arbitrary resolution during inference with an easily implementable local refinement procedure, while training is performed stochastically over small subset of sampled points. Following this approach, Santa Cruz et al. [61] proposed DeepCSR, the first geometric deep learning model for cortical surface reconstruc- tion. Its main limitation is the difficulty to control the topology and mesh quality of the reconstructed surfaces, which hampers atrophy estimation used for neurodegeneratiie disease diagnosis [24 58 62] As result, DeepCSR resorts to computationally expensive topology correction algorithm produce a final cortical surface almost free of artifacts and with a spherical topology.
Methods based on geometric primitives build a surface representation to approximate complex shapes as the union of these primitives. In this category, we can highlight the works of Niu et al. [51) and Groueix et al. [28) which propose to approximate complex object shapes with collection of

---

"cuboids" or "surface patches". Recent works by [10 17 revisit the convex decomposition idea and propose reconstruct complex object shapes by predicting collections of convex parts. The former predicts a set of localized convex polytopes formed by their hyperplane parameters and translation vector, while the latter predicts a binary space partitioning tree to reconstruct the target shape. While very promising in terms of information compactness, these approaches are challenged to generate cortical surfaces due to their varied curvatures, which require a large number of convex parts to produce accurate results.
Finally, geometric methods comprise techniques that allow estimating a resolutsotttion mesh by transforming a known template mesh [52] 53 65 [7]). Following this approach, Wang et al. [7] propose a graph-convolution network predict vertex wise deformations of a spherical mesh while dynamically increasing its resolution with point pooling process. Wickramasinghe et al. [73] extended this model for the reconstruction of smooth anatomical surfaces such as the liver or hippocampus for different image modalities. Topological errors are reduced using three different penalty functions in the loss function. Recently, Gupta and Chandraker (30] leverage NODE blocks [9] to parameterize regular deformations that allow conserving the two-manifoldness property of the input template.
Indeed, the "manifoldness" measures as non-manifold edges or non-manifole vertices, and defined by Gupta and Chandraker [30]. are conserved by a deformable model which is not generating new vertices since those properties are inherited from the template mesh (only the vertices' posi- tions are affected). However, the normal consistency mnnn-mniffodd faces) is only conserved by homeomorphisms which can at most flip globally the faces' normal orientation.
2.2 Generation of diffeomorphic mappings
b.
The reconstruction of regular surfaces from deformable model is a subtle trade off between finding the right pa- rameterization suitable level of regularization during training. The delicacy of this problem illustrated in Figure 1. First, one employ multiple penalty func- tions as Chamfer normal, normal consistency, Laplacian loss, or edge length loss [71 73] It is worth mentioning that without such penalizations, a deformable model will learn non-smooth deformations, which leads irregular meshes (as shown in Figure 1.c). However, those penal- izations simply encourage the reconstructed surface to be regular. The second approach consists of parameter- izing the set of learned deformations; this approach is favored in our paper since it allows stronger theoretical guarantees and A natural framework to generate invertible deformations is to consider a flow ODE 20 69 This framework has been successfully applied in pattern recognition and im- age registration [2] 19]. The main idea is to consider the mapping the solution at time of initial value problem (IVP) of the form,
c.
subject typer-parameter tuning.
Figure 1: Generating regular meshes of the left hemisphere of the brain cortical surface. a. CorticalFlow, b. NMF with global feature descriptor (30. e. UNer without diffeomor- phic parametrization geometric penaliza- tion.
d x)
(8)) with (000;) x) =x,
ds
under regularity hypothesis and upon boundedness of its support, and using the Picard-Lidelöf theorem, one can show that a unique solution of this problem exists for т E IR. In addition, the mapping x (s; defines family of diffeomorphisms can be computed through a backward integration.
6 for all time [0. r] whose inverse
When the vector field is constant over time i.e. v: R - 3 Equation ① describes the Stationary If u a time- mareingryigg vector field, the framework described
Velocity Field (SVF) framework [
in becomes the LDDMM model
5
8 64 70)
3

---

This generic framework has been successfully applied within deep learning methods for diffeomorphic
image registration [T5] 47), point-cloud completion (49
single view reconstruction [30 and to
parameterize set of deformations 37
The SVF formulation has been particularly fecund in medical deep-learning registration 15
40 47
761, where the resolution of Equation ( is performed using the scaling and squaring method 22. 3
to predict the displacement of each voxel-center and compute the registered image. However, this technique is not suitable for points that lie on non-regular coordinates. Naively, one can compute these mappings on a dense grid and then interpolate the deformation non-regular coordinates. This simple approach is subject to two main limitations. Firstly, Equation ( has to be solved in our context for millions of grid points where only a few hundred thousand vertices are displaced. Secondly, one cannot guarantee the invertibility of such a mapping with linear interpolation and one cannot compose provably several of those approximated mappings.
In 30 49], the problem solved using a black-box neural ODE (9) and by learning a neural vector field v. Despite allowing learning a time-dependent vector field, this approach has shown its limitations in our targeted application. Cortical surfaces are unique to each individual; indeed, the cortical folding patterns are similar to a fingerprint +4 and constitute a distinctive biometric for each individual. Moreover, we observe that the classical approach, which consists of conditioning the neural ODE on global feature descriptor of the input image, fails to provide satisfactory results for cortical surface reconstruction (see Figure 1.b. and the supplementary material). Instead, one has to equip each moving vertex with a local feature descriptor of the input image, limiting the number of vertices of the resulting mesh.
Our work lies the intersection of these methods. We propose to extend the SVF framework for points lying in real-coordinates, with particular care given to the numerical affordability of the ODE solver. We define a multi-scale approach, that the final deformation the result of the composition of three successive deformations that allow to approach more complex mappings and alleviate the limitations of the one vector field SVF framework [23 43) This framework is memory efficient, theoretically tractable, and can seamlessly handle large template meshes (z2 450k vertices).
3 Method
CorticalFlow (CF) is multi-level deep learning architecture composed of several Diffeomorphic Mesh Deformation (DMD) modules. It takes as input a 3-dimensional Magnetic Resonance Image (MRI) of patient brain denoted (tensor of dimensions H x W D) and a template T (where i represents the degree of refinement of the template). CorticalFlow outputs the surface representation of an anatomical substructure by composing stackable diffeomorphic deformations generated by DMD modules. CorticalFlow with & deformations (CF*) can be written using the following recurrence,
CF2, 0, (I) = (MDe),er), (I) Ti))
CF, T++1) for i 2 1, CF B.4. T++1) DMNet i+1 (UT ... U, I), ..,
with A B the channel-wise concatenation of the tensors A and B and where Uj denotes the output of the k-th UNet", parameterized by 0g..
In our paper describe CF², a version of CorticalFlow with three stages where each stage is learned successively. CorticalFlow is trained in a supervised fashion, given a dataset D composed of pairs of MR-image I and triangle mesh S representing cortical structure and for i E (1, optimize the following objective,
arg min
Σ (LS)
C(CF') e, (T)
As training loss C(.,.), we minimize the mesh edge loss and Chamfer distance computed on point clouds of 150k points sampled from the predicted and ground-truth surfaces using random uniform sampling. The implementation of these losses and sampling algorithm are provided in the PyTorch3D library [56).

---

3.1 DMD Diffeomorphic Mesh Deformation module
The introduction of a Diffeomorphic Mesh Deformation module (DMD) is driven by the following classification of surfaces in 3 dimensions:
Theorem 3.1. Suppose that B is smooth closed manifold of dimension 2 embedded in RR. Suppose that ④ : |0, r| x RR - TR' is a family of homeomorphisms (continuous map such that for each the mapping + (L,, ) is bijective with continuous inverse), with (00; x) x. Then, for each the homotopy classes of and BB (t) {1 y).y y B3 are the same.
This theorem means that if B is sphere, the surface (t)(() is of genus 0 with self-intersection.
Existence and uniqueness of solution to the continuous problem
The DMD generation of diffeomorphic mappping relies on the resolution of continuous flow ODE. For that purpose, let x E Ω + v(x) E R be constant over time vector field supported on the MRI space ! and obtained by tri-linear interpolation of a feature map U R H XTxxxxxx3 Suppose that v = 0 on An The image origin is denoted by O = (01, 02, 0a) and denote by d; the interpolation spacing in the i-th direction such that (0n, or + di (H 1)] x [02, 02 d2(W (W 1)] (os, 03 dD( - -)).
Theorem 3.2. Existence and uniqueness of the solution. Define t through the autonomous ODE,
db{(s; x) x)), with + =
(4)
ds
Then is uniquely defined on Q, is Lipschitz and for each t, the mapping x &tt, x) is bijective with Lipschitz inverse. The proof of this result can be found in the supplementary material.
Being Lipschitz is more difficult to achieve than being merely continuous. Less formally, Theorem 3.2 ensures that if B is smooth, then for all t, the surface 4y (t) may, in the worst case, have kinks. fh only continuous and not Lipschitz, then the surface Pg (t) might have cusps that are more irregular than kinks. Note well that the choice of the interpolation technique used to generate is pivotal since it drives the regularity of the right-hand hand side of Equation (4) Indeed, Lipschitz'' 's constant boundedness allows the definition of solution the continuous problem. More importantly, and as described in the next section, it rules the step-size to use for obtaining a stable numeric method.
Numerical resolution of ODE
The DMD module solves for each vertices' position a flow ODE defined in Equation ( defined by v the numeric approximation of & by explicit forward method, the invertibility of this discretisation is given by the following theorem
Theorem 3.3. Let v be L-Lipschitz. Define as the Forward Euler approximation,
(h x.) = x. + hw(x).
(5)
is a Lipschitz homeomorphism for each h < L-1
As result, in combination with Theorem 3.2 the surface Vyn (h) is smooth as long as hL 1. We note that the stability condition h. < 1 is commonly used in Computational Fluid Dynamics [39.
Less formally, this estimate tells us that the less regular (high gradient) is, the smaller the integration step should be chosen.
Proof. Let < - and denote f + x + he(x). Then f is a Lipschitz mapping and it is sufficient to prove the bijectivity of f. The injectivity stems from, for all x # y,
[s(x)) s())I 2 || vll h))e(())))))))))))))()))))) r(v) 2 || v - hT). > 0.
The surjectivity comes from the fact that for each y, the mapping 9 + y- kv(x)) is hL-Lipschitz with hL < 1, hence is a contraction. By a fixed point theorem, g admits unique x= solution to g(x*) = x², equivalently y fxx)). Hence, is surjective.
5

---

UNet,
UNetz
UNetz
Output Mesh
Image
Flow Field
Flow Field 2
Flow Field 3
DMD
DMD
DMD
Template Mesh
Figure 2: CorticalFlow architecture for the cortical surface reconstruction problem. Following the notations of Equation (2). UNete, implements a UNet, layer and UNer implements UNetz layer. The architecture of UNet, and UNet, are described in the supplementary materials.
Theorem 3.3 ensures that (h) is non- intersecting manifold. Unfortunately, there exists another layer of numerical approx- imation that prevents V to have the desired topological properties. Indeed, sup- pose that B is sphere, one never computes V aB (h), but triangulates sphere with tices Vi, and evaluates the image of the vertices using Algorithm [ A new mesh is formed by using the image of the ver- tices and the original connectivity. Since the perturbed edges of the mesh are not the images of the original edges by the For- ward Euler scheme, the resulting mesh may self seff-intersect (we refer interested readers to Section 2.3 of our supplementary material). Notwithstanding this limitation, we use the rule of thumb hL < and check that for all considered examples, and we have hL < 1.
Algorithm DMD module pseudo-code
Input: U [B
Discrete flow T with vertices (Vi)iet..m b Triangle mesh
Input:
Input:
Number of integration steps Output: Updated (Vi)ie1..m
N'
h+
Ensure: h < + for i E [1,m] ml do
for j E [1. [1. n| do
V/ (h, Vi)
end for end for
3.2 Network architecture
As shown in Figure 2 CorticalFlow consists of chain of three deformations. Note that more deformation modules could used, we focus on three modules to have a fair comparison with existing techniques. The first deformation module receives input volumetric image and outputs flow vector field with the same dimensions using UNet-3D [59]. This discrete flow vector field is integrated by the DMD module compute smooth deformations explained in Section 3.1 The subsequent UNet-3D receives as input the image and the flow vector fields predicted by the previous deformation modules. The set of resulting mappings composed produce the final mesh.
For training CorticalFlow, we adopted sequential approach where we train one deformation module at time while freezing the weights of the others. We train the first deformation with low-resolution template (Ti) of 30k vertices and a UNet-3D architecture with four down/up-sampling levels. To train the second and third deformations, we increase the resolution of the template mesh to 135k and 435k vertices, for Tz and Ts respectively, and reduce the UNet-3D hown/up-samplinn levels to two. These networks are respectively labeled as UNet, and UNet, in Figure 2 while their layer details described in our supplementary material. The choice of the architecture depth motivated by the numerical conditions on the integration step-size derived in Section 3.1
To verify the conditions of Theorem 3.3 it is essential to mention the use of template meshes with different resolutions and different UNet architectures. Indeed, the first block has to provide large deformation resulting in high |w|] To keep the Lipschitz constant small, one observes that the use of a low-resolution template with 30k vertices along with a deeper convolutional architecture forces the UNet to recover only coarse details and thus produces a flow vector field with small Vv. During the second and third deformation, more details and higher resolution folds can be learned,

---

with templates composed of 135k and 435k vertices, respectively (see the ablation study presented in Section 1.1 of our supplementary material). We empirically verify that this hierarchy of deformations was beneficial for producing flow vector field e with a small Lipschitz constant. This multi-step approach allows attaining up to 14.5 times less self-intersection in comparison with Neural Mesh Flow (see T.
To generate the template mesh, we take the convex hull of all surfaces contained in the training dataset and remesh them uniformly using JIGSAW [2I] To achieve a different order of refinement, we use the midpoint subdivision algorithm implemented in MeshLab [12]. Model syper-parameters and further implementation details are provided in the supplementary material.
4 Experiments
We benchmark Cor and other existing deep learning techniques on the cortical surface reconstruction problem. The goal is to estimate geometrically accurate and topologically correct triangular meshes for the inner and outer cortical surfaces from a given MRI. Like previous works [14 16 31 34 38 61), these surfaces are further divided into the left and right brain hemispheres. See below a summary of the dataset, evaluated methods, and metrics used in our benchmark, in addition to the detailed discussion of the results summarized in Table
Oeter
Surface
Kight Chuzer Surince
CHimm)) 1
HIXvom)
CHN
+SSIF
DSC1
VS1
CHisai 1
HDDmmn' 1
CHN
15 SIF
DSC
VS-
CorticalHow
(303)
0.307
10230
03277
0200
11315
1,20
11000
1.148 /40/3 GB
(40,008)
(=0.040
(40.000)
[#0.669)
(=0000)
(46.006)
(20.01)
[40.045)
[60006]
(40.820)
[60.00)
(30.00)
DSepCSR
0.925
0.98:
0.912
0.955
0.33
0.000
C.003
(-0135)
(10011)
(+6.005)
NMP
0
1.772
1.588
0.811
1,510
6.983
42 14.331 GB
(20,200)
(=11116
(20000)
(40.545
(=000G)
(40,088)
(60.177)
(40.301)
(50.088
(20.202)
(50.000)
(20200))
QuickNAT
5.722
2,582
0.511
0.030
0,966
0.775
4.60
2,00
0,800
0.000
0.954
0
13.00 003 see 9,627 GB
(+1.971)
(=1001
(240224)
(60,000)
(=0100)
(40.013)
(62250)
(61.00))
(5M1011)
Vose22Mesh
7,185
3,80
0.721
0.557
0,895
03576
7.461
3150
0.77
0,833
0.888
20340 wne GB
(40.500)
(=0218
(30006))
(40.250)
(=0000)
(10.013)
(20771)
(4002277
(10.0LT)
(20.200)
[60.00))
(300000)
Lett Inner Surface
Right Inner Surface
CHirm)) 1 HiXoom)
CHN
SIF
DSC1
VSt
CHimao 4
HD 1
CHN
SIF
DSC +
VS
CotticalFlew
0.233
0,783
0392
BUST
0,300
CRR
1.14 105522/33000 33071 GB
(40.006)
(=00600
(40000)
(60.060)
(=000))
(10.010)
(5008)
(60.009)
(500007)
(20.200
(60.05)
(30.00)
DSepCSR
0.658
0.966
0,963
0.966
0.694
0.500
0.566
0.000
0.086
(40000)
[60.00))
(40.013)
NMF
1.404
1.477
0,025
14844
14453
0,553
0,435
0.27
C.064
GB
(40.185)
(-0110)
(-0011)
[+0.105)
(-000)
(+0.015)
(60158)
(60.125)
(+0.000)
(10.188)
[H0006)
(90.010)
QUIcKNAT
2,570
1.595
6,906
0.000
0,905
0921
2.285
1.788
0.68
0,000
0,908
0.951
13 003 see 19627 GR
(#1.80))
(34)
(2002))
(+0000))
(#1.9))
(25)7155
(=0000)
[SQUUTT
Mosc
5,005
3,46
0,713
D.G21
0,550
6,172
3.540
0.707
0,761
0.542
2o.40.az//334000 GB
(40.467)
(=02477
(-00017)
[#0.22))
(=0.000)
(#0.014)
(20.71))
(400272)
(5G.OLT)
(-0.2771)
[40.0L0)
(20.018)
Table I: Cortical Surface Reconstruction Benchmark. Consider the evaluation metrics: chamfer
distance (CH), Hausdorff distance (HD), chamfer normals (CHN), percentage of self-intersecting
faces (%SIF), Dice Score (DSC), and Volume Similarity (VS). | indicates smaller metric value is
better, while 1 indicates greater metric value is better. We also report the inference runtime and GPU memory footprint required by the compared algorithms.
Dataset. We used the same MRIs, pseudo-ground-trutte surfaces, and data splits as (6]. This dataset consists of 3,876 MRI images extracted from the Alzheimer's 's Disease Neuroimaging Initiative (ADNI) (36 and their respective pseudo- sonudd--roun--ruute surfaces generated with the FreeSurfer V6.0 cross-sectional pipeline (25). We train all methods on the training set ( 60%) until their losses plateau on the validation set (+= 10%) and report their performance on the test set (ss 30%). We refer the reader to [61] and our supplementary material for full details on the dataset.
Evaluated Methods. We compare CorticalFlow to the following methods: DeepCSR 6 Voxel2Mesh (73. NMF (30, and QuickNAT[I [60]. As discussed in Section DeepCSR the state-of-the-ard art geometric deep learning model for cortical surface reconstruction, while Voxe12Mesh is deformation- based model proposed to retrieve generic anatomical surfaces from volumetric medical images like MRIs and CT scans. Differently, NMF is a deformation-baan based model for single-view object reconstruction from 2D images. To adapt this model to our task whose input is
DSepCSRR official implementation retrieved from http:p//sssttooeeec...c..aaap/pp
r repos/ peeps//eeeparr
Vexee22Mssh official implementation retrieved from httppp/s/gggtta..oooooo/aae////a
e NMF official implementation retrieved from httppppppp..pphacooo///a/aaaa/aa
'QuickNAT official implementation retrieved from httppp///gtthhb..cooo//////////t
7

---

3D MRI. we evaluate different 3D convolutional network backbones based on UNet [59, ResNet B3], and Hypercolumn [6]] The Hypercolumn backbone provides the best results thanks to its vertex cetexxdeeendent features. As such, this is used as the NMF backhone in our benchmark. At the same time, the results for the other backbones are presented in our supplementary material. We also evaluate baseline composed of state-of-the art brain segmentation model QuickNAT (60] followed with the marching cubes to evaluate the surface. All of these methods were trained and evaluated using a NVIDIA P100 GPU and Intel Xeon (E5-2690) CPU, except Voxel2Mesh which required NVIDIA RTX 3090 GPU due to its GPU memory requirements.
Evaluation Metrics. We compare these methods for their geometric accuracy and surface regularity, as well as their time and space complexity. As measure of geometric accuracy. we report the standard Chamfer distance (CH), Hausdorff distance (HD), and Chamfer normals (CHN), We compute these distances for point clouds of 200k points uniformly sampled from the predicted and target surfaces. As a measure of regularity, we compute the percentage of self-intersecting faces (&SIF) using PyMeshLab [48]. We also report volumetric overlap metries [67] including Dice Score (DSC) and Volume Similarity (VS) computed on the high-resolution rasterization (4xthe input MRI resolution) of the generated and ground-truth surfaces.
For the time and space complexity of the evaluated methods, we report their average inference time (in seconds) and inference GPU memory footprint (in GB) to reconstruct the four cortical surfaces, respectively.
CorticalFlow
DeepCSR
NMF
088
Distance
Figure 3: Predicted outer cortical surfaces color-coded with the distance to the pseudo-ground-ruuth surfaces. Here we present the results for the three best-performing methods in terms of geometric accuracy: CorticalFlow, DeepCSR, and NMF, See our supplementary materials for more examples.
Results & Discussion. In our experiments, we noticed that CorticalFlow produces more geomet- rically accurate surfaces than the other methods. On average, it presents better geometric metrics across all the cortical surfaces. In addition, as shown in Figure 3. CorticalFlow errors are smaller (S 0.2mmm) and evenly spread across the surface compared to the other methods. In contrast, NMF and DeepCSR can present substantial errors (2 1mm). The former has its error spread across the entire surface, while the latter can produce large errors at specific regions.
CorticalFlow is also more robust than the competitors presenting lower error variation across individ- uals as suggested by the smaller standard deviation of the geometric metrics computed. Interestingly, CorticalFlow is also more robust to MRI artifacts even when the sseudo-ground-teeg poor quality. For instance, in Figure CorticalFlow predictions are still plausible for blurry input MRI while FreeSurfer fails significantly to generate appropriate surfaces for the same input. These examples support our claim that a regular parametrization allows us to reduce non-plausible and non-diffeomorphii predictions that our model cannot learn by construction.
surface has
CorticalFlow also generates triangular meshes with better properties than the evaluated methods. Compared to the deformation
methods NMF and Voxe12Mesh, CorticalFlow predicted meshes are genus-zero surfaces and present a lower percentage of self-intersectie cortical surfaces). Figure presents examples of self-intersecting faces produced by CorticalFlow, which are contrasted with the NMF predicted mesh for the same input MRI. The implicit-surface
faces (mainly for the inner
8

---

Figure 4: Slices of blurry MRI and the outer surface delineation generated by FreeSurfer V6 (yellow contour) and CorticalFlow (blue contour). Orange circles highlight blurry MRI regions, green circles highlight FreeSurfer's underestimated areas, while red circles highlight non-plausible predictions avoided by CorticalFlow thanks to the diffeomorphism of its predicted deformations.
CorticalFlow
NMF
Inner Surface
Outer Surface
(a)
(b)
Figure a Predicted cortical surfaces by CorticalFlow and NMF with eeff-intersecting faces highlighted in red. b) Significant mistakes generated by the topology correction algorithm used in the DeepCSR method.
based DeepCSR method does not produce a single self-intersecting face since it employs computa- tionally expensive post- processing routines like topology correction 4 and iso-surface extraction. However, these post-processing routines do not take into account the input MRI which can generate non-plausible corrections exemplified in Figure 56 Similarly, the voxel-wise segmentation baseline (i.e., QuickNAT) is free of self-intersecting faces, but it does not produce genus-zero surfaces. Indeed, QuickNAT's predicted surfaces are composed multiple connected components presenting many handles and holes which not acceptable for the purpose of cortical surface reconstruction. Some examples of QuickNAT reconstructed cortical surfaces are presented our supplementary material. Therefore, we argue that CorticalFlow is the method of choice reconstruct regular surfaces from volumetric images.
the output mesh previously observed in Segonne et al. [63 and
Due to its elemental construction (three UNet-3D backbones and an interpolation module for the integration), CorticalFlow remains highly efficient. It has minimal GPU memory footprint and faster inference runtime while handling larger surfaces with more vertices both during training and inference. This feature allows its deployment low-end computers and embedded devices which is pivotal in many scenarios across public health and for commercialization of affordable Al healthcare solutions B (55)
Finally, as a by-product of CorticalFlow's deformable and diffeomorphic nature, one can seamlessly obtain sub-voxel resolution segmentation by applying voxelization engine. This can capture variations below the image resolution while traditional segmentation methods 60) are restrained from working at the image resolution (see Figure 6a). Additionally, essential component of

---

-030
(a)
(b)
QuickNAT segmentation (inner surface volume in pink and outer surface volume in yellow) and CorticalFlow generated surfaces (inner surface in light blue and outer surface in dark blue). Note that the highlighted artifacts in QuickNAT segmentation can be easily surpassed by CorticalFlow thanks its continuous representation of the region of interest. b Outer cortical surface color-coded with the curvature of its cortical foldings. Our method allows us to map back this measure and any other biomarker to the input template using the composition of the inverse deformations
Figure 6:
computational neuro-anatomy consists of computing local shape descriptors for different individuals
and transferring them to the same reference space using conformal mappings [29] [66. For the
proposed model, one can efficiently compute the inverse transformation t shown in Figure 6b for the surface curvature descriptor.
5 Conclusion
This paper introduces CorticalFlow a geometric deep learning model for efficiently reconstructing high-resolution, accurate, and regular triangular meshes from volumetric images. We develop lightweight neural network to predict dense 3D flow vector field from a volumetric image. Then, describe new Diffeomorphic Mesh Deformation (DMD) module, which is parameterized by set of diffeomorphic mappings. This includes the derivation of numerical conditions for recasting the continuous flow ODE problem into efficient discrete solver. Finally, we extensively verify that the proposed model achieves state-of taite-of-theaate tithhe-art art performance in the challenging brain cortical surface reconstruction problem. This benchmark reveals that CorticalFlow is more accurate and, by construction, more robust to image artifacts providing anatomically plausible surfaces. Thanks also its low space and time complexity, the proposed method can facilitate large-scale medical studies and support new healthcare applications.
6 Compliance with Ethical Standards
This research was approved by CSIRO ethics 2020 068 LR.
Acknowledgements
This work was funded in part through Australian Department of Industry, Energy and Resources CRC-P project between CSIRO, Maxwell Plus and I-Med Radiology Network.
10

---

References
[1] Vincent Arsigny. Processing data in lie groups: an algebraic approach. application to non- linear registration and diffusion tensor mri. PhD thesis, Citeseer, 2004.
[2] Vincent Arsigny, Xavier Pennec, and Nicholas Ayache. Polyrigid and polyaftine transformations: a novel geometrical tool deal with non-rigid cpppmmatttoonsapappaatioooo the registration of histological slices. Medical image analysis, 96):507-523, 2005.
[3] John Ashburner. A fast diffeomorphic image registration algorithm. Neuroimage, 38(1):95-113,
2007.
[4] Pierre-Louis Bazin and Dzung L Pham. Topology correction of segmented medical images using fast marching algorithm. Computer methods and programs in biomedicine, 88(2): 182-190. 190, 2007.
[5] M Faisal Beg, Michael I Miller, Alain Trouvé, and Laurent Younes. Computing large deforma- tion metric mappings via geodesic flows of diffeomorphisms. International journal of computer vision, (1): 2005.
[6] Marcel Berger and Bernard Gostiaux. Differential Geometry: Manifolds, Curves, and Surfaces: Manifolds, Curves, and Surfaces, volume 115. Springer Science & Business Media, 2012.
[7] Vincent Camion and Laurent Younes. Geodesic interpolating splines. International workshop on energy minimization methods in computer vision and pattern recognition, pages 513-527, Springer, 2001.
[8] Nicolas Charon. Analysis of geometric and functional shapes with extensions of currents: applications to registration and atlas estimation. Phl) thesis, École normale supérieure de Cachan-ENS Cachan, 2013.
[9] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
[10] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 45-54, 2020.
[11] Christopher Choy, Danfci Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference Springer, 2016.
on computer vision, pages
[12] Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. MeshLab: an Open-Source Mesh Processing Tool. In Vittorio Scarano, Rosario De Chiara, and Ugo Erra, editors, Eurographics Italian Chapter Conference. The Eurographics Association, 2008. ISBN 978-3-905673-68-5 doi: 10.23 102211//eeal MtalCha/itaiaanChapCoo2202 136.
[13] Clarissa Z Cooley, Patrick C McDaniel, Jason P Stockmann, Sai Abitha Srinivas, Stephen Cauley, Monika Sliwiak, Charlotte R Sappo, Christopher F Vaughn, Bastien Guerin, Matthew Rosen, et al. A portable brain mri scanner for underserved settings and point-of-cant imaging. arXiv preprint N0KV::2004.118833 2020.
[14] Robert Dahnke, Rachel Aine Yotter, and Christian Gaser. Cortical thickness and central surface estimation. Neurolmage, 65:336-348, 2013.
[15] Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu. Unsupervised learning of probabilistic diffeomorphie registration for images and surfaces. Medical image analysis, 57: 226-236, 2019.
[16] Anders M Dale, Bruce Fischl, and Martin I Sereno. Cortical surface-based analysis: segmen- tation and surface reconstruction. Neurolmage, 9(2):179-194, 1999.

---

[17] Boyang Deng, Kyle Genova, Soroosh Yazdani. Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 31-44, 2020.
[18] An-Tao Du, Norbert Schuff. Joel H Kramer, Howard J Rosen, Maria Luisa
Katherine Rankin, Bruce Miller, and Michael W Weiner. Different regional patterns of cortical thinning in alzheimer's s disease and frontotemporal dementia. Brain, 130(4): 159:::11111111 11911199661991 1166, 2007.
[19] Paul Dupuis, Ulf Grenander, and Michael I Miller. Variational problems on flows of diffcomor- phisms for image matching. Quarterly of applied mathematics, pages 587-600, 1998.
[20] David G Ebin and Jerrold Marsden. Groups of diffeomorphisms and the motion of incom- pressible fluid. Annals of Mathematics, pages 102-163, 1970.
[21] Darren Engwirda and David Ivers. Off-centre steiner points for delaunay-refinement on curved surfaces. Computer-Aided Design, 72:157-171, 2016.
[22] George Fahim, Khalid Amin, and Sameh Zarif. Single-view 3d reconstruction: A survey of deep learning methods. Computers & Graphics, 94:164-190, 2021.
[23] Jean Feydy. Analyse de données géométriques, au delà des convolutions. PhD thesis, Université Paris-Saclay, 2020.
[24] Bruce Fischl, Martin I Sereno, and Anders M Dale. Cortical buseece-bbee analysis: Ii: inflation, flattening, and a surface-based coordinate system. Neuroimage, 9(2):195-207) 1999.
[25] Bruce Fischl, David H Salat, Evelina Busa, Marilyn Albert, Megan Dieterich, Christian Hasel- grove, Andre Van Der Kouwe, Ron Killiany, David Kennedy, Shuna Klaveness, et al. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33(3):341-33) 335, 2002.
[26] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154-7164, 2019.
[27] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning predictable and generative vector representation for objects. In European Conference on Computer Vision, pages 484-499. Springer, 2016.
[28] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-måché approach learning 3d surface generation. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 216-224, 2018.
(29) Xianfeng Gu, Yalin Wang, Tony Chan, Paul M Thompson, and Shing-Tung Yau. Genus zero surface conformal mapping and its application brain surface mapping. IEEE transactions on medical imaging, 23(8)9949958, 2004.
[30] Kunal Gupta and Manmohan Chandraker. Neural mesh flow: 3d manifold mesh generation via diffeomorphic flows. In Advances in Neural Information Processing Systems, volume 33, pages 1747-1758, 2020.
[31] Xiao Han. Dzung L Pham. Duygu Tosun, Maryam E Rettmann, Chenyang Xu, and Jerry Prince. Cruise: cortical reconstruction using implicit surface evolution. Neurolmage, 23(3): 997-1012, 2004.
[32] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: A self-prior for deformable meshes. ACM Trans. Graph., 39(4), 2020.
[33] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] Leonie Henschel, Sailesh Conjeti, Santiago Estrada, Kersten Diers, Bruce Fischl, and Martin Reuter. Fastsurfer-a fast and accurate deep learning based neuroimaging pipeline. Neurolmage, page 117012, 2020.
12

---

[35] Christian Häne, Shubham Tulsiani, and Jitendra Malik. Hierarchical surface prediction for 3d object reconstruction. In 2017 International Conference on 3D Vision (3DV), pages 412-420, 2017. doi: 10. 109/90V.2211..0000.
[36] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L. Whitwell, Chadwick Ward, et al. The alzheimer's s disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Reso- nance Imaging, 27(())685 2008.
[37] Chiyu Jiang. Jingwei Huang, Andrea Tagliasacchi, Leonidas Guibas, et al. Shapeflow; Learnable deformations among 3d shapes. arXiv preprini arXiv:2006.07982. 2020.
[38] June Sic Kim, Vivek Singh, Jun Ki Lee, Jason Lerch, Yasser Ad-Dab'bar David MacDonald, Jong Min Lee, Sun Kim, and Alan C Evans. Automated 3-d extraction and evaluation of the inner and outer cortical surfaces using a laplacian map and partial volume effect classification. Neurolmage, 27(1):210-221, 2005.
[39] Petros Koumoutsakos, Georges-Hene Cottet, and Diego Rossinelli. Flow simulations us- ing particles-bridging computer graphics and cfd. In SIGGRAPH 2008-35th International Conference on Computer Graphics and Interactive Techniques, pages 1-73. ACM, 2008.
[40] Julian Krebs, Hervé Delingette, Boris Mailhé, Nicholas Ayache, and Tommaso Mansi. Learning a probabilistic model for diffeomorphie registration. IEEE transactions on medical imaging, 38 (9):2165-4)) 2176, 2019.
[41] Thomas Lewiner, Hélio Lopes, Antônio Wilson Vieira, and Geovan Tavares. Efficient imple- mentation of marching cubes' cases with topological guarantees. Journal of graphics tools, (2):1-15, 2003.
[42] Minghua Liu, Xiaoshuai Zhang, and Hao Su. Meshing point clouds with predicted intrinsic- extrinsic ratio guidance. In European Conference on Computer Vision, pages 68 84. Springer, 2020.
[43] Marco Lorenzi and Xavier Pennec. Geodesics, parallel transport & one-parameter subgroups for diffeomorphic image registration. International journal of computer vision, 105(2):111-127, 2013.
[44] J-F Mangin, Denis Riviere, Arnaud Cachia, Edouard Duchesnay, Yves Cointepas, Dimitri Papadopoulos- Orranoo,,-- Paola Scifo, Ochiai, Francis Brunelle, and Jean Régis. framework to study the cortical folding patterns. Neuroimage, 23:S129-SI S138, 2004.
[45] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[46] Mateusz Michalkiewicz, Jhony K Pontes, Dominie Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4743 4752, 2019.
[47] Tony CW Mok and Albert Chung. Fast symmetric diffeomorphic image registration with convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4644 4653, 2020.
[48]
Alessandro Muntoni and Paolo Cignoni. PyMeshLab, January 2021.
[49] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Occupancy flow:
4d reconstruction by learning particle dynamics. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 5379-5389, 2019.
[50] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volu- metric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515,
2020.
13

---

[51] Chengjie Niu, Jun Li, and Kai Xu. 1m2struct: Recovering 3d shape structure from a single rgb image. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4521-4529, 2018.
[52] Junyi Pan, Jun Li, Xiaoguang Han, and Kui Jia. Residual meshnet: Learning to deform meshes for single-vier view 3d reconstruction. In 2018 International Conference on 3D Vision (3DV), pages 719-727. IEEE, 2018.
[53] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[54] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[55] Magdalini Paschali, Stefano Gasperini, Abhijit Guha Roy, Michael Y-S Fang, and Nassir Navab. 3dq: Compact quantized neural networks for volumetric whole brain segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention. pages 438-446. Springer, 2019.
[56] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. urXiv:2007.085000 2020.
[57] Lars M Rimol, Ragnar Nesväg, Don J Hagler Jr, Ørjan Bergmann, Christine Cennema-Notestine Cecilie B Hartberg, Unn K Haukvik, Elisabeth Lange, Chris J Pung, Andres Server, al. Cortical volume, surface area, and thickness in schizophrenia and bipolar disorder. Biological psychiatry, 7166)5552-560, 2012.
[58] Claudia E Rodriguez-Carran Studholme. A framework for in vivo quantification of regional brain folding in premature neonates. Neuroimage, 41(2):462-478, 2008
Pratik Mukherjee, D Vigneron, J Barkovich, and Colin
[59] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer-assisted intervention, pages 234-241, Springer, 2015.
[60] Abhijit Guha Roy, Sailesh Conjeti, Nassir Navab, Christian Wachinger, Alzheimer's Dis- ease Neuroimaging Initiative, al. Quicknat: A fully convolutional network for quick and accurate segmentation of neuroanatomy. Neurolmage, 186:713-722 2019.
[61] Rodrigo Santa Cruz, Leo Lebrat, Pierrick Bourgeat, Clinton Fookes. Jurgen Fripp, and Olivier Salvado. Deepesr: A 3d deep learning approach for cortical surface reconstruction. In Proceed- ings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 806-815, 2021.
[62] Marie Schaer, Meritxell Bach Cuadra, Lucas Tamarit, François Lazeyras, Stephan Eliez, and Jean-Philippe Thiran. A surface-based approach quantify local cortical gyrification. IEEE transactions on medical imaging, 27(2):161-170, 2008.
[63] Florent Segonne, Jenni Pacheco, and Bruce Fischl. Geometrically accurate topology-correctoo of cortical surfaces using nonseparating loops. IEEE Transactions on Medical Imaging, 26(4): 518-529, 2007.
[64] Zhengyang Shen, François-Xavier Vialard, and Marc Niethammer. Region-specific diffeomor- phic metric mapping. arXiv preprint arXiv:/906.0 /006.0006010010 00139, 2019.
[65] Edward Smith, Scott Fujimoto. Adriana Romero, and David Meger. GEOMetrics: Exploiting geometric structure for graph-encoded objects. Kamalika Chaudhuri and Ruslan Salakhutdi- nov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97
of Proceedings of Machine Learning Research, pages 5866-5876, Long Beach, California, USA, 09-15 Jun 2019. PMLR
14

---

[66] Zhengyu Su, Wei Zeng. Yalin Wang, Zhong-Ling Lin Lu, and Xianfeng Gu. Shape classification using wasserstein distance for brain morphometry analysis. International Conference on Information Processing in Medical Imaging, pages 411 413-423. Springer, 2015.
[67] Abdel Aziz Taha and Allan Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging. 15(1):1-28, 2015.
[68] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE International Conference on Computer Vision, pages 2088-2096, 2017.
[69] Alain Trouvé. An infinite dimensional group approach for physics based models in pattern recognition. preprint, 1995.
[70] François-Xavier Vialard, Laurent Risser, Daniel Rueckert, and Colin Cotter. Diffeomorphic 3d image registration via geodesic shooting using an efficient adjoint calculation. International Journal of Computer Vision, 97(2):229 241, 2012.
[71] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu- Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018.
[72] Peng-Shuai Wang, Chun- Yu Sun, Yang Liu, and Xin Tong. Adaptive o-cnn: a patch-based deep representation of 3d shapes. ACM Transactions on Graphics (TOG), 37(6):1-11, 2018.
[73] Udaranga Wickramasinghe, Edoardo Remelli, Graham Knott, and Pascal Fua. Voxel2mesh: 3d mesh model generation from volumetric data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 199-308. Springer, 2020.
[74] Jiajun Wu, Chengkai Zhang. Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum. Learning shape priors for single-view 3d completion and reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 646-662, 2018.
[75] Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems 32, pages 492-502, 2019.
[76] Shuo Zhang, Peter Xiaoping Liu, Minhua Zheng, and Wen Shi. A diffeomorphic unsupervised method for deformable soft tissue image registration. Computers in biology and medicine, 120: 103708, 2020.
[77] Nikola Zubić and Pietro Liò. An effective loss function for generating 3d models from single 2d image without rendering. arXiv preprint arXiv:2 2021.
15


Reviews:
[{"review_id": "rUHhgPpqs_j", "reviewer": "Reviewer_xE8n", "summary": "This paper focuses on the **segmentation of the cortex** in 3D brain MRI volumes. The proposed method combines a **diffeomorphic registration layer** with **volumetric U-Net** modules to iteratively register a (very smooth, sphere-like) template mesh onto the target “level set” of the 3D MRI. Performances are evaluated on the **ADNI dataset,** using FreeSurfer results as pseudo ground truth.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written, with clear attention paid to details. **Figures are well-chosen, informative and of the highest quality.** Performances on the ADNI dataset are highly competitive with a relatively simple architecture, fast run times and strong guarantees on the topology of the output surfaces. The generalisation capabilities of the trained model remain to be investigated, but there is no doubt that the present work is of **significant interest to the field of computational neuroanatomy.**\n\n\nNevertheless, I have two reservations about “warmly” recommending this paper for acceptance at NeurIPS 2021:\n\n1 — The paper brings relatively **few novel ideas** to the field.\n \nIn Section 3.1, the authors present their “Diffeomorphic Mesh Transformer” layer as a fundamental contribution. (I would suggest replacing “Transformer” by e.g. “Deformation” to avoid confusions with “attention” modules.)\n\nHowever, the content of this part of the paper has been **standard in the field of computational anatomy for about ten years.** The generation of diffeomorphisms for images and meshes through the integration of stationary (resp. time-varying) vector fields is usually known as the Stationary Velocity Fields “SVF” (resp. Large Deformations Diffeomorphic Metric Mapping “LDDMM”) framework. \nI refer the authors to e.g. Chapter 5 of Jean Feydy’s PhD thesis (“Geometric data analysis, beyond convolutions”, 2020) for a recent overview of the topic and to e.g. the Deformetrica software for a user-friendly implementation of such methods on 3D meshes.\n\nIn the modern literature, “Diffeomorphic Mesh Transformers” are usually referred to as standard **“SVF exponentials” and used without problems on both images and meshes.** In this context, the main originality of the present paper lies in its network architecture, which is novel but **in line with recent works in the field** — see e.g. “Networks for Joint Affine and Non-parametric Image Registration“, Shen et al, (CVPR 2019) for a similar combination of modules in the case of image registration. \n\nI appreciate the fact that this paper makes a **strong case for a deep diffeomorphic approach** to cortex segmentation, but **I would not overstate the significance of this work:** it does not bring much on the theoretical side, and does not provide an “outstanding” experimental evaluation on e.g. multiple datasets that would unambiguously establish the proposed algorithm as a reference method in the field of neuroanatomy.\n\n2 — I do not know if this paper is a **right fit for NeurIPS.**\n\nThe present work is **clearly solid and useful to practitioners:** I have no doubt that this paper would be very well received at e.g. MICCAI (possibly as an oral), IPMI or in a neuroimaging journal. On the other hand, this work does not really bring forward any deep methodological idea for the wider community in shape analysis and computer vision. \n\nAll in all, this work reads like an **excellent application paper:** I personally enjoyed reading it and believe that “grounded” applicative works **bring a lot of value** to generalist conferences such as NeurIPS. On the other hand, I understand that the organizing committee may also want to keep the conference focused on papers that have a broader appeal to the wider ML community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact of this work have been adequatly addressed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "nzQl68PmcS", "reviewer": "Reviewer_wgwm", "summary": "The paper proposes a template registration approach for meshes that is fast to compute, and that maintains constraints on the topology of the mesh / structure to be segmented. Applications on cortex segmentation in 3D images are demonstrated.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Plus:\n* well embedded into the existing literature\n* well chosen and fair baselines\n* still: good and competitive results, qualitatively and quantitatively\n\nCon: \n* theorem 3.3 assumes v is L-Lifshitz, while v is only a trilinear interpolation (L172) and, hence, not differentiable at the voxel boundaries. This would imply L is infinite... which may be seen as a undesired property. (Could you choose h so that h L <1?) Please comment.\n* a quantitative comparison with \"pseudo-ground truth\" is dangerous - you may only reproduce the FreeSurfer bias better than others do. \n* I might have missed it but can you comment on the loss function? (Two sided chamfer distance?)\n* shapeflow would be worth to be mentioned. I doesn't start with images, so it doesn't have to be added to the baseline comparison, but its concept is similar and should be discussed.\n* to me theorem 3.1 is not obvisous. Would it be worth to present a proof?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: are addressed\n\nSocietal impact: does not apply", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "ghMDQuzFyCz", "reviewer": "Reviewer_EX4h", "summary": "\nThe paper presents a diffeomorphic surface fitting framework. The method takes in a volume and fits a surface by predicting stacked diffeomorphic flow fields, interpolating these to warp a template mesh in a stacked fashion, and warping to obtain the surface required. The authors use a scale-space/pyramid of deformations to obtain accurate deformations, and perform a series of experiments showing that in some metrics the method outperforms existing strategies.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nIt is hard for me to discern the explicit contribution of the papers. The authors review literature and note several methods that are very closely related. As far as I can tell, there are three parts to the method: a volume-to-SVF-network (which of course has been widely done), a SVF-to-mesh Transformer (which I don't fully understand, more below), and a multi-scale approach to this (which of course has been widely done). Perhaps another contribution the authors claim are the various Theorems in 3.1?\n\nMy best estimate is that the authors are claiming 3.1 (the mesh transformer and related theorems) as the main contribution, and perhaps putting the framework together. Unfortunately, although I focused on 3.1 quite a bit, I am having a hard time understanding a couple of things. The authors intersperse Theorems with some practical implementations -- in the sense that the theorems are nice but cannot be executed in practice where approximations are needed, but what the authors do in practice is a bit of a mystery to me still. \n\nI strongly encourage the authors to clarify this section, perhaps separating theory from practical implementation/execution. I assume that lines 185-186 and Eq (4) are what the authors do to approximate the ODE, but it's unfortunately unclear to me how the V_k^i are estimated for k > 0 -- is it simply with tri-linear interpolation on demand? If so, how is this linear approximation more appropriate than scaling and squaring, which has been done for surfaces with networks (see e.g. [13]) and of course for volumes by many methods including several of those cited). To me, this is a crux of the method -- is there a practical contribution in this section, or is the contribution in combining the 3 parts of the framework and the theorems in 3.1? While I appreciate the Theorems, they do not seem like a substantial contribution by themselves.\n\nIn the experiments, as far as I can tell the authors implement a thorough set of experiments, and I appreciate the separation of train/val/test and including of error/variance measures. However, the results of CorticalFlow are very close to DeepCSR in most measures, perhaps other than CH. Interestingly, they are quite close in terms of HD, which is in some sense close to CH in what it tried to capture. Since the improvements are minimal, the authors analyze the specific types of improvements a bit more qualitatively -- that the improvements are more in robustness (no/fewer outliers) and when zooming in on regions they find instances where they have better topological fitting. I think what would be interesting or important is to really evaluate where the method contributes to insight -- which part of the framework is important, is it the ODE approximation? If so, can they replace it by a standard method that is currently used (e.g. scaling and squaring)? Is it the pyramid/scale-space (I'm not sure if the other methods like DeepCSR do this). I believe investigating this is crucial to the proposed method and would make the paper much stronger.\n\nOverall, I think there is potential in the method, but I have a hard time understanding the root of the contribution -- is there an interesting insightful nugget, or is the main contribution (other than 3.1 Theorems) that the authors smartly put the pieces together (which is fine, but different than if the DMT is somehow completely novel). This can be better explained in the intro (after explaining the literature by having a list of contributions), clarified in 3.1 (by clarifying the DMT model in practice, how it's different from other SVF integrations that are used, and separating the Theorems) and in the experiments (via ablation studies or proper analysis). It seems to me that it is central to the paper to take these steps to clarify to the reader what the take-away is and what we are learning from this.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors don't seem to discuss these. I provide several ideas for experiments to extract a clarification of what the contributions are -- these experiments will hopefully also indicate some of the limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0xk5pRBx4P2", "reviewer": "Reviewer_4s5o", "summary": "This paper introduces CorticalFlow – a geometric deep learning system to reconstruct topologically correct inner and outer cortical surface meshes from 3D MRI images. The model possesses two blocks: 1) a UNet architecture that given a 3D MRI image predicts a flow vector field and 2) a diffeomeorphic mesh transformer (DMT) that given the flow vector field, outputs a diffeomorphic mapping that may be applied to an unfeatured template mesh. The overall architecture has three stages at increasing mesh resolution, trained separately. The resulting surfaces lack geometric artifacts seen in other approaches. The inference time is also significantly faster than the baseline methods with lower memory footprint.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the results are very impressive (Table 1, Figures 3,4,5). The method even improves upon FreeSurfer, the software used to generate the pseudo-ground truth surfaces, in cases where the underlying MRI is blurry/has artifacts (Figure 5). I have a few minor comments/clarifications to the authors:\n\n* If the only allowable transformations are diffeomorphic, why are there self-intersecting faces (nonzero SIF% in Table 1)? \n* For clarity, the authors should bold the best values in Table 1.\n* Are some of the arrows incorrect in Figure 5? (0 arrows from Cortical Flow, 2 arrows from NMF)\n* How do the number of parameters in the model compare to the parameters in the voxel representation?\n* The authors should discuss potential applications or connections to other areas for a general ML audience\n* The authors should discuss limitations of this method\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should add more discussion on the limitations of their method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a Geometric deep learning model that learns to diffeomorphic deform a regular
template mesh towards a targeted object.
The paper was found interesting from a methods perspective and it lead to impressive results. The
methodology is strong and elegant, making a strong case for a "deep diffeomorphic" approach. The
experimental baselines are well chosen.
The only controversy concerned the state of the art, that may not have been sufficiently well
acknowledged. Part of the contributions are grounded in a set of now classical contributions on LDDMM,
which was not made clear in the writing. The authors however received this comment well and should
address it.
In summary, there is a clear accept here. But the updated version should dedicate more space to
acknowledging prior work.
Decision: Accept (Poster)