CorticalFlow: A Diffeomorphic Mesh Transformer Network for Cortical Surface Reconstruction
Authors: Leo Lebrat
Rodrigo Santa Cruz
Frederic de Gournay
Darren Fu
Pierrick Bourgeat
Jurgen Fripp
Clinton Fookes
Olivier Salvado
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Deep Learning
Geometric Deep Learning
Regular Surface Recosntruction
Cortical Surface Reconstruction

Abstract: In this paper, we introduce CorticalFlow, a new geometric deep-learning model that, given a 3-dimensional image, learns to deform a reference template towards a targeted object. To conserve the template mesh’s topological properties, we train our model over a set of diffeomorphic transformations. This new implementation of a flow Ordinary Differential Equation (ODE) framework benefits from a small GPU memory footprint, allowing the generation of surfaces with several hundred thousand vertices. To reduce topological errors introduced by its discrete resolution, we derive numeric conditions which improve the manifoldness of the predicted triangle mesh. To exhibit the utility of CorticalFlow, we demonstrate its performance for the challenging task of brain cortical surface reconstruction. In contrast to the current state-of-the-art, CorticalFlow produces superior surfaces while reducing the computation time from nine and a half minutes to one second. More significantly, CorticalFlow enforces the generation of anatomically plausible surfaces; the absence of which has been a major impediment restricting the clinical relevance of such surface reconstruction methods.

CorticalFlow: A Diffeomorphic Mesh Deformation Module for Cortical Surface Reconstruction
Léo Lebrat CSIRO, QUT leb0260csiro.. au
Rodrigo Santa Cruz CSIRO, QUT
Frédéric de Gournay IMT UMRS219 degourna@insa-tool..oo
Darren Fu UQ
fon0220csiro.
Pierrick Bourgeat CSIRO
Jurgen Fripp CSIRO
Clinton Fookes
Olivier Salvado
QUT
CSIRO, Data61.
Abstract
In this paper introduce CorticalFlow, geometric deep-learning model that, given a 3-dimensional image, learns to deform a reference template towards a targeted object. To conserve the template mesh's topological properties, train our model over a set of diffeomorphic transformations. This new implementa- tion of flow Ordinary Differential Equation (ODE) framework benefits from a small GPU memory footprint, allowing the generation of surfaces with several hundred thousand vertices. To reduce topological errors introduced by its discrete resolution, we derive numeric conditions which improve the manifoldness of the predicted triangle mesh. To exhibit the utility of CorticalFlow, we demonstrate its performance for the challenging task of brain cortical surface reconstruction. In contrast current state-of-the-ar,, CorticalFlow produces superior surfaces while reducing the computation time from nine and half minutes to one second. More significantly, CorticalFlow enforces the generation of anatomically plausible surfaces; the absence of which has been major impediment restricting the clinical relevance such surface reconstruction methods.
Introduction
The field of 3D shape reconstruction using deep learning techniques has attracted much attention. Recently, a plethora of methods have been developed for problems such as single-view object
reconstruction 22 50 77]. surface generation (27 74. and meshing noisy point clouds 332 42 At
first, these methods solely aimed to retrieve surface meshes geometrically close as possible to the target shape. However, recent applications require generating regular meshes with a known genus, such physics simulation, 3D-printing, and clinical analysis of anatomical surfaces [24] 58 62).
In this direction, three approaches in the literature stand DeepCSR 61. Voxel2Mesh [[]] and Neural Mesh Flow (NMF) [30]. DeepCSR first predicts implicit surface functions and then employs an iso-surface extraction method along with topology correction algorithm obtain genus- zero surfaces without handles or holes. Voxel2Mesh extends the vertex-wise template deformation approach of Wang et al. (7 by optimizing several mesh-smoothing penalty functions. contrast, NMF builds an invertible mapping that enforces topology conservation upon the resolution of an Ordinary Differential Equation (ODE) through sequence of residual blocks called Neural Ordinary Differential Equation (NODE) (9] However, these methods come with several limitations. The topology correction algorithm employed by DeepCSR computationally expensive and blind
Equal contribution
Check our project web-page http::///ebrreggggttoo.ooooorrre
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

towards the anatomical validity of its reconstructions, which can result in implausible corrections and mesh artifacts. On the other hand, Voxel2Mesh and NMF rely on deme-demnnding and vertex- dependent building blocks such as graph convolution that do not scale up well as the number of vertices in the template mesh increases to accommodate complex shapes. In addition, the dynamics learned by the NODE model in NMF can be very complex and may lead to a non-diffeomorphic mapping resulting in self-intersections in the reconstructed mesh.
This paper introduces CorticalFlow (CF), a new geometric deep learning model that smoothly deforms a template mesh towards complex shapes producing high-resolution regular meshes. First, simple 3D convolution neural network predicts a dense 3D flow field from volumetric image with a modest GPU memory footprint. Second, we formulate a tractable mathematical framework to compute diffeomorphic mapping for each vertex by solving a flow ODE. We derive sufficient and comprehensible conditions for meeting the diffeomorphic properties of these transformations. Finally, sequence of these diffeomorphic mappings is composed to produce accurate high-resolution genus-7ero regular meshes.
To evaluate our approach and compare it to existing techniques for regular surface reconstruction, we consider the problem of brain cortical surface reconstruction, which is an essential step for the analysis of brain morphometry in neurodegenerative diseases [18] and psychological disorders [57]. Given a
3D MRI of the brain, the goal to describe the inner and outer surfaces of the brain cortex, which are both homeomorphic to a sphere. Cortical surface reconstruction is challenging given the complexity, high resolution, and regularity required for the predicted meshes. In our experiments, CorticalFlow more accurate than ofthh-of-the-rt methods, providing an average reduction of 17.38% in Chamfer distance across all cortical surfaces compared to DeepCSR (the second-best performing method in this criteria). In terms of surface regularity, it surpasses NMF or Voxel2Mesh with an average reduction of at least 32.58% of self-intersecting faces while handling template meshes with many more vertices. It is also faster and more memory-efficied than all of these competitors.
2
Related Works
2.1 Geometric deep learning for surface reconstruction
Supervised surface reconstruction can be broadly categorized according to the 3D shape representation used to encode the prediction as either volumetric, implicit surfaces representation, novel geometric primitives, or geometric [22].
Volumetric methods predict shapes encoded as 3D grid of voxels containing discretized surface representations such as occupancy [I] and level-sets (46). From this representation, surfaces are obtained using iso-surface extraction methods, such as marching cubes [41] While 3D volumetric processing amenable to a convolutional neural network, the memory requirements are often a limitation to attain high high-resolution reconstructions (it grows cubically with the voxel-grid resolution). To overcome this issue, approaches based on octrees [35 68 72 output resolution from voxel-grid of 323 to 2563. Unfortunately, these approaches sacrifice speed and necessitate the redefinition of standard network operations such as convolution, pooling, and unpooling for this hierarchical data structure. Furthermore, as presented in [61], even this level of resolution, the precision is too coarse to capture the highly curved regions of the cortical surfaces.
have been proposed to increase the
Implicit surface methods alleviate resolution limitations of the volumetric methods by directly pre- dicting surface representations like occupancy (45), signed distance [54 755,, and 3D Gaussians (26) for points with continuous coordinates. This formulation allows synthesizing grids at an arbitrary resolution during inference with an easily implementable local refinement procedure, while training is performed stochastically over small subset of sampled points. Following this approach, Santa Cruz et al. [61] proposed DeepCSR, the first geometric deep learning model for cortical surface reconstruc- tion. Its main limitation is the difficulty to control the topology and mesh quality of the reconstructed surfaces, which hampers atrophy estimation used for neurodegeneratiie disease diagnosis [24 58 62] As result, DeepCSR resorts to computationally expensive topology correction algorithm produce a final cortical surface almost free of artifacts and with a spherical topology.
Methods based on geometric primitives build a surface representation to approximate complex shapes as the union of these primitives. In this category, we can highlight the works of Niu et al. [51) and Groueix et al. [28) which propose to approximate complex object shapes with collection of

---

"cuboids" or "surface patches". Recent works by [10 17 revisit the convex decomposition idea and propose reconstruct complex object shapes by predicting collections of convex parts. The former predicts a set of localized convex polytopes formed by their hyperplane parameters and translation vector, while the latter predicts a binary space partitioning tree to reconstruct the target shape. While very promising in terms of information compactness, these approaches are challenged to generate cortical surfaces due to their varied curvatures, which require a large number of convex parts to produce accurate results.
Finally, geometric methods comprise techniques that allow estimating a resolutsotttion mesh by transforming a known template mesh [52] 53 65 [7]). Following this approach, Wang et al. [7] propose a graph-convolution network predict vertex wise deformations of a spherical mesh while dynamically increasing its resolution with point pooling process. Wickramasinghe et al. [73] extended this model for the reconstruction of smooth anatomical surfaces such as the liver or hippocampus for different image modalities. Topological errors are reduced using three different penalty functions in the loss function. Recently, Gupta and Chandraker (30] leverage NODE blocks [9] to parameterize regular deformations that allow conserving the two-manifoldness property of the input template.
Indeed, the "manifoldness" measures as non-manifold edges or non-manifole vertices, and defined by Gupta and Chandraker [30]. are conserved by a deformable model which is not generating new vertices since those properties are inherited from the template mesh (only the vertices' posi- tions are affected). However, the normal consistency mnon-mniifodd faces) is only conserved by homeomorphisms which can at most flip globally the faces' normal orientation.
2.2 Generation of diffeomorphic mappings
b.
The reconstruction of regular surfaces from deformable model is a subtle trade off between finding the right pa- rameterization suitable level of regularization during training. The delicacy of this problem illustrated in Figure 1. First, one employ multiple penalty func- tions as Chamfer normal, normal consistency, Laplacian loss, or edge length loss [71 73] It is worth mentioning that without such penalizations, a deformable model will learn non-smooth deformations, which leads irregular meshes (as shown in Figure 1.c). However, those penal- izations simply encourage the reconstructed surface to be regular. The second approach consists of parameter- izing the set of learned deformations; this approach is favored in our paper since it allows stronger theoretical guarantees and A natural framework to generate invertible deformations is to consider a flow ODE 20 69 This framework has been successfully applied in pattern recognition and im- age registration [2] 19]. The main idea is to consider the mapping the solution at time of initial value problem (IVP) of the form,
c.
subject typer-parameter tuning.
Figure 1: Generating regular meshes of the left hemisphere of the brain cortical surface. a. CorticalFlow, b. NMF with global feature descriptor (30. e. UNer without diffeomor- phic parametrization geometric penaliza- tion.
d x)
(f)) with (000;) x) =x
ds
under regularity hypothesis and upon boundedness of its support, and using the Picard-Lidelöf theorem, one can show that a unique solution of this problem exists for т E IR. In addition, the mapping x (s; defines family of diffeomorphisms can be computed through a backward integration.
6 for all time [0. r] whose inverse
When the vector field is constant over time i.e. v: R - 3 Equation ① describes the Stationary If u a time- mareingryigg vector field, the framework described
Velocity Field (SVF) framework [
in becomes the LDDMM model
5
8 64 70)
3

---

This generic framework has been successfully applied within deep learning methods for diffeomorphic
image registration [T5] 47), point-cloud completion (49
single view reconstruction [30 and to
parameterize set of deformations 37
The SVF formulation has been particularly fecund in medical deep-learning registration 15
40 47
761, where the resolution of Equation ( is performed using the scaling and squaring method 22. 3
to predict the displacement of each voxel-center and compute the registered image. However, this technique is not suitable for points that lie on non-regular coordinates. Naively, one can compute these mappings on a dense grid and then interpolate the deformation non-regular coordinates. This simple approach is subject to two main limitations. Firstly, Equation ( has to be solved in our context for millions of grid points where only a few hundred thousand vertices are displaced. Secondly, one cannot guarantee the invertibility of such a mapping with linear interpolation and one cannot compose provably several of those approximated mappings.
In 30 49], the problem solved using a black-box neural ODE (9) and by learning a neural vector field v. Despite allowing learning a time-dependent vector field, this approach has shown its limitations in our targeted application. Cortical surfaces are unique to each individual; indeed, the cortical folding patterns are similar to a fingerprint +4 and constitute a distinctive biometric for each individual. Moreover, we observe that the classical approach, which consists of conditioning the neural ODE on global feature descriptor of the input image, fails to provide satisfactory results for cortical surface reconstruction (see Figure 1.b. and the supplementary material). Instead, one has to equip each moving vertex with a local feature descriptor of the input image, limiting the number of vertices of the resulting mesh.
Our work lies the intersection of these methods. We propose to extend the SVF framework for points lying in real-coordinates, with particular care given to the numerical affordability of the ODE solver. We define a multi-scale approach, that the final deformation the result of the composition of three successive deformations that allow to approach more complex mappings and alleviate the limitations of the one vector field SVF framework [23 43) This framework is memory efficient, theoretically tractable, and can seamlessly handle large template meshes (z2 450k vertices).
3 Method
CorticalFlow (CF) is multi-level deep learning architecture composed of several Diffeomorphic Mesh Deformation (DMD) modules. It takes as input a 3-dimensional Magnetic Resonance Image (MRI) of patient brain denoted (tensor of dimensions H x W D) and a template T (where i represents the degree of refinement of the template). CorticalFlow outputs the surface representation of an anatomical substructure by composing stackable diffeomorphic deformations generated by DMD modules. CorticalFlow with & deformations (CF*) can be written using the following recurrence,
CF2, 0, (I) = (MDe),er), (I) Ti))
CF, T++1) for i 2 1, CF++1 B.4. T++1) DMNet i+1 (UT ... U, I), ..,
with A B the channel-wise concatenation of the tensors A and B and where Uj denotes the output of the k-th UNet", parameterized by 0g..
In our paper describe CF², a version of CorticalFlow with three stages where each stage is learned successively. CorticalFlow is trained in a supervised fashion, given a dataset D composed of pairs of MR-image I and triangle mesh S representing cortical structure and for i E (1, optimize the following objective,
arg min
Σ (LS)
C(CF') e, (T)
As training loss C(.,.), we minimize the mesh edge loss and Chamfer distance computed on point clouds of 150k points sampled from the predicted and ground-truth surfaces using random uniform sampling. The implementation of these losses and sampling algorithm are provided in the PyTorch3D library [56).

---

3.1 DMD Diffeomorphic Mesh Deformation module
The introduction of a Diffeomorphic Mesh Deformation module (DMD) is driven by the following classification of surfaces in 3 dimensions:
Theorem 3.1. Suppose that B is smooth closed manifold of dimension 2 embedded in RR. Suppose that ④ : |0, r| x RR - TR' is a family of homeomorphisms (continuous map such that for each the mapping + (L,, ) is bijective with continuous inverse), with (00; x) x. Then, for each the homotopy classes of and BB (t) {1 y).y y B3 are the same.
This theorem means that if B is sphere, the surface (t)(() is of genus 0 with self-intersection.
Existence and uniqueness of solution to the continuous problem
The DMD generation of diffeomorphic mappping relies on the resolution of continuous flow ODE. For that purpose, let x E Ω + v(x) E R be a constant over time vector field supported on the MRI space ! and obtained by tri-linear interpolation of a feature map U R H <Txx>xxx Suppose that v = 0 on An The image origin is denoted by O = (01, 02, 0a) and denote by d; the interpolation spacing in the i-th direction such that (0n, or + di (H 1)] x [02, 02 d2(W (W 1)] (os, 03 d - -)).
Theorem 3.2. Existence and uniqueness of the solution. Define t through the autonomous ODE,
db{(s; x) x)), with + =
(4)
ds
Then is uniquely defined on Q, is Lipschitz and for each t, the mapping x &tt, x) is bijective with Lipschitz inverse. The proof of this result can be found in the supplementary material.
Being Lipschitz is more difficult to achieve than being merely continuous. Less formally, Theorem 3.2 ensures that if B is smooth, then for all t, the surface 4y (t) may, in the worst case, have kinks. fh only continuous and not Lipschitz, then the surface Pg (t) might have cusps that are more irregular than kinks. Note well that the choice of the interpolation technique used to generate is pivotal since it drives the regularity of the right-hand hand side of Equation (4) Indeed, Lipschitz'' 's constant boundedness allows the definition of solution the continuous problem. More importantly, and as described in the next section, it rules the step-size to use for obtaining a stable numeric method.
Numerical resolution of ODE
The DMD module solves for each vertices' position a flow ODE defined in Equation ( defined by v the numeric approximation of & by explicit forward method, the invertibility of this discretisation is given by the following theorem
Theorem 3.3. Let v be L-Lipschitz. Define as the Forward Euler approximation,
(h x.) = x. + hw(x).
(5)
is a Lipschitz homeomorphism for each h < L-1
As result, in combination with Theorem 3.2 the surface Vyn (h) is smooth as long as hL 1. We note that the stability condition h. < 1 is commonly used in Computational Fluid Dynamics [39.
Less formally, this estimate tells us that the less regular (high gradient) is, the smaller the integration step should be chosen.
Proof. Let < - and denote f + x + he(x). Then f is a Lipschitz mapping and it is sufficient to prove the bijectivity of f. The injectivity stems from, for all x # y,
[s(x)) s())I 2 || vll h))e(())))))))))))))()))))) r(v) 2 || v - hT). > 0.
The surjectivity comes from the fact that for each y, the mapping 9 + y- kv(x)) is hL-Lipschitz with hL < 1, hence is a contraction. By a fixed point theorem, g admits unique x= solution to g(x*) = x², equivalently y fxx)). Hence, is surjective.
5

---

UNet,
UNetz
UNetz
Output Mesh
Image
Flow Field
Flow Field 2
Flow Field 3
DMD
DMD
DMD
Template Mesh
Figure 2: CorticalFlow architecture for the cortical surface reconstruction problem. Following the notations of Equation (2). UNete, implements a UNet, layer and UNer implements UNetz layer. The architecture of UNet, and UNet, are described in the supplementary materials.
Theorem 3.3 ensures that (h) is non- intersecting manifold. Unfortunately, there exists another layer of numerical approx- imation that prevents V to have the desired topological properties. Indeed, sup- pose that B is sphere, one never computes V aB (h), but triangulates sphere with tices Vi, and evaluates the image of the vertices using Algorithm [ A new mesh is formed by using the image of the ver- tices and the original connectivity. Since the perturbed edges of the mesh are not the images of the original edges by the For- ward Euler scheme, the resulting mesh may self seff-intersect (we refer interested readers to Section 2.3 of our supplementary material). Notwithstanding this limitation, we use the rule of thumb hL < and check that for all considered examples, and we have hL < 1.
Algorithm DMD module pseudo-code
Input: U [B
Discrete flow T with vertices (Vi)iet..m b Triangle mesh
Input:
Input:
Number of integration steps Output: Updated (Vi)ie1..m
N'
h+
Ensure: h < + for i E [1,m] ml do
for j E [1. [1. n| do
V/ (h, Vi)
end for end for
3.2 Network architecture
As shown in Figure 2 CorticalFlow consists of chain of three deformations. Note that more deformation modules could used, we focus on three modules to have a fair comparison with existing techniques. The first deformation module receives input volumetric image and outputs flow vector field with the same dimensions using UNet-3D [59]. This discrete flow vector field is integrated by the DMD module compute smooth deformations explained in Section 3.1 The subsequent UNet-3D receives as input the image and the flow vector fields predicted by the previous deformation modules. The set of resulting mappings composed produce the final mesh.
For training CorticalFlow, we adopted sequential approach where we train one deformation module at time while freezing the weights of the others. We train the first deformation with low-resolution template (Ti) of 30k vertices and a UNet-3D architecture with four down/up-sampling levels. To train the second and third deformations, we increase the resolution of the template mesh to 135k and 435k vertices, for Tz and Ts respectively, and reduce the UNet-3D hown/up-samplinn levels to two. These networks are respectively labeled as UNet, and UNet, in Figure 2 while their layer details described in our supplementary material. The choice of the architecture depth motivated by the numerical conditions on the integration step-size derived in Section 3.1
To verify the conditions of Theorem 3.3 it is essential to mention the use of template meshes with different resolutions and different UNet architectures. Indeed, the first block has to provide large deformation resulting in high |w|] To keep the Lipschitz constant small, one observes that the use of a low-resolution template with 30k vertices along with a deeper convolutional architecture forces the UNet to recover only coarse details and thus produces a flow vector field with small Vv. During the second and third deformation, more details and higher resolution folds can be learned,

---

with templates composed of 135k and 435k vertices, respectively (see the ablation study presented in Section 1.1 of our supplementary material). We empirically verify that this hierarchy of deformations was beneficial for producing flow vector field e with a small Lipschitz constant. This multi-step approach allows attaining up to 14.5 times less self-intersection in comparison with Neural Mesh Flow (see T.
To generate the template mesh, we take the convex hull of all surfaces contained in the training dataset and remesh them uniformly using JIGSAW [2I]. To achieve a different order of refinement, we use the midpoint subdivision algorithm implemented in MeshLab [12]. Model syper-parameters and further implementation details are provided in the supplementary material.
4 Experiments
We benchmark Cor and other existing deep learning techniques on the cortical surface reconstruction problem. The goal is to estimate geometrically accurate and topologically correct triangular meshes for the inner and outer cortical surfaces from a given MRI. Like previous works [14 16 31 34 38 61), these surfaces are further divided into the left and right brain hemispheres. See below a summary of the dataset, evaluated methods, and metrics used in our benchmark, in addition to the detailed discussion of the results summarized in Table
Oeter
Surface
Kight Chuzer Surince
CHimm)) 1
HIXvom)
CHN
+SSIF
DSC1
VS1
CHisai 1
HDDmmn' 1
CHN
15 SIF
DSC
VS-
CorticalHow
(303)
0.307
10230
03277
0200
11315
1,20
11000
1.148 /40/3 GB
(40,008)
(=0.040
(40.000)
[#0.669)
(=0000)
(46.006)
(20.01)
[40.045)
[60006]
(40.820)
[60.00)
(30.00)
DSepCSR
0.925
0.98:
0.912
0.955
0.33
0.000
C.003
(-0135)
(10011)
(+6.005)
NMP
0
1.772
1.588
0.811
1,510
6.983
42 14.331 GB
(20,200)
(=11116
(20000)
(40.545
(=000G)
(40,088)
(60.177)
(40.301)
(50.088
(20.202)
(50.000)
(20200))
QuickNAT
5.722
2,582
0.511
0.030
0,966
0.775
4.60
2,00
0,800
0.000
0.954
0
13.00 003 see 9,627 GB
(+1.971)
(=1001
(240224)
(60,000)
(=0100)
(40.013)
(62250)
(61.00))
(5M1011)
Vose22Mesh
7,185
3,80
0.721
0.557
0,895
03576
7.461
3150
0.77
0,833
0.888
20340 wne GB
(40.500)
(=0218
(30006))
(40.250)
(=0000)
(10.013)
(20771)
(4002277
(10.0LT)
(20.200)
[60.00))
(300000)
Lett Inner Surface
Right Inner Surface
CHirm)) 1 HiXoom)
CHN
SIF
DSC1
VSt
CHimao 4
HD 1
CHN
SIF
DSC +
VS
CotticalFlew
0,233
0,783
0392
BSST
0,300
CRRT
1.14 105522/33000 33071 GB
(40.006)
(=00600
(40000)
(60.060)
(=000))
(10.010)
(5008)
(60.009)
(500007)
(20.200
(60.05)
(30.00)
DSepCSR
0.658
0.966
0,963
0.966
0.694
0.500
0.566
0.000
0.086
(40000)
[60.00))
(40.013)
NMF
1.404
1.477
0,025
14844
14453
0,553
0,435
0.27
C.064
GB
(40.185)
(-0110)
(-0011)
[+0.105)
(-000)
(+0.015)
(60158)
(60.125)
(+0.010)
(10.188)
[H0006)
(90.010)
QUIcKNAT
2,570
1.595
6,906
0.000
0,905
0921
2.285
1.788
0.68
0,000
0,908
0.951
13 003 see 19627 GR
(#1.80))
(34)
(2002))
(+0000))
(#1.9))
(25)7155
(=0000)
[SQUUTT
Mosc
5,005
3,46
0,713
D.G21
0,550
6,172
3.540
0.707
0,761
0.542
2o.40.az//334000 GB
(40.467)
(=02477
(-00017)
[#0.22))
(=0.000)
(#0.014)
(20.71))
(400272)
(5G.OLT)
(-0.2771)
[40.0L0)
(20.018)
Table I: Cortical Surface Reconstruction Benchmark. Consider the evaluation metrics: chamfer
distance (CH), Hausdorff distance (HD), chamfer normals (CHN), percentage of self-intersecting
faces (%SIF), Dice Score (DSC), and Volume Similarity (VS). | indicates smaller metric value is
better, while 1 indicates greater metric value is better. We also report the inference runtime and GPU memory footprint required by the compared algorithms.
Dataset. We used the same MRIs, pseudo-ground-trutte surfaces, and data splits as (6]. This dataset consists of 3,876 MRI images extracted from the Alzheimer's 's Disease Neuroimaging Initiative (ADNI) (36 and their respective pseudo- sonudd--roun--ruute surfaces generated with the FreeSurfer V6.0 cross-sectional pipeline (25). We train all methods on the training set ( 60%) until their losses plateau on the validation set (+= 10%) and report their performance on the test set (ss 30%). We refer the reader to [61] and our supplementary material for full details on the dataset.
Evaluated Methods. We compare CorticalFlow to the following methods: DeepCSR 6 Voxel2Mesh (73. NMF (30, and QuickNAT[I [60]. As discussed in Section DeepCSR the state-of-the-ard art geometric deep learning model for cortical surface reconstruction, while Voxe12Mesh is deformation- based model proposed to retrieve generic anatomical surfaces from volumetric medical images like MRIs and CT scans. Differently, NMF is a deformation-baan based model for single-view object reconstruction from 2D images. To adapt this model to our task whose input is
DSepCSRR official implementation retrieved from http::////sssuuteee......r/////p
r repos/ peeps//eeeparr
Vexee22Mssh official implementation retrieved from httppp/s/gggtta..oooooo/aae////a
e NMF official implementation retrieved from httppppppp..pphacooo///a/aaaa/aa
'QuickNAT official implementation retrieved from httppp///gtthhb..cooo/////////lt
7

---

3D MRI. we evaluate different 3D convolutional network backbones based on UNet [59, ResNet B3], and Hypercolumn [6]] The Hypercolumn backbone provides the best results thanks to its vertex cetexxdeeendent features. As such, this is used as the NMF backhone in our benchmark. At the same time, the results for the other backbones are presented in our supplementary material. We also evaluate baseline composed of state-of-the art brain segmentation model QuickNAT (60] followed with the marching cubes to evaluate the surface. All of these methods were trained and evaluated using a NVIDIA P100 GPU and Intel Xeon (E5-2690) CPU, except Voxel2Mesh which required NVIDIA RTX 3090 GPU due to its GPU memory requirements.
Evaluation Metrics. We compare these methods for their geometric accuracy and surface regularity, as well as their time and space complexity. As measure of geometric accuracy. we report the standard Chamfer distance (CH), Hausdorff distance (HD), and Chamfer normals (CHN), We compute these distances for point clouds of 200k points uniformly sampled from the predicted and target surfaces. As a measure of regularity, we compute the percentage of self-intersecting faces (&SIF) using PyMeshLab [48]. We also report volumetric overlap metries [67] including Dice Score (DSC) and Volume Similarity (VS) computed on the high-resolution rasterization (4xthe input MRI resolution) of the generated and ground-truth surfaces.
For the time and space complexity of the evaluated methods, we report their average inference time (in seconds) and inference GPU memory footprint (in GB) to reconstruct the four cortical surfaces, respectively.
CorticalFlow
DeepCSR
NMF
088
Distance
Figure 3: Predicted outer cortical surfaces color-coded with the distance to the pseudo-ground-ruuth surfaces. Here we present the results for the three best-performing methods in terms of geometric accuracy: CorticalFlow, DeepCSR, and NMF, See our supplementary materials for more examples.
Results & Discussion. In our experiments, we noticed that CorticalFlow produces more geomet- rically accurate surfaces than the other methods. On average, it presents better geometric metrics across all the cortical surfaces. In addition, as shown in Figure 3. CorticalFlow errors are smaller (S 0.2mmm) and evenly spread across the surface compared to the other methods. In contrast, NMF and DeepCSR can present substantial errors (2 1mm). The former has its error spread across the entire surface, while the latter can produce large errors at specific regions.
CorticalFlow is also more robust than the competitors presenting lower error variation across individ- uals as suggested by the smaller standard deviation of the geometric metrics computed. Interestingly, CorticalFlow is also more robust to MRI artifacts even when the sseudo-ground-teeg poor quality. For instance, in Figure CorticalFlow predictions are still plausible for blurry input MRI while FreeSurfer fails significantly to generate appropriate surfaces for the same input. These examples support our claim that a regular parametrization allows us to reduce non-plausible and non-diffeomorphii predictions that our model cannot learn by construction.
surface has
CorticalFlow also generates triangular meshes with better properties than the evaluated methods. Compared to the deformation
methods NMF and Voxe12Mesh, CorticalFlow predicted meshes are genus-zero surfaces and present a lower percentage of self-intersectie cortical surfaces). Figure presents examples of self-intersecting faces produced by CorticalFlow, which are contrasted with the NMF predicted mesh for the same input MRI. The implicit-surface
faces (mainly for the inner
8

---

Figure 4: Slices of blurry MRI and the outer surface delineation generated by FreeSurfer V6 (yellow contour) and CorticalFlow (blue contour). Orange circles highlight blurry MRI regions, green circles highlight FreeSurfer's underestimated areas, while red circles highlight non-plausible predictions avoided by CorticalFlow thanks to the diffeomorphism of its predicted deformations.
CorticalFlow
NMF
Inner Surface
Outer Surface
(a)
(b)
Figure a Predicted cortical surfaces by CorticalFlow and NMF with eeff-intersecting faces highlighted in red. b) Significant mistakes generated by the topology correction algorithm used in the DeepCSR method.
based DeepCSR method does not produce a single self-intersecting face since it employs computa- tionally expensive post- processing routines like topology correction 4 and iso-surface extraction. However, these post-processing routines do not take into account the input MRI which can generate non-plausible corrections exemplified in Figure 56 Similarly, the voxel-wise segmentation baseline (i.e., QuickNAT) is free of self-intersecting faces, but it does not produce genus-zero surfaces. Indeed, QuickNAT's predicted surfaces are composed multiple connected components presenting many handles and holes which not acceptable for the purpose of cortical surface reconstruction. Some examples of QuickNAT reconstructed cortical surfaces are presented our supplementary material. Therefore, we argue that CorticalFlow is the method of choice reconstruct regular surfaces from volumetric images.
the output mesh previously observed in Segonne et al. [63 and
Due to its elemental construction (three UNet-3D backbones and an interpolation module for the integration), CorticalFlow remains highly efficient. It has minimal GPU memory footprint and faster inference runtime while handling larger surfaces with more vertices both during training and inference. This feature allows its deployment low-end computers and embedded devices which is pivotal in many scenarios across public health and for commercialization of affordable Al healthcare solutions B (55)
Finally, as a by-product of CorticalFlow's deformable and diffeomorphic nature, one can seamlessly obtain sub-voxel resolution segmentation by applying voxelization engine. This can capture variations below the image resolution while traditional segmentation methods 60) are restrained from working at the image resolution (see Figure 6a). Additionally, essential component of

---

-030
(a)
(b)
QuickNAT segmentation (inner surface volume in pink and outer surface volume in yellow) and CorticalFlow generated surfaces (inner surface in light blue and outer surface in dark blue). Note that the highlighted artifacts in QuickNAT segmentation can be easily surpassed by CorticalFlow thanks its continuous representation of the region of interest. b Outer cortical surface color-coded with the curvature of its cortical foldings. Our method allows us to map back this measure and any other biomarker to the input template using the composition of the inverse deformations
Figure 6:
computational neuro-anatomy consists of computing local shape descriptors for different individuals
and transferring them to the same reference space using conformal mappings [29] [66. For the
proposed model, one can efficiently compute the inverse transformation t shown in Figure 6b for the surface curvature descriptor.
5 Conclusion
This paper introduces CorticalFlow a geometric deep learning model for efficiently reconstructing high-resolution, accurate, and regular triangular meshes from volumetric images. We develop lightweight neural network to predict dense 3D flow vector field from a volumetric image. Then, describe new Diffeomorphic Mesh Deformation (DMD) module, which is parameterized by set of diffeomorphic mappings. This includes the derivation of numerical conditions for recasting the continuous flow ODE problem into efficient discrete solver. Finally, we extensively verify that the proposed model achieves state-of taite-of-the-ate tithhe-art art performance in the challenging brain cortical surface reconstruction problem. This benchmark reveals that CorticalFlow is more accurate and, by construction, more robust to image artifacts providing anatomically plausible surfaces. Thanks also its low space and time complexity, the proposed method can facilitate large-scale medical studies and support new healthcare applications.
6 Compliance with Ethical Standards
This research was approved by CSIRO ethics 2020 068 LR.
Acknowledgements
This work was funded in part through Australian Department of Industry, Energy and Resources CRC-P project between CSIRO, Maxwell Plus and I-Med Radiology Network.
10

---

References
[1] Vincent Arsigny. Processing data in lie groups: an algebraic approach. application to non- linear registration and diffusion tensor mri. PhD thesis, Citeseer, 2004.
[2] Vincent Arsigny, Xavier Pennec, and Nicholas Ayache. Polyrigid and polyaftine transformations: a novel geometrical tool deal with non-rigid cpppmmatttoonsapappaatioooo the registration of histological slices. Medical image analysis, 96):507-523, 2005.
[3] John Ashburner. A fast diffeomorphic image registration algorithm. Neuroimage, 38(1):95-113,
2007.
[4] Pierre-Louis Bazin and Dzung L Pham. Topology correction of segmented medical images using fast marching algorithm. Computer methods and programs in biomedicine, 88(2): 182-190. 190, 2007.
[5] M Faisal Beg, Michael I Miller, Alain Trouvé, and Laurent Younes. Computing large deforma- tion metric mappings via geodesic flows of diffeomorphisms. International journal of computer vision, (1): 2005.
[6] Marcel Berger and Bernard Gostiaux. Differential Geometry: Manifolds, Curves, and Surfaces: Manifolds, Curves, and Surfaces, volume 115. Springer Science & Business Media, 2012.
[7] Vincent Camion and Laurent Younes. Geodesic interpolating splines. International workshop on energy minimization methods in computer vision and pattern recognition, pages 513-527, Springer, 2001.
[8] Nicolas Charon. Analysis of geometric and functional shapes with extensions of currents: applications to registration and atlas estimation. Phl) thesis, École normale supérieure de Cachan-ENS Cachan, 2013.
[9] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
[10] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 45-54, 2020.
[11] Christopher Choy, Danfci Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference Springer, 2016.
on computer vision, pages
[12] Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. MeshLab: an Open-Source Mesh Processing Tool. In Vittorio Scarano, Rosario De Chiara, and Ugo Erra, editors, Eurographics Italian Chapter Conference. The Eurographics Association, 2008. ISBN 978-3-905673-68-5 doi: 10.23 1022 MtalCha/itaiaanChapCoo2202 136.
[13] Clarissa Z Cooley, Patrick C McDaniel, Jason P Stockmann, Sai Abitha Srinivas, Stephen Cauley, Monika Sliwiak, Charlotte R Sappo, Christopher F Vaughn, Bastien Guerin, Matthew Rosen, et al. A portable brain mri scanner for underserved settings and point-of-cant imaging. arXiv preprint N0KV::2004.118833 2020.
[14] Robert Dahnke, Rachel Aine Yotter, and Christian Gaser. Cortical thickness and central surface estimation. Neurolmage, 65:336-348, 2013.
[15] Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu. Unsupervised learning of probabilistic diffeomorphie registration for images and surfaces. Medical image analysis, 57: 226-236, 2019.
[16] Anders M Dale, Bruce Fischl, and Martin I Sereno. Cortical surface-based analysis: segmen- tation and surface reconstruction. Neurolmage, 9(2):179-194, 1999.

---

[17] Boyang Deng, Kyle Genova, Soroosh Yazdani. Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 31-44, 2020.
[18] An-Tao Du, Norbert Schuff. Joel H Kramer, Howard J Rosen, Maria Luisa
Katherine Rankin, Bruce Miller, and Michael W Weiner. Different regional patterns of cortical thinning in alzheimer's s disease and frontotemporal dementia. Brain, 130(4): 159:::11111111 11911199661991 1166, 2007.
[19] Paul Dupuis, Ulf Grenander, and Michael I Miller. Variational problems on flows of diffcomor- phisms for image matching. Quarterly of applied mathematics, pages 587-600, 1998.
[20] David G Ebin and Jerrold Marsden. Groups of diffeomorphisms and the motion of incom- pressible fluid. Annals of Mathematics, pages 102-163, 1970.
[21] Darren Engwirda and David Ivers. Off-centre steiner points for delaunay-refinement on curved surfaces. Computer-Aidedt Lindut Design, 72:157-171, 2016.
[22] George Fahim, Khalid Amin, and Sameh Zarif. Single-view 3d reconstruction: A survey of deep learning methods. Computers & Graphics, 94:164-190, 2021.
[23] Jean Feydy. Analyse de données géométriques, au delà des convolutions. PhD thesis, Université Paris-Saclay, 2020.
[24] Bruce Fischl, Martin I Sereno, and Anders M Dale. Cortical buseece-bbee analysis: Ii: inflation, flattening, and a surface-based coordinate system. Neuroimage, 9(2):195-207) 1999.
[25] Bruce Fischl, David H Salat, Evelina Busa, Marilyn Albert, Megan Dieterich, Christian Hasel- grove, Andre Van Der Kouwe, Ron Killiany, David Kennedy, Shuna Klaveness, et al. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33(3):341-33) 335,, 2002.
[26] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154-7164, 2019.
[27] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning predictable and generative vector representation for objects. In European Conference on Computer Vision, pages 484-499. Springer, 2016.
[28] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-måché approach learning 3d surface generation. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 216-224, 2018.
(29) Xianfeng Gu, Yalin Wang, Tony Chan, Paul M Thompson, and Shing-Tung Yau. Genus zero surface conformal mapping and its application brain surface mapping. IEEE transactions on medical imaging, 23(8)9949958, 2004.
[30] Kunal Gupta and Manmohan Chandraker. Neural mesh flow: 3d manifold mesh generation via diffeomorphic flows. In Advances in Neural Information Processing Systems, volume 33, pages 1747-1758, 2020.
[31] Xiao Han. Dzung L Pham. Duygu Tosun, Maryam E Rettmann, Chenyang Xu, and Jerry Prince. Cruise: cortical reconstruction using implicit surface evolution. Neurolmage, 23(3): 997-1012, 2004.
[32] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: A self-prior for deformable meshes. ACM Trans. Graph., 39(4), 2020.
[33] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] Leonie Henschel, Sailesh Conjeti, Santiago Estrada, Kersten Diers, Bruce Fischl, and Martin Reuter. Fastsurfer-a fast and accurate deep learning based neuroimaging pipeline. Neurolmage, page 117012, 2020.
12

---

[35] Christian Häne, Shubham Tulsiani, and Jitendra Malik. Hierarchical surface prediction for 3d object reconstruction. In 2017 International Conference on 3D Vision (3DV), pages 412-420, 2017. doi: 10.1 109/90V.2211..0000.
[36] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L. Whitwell, Chadwick Ward, et al. The alzheimer's s disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Reso- nance Imaging, 27(())685 2008.
[37] Chiyu Jiang. Jingwei Huang, Andrea Tagliasacchi, Leonidas Guibas, et al. Shapeflow; Learnable deformations among 3d shapes. arXiv preprini arXiv:2006.07982. 2020.
[38] June Sic Kim, Vivek Singh, Jun Ki Lee, Jason Lerch, Yasser Ad-Dab'bar David MacDonald, Jong Min Lee, Sun Kim, and Alan C Evans. Automated 3-d extraction and evaluation of the inner and outer cortical surfaces using a laplacian map and partial volume effect classification. Neurolmage, 27(1):210-221, 2005.
[39] Petros Koumoutsakos, Georges-Hene Cottet, and Diego Rossinelli. Flow simulations us- ing particles-bridging computer graphics and cfd. In SIGGRAPH 2008-35th International Conference on Computer Graphics and Interactive Techniques, pages 1-73. ACM, 2008.
[40] Julian Krebs, Hervé Delingette, Boris Mailhé, Nicholas Ayache, and Tommaso Mansi. Learning a probabilistic model for diffeomorphie registration. IEEE transactions on medical imaging, 38 (9):2165-4)) 2176, 2019.
[41] Thomas Lewiner, Hélio Lopes, Antônio Wilson Vieira, and Geovan Tavares. Efficient imple- mentation of marching cubes' cases with topological guarantees. Journal of graphics tools, (2):1-15, 2003.
[42] Minghua Liu, Xiaoshuai Zhang, and Hao Su. Meshing point clouds with predicted intrinsic- extrinsic ratio guidance. In European Conference on Computer Vision, pages 68 84. Springer, 2020.
[43] Marco Lorenzi and Xavier Pennec. Geodesics, parallel transport & one-parameter subgroups for diffeomorphic image registration. International journal of computer vision, 105(2):111-127, 2013.
[44] J-F Mangin, Denis Riviere, Arnaud Cachia, Edouard Duchesnay, Yves Cointepas, Dimitri Papadopoulos- Orranoo,,-- Paola Scifo, Ochiai, Francis Brunelle, and Jean Régis. framework to study the cortical folding patterns. Neuroimage, 23:S129-SI S138, 2004.
[45] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[46] Mateusz Michalkiewicz, Jhony K Pontes, Dominie Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4743 4752, 2019.
[47] Tony CW Mok and Albert Chung. Fast symmetric diffeomorphic image registration with convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4644 4653, 2020.
[48]
Alessandro Muntoni and Paolo Cignoni. PyMeshLab, January 2021.
[49] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Occupancy flow:
4d reconstruction by learning particle dynamics. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 5379-5389, 2019.
[50] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volu- metric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515,
2020.
13

---

[51] Chengjie Niu, Jun Li, and Kai Xu. 1m2struct: Recovering 3d shape structure from a single rgb image. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4521-4529, 2018.
[52] Junyi Pan, Jun Li, Xiaoguang Han, and Kui Jia. Residual meshnet: Learning to deform meshes for single-vier view 3d reconstruction. In 2018 International Conference on 3D Vision (3DV), pages 719-727. IEEE, 2018.
[53] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[54] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[55] Magdalini Paschali, Stefano Gasperini, Abhijit Guha Roy, Michael Y-S Fang, and Nassir Navab. 3dq: Compact quantized neural networks for volumetric whole brain segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention. pages 438-446. Springer, 2019.
[56] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. urXiv:2007.085000 2020.
[57] Lars M Rimol, Ragnar Nesväg, Don J Hagler Jr, Ørjan Bergmann, Christine Cennema-Notestine Cecilie B Hartberg, Unn K Haukvik, Elisabeth Lange, Chris J Pung, Andres Server, al. Cortical volume, surface area, and thickness in schizophrenia and bipolar disorder. Biological psychiatry, 7166)5552-560, 2012.
[58] Claudia E Rodriguez-Carran Studholme. A framework for in vivo quantification of regional brain folding in premature neonates. Neuroimage, 41(2):462-478, 2008
Pratik Mukherjee, D Vigneron, J Barkovich, and Colin
[59] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer-assisted intervention, pages 234-241, Springer, 2015.
[60] Abhijit Guha Roy, Sailesh Conjeti, Nassir Navab, Christian Wachinger, Alzheimer's Dis- ease Neuroimaging Initiative, al. Quicknat: A fully convolutional network for quick and accurate segmentation of neuroanatomy. Neurolmage, 186:713-722 2019.
[61] Rodrigo Santa Cruz, Leo Lebrat, Pierrick Bourgeat, Clinton Fookes. Jurgen Fripp, and Olivier Salvado. Deepesr: A 3d deep learning approach for cortical surface reconstruction. In Proceed- ings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 806-815, 2021.
[62] Marie Schaer, Meritxell Bach Cuadra, Lucas Tamarit, François Lazeyras, Stephan Eliez, and Jean-Philippe Thiran. A surface-based approach quantify local cortical gyrification. IEEE transactions on medical imaging, 27(2):161-170, 2008.
[63] Florent Segonne, Jenni Pacheco, and Bruce Fischl. Geometrically accurate topology-correctoo of cortical surfaces using nonseparating loops. IEEE Transactions on Medical Imaging, 26(4): 518-529, 2007.
[64] Zhengyang Shen, François-Xavier Vialard, and Marc Niethammer. Region-specific diffeomor- phic metric mapping. arXiv preprint arXiv:/906.0 /006.0006010010 00139, 2019.
[65] Edward Smith, Scott Fujimoto. Adriana Romero, and David Meger. GEOMetrics: Exploiting geometric structure for graph-encoded objects. Kamalika Chaudhuri and Ruslan Salakhutdi- nov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97
of Proceedings of Machine Learning Research, pages 5866-5876, Long Beach, California, USA, 09-15 Jun 2019. PMLR
14

---

[66] Zhengyu Su, Wei Zeng. Yalin Wang, Zhong-Ling Lin Lu, and Xianfeng Gu. Shape classification using wasserstein distance for brain morphometry analysis. International Conference on Information Processing in Medical Imaging, pages 411 413-423. Springer, 2015.
[67] Abdel Aziz Taha and Allan Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging. 15(1):1-28, 2015.
[68] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE International Conference on Computer Vision, pages 2088-2096, 2017.
[69] Alain Trouvé. An infinite dimensional group approach for physics based models in pattern recognition. preprint, 1995.
[70] François-Xavier Vialard, Laurent Risser, Daniel Rueckert, and Colin Cotter. Diffeomorphic 3d image registration via geodesic shooting using an efficient adjoint calculation. International Journal of Computer Vision, 97(2):229 241, 2012.
[71] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu- Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018.
[72] Peng-Shuai Wang, Chun- Yu Sun, Yang Liu, and Xin Tong. Adaptive o-cnn: a patch-based deep representation of 3d shapes. ACM Transactions on Graphics (TOG), 37(6):1-11, 2018.
[73] Udaranga Wickramasinghe, Edoardo Remelli, Graham Knott, and Pascal Fua. Voxel2mesh: 3d mesh model generation from volumetric data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 199-308. Springer, 2020.
[74] Jiajun Wu, Chengkai Zhang. Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum. Learning shape priors for single-view 3d completion and reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 646-662, 2018.
[75] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems 32, pages 492-502, 2019.
[76] Shuo Zhang, Peter Xiaoping Liu, Minhua Zheng, and Wen Shi. A diffeomorphic unsupervised method for deformable soft tissue image registration. Computers in biology and medicine, 120: 103708, 2020.
[77] Nikola Zubić and Pietro Liò. An effective loss function for generating 3d models from single 2d image without rendering. arXiv preprint arXiv:2 2021.
15


Reviews:
[{"review_id": "KRcN48TOliR", "reviewer": "Reviewer_FY8L", "summary": "This paper proposes a method called CorticalFlow, which is a geometric deep learning model to acquire brain surface mesh out of 3D volume images. It preserves topological property of the extracted mesh with relatively small computation using diffeomorphic transformations. Various results are demonstrated from experiments using ADNI dataset. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a nicely working method for brain surface deformation and prediction. \nIt provides theoretical contents and qualitatively/quantitatively good results. \nHowever, I have concerns regarding the completeness and self-containment of the manuscript. \nAlso, it is quite difficult to notice where the novelty of this work is, especially the explanation of core ideas are missing in the introduction. Also, deffeomorphic image/surface registration has been done as the authors have referenced in [13, 41]. \nWhile good empirical results demonstrate the effectiveness of the proposed method, I think it should be better elaborated in the manuscript. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Below are my comments:\n- Perhaps it is better to include a brief description of diffeomorphism for broader audience. A reader who do not know much about diffeomorphism will not be able to read and understand this paper well. \n- Eq (1) should be elaborated much better. What are x, s, \\Phi, \\tau and v? Again, to make this paper self-contained, kinder explanation on diffeomorphism, diffeomorphic mapping, its property, constrains and so on should be better explained. \n- Section 3 should start explaining the basic setting. Target task is well described but the initial settings are not explained. Are we given with a set of image I and surface S pairs for training? What do the surfaces look like? What are we exactly comparing in the loss function?\n- In Eq (2), the use of composition operation looks a bit strange without an input. If the \\circle is not a composition, it should be properly explained. \n- I did not find a proof for Theorem 3.1 in the paper nor in the supplement. \n- Functions/Parameters given in the Theorems should be mentioned in detail in the Theorems. \n- Based on the results from NMF given in Fig 1 and 3, it seems like NMF is not even close to getting the right surface. Is there a reason why NMF produces such smooth surfaces?\n- In Fig. 5, I cannot find highlighted regions... I just found them but they seem very very tiny. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "DCz8P1PHlSy", "reviewer": "Reviewer_ZUMJ", "summary": "This paper presents a cortical surface reconstruction method called CorticalFlow. The method uses a neural network to predict diffeomorphic deformation in the image space and then uses a Diffeomorphic Mesh Transformer (DMT) to generate the deformation field on the mesh. In practice, a cascade of three networks with different U-Net architecture and input resolution are used to predict the final mesh. The method is validated on ADNI with free-surfer output as groud-truth and compared with several other mesh construction methods. ", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method produces visually appealing results, which seem to improve the Freesurfer output and results in more regular meshes compared to the baseline approaches presented herein.  The main concern of mine is the clarity of writing and missing comparison with an important baseline. Here are my suggestions:\n\nIn general, I feel the method sections are written in a loose language and lack important connections and details. \n\nSection 3.1 describes the core DMT, but it only presents three theorems without specifying how DMT is defined. Is it a neural network or just a tri-linear interpolation from image space to mesh? Continuous math formulation, discrete solution, and implementation are intertwined together, making it hard to read.  Some sentences are out of context, e.g., \"obtained by tri-linear interpolation of a feature map U\", \"check that for all considered examples, and we have $hL \\leq 1$\". Moreover, Eq. 3 is in image space but Eq. 4 is on the discrete mesh. The connection here is not clear ($\\tau$ is not defined).  The implication of Theorem 3.3 on diffeomorphic properties is not clear, and how the criteria $hL \\leq 1$ relates to the DMT operation?\n\nSection 3.2 then introduces a series of engineering choices, which seem to be arbitrary and do not reflect the theory in Section 3.1. They seem to be specifically tailored for the dataset (number of networks, architecture, resolution). It is unclear why the U-Net parameterization can guarantee diffeomorphism. From my reading, $hL \\leq 1$ is the main mechanism for deriving smoothness but how $h$ and Lipchitz-$L$ are integrated into the implementation? Why is the template an inflated brain rather than the cortical surface of a template image (Fig.2)?\n\nAn important baseline is missing. VoxelMorph is the STOA approach for computing diffeomorphic velocity fields from CNN.  In my opinion, it is the closest baseline to the proposed approach. The comment on Voxelmorph in Section 2.2 is unfair. The velocity field is also constant for the proposed approach (section 3.1). Moreover, the velocity field in VoxelMorph is the initial shooting tangent vector associated with the Lie group of diffeomorphism, which relies on a  totally different concept. VoxelMorph can also directly work on high resolution meshes so the claim in Section 2.2 is not stringent.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As written above, clarity and missing comparison are two major concerns. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "7RDHk_NVwnH", "reviewer": "Reviewer_iY4e", "summary": "The paper presents a geometric deep learning model that predicts a high-resolution deformation field of a predefined template mesh given a 3D image. The model leverages the flow ODE framework to provide a scalable model that can deform high-resolution meshes with a small GPU memory footprint. The work is motivated by reconstructing/estimating brain cortical surfaces from MRI images. The model is trained using a surface matching loss where the deformed template is compared to the ground truth surface. The model has a hierarchical network structure to provide progressively higher resolution deformations. The proposed model is compared to SOTA surface reconstruction methods and a segmentation baseline method. The proposed model is faster, more memory-efficient, and provides anatomically plausible surfaces compared to SOTA methods.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses a significant problem and is motivated by a challenging task of brain cortical surface reconstruction. A comprehensive set of experiments comparing the proposed model with SOTA methods shows the superiority of the proposed method to produce regular, anatomically plausible, high resolution surfaces with less computation time and GPU memory footprint.\n\nThe work can be improved by addressing the below concerns:\n- The impact of template selection is not discussed. Does the predefined template induce bias? especially if the resulting surfaces are used for brain morphometry and statistical analyses.\n- The network is trained using a surface matching loss between pseudo-ground truth surfaces from FreeSurfer and the template mesh, which assumes vertex-to-vertex correspondence. How would the proposed model handle the case where this type of correspondence is not available in the training data?\n- In order for this model to work, training/testing samples should be \"roughly\" aligned to the template surface. The alignment assumption (rigid, similarity, or affine) is neither discussed nor acknowledged. It would provide more insights to explore the impact of misalignments in both training and testing. \n- It not clear how the template of inner and outer cortical surfaces are constructed to maintain their concentric nature using the convex hull of training surfaces, i.e., is there any guarantee that inner and outer template surfaces are not cross-penetrating each other? \n- Are different models training for the four surfaces (left inner, left outer, right inner, and right outer)? Or a single model for a pair of inner and outer surfaces? in either case, how is the concentric nature of these surfaces preserved by the network?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not acknowledge any potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0tF_i__8-6r", "reviewer": "Reviewer_oT7u", "summary": "This paper presents a new model for reconstructing cortical surfaces.\nThe method comprises multiple parts: first, dense 3D flow vector fields\nare learned from input volumetric images. The flow fields are then\nsubjected to a diffeomorphic mesh transform, which ensures ensures\nquality constraints of the resulting mesh. These transforms are used to\nimprove the geometry of a template mesh in different resolutions, until\nan output mesh of satisfactory quality is produced.\n\nGiven an experimental suite that compares different methods in this\ndomain, the proposed method turns out to perform favourably, exhibiting\nlower 'mesh defects' as well as closer similarity to template meshes.\nMoreover, the new method is significantly more efficient in terms of\ncomputational resources than prior work. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary of the review\n\nThe main contribution of this paper, i.e. a new mesh transformer that\naids in surface reconstruction tasks, is highly relevant for healthcare\napplications. I appreciate the connections between differential\nequations and diffeomorphisms as outlined in the main text. However, at\npresent, I cannot fully endorse this paper for publication yet because\nof certain issues:\n\n1. Lack of clarity: the current version of the manuscript lacks a clear\n   description of the method and required concepts. Theoretical\n   considerations are interspersed with algorithmic description, and\n   several key concepts are left undefined, leaving me to guess them\n   based on the context and/or related work. These shortcomings have to\n   be addressed prior to publication. (see below for more details)\n\n2. Issues with experimental setup: from reading some of the related work\n   cited by this manuscript, I understand that the experimental setup\n   might be lacking certain comparison metrics, such as the Earth\n   Mover's Distance (EMD), which features prominently in the 'DeepCSR'\n   paper, for instance. The current metrics shown in the paper appear to\n   be restricted to 'static' geometries, whereas EMD would actually try\n   to check for an optimised alignment. Plus, I am not sure about the\n   stability of the depicted metrics; the Hausdorff distance, for\n   instance, can easily be changed by a few outliers, even if the rest\n   of the mesh is of high quality. I therefore strongly suggest to\n   incorporate this additional metric (EMD) or discuss why it is not\n   appropriate here.\n\n## Comments concerning clarity\n\n- When introducing diffeomorphism for the first time, the paper should\n  ensure that the domain and the range of the diffeomorphism are clear.\n  I had to read into related work to gain a better understanding here,\n  so a more succinct definition would be appreciated.\n\n- Section 2.1 could provide some more details on the computational\n  issues plaguing existing methods. I think this would be appreciate by\n  readers.\n\n- How is the 'manifoldness' measure defined?\n\n- Section 2.2 requires more background information. It goes directly\n  into the details of the method, but details and the motivation are\n  left unclear. For instance, it would be helpful if some more details\n  about the penalty functions were being provided.\n\n- What does it mean for the reconstructed surface to be regular? Is this\n  the standard definition from differential geometry (i.e. there must be\n  a local smooth homeomorphism etc.)? Please provide a cursory\n  definition of this.\n\n- Equation 1 needs more explanations. What are the different functions\n  involved in this equation? What is their image, their range, etc.\n  I found the paper by Niemeyer et al. [43] very useful to learn more\n  about the background. I would recommend to update this manuscript\n  according to their background section (I am using this paper only\n  because it was the first one I encountered when reading more deeply\n  into this topic).\n\n- Why are there no other comparison partners in Figure 1? From\n  subsequent figures, it seems that other methods are doing reasonably\n  well on such data; Figure 1 makes it seem as if the proposed method is\n  the only one capable of addressing the problem of cortical surface\n  reconstruction.\n\n- At the end of Section 2.2, please provide more background details on\n  neural ODEs; the subsequent discussions require more details because\n  they are hard to understand otherwise.\n\n- In Equation 2, I find the $\\circ$ notation to be slightly misleading.\n  $\\mathcal{T}$ is *not* a function, but $\\Phi \\circ \\mathcal{T}$ seems\n  to suggest that $\\Phi$ is applied after applying a function\n  $\\mathcal{T}$, when the intent is rather that $\\Phi_{\\Theta}$ is\n  a function receiving an input image and resulting in a transformation\n  from $\\mathcal{T}$ to $\\mathbf{S}$. Essentially, $\\Phi$ thus returns\n  a *function*, which the notation should correctly indicate.\n\n- Theorem 3.1 requires more precision. $\\Phi$ is not technically\n  a homeomorphism, but rather a sequence of homeomorphisms. This is\n  correctly indicated in the parenthetical remark given in this theorem,\n  but should be reworded nonetheless. I would suggest to clarify that\n  this theorem is a well-known results in differential geometry;\n  essentially, it states that the homotopy type of a space remains\n  unchanged under a homeomorphism. Please clarify this and provide an\n  adequate citation (e.g. from a textbook).\n\n- Theorem 3.2 reiterates Equation 1; I would suggest to show the same\n  equation only once and provide more information (as mentioned above,\n  the individual components of this equation need to be discussed).\n  Specifically, I am wondering: what are the *properties* of the\n  vector field $v(\\cdot)$ shown in Equations 1 and 3?\n\n- I am wondering whether the proof of THeorem 3.3 has to be provided in\n  the main material. Given that it glosses over certain details (for\n  instance, it was initially unclear to me why $f$ should be a Lipschitz\n  mapping; one or two sentences more would be appreciated here, lest the\n  reader has to reconstruct this property for themselves).\n\n- The paper often appears to 'jump' between the 'mesh' perspective and\n  the 'manifold' perspective. This needs to be made more precise. For\n  instance, the input to $\\Psi$ can be point arising from a manifold,\n  but the paper states that $\\Psi(h, B)$ is a non-intersecting manifold.\n  However, if a space is self-intersecting it cannot be a manifold by\n  definition. I guess that the statement rather refers to a mesh\n  satisfying the manifold condition (i.e. regular neighbourhoods of\n  faces etc.), which *still* intersects geometrically (whereas the\n  manifold condition of a mesh is a purely topological property). Is\n  this correct? This needs additional clarification.\n\n- What is the meaning of $\\|\\|v\\|\\|$ being high as discussed in l. 228?\n\n- When describing the training, how is the loss function defined?\n  I understand that ground truth data exists, but I could not find\n  a succinct explanation of the actual training process in the main\n  manuscript.\n\n## Style comments\n\n- I would suggest not to use citations as nouns. Please use `\\citet`\n  when citing something in the text.\n\n- Instead of stating 'has a 0 genus', I would say 'is of genus 0'.\n\n- 'Lipchitz' --> 'Lipschitz'\n\n- 'Denote $O = $...': I would rephrase this to state (for instance):\n  'The image origin is denoted by...'", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not adequately discussed in the current manuscript.\nI understand that the proposed method is beneficial for cortical surface\nreconstruction. Nevertheless, a discussion of error cases or limitations\nwould be necessary. For instance, are there types of meshes where the\nproposed method does not result in satisfactory results? How much data\nis required in order to obtain suitable results? Such questions should\nat least be briefly addressed and discussed in a revision of the paper. \n\nBroader societal ramifications are not directly addressed in the text,\nbut given the positioning in a specific topic of healthcare, I foresee\nno adverse effects arising out of this particular application.\nA discussion of reconstruction quality would nevertheless be helpful\nhere; I wonder if (or to what extent) current medical databases are\nbiased.\n\n# Update after rebuttal\n\nWhile I appreciate the clarifications of the authors (thus raising my score to 5), I am still convinced that the changes we discussed require a major revision that is outside the scope of a conference cycle.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a geometric deep learning approach to process volumetric images to extract mesh representations from them via a set of learned deformations aimed to capture a manifold (namely a surface) from the 3D input image. As the principal application and motivation for this work, it is demonstrated on cortical surface reconstruction. While the idea and approach may be interesting, the reviewers have all raised some significant concerns regarding several aspects of this work. It seems not all of these concerns have been addressed by author responses, and in any case the revisions that would be required in order to address them would be sufficiently substantial to necessitate another round of reviews, thus not fitting within the scope allowed by the conference review process. Therefore, following discussion, a unanimous decision has been reached to recommended this manuscript be rejected at this stage. 

To the authors: while the outcome of this review may be disappointing, I hope the feedback you get from it will help you to improve your manuscript, and I would recommend that you go over the reviewer remarks and revise it accordingly in preparing future submissions.
Decision: Accept (Poster)