Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a method called CorticalFlow, which is a geometric deep learning model to acquire brain surface mesh out of 3D volume images. It preserves topological property of the extracted mesh with relatively small computation using diffeomorphic transformations. Various results are demonstrated from experiments using ADNI dataset. 
main_review: This paper proposes a nicely working method for brain surface deformation and prediction. 
It provides theoretical contents and qualitatively/quantitatively good results. 
However, I have concerns regarding the completeness and self-containment of the manuscript. 
Also, it is quite difficult to notice where the novelty of this work is, especially the explanation of core ideas are missing in the introduction. Also, deffeomorphic image/surface registration has been done as the authors have referenced in [13, 41]. 
While good empirical results demonstrate the effectiveness of the proposed method, I think it should be better elaborated in the manuscript. 

limitations_and_societal_impact: Below are my comments:
- Perhaps it is better to include a brief description of diffeomorphism for broader audience. A reader who do not know much about diffeomorphism will not be able to read and understand this paper well. 
- Eq (1) should be elaborated much better. What are x, s, \Phi, \tau and v? Again, to make this paper self-contained, kinder explanation on diffeomorphism, diffeomorphic mapping, its property, constrains and so on should be better explained. 
- Section 3 should start explaining the basic setting. Target task is well described but the initial settings are not explained. Are we given with a set of image I and surface S pairs for training? What do the surfaces look like? What are we exactly comparing in the loss function?
- In Eq (2), the use of composition operation looks a bit strange without an input. If the \circle is not a composition, it should be properly explained. 
- I did not find a proof for Theorem 3.1 in the paper nor in the supplement. 
- Functions/Parameters given in the Theorems should be mentioned in detail in the Theorems. 
- Based on the results from NMF given in Fig 1 and 3, it seems like NMF is not even close to getting the right surface. Is there a reason why NMF produces such smooth surfaces?
- In Fig. 5, I cannot find highlighted regions... I just found them but they seem very very tiny.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a new model for reconstructing cortical surfaces.
The method comprises multiple parts: first, dense 3D flow vector fields
are learned from input volumetric images. The flow fields are then
subjected to a diffeomorphic mesh transform, which ensures ensures
quality constraints of the resulting mesh. These transforms are used to
improve the geometry of a template mesh in different resolutions, until
an output mesh of satisfactory quality is produced.

Given an experimental suite that compares different methods in this
domain, the proposed method turns out to perform favourably, exhibiting
lower 'mesh defects' as well as closer similarity to template meshes.
Moreover, the new method is significantly more efficient in terms of
computational resources than prior work. 

main_review: ## Summary of the review

The main contribution of this paper, i.e. a new mesh transformer that
aids in surface reconstruction tasks, is highly relevant for healthcare
applications. I appreciate the connections between differential
equations and diffeomorphisms as outlined in the main text. However, at
present, I cannot fully endorse this paper for publication yet because
of certain issues:

1. Lack of clarity: the current version of the manuscript lacks a clear
   description of the method and required concepts. Theoretical
   considerations are interspersed with algorithmic description, and
   several key concepts are left undefined, leaving me to guess them
   based on the context and/or related work. These shortcomings have to
   be addressed prior to publication. (see below for more details)

2. Issues with experimental setup: from reading some of the related work
   cited by this manuscript, I understand that the experimental setup
   might be lacking certain comparison metrics, such as the Earth
   Mover's Distance (EMD), which features prominently in the 'DeepCSR'
   paper, for instance. The current metrics shown in the paper appear to
   be restricted to 'static' geometries, whereas EMD would actually try
   to check for an optimised alignment. Plus, I am not sure about the
   stability of the depicted metrics; the Hausdorff distance, for
   instance, can easily be changed by a few outliers, even if the rest
   of the mesh is of high quality. I therefore strongly suggest to
   incorporate this additional metric (EMD) or discuss why it is not
   appropriate here.

## Comments concerning clarity

- When introducing diffeomorphism for the first time, the paper should
  ensure that the domain and the range of the diffeomorphism are clear.
  I had to read into related work to gain a better understanding here,
  so a more succinct definition would be appreciated.

- Section 2.1 could provide some more details on the computational
  issues plaguing existing methods. I think this would be appreciate by
  readers.

- How is the 'manifoldness' measure defined?

- Section 2.2 requires more background information. It goes directly
  into the details of the method, but details and the motivation are
  left unclear. For instance, it would be helpful if some more details
  about the penalty functions were being provided.

- What does it mean for the reconstructed surface to be regular? Is this
  the standard definition from differential geometry (i.e. there must be
  a local smooth homeomorphism etc.)? Please provide a cursory
  definition of this.

- Equation 1 needs more explanations. What are the different functions
  involved in this equation? What is their image, their range, etc.
  I found the paper by Niemeyer et al. [43] very useful to learn more
  about the background. I would recommend to update this manuscript
  according to their background section (I am using this paper only
  because it was the first one I encountered when reading more deeply
  into this topic).

- Why are there no other comparison partners in Figure 1? From
  subsequent figures, it seems that other methods are doing reasonably
  well on such data; Figure 1 makes it seem as if the proposed method is
  the only one capable of addressing the problem of cortical surface
  reconstruction.

- At the end of Section 2.2, please provide more background details on
  neural ODEs; the subsequent discussions require more details because
  they are hard to understand otherwise.

- In Equation 2, I find the $\circ$ notation to be slightly misleading.
  $\mathcal{T}$ is *not* a function, but $\Phi \circ \mathcal{T}$ seems
  to suggest that $\Phi$ is applied after applying a function
  $\mathcal{T}$, when the intent is rather that $\Phi_{\Theta}$ is
  a function receiving an input image and resulting in a transformation
  from $\mathcal{T}$ to $\mathbf{S}$. Essentially, $\Phi$ thus returns
  a *function*, which the notation should correctly indicate.

- Theorem 3.1 requires more precision. $\Phi$ is not technically
  a homeomorphism, but rather a sequence of homeomorphisms. This is
  correctly indicated in the parenthetical remark given in this theorem,
  but should be reworded nonetheless. I would suggest to clarify that
  this theorem is a well-known results in differential geometry;
  essentially, it states that the homotopy type of a space remains
  unchanged under a homeomorphism. Please clarify this and provide an
  adequate citation (e.g. from a textbook).

- Theorem 3.2 reiterates Equation 1; I would suggest to show the same
  equation only once and provide more information (as mentioned above,
  the individual components of this equation need to be discussed).
  Specifically, I am wondering: what are the *properties* of the
  vector field $v(\cdot)$ shown in Equations 1 and 3?

- I am wondering whether the proof of THeorem 3.3 has to be provided in
  the main material. Given that it glosses over certain details (for
  instance, it was initially unclear to me why $f$ should be a Lipschitz
  mapping; one or two sentences more would be appreciated here, lest the
  reader has to reconstruct this property for themselves).

- The paper often appears to 'jump' between the 'mesh' perspective and
  the 'manifold' perspective. This needs to be made more precise. For
  instance, the input to $\Psi$ can be point arising from a manifold,
  but the paper states that $\Psi(h, B)$ is a non-intersecting manifold.
  However, if a space is self-intersecting it cannot be a manifold by
  definition. I guess that the statement rather refers to a mesh
  satisfying the manifold condition (i.e. regular neighbourhoods of
  faces etc.), which *still* intersects geometrically (whereas the
  manifold condition of a mesh is a purely topological property). Is
  this correct? This needs additional clarification.

- What is the meaning of $\|\|v\|\|$ being high as discussed in l. 228?

- When describing the training, how is the loss function defined?
  I understand that ground truth data exists, but I could not find
  a succinct explanation of the actual training process in the main
  manuscript.

## Style comments

- I would suggest not to use citations as nouns. Please use `\citet`
  when citing something in the text.

- Instead of stating 'has a 0 genus', I would say 'is of genus 0'.

- 'Lipchitz' --> 'Lipschitz'

- 'Denote $O = $...': I would rephrase this to state (for instance):
  'The image origin is denoted by...'
limitations_and_societal_impact: Limitations are not adequately discussed in the current manuscript.
I understand that the proposed method is beneficial for cortical surface
reconstruction. Nevertheless, a discussion of error cases or limitations
would be necessary. For instance, are there types of meshes where the
proposed method does not result in satisfactory results? How much data
is required in order to obtain suitable results? Such questions should
at least be briefly addressed and discussed in a revision of the paper. 

Broader societal ramifications are not directly addressed in the text,
but given the positioning in a specific topic of healthcare, I foresee
no adverse effects arising out of this particular application.
A discussion of reconstruction quality would nevertheless be helpful
here; I wonder if (or to what extent) current medical databases are
biased.

# Update after rebuttal

While I appreciate the clarifications of the authors (thus raising my score to 5), I am still convinced that the changes we discussed require a major revision that is outside the scope of a conference cycle.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper presents a cortical surface reconstruction method called CorticalFlow. The method uses a neural network to predict diffeomorphic deformation in the image space and then uses a Diffeomorphic Mesh Transformer (DMT) to generate the deformation field on the mesh. In practice, a cascade of three networks with different U-Net architecture and input resolution are used to predict the final mesh. The method is validated on ADNI with free-surfer output as groud-truth and compared with several other mesh construction methods. 
main_review: The method produces visually appealing results, which seem to improve the Freesurfer output and results in more regular meshes compared to the baseline approaches presented herein.  The main concern of mine is the clarity of writing and missing comparison with an important baseline. Here are my suggestions:

In general, I feel the method sections are written in a loose language and lack important connections and details. 

Section 3.1 describes the core DMT, but it only presents three theorems without specifying how DMT is defined. Is it a neural network or just a tri-linear interpolation from image space to mesh? Continuous math formulation, discrete solution, and implementation are intertwined together, making it hard to read.  Some sentences are out of context, e.g., "obtained by tri-linear interpolation of a feature map U", "check that for all considered examples, and we have $hL \leq 1$". Moreover, Eq. 3 is in image space but Eq. 4 is on the discrete mesh. The connection here is not clear ($\tau$ is not defined).  The implication of Theorem 3.3 on diffeomorphic properties is not clear, and how the criteria $hL \leq 1$ relates to the DMT operation?

Section 3.2 then introduces a series of engineering choices, which seem to be arbitrary and do not reflect the theory in Section 3.1. They seem to be specifically tailored for the dataset (number of networks, architecture, resolution). It is unclear why the U-Net parameterization can guarantee diffeomorphism. From my reading, $hL \leq 1$ is the main mechanism for deriving smoothness but how $h$ and Lipchitz-$L$ are integrated into the implementation? Why is the template an inflated brain rather than the cortical surface of a template image (Fig.2)?

An important baseline is missing. VoxelMorph is the STOA approach for computing diffeomorphic velocity fields from CNN.  In my opinion, it is the closest baseline to the proposed approach. The comment on Voxelmorph in Section 2.2 is unfair. The velocity field is also constant for the proposed approach (section 3.1). Moreover, the velocity field in VoxelMorph is the initial shooting tangent vector associated with the Lie group of diffeomorphism, which relies on a  totally different concept. VoxelMorph can also directly work on high resolution meshes so the claim in Section 2.2 is not stringent.
limitations_and_societal_impact: As written above, clarity and missing comparison are two major concerns.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a geometric deep learning model that predicts a high-resolution deformation field of a predefined template mesh given a 3D image. The model leverages the flow ODE framework to provide a scalable model that can deform high-resolution meshes with a small GPU memory footprint. The work is motivated by reconstructing/estimating brain cortical surfaces from MRI images. The model is trained using a surface matching loss where the deformed template is compared to the ground truth surface. The model has a hierarchical network structure to provide progressively higher resolution deformations. The proposed model is compared to SOTA surface reconstruction methods and a segmentation baseline method. The proposed model is faster, more memory-efficient, and provides anatomically plausible surfaces compared to SOTA methods.
main_review: The paper addresses a significant problem and is motivated by a challenging task of brain cortical surface reconstruction. A comprehensive set of experiments comparing the proposed model with SOTA methods shows the superiority of the proposed method to produce regular, anatomically plausible, high resolution surfaces with less computation time and GPU memory footprint.

The work can be improved by addressing the below concerns:
- The impact of template selection is not discussed. Does the predefined template induce bias? especially if the resulting surfaces are used for brain morphometry and statistical analyses.
- The network is trained using a surface matching loss between pseudo-ground truth surfaces from FreeSurfer and the template mesh, which assumes vertex-to-vertex correspondence. How would the proposed model handle the case where this type of correspondence is not available in the training data?
- In order for this model to work, training/testing samples should be "roughly" aligned to the template surface. The alignment assumption (rigid, similarity, or affine) is neither discussed nor acknowledged. It would provide more insights to explore the impact of misalignments in both training and testing. 
- It not clear how the template of inner and outer cortical surfaces are constructed to maintain their concentric nature using the convex hull of training surfaces, i.e., is there any guarantee that inner and outer template surfaces are not cross-penetrating each other? 
- Are different models training for the four surfaces (left inner, left outer, right inner, and right outer)? Or a single model for a pair of inner and outer surfaces? in either case, how is the concentric nature of these surfaces preserved by the network?
limitations_and_societal_impact: The authors did not acknowledge any potential negative societal impact of their work.
review_title: null

Meta-review: This paper proposes a geometric deep learning approach to process volumetric images to extract mesh representations from them via a set of learned deformations aimed to capture a manifold (namely a surface) from the 3D input image. As the principal application and motivation for this work, it is demonstrated on cortical surface reconstruction. While the idea and approach may be interesting, the reviewers have all raised some significant concerns regarding several aspects of this work. It seems not all of these concerns have been addressed by author responses, and in any case the revisions that would be required in order to address them would be sufficiently substantial to necessitate another round of reviews, thus not fitting within the scope allowed by the conference review process. Therefore, following discussion, a unanimous decision has been reached to recommended this manuscript be rejected at this stage. 

To the authors: while the outcome of this review may be disappointing, I hope the feedback you get from it will help you to improve your manuscript, and I would recommend that you go over the reviewer remarks and revise it accordingly in preparing future submissions.
Decision: Accept (Poster)