SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mesh loss. It aims to minimize distances of the vertices between the predicted surface meshes  $S_W$  (and  $S_G$ ) and their corresponding ground truth (GT) meshes  $S_*$  by the bidirectional Chamfer distance [26]:
$$
\mathcal {L} _ {c h W} = \sum_ {\mathbf {p} \in \mathcal {S} _ {W}} \min  _ {\mathbf {p} _ {*} \in \mathcal {S} _ {W _ {*}}} \| \mathbf {p} - \mathbf {p} _ {*} \| _ {2} ^ {2} + \sum_ {\mathbf {p} _ {*} \in \mathcal {S} _ {W _ {*}}} \min  _ {\mathbf {p} \in \mathcal {S} _ {W}} \| \mathbf {p} _ {*} - \mathbf {p} \| _ {2} ^ {2}, \tag {3}
$$
where  $\mathbf{p}$  and  $\mathbf{p}_*$  are the coordinates of vertices on meshes. We can compute  $\mathcal{L}_{chG}$  analogously. The mesh loss is  $\mathcal{L}_{ch} = \mathcal{L}_{chW} + \mathcal{L}_{chG}$ .
Trajectory loss. Starting from the midthickness surface, the trajectory length of the vertex moving to the WM and the pial surfaces should be equal. We propose to compute the mean square difference of the vertex's trajectories:
$$
\mathcal {L} _ {\text {d i s t}} = \frac {1}{N} \sum_ {\mathbf {p} \in \Omega} \| L _ {\text {M i d} \rightarrow G M} (\mathbf {p}) - L _ {\text {M i d} \rightarrow W M} (\mathbf {p}) \| _ {2} ^ {2}, \tag {4}
$$
where  $L_{Mid\rightarrow GM}(\mathbf{p}) = \sum_{t = 0}^{T}\Delta \phi_{G,t}(\mathbf{p})$  is the accumulated Euclidean distance (i.e., trajectory length) of  $T$  steps of deformation. This term encourages the midthickness surface can be deformed to the inner and outer cortical surfaces with the same deformation path length. In other words, the midthickness surface should lie halfway between the inner and outer cortical surfaces.
Symmetric cycle loss. We formulate it as Eq. 2 to encourage  $\phi_W$  and  $\phi_G$  to be invertible.
Symmetric similarity loss. To optimize the midthickness surface to lie halfway between the inner and outer cortical surfaces, a magnitude difference constraint is adopted directly on the SVFs:
$$
\mathcal {L} _ {s s} = \left\| \mathbf {v} _ {G} - \overline {{\mathbf {v}}} _ {W} \right\| _ {2} ^ {2}, \tag {5}
$$
where  $\overline{\mathbf{v}}_W$  represents reverse-directional  $\mathbf{v}_W$ .
Normal consistency loss. We also incorporate a normal consistency regularization term to promote robust learning of the surfaces and ensure their smoothness:
$$
\mathcal {L} _ {n c} = \sum_ {e \in E, f _ {0} \cap f _ {1} = e} (1 - \cos \left(\mathbf {n} _ {f _ {0}}, \mathbf {n} _ {f _ {1}}\right)), \tag {6}
$$
where  $e$  is an edge,  $f_0$  and  $f_1$  are  $e$ 's two neighboring faces with their unit normals  $\mathbf{n}_{f_0}$  and  $\mathbf{n}_{f_1}$ .
In summary, we combine all the losses to jointly optimize our DL model:  $\mathcal{L} = \lambda_1\mathcal{L}_{ch} + \lambda_2\mathcal{L}_{dist} + \lambda_3\mathcal{L}_{cyc} + \lambda_4\mathcal{L}_{ss} + \lambda_5\mathcal{L}_{nc}$ , where  $\{\lambda_i\}_{i=1,\dots,5}$  are weights to balance the loss terms. We empirically set  $\lambda_i = 1$  ( $i = 1,\dots,4$ ) and  $\lambda_5 = 0.001$ .
We evaluated our method for reconstructing both white-matter (WM) and pial surfaces on two large-scale datasets, (ADNI) [22] and OASIS [33], and compared it with state-of-the-art (SOTA) DL-based CSR methods. We also tested its robustness and performed ablation analyses.
Datasets. The ADNI-1 [22] dataset consists of 817 subjects and we randomly split it into 654, 50, and 113 subjects for training, validation, and testing, respectively. The OASIS-1 [33] dataset consists of 413 subjects and we randomly split it into 330, 25, and 58 for training, validation, and testing, respectively. The models were trained on the training set until they reached a loss plateau on the validation set, after which their performance was evaluated on the test set. We followed pre-processing protocols in previous works [10, 13, 26, 30] for fair comparison. The T1-weighted MRI scans were aligned rigidly to the MNI152 template and clipped to the size of  $192 \times 224 \times 192$  at  $1mm^3$  isotropic resolution. The pseudo ground-truth of ribbon segmentation and cortical surfaces were generated using FreeSurfer v7.2.0 [17]. The intensity values of MRI scans, ribbon segmentation maps, and SDFs were normalized to  $[0, 1]$  and the coordinates of the vertices were normalized to  $[-1, 1]$ . The WM and GM in the cortical ribbon segmentation maps were assigned values of 0.5 and 1 respectively.
Implementation details. Our framework was implemented in PyTorch [40] and trained on an NVIDIA P100 GPU of 16 GB memory. The 3D U-Net [44] for ribbon segmentation was trained for 200 epochs using Adam [23] optimization and achieved an average Dice index of 0.96 on the testing set. The CSR model was trained for 400 epochs using Adam ( $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 1e^{-10}$ , learning rate  $1e^{-4}$ ) to optimize the midthickness surface and reconstruct the WM and pial surfaces for each hemisphere. The surface meshes had  $\sim 130K$  vertices.
Evaluation metrics. We utilized three distance-based metrics to measure the CSR accuracy, including Chamfer distance (CD), average symmetric surface distance (ASSD), and Hausdorff distance (HD). In particular, CD measures the mean distance between two sets of vertices [16, 52]; ASSD and HD measure the average and maximum distances between two surfaces [13, 50]. They are computed bidirectionally over  $\sim 130K$  points uniformly sampled from the predicted and target surfaces. A lower distance indicates a better result. We used the 90th percentile instead of the maximum because HD is sensitive to outliers [21]. We also utilized the ratio of self-intersection faces (SIF) to measure surface quality [13, 30].
4.1 Comparison with Related Works
From the two categories of existing DL-based cortical surface reconstruction methods described in Section 1, we selected representative ones from each category for comparison. The experimental results are summarized in Table 2 and illustrated in Figure 4.
Main Results & Analysis. It is evident that our method achieved substantial improvement on both the WM and pial surface reconstruction over other approaches. Since DeepCSR [13] predicts an SDF-based implicit surface and requires post-processing to correct topology and extract a mesh, its results may contain no SIFs but were less accurate compared with the explicit CSR methods. Starting from the WM surface, PialNN [31] can achieve sub-voxel accuracy but its SIF ratio was relatively high. Vox2Cortex [30] can generate multiple surfaces from different template meshes. It employs complex CNN and GNN models to model the deformation for each vertex but has no diffeomorphism guarantee. The promising results of [26, 30, 47] indicated that using neural networks to parameterize the ODE can facilitate the diffeomorphic deformation, yielding better CSR accuracy. Our method
Table 2: Quantitative analysis of cortical surface reconstruction on geometric accuracy and surface quality. The Chamfer distance (CD), average symmetric surface distance (ASSD), Hausdorff distance (HD), and the ratio of the self-intersecting faces (SIF) were measured for WM and pial surfaces on the two datasets. The mean value and standard deviation are reported. The best ones are in bold.
Method	L-Pial Surface	L-WM Surface
CD (mm)	ASSD (mm)	HD (mm)	SIF (%)	CD (mm)	ASSD (mm)	HD (mm)	SIF (%)
ADNI	DeepCSR [13]	0.945±0.078	0.593±0.065	1.149±0.203	\	0.938±0.076	0.587±0.064	1.137±0.193	\
PiaiNN [31]	0.621±0.035	0.465±0.044	1.002±0.106	0.137±0.093	\	\	\	\
CorticalFlow [26]	0.691±0.043	0.497±0.049	1.106±0.115	0.149±0.087	0.641±0.037	0.465±0.042	0.996±0.100	0.108±0.073
CorticalFlow++ [47]	0.545±0.036	0.410±0.033	0.886±0.069	0.098±0.067	0.544±0.034	0.401±0.030	0.878±0.066	0.069±0.042
cortexODE [30]	0.476±0.017	0.214±0.020	0.455±0.058	0.022±0.012	0.458±0.016	0.192±0.015	0.436±0.014	0.015±0.011
Vox2Cortex [10]	0.582±0.028	0.370±0.025	0.746±0.057	0.059±0.039	0.577±0.027	0.353±0.022	0.722±0.055	0.043±0.023
Ours	0.410±0.016	0.136±0.012	0.293±0.026	0.035±0.021	0.213±0.008	0.071±0.005	0.155±0.012	0.007±0.010
OASIS	DeepCSR [13]	0.986±0.085	0.617±0.070	1.331±0.212	\	0.975±0.081	0.594±0.067	1.151±0.197	\
PiaiNN [31]	0.635±0.032	0.460±0.038	0.993±0.082	0.141±0.096	\	\	\	\
CorticalFlow [26]	0.687±0.040	0.495±0.047	1.082±0.110	0.147±0.086	0.637±0.035	0.462±0.040	0.992±0.097	0.101±0.070
CorticalFlow++ [47]	0.531±0.035	0.399±0.030	0.812±0.057	0.088±0.045	0.529±0.033	0.398±0.030	0.810±0.055	0.086±0.042
cortexODE [30]	0.481±0.019	0.218±0.021	0.461±0.062	0.026±0.015	0.463±0.018	0.207±0.017	0.435±0.015	0.018±0.010
Vox2Cortex [10]	0.588±0.032	0.381±0.030	0.750±0.063	0.061±0.037	0.581±0.028	0.375±0.027	0.731±0.059	0.046±0.027
Ours	0.442±0.014	0.161±0.012	0.348±0.025	0.037±0.023	0.218±0.007	0.073±0.006	0.159±0.013	0.008±0.011
achieved the overall best performance due to its explicit regularizations on the deformation trajectory of vertices and its better initialization surface. On the ADNI dataset, our method achieved  $\sim 48.8\%$  improvement in mean ASSD (of WM and pial surfaces) compared to the second best CortexODE (i.e.,  $0.104mm$  v.s  $0.203mm$ ) with competitive self-intersection ratio (average  $\sim 0.021\%$ ). On the OASIS dataset, our method achieved similar performance improvement. More quantitative results on the WM surface are reported in the Supplementary Materials. As shown in Fig. 4, the CSR results obtained by our method had uniformly smaller errors across the whole surfaces.
Figure 4: Visualization of the reconstructed surfaces with distance compared to ground truth.
Runtime Analysis. It took  $\sim 0.8s$  for our method to obtain a ribbon segmentation map and another  $\sim 2s$  for the topology correction and surface initialization. For the surface reconstruction, the inference time for simultaneously reconstructing three (i.e., WM, pial, and midthickness) surfaces was  $\sim 1.5s$ . In comparison, the SOTA explicit CSR method CortexODE [30] needed  $\sim 2s$  to reconstruct two surfaces sequentially. Overall, our method is computationally as efficient as the SOTA alternatives.
Input. Our method takes as input a multi-channel 3D images  $I_{comb}$  and an initialization midthickness surface  $S_0$ . We conducted two experiments to analyze the influence of the input components on the CSR performance (Table 3 Top). First, by gradually removing components from  $I_{comb}$ , we observed a significant drop in accuracy, indicating the contribution of both the SDF and segmentation maps to the final results. Second, we investigated the impact of the fine structure of the initialization surface by applying Laplacian smoothing on  $S_0$  to generate oversmoothed initialization surfaces. The results revealed that the accuracy of the model decreased with the increasing of smoothing steps which resulted in a coarser initialization surface  $S_0$ .
Loss functions. We evaluated the contribution of different losses of our method to the surface reconstruction performance in terms of both accuracy (CD, ASSD, HD) and topological correctness (SIF), with the results summarized in Table 3 Middle. Through the ablation studies, we observed
Table 3: Ablation studies with the surface reconstruction results quantified in terms of CD, ASSD, HD, and SIF. The setting I0, S0, and  $T = 5$  all refer to our complete setting (cf. Table 2). Top: The impact of the input image and initialization surface. Middle: The impact of the loss functions. Bottom: The impact of the deformation steps in DDM.
Setting	Input	Initial mesh # of Lap. Sm.	L-Pial Surface	L-WM Surface
I	SDF	Seg	CD (mm)	ASSD (mm)	HD (mm)	CD (mm)	ASSD (mm)	HD (mm)
I0	✓	✓	✓	0	0.410±0.016	0.136±0.012	0.293±0.026	0.213±0.008	0.071±0.005	0.155±0.012
I1	✓	✓	0	0.426±0.017	0.167±0.017	0.358±0.038	0.222±0.011	0.075±0.006	0.164±0.013
I2	✓	0.453±0.021	0.201±0.026	0.438±0.074	0.250±0.013	0.085±0.008	0.184±0.016
M1	✓	✓	✓	10	0.416±0.016	0.147±0.013	0.315±0.028	0.225±0.010	0.084±0.007	0.184±0.015
M2	✓	✓	✓	20	0.429±0.018	0.163±0.017	0.361±0.040	0.235±0.012	0.091±0.009	0.190±0.017
Setting	Loss	L-Pial Surface	L-WM Surface
LCH	Ldist	Lcyc	Lss	Lnc	CD (mm)	ASSD (mm)	HD (mm)	SIF (%)	CD (mm)
S0	✓	✓	✓	✓	✓	0.410±0.016	0.136±0.012	0.293±0.026	0.035±0.021	0.213±0.008
S1	✓	✓	✓	✓	✓	0.412±0.016	0.138±0.012	0.299±0.026	0.036±0.021	0.213±0.010
S2	✓	✓	✓	0.412±0.016	0.139±0.012	0.302±0.027	0.037±0.022	0.211±0.009
S3	✓	✓	0.409±0.016	0.135±0.012	0.300±0.027	0.275±0.100	0.209±0.009
S4	✓	0.404±0.015	0.129±0.011	0.278±0.024	2.522±0.791	0.203±0.009
Number of steps (T) in DDM	Surface	Surface
CD (mm)	ASSD (mm)	HD (mm)	SIF(%)	CD (mm)	ASSD (mm)	HD (mm)
5	0.410±0.016	0.136±0.012	0.293±0.026	0.035±0.021	0.213±0.008	0.071±0.005	0.155±0.012
4	0.411±0.015	0.137±0.012	0.295±0.026	0.038±0.022	0.208±0.009	0.072±0.006	0.157±0.012
3	0.411±0.016	0.138±0.012	0.297±0.026	0.044±0.025	0.205±0.009	0.072±0.006	0.158±0.013
2	0.415±0.016	0.143±0.013	0.309±0.028	0.432±0.110	0.204±0.008	0.075±0.005	0.166±0.012
that each component played its own role in a complementary way. The first row (referred to as S0) corresponds to our complete network setting, while the last row (S4) represents using Chamfer distance alone. The results of setting S4 indicated that the model generated surfaces well matched to the ground truth data at the cost of high topological errors, particularly in highly curved regions, reflected by the results that the SIF ratio was worse on the pial surface than on the WM surface. Enforcing equality of the trajectories from the midthickness surface to the WM and pial surfaces (S3,  $\mathcal{L}_{dist}$ ) helped optimize the midthickness surface, thereby preventing deformation in an arbitrary direction and reducing self-intersection. However, the geometric accuracy slightly decreased, which might be caused by the difficulty in accessing highly curved regions or deep sulci under such strong topology constraints. The proposed symmetric cycle loss (S2,  $\mathcal{L}_{cyc}$ ) promoted the invertibility of deformations, yielding a significant reduction of self-intersections on the meshes since our method jointly reconstructs both the inner and outer surfaces by deforming the midthickness surface inward and outward with two VFs. Such invertibility also facilitates accurate estimation of the cortical thickness from the trajectory, as illustrated by a sample vertex deformation trajectory in Fig. 2 obtained with settings of S2 and S3. Moreover, the inclusion of regularization terms on the smoothness of SVFs (S1,  $L_{ss}$ ) and surfaces (S0,  $L_{nc}$ ) contributed to enhancement in surface quality. Overall, our proposed method struck a balance between geometric accuracy and topology quality.
Deformation steps in DDM. Table 3 Bottom shows the evaluation results of different numbers of deformation steps  $(T)$  in DDM. As  $T$  increased, the performance first improved and then saturated, indicating that five steps were sufficient to deform the midthickness surface to the inner or outer surfaces.
4.3 Reproducibility
We carried out two experiments on two datasets: a paired ADNI $_{1.5\&3\mathrm{T}}$  dataset [22] consisting of 1.5T and 3T MRIs of the same subjects, and the Test-Retest dataset [32] comprising 40 MRIs collected within a short period for each of the 3 subjects. In these scenarios, the cortical surfaces of the same subject should be nearly identical. Following the experimental setup outlined in [10, 13, 30], we utilized the iterative closest-point algorithm (ICP) to align image pairs and computed the geometric distance between surfaces. The
Table 4: Reproducibility analysis.
Method	L-WM Surface
CD (mm)	ASSD (mm)	HD (mm)
ADNI-pair	Ours	0.520±0.053	0.337±0.058	0.738±0.151
CortexODE	0.521±0.056	0.340±0.060	0.741±0.154
DeepCSR	0.618±0.103	0.397±0.080	0.823±0.211
FreeSurfer	0.556±0.049	0.364±0.054	0.764±0.118
TRT	Ours	0.451±0.019	0.235±0.030	0.492±0.059
CortexODE	0.457±0.021	0.238±0.031	0.504±0.071
DeepCSR	0.505±0.047	0.297±0.053	0.610±0.100
FreeSurfer	0.476±0.015	0.253±0.022	0.519±0.048
results for the left WM surfaces are presented in Table 4 (more in Supplementary Materials),
demonstrating that our method obtained superior reproducibility compared with FreeSurfer and was comparable to the SOTA DL methods.
4.4 Cortical Thickness
In contrast to the alternative methods that rely on the ICP algorithm for registering WM and pial surfaces prior to calculating the Euclidean distance [10], our proposed method directly provides vertex-wise cortical thickness estimation. To validate the cortical thickness estimation, we compared our method with FreeSurfer for estimating the cortical thickness. We identified 200 subjects from the ADNI-2GO [22] dataset (100 are diagnosed with Alzheimer's disease and 100 are normal controls) and computed the average cortical thickness across 35 cortical regions based on a surface parcellation provided by FreeSurfer [17]. Fig. 5 shows the correlation between ours and FreeSurfer's results, showcasing the effectiveness of our proposed framework in accurately capturing cortical thickness.
Figure 5: Correlation between prediction (Y-axis) and FreeSurfer's thickness (X-axis) on 35 cortical regions  $(mm)$ .
Despite achieving improved CSR accuracy and a low SIF ratio, our method can be further improved by adopting post-processing methods and new loss functions in order to minimize the SIF ratio and improve surface quality. While focusing on cortical thickness estimation in this paper, we recognize the value of incorporating other cortical attributes like surface area and sulci depth into CSR and analysis tasks. These attributes could serve as complementary constraints, enhancing overall performance. It should be noted that further analysis is merited to thoroughly evaluate the proposed method on a large cohort of subjects (e.g., subjects in different stages of AD) although we have demonstrated the correlation between the estimated cortical thickness and that of FreeSurfer on a balanced dataset.
We introduce a new DL framework for cortical surface reconstruction by generating a midthickness surface to initialize a coupled reconstruction of both the WM and pial surfaces. Specifically, the midthickness surface is estimated from a 3D distance map from each MRI by generating a cortical ribbon segmentation map that encodes structural information of the cerebral cortex. The estimated midthickness surface is represented as a triangular mesh with spherical topology, and the mesh is optimized to lie at the center of the inner and outer cortical surfaces and deformed to the inner and outer cortical surfaces by three diffeomorphic flows that are learned jointly with CNNs optimized with a multi-channel input consisting of the brain MRI, the 3D distance map of midthickness surface, and the cortical ribbon segmentation map. Our proposed symmetric cycle loss helps learn diffeomorphic deformation and the numerical solution of DDM improves CSR accuracy and computation efficiency. Experiments on two large-scale neuroimage datasets have demonstrated the superior performance of our method. Moreover, our method generates an estimation of cortical thickness, facilitating statistical analyses of brain atrophy.
This work was supported in part by the NIH grants AG066650 and EB022573.
[1] V. I. Arnold. Ordinary differential equations. Springer Science & Business Media, 1992.
[54] X. Wen, J. Zhou, Y.-S. Liu, H. Su, Z. Dong, and Z. Han. 3d shape reconstruction from 2d images with disentangled attribute flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3803-3813, 2022.
[55] U. Wickramasinghe, E. Remelli, G. Knott, and P. Fua. Voxel2Mesh: 3D mesh model generation from volumetric data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 299-308. Springer, 2020.
[56] Z. Wu, F. Zhao, F. Wang, W. Lin, L. Wang, D. Shen, and G. Li. Surface-based analysis of the developing cerebral cortex. In Advances in Magnetic Resonance Technology and Applications, volume 2, pages 287-307. Elsevier, 2021.
[57] H. Xie, H. Yao, S. Zhang, S. Zhou, and W. Sun. Pix2Vox++: Multi-scale context-aware 3d object reconstruction from single and multiple images. International Journal of Computer Vision, 128(12):2919-2935, 2020.
[58] S. Yang, M. Xu, H. Xie, S. Perry, and J. Xia. Single-view 3d object reconstruction from shape priors in memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3152-3161, 2021.
[59] H. Zheng, H. Li, and Y. Fan. SurfNN: Joint reconstruction of multiple cortical surfaces from magnetic resonance images. In International Symposium on Biomedical Imaging, pages 1-4, 2023.