SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate different registration methods on our dataset, including conventional registration without learning, image-based deep learning (DL) methods, and point cloud-based DL methods. As detailed above, the DIR-LAB COPDgene dataset with its 20 CT scans of 10 COPD patients at breathhold inspiration and expiration serves as test set for the proposed benchmark. For each scan pair, 300 manual landmarks with high inter-rater agreement are available for quantifying registration accuracy. Since the scan pairs have already been cropped based on the lung masks (bounding boxes) the initial registration error of the COPDgene dataset is lowered after this implicit translation alignment from  $23.36\mathrm{mm}$  to  $16.25\mathrm{mm}$ . First, we evaluate the quality of the provided pseudo-labels, by extrapolating the sparse corrField keypoint correspondences [20] of the test images to the manual landmark coordinates using a thin-plate-spline transform. This results in an accuracy of  $1.45\mathrm{mm}$ . Next, we apply the conventional 3D image registration method deeds [19] that was designed for deformable lung alignment and performed best in a comparative study of six SOTA methods for abdominal registration [59]. The only modification was to supply 3D scans where everything outside the lungs is masked out. It runs for about 10 minutes on a 28-core CPU and yields a target registration error (TRE) of  $1.53\mathrm{mm}$ .
Since the very popular baseline of VoxelMorph [2] does not achieve satisfactory performance on highly deformable 3D registration, we employ the VoxelMorph+ adaptation of [18] that adapts the
$^{9}$ https://grand-challenge.org/algorithms/corrfield/
Table 2: Quantitative results on the COPD test cases of image-based (left) and point-based (right) methods, reported as mean TRE and  ${25}/{50}/{75}\%$  percentiles in mm. IO: Instance optimisation
Method	TRE	25%	50%	75%	Method	TRE	25%	50%	75%
initial	16.25	10.14	15.94	21.76	initial	16.25	10.14	15.94	21.76
corrField	1.45	0.83	1.23	1.77	CPD	3.13	1.51	2.28	3.58
deeds	1.53	0.83	1.22	1.79	CPD w/ labels	2.59	1.36	2.01	3.16
VM+ w/o IO	6.53	3.38	5.82	8.50	PPWC sup.	2.85	1.52	2.33	3.54
VM+ w/ IO	4.31	0.87	1.55	7.42	PPWC syn.	2.73	1.52	2.28	3.45
VM++ w/o IO	4.47	2.41	3.74	5.69
VM++ w/ IO	2.26	0.75	1.16	1.90
Table 3: Quantitative results of Voxelmorph++ (with instance optimisation) on the test cases with different sub-sets of the dataset as training data, reported as mean TRE and  $25/50/75\%$  percentiles in mm.
Setup	TRE	25%	50%	75%
initial	16.25	10.14	15.94	21.76
full dataset	2.26	0.75	1.16	1.90
TCIA-NSCLC	5.07	1.07	2.11	8.93
TCIA-NLST	3.30	0.90	1.46	3.27
Empire10	3.35	0.90	1.47	3.57
TCIA-Ventilation	3.19	0.89	1.42	2.86
L2R-LungCT	2.90	0.88	1.41	2.63
loss to using a combination of MIND features and a Laplace regularisation. The network extends the basic U-Net with a heatmap prediction head. When trained for approx. 6 hours on the proposed dataset, it achieves a TRE of  $6.53\mathrm{mm}$  (feed-forward network only) or  $4.31\mathrm{mm}$  (when followed by test-time instance optimisation (IO)) with a runtime of only 2 s per case.
As a final benchmark method for image-based approaches, we explore the benefits of incorporating the keypoint supervision provided for this new dataset and train the complete VoxelMorph++ approach [18]. This yields a TRE of  $4.47\mathrm{mm}$  (network only) and  $2.26\mathrm{mm}$  (followed by IO), which is an improvement of nearly  $50\%$  solely based on the stronger supervision that is provided.
For point cloud-based registration, we implement the PointPWC-Net [56] as it is part of the current SOTA method for point cloud-based lung registration on the PVT dataset [48]. We train the network with two different supervision strategies on the subsampled clouds with 8k points: 1) Training on real inhale-exhale pairs supervised with the automatic correspondences. 2) Similar to [56], training pairs consist of one real cloud and a synthetic deformation (by a 2-scale random field) such that point correspondences are precisely known. The two approaches achieve  $2.85\mathrm{mm}$  and  $2.73\mathrm{mm}$  TRE, respectively, with an inference time of  $0.2\mathrm{s}$ . In addition, we employ the optimisation-based Coherent Point Drift (CPD) with implementation choices detailed in the provided source code. Running for 50 iterations in approx.  $14\mathrm{s}$  it reaches  $3.13\mathrm{mm}$  TRE and can be further improved by incorporating the provided vein/artery labels to  $2.60\mathrm{mm}$  (label CPD).
To evaluate how training on the full Lung250M-4B training dataset compares to the use of singular sub-datasets, we performed ablation experiments using Voxelmorph++ with different training setups, using only one of the sub-datasets for each training. Results are presented in Tab. 3 and range from  $5.07\mathrm{mm}$  to  $2.90\mathrm{mm}$  TRE with each setup being outperformed by training on the full dataset. This demonstrates the benefits of our large-scale combined dataset from multiple sources.
All above described experiments are performed with the proposed fixed train/Validation/test split. To evaluate the remaining domain gap between training and test data and the suitability of this split, we trained Voxelmorph++ on DIR-LAB COPDgene in a 2-fold cross-validation manner in addition to the full dataset as training data. This setup results in a TRE of  $2.10\mathrm{mm}$ , only slightly better than the achieved  $2.26\mathrm{mm}$  TRE, when training with the proposed split.
Comparison to Published Work One objective of our work is to provide a more comparable benchmark for evaluating 3D deformable (lung) registration on both images and point clouds. Hence, the two previously separated research fields are joined more closely. Current SOTA image-based methods achieved  $7.98\mathrm{mm}$  [2],  $3.83\mathrm{mm}$  [38],  $1.34\mathrm{mm}$  [16] and  $0.82\mathrm{mm}$  [47] on the same test data, where the last method is a non-learning based approach that requires a runtime of 5 minutes. Our best benchmark method yielding  $2.24\mathrm{mm}$  comes close to the current best DL-network. For point cloud SOTA two different preprocessing strategies were proposed in prior work: 1) small but expressive Förstner keypoint clouds that yield a TRE of  $2.44\mathrm{mm}$  with a graph network plus optimiser [15] and 2) large-scale vessel clouds with additional geometric but no semantic features (PVT dataset) that yield a TRE of  $2.86\mathrm{mm}$  using a PointPWC-Net with optimal transport and postprocessing [48]. A PointPWC-Net trained on the PVT1010 dataset under a training setup comparable to the one we employed in our experiments achieves a TRE of  $4.50\mathrm{mm}$  [6].
This highlights that Lung250M-4B dataset is very suitable for both image-based and/or point-cloud-based approaches and hybrid approaches that combine convolutional feature extraction with graph neural networks are likely to benefit. Interestingly, our almost naive coherent point drift (CPD) baseline can substantially outperform the CPD results in [48]  $(9.30\mathrm{mm})$ , which indicates that our data generation that enables both semantic labels and relatively expressive point clouds is useful.
6 Technical Limitations
The wide scope of potential methods to extract point clouds from 3D medical volumes and generate labels and correspondences for supervision make it impossible to study the influence of all implementation choices on our provided dataset. The chosen nnUNet architecture has been proven to be very robust for 3D medical segmentation and the 100 manually labelled pulmonary artery trees appear sufficient to train a reliable model. Nevertheless, uncommon anatomical variations or pathologies may impede the quality of point cloud extraction and subsequent analysis of derived measures, e.g. motion vectors and volume change. The provided automatic correspondences are well distributed (see Fig. 3) and yield an excellent agreement with manual landmarks (see Tab. 2) but nevertheless constitute noisy labels. Learning from "perfect" synthetic transformations or at least adding them to a combined supervision may further improve performance (see results of PointPwC in Tab. 2).
The size of our dataset is about  $3 - 4 \times$  larger than previous medical registration benchmarks (apart from inter-subject brain databases) but still considered small in comparison to computer vision. Please note that nearly no other 3D+time datasets with deformable motion and (pseudo) ground truth motion exists otherwise and dynamic FAUST [8] comprises only 10 different subjects. Finally, the set of benchmark methods is not complete, but we strongly believe the provision of Lung250M-4B will encourage researchers from both 3D medical imaging and geometric deep learning (for motion) to challenge their latest methods and to set new best scores.
We introduced the Lung250M-4B dataset as a public benchmark for large motion exhale-to-inhale lung registration. The dataset stands out from existing benchmarks in three aspects. 1) It is the first combined benchmark for point- and image-based registration. 2) It features 124 images, which is roughly one order of magnitude more than previous large-motion lung datasets. 3) It provides accurate keypoint correspondences for supervised learning.
In our experiments, we evaluated diverse registration methods, including classical and deep learning-based approaches for point- and image-based registration, on the dataset to provide strong baselines. The experiments revealed several insights, pointing to promising future research directions. 1) Classical optimisation methods achieved the best results for both image- and point-based registration, generally demonstrating room for improvement of learning-based methods. 2) DL-based point cloud registration is comparable with DL-based image registration even though point-based methods do not employ IO. Point-based IO is to date barely studied in the literature and could potentially make point-based methods surpass image-based DL methods. 3) The inclusion of artery and vein labels could improve performance of a classical optimization approach (CPD). Hence, they could also be a valuable source of supervision for future training of DL-based methods.
Overall, our work highlights great potential of point cloud-based lung registration. However, to fully understand the potential of point-based methods in medical registration, similar analyses and benchmarks for other anatomies are needed in future work. If our findings transfer to such other anatomies, point-based registration, with the inherent benefits of anonymity preservation, robustness, and computational efficiency, might become an increasingly important component in medical registration.
We are grateful to Rohit Jena and Kayhan Batmanghelich for providing us with a pre-trained version of their vessel segmentation model [26]. We also thank Lasse Hansen for sharing his efficient coherent point drift implementation.
[1] B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1):26-41, 2008.
[2] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. Dalca. Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging, 38(8):1788-1800, 2019.
[3] S. Balik, E. Weiss, N. Jan, N. Roman, W. C. Sleeman, M. Fatyga, G. E. Christensen, C. Zhang, M. J. Murphy, J. Lu, et al. Evaluation of 4-dimensional computed tomography to 4-dimensional cone-beam computed tomography deformable image registration for lung cancer adaptive radiation therapy. International Journal of Radiation Oncology* Biology* Physics, 86(2): 372-379, 2013.
[4] Z. M. Baum, Y. Hu, and D. C. Barratt. Real-time multimodal image registration with partial intraoperative point-set data. Medical image analysis, 74:102231, 2021.
[5] P. J. Besl and N. D. McKay. Method for registration of 3-D shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[6] A. Bigalke and M. P. Heinrich. A denoised mean teacher for domain adaptive point cloud registration. arXiv preprint arXiv:2306.14749, 2023.
[7] F. Bogo, J. Romero, M. Loper, and M. J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3794-3801, 2014.
[8] F. Bogo, J. Romero, G. Pons-Moll, and M. J. Black. Dynamic FAUST: Registering human bodies in motion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6233-6242, 2017.
[9] R. Castillo, E. Castillo, D. Fuentes, M. Ahmad, A. M. Wood, M. S. Ludwig, and T. Guerrero. A reference dataset for deformable image registration spatial accuracy evaluation using the COPDgene study archive. Physics in Medicine & Biology, 58(9):2861, 2013.
[10] K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle, et al. The cancer imaging archive (TCIA): maintaining and operating a public information repository. Journal of digital imaging, 26:1045-1057, 2013.
[11] E. M. Eslick, J. Kipritidis, D. Gradinscak, M. J. Stevens, D. L. Bailey, B. Harris, J. T. Booth, and P. J. Keall. CT ventilation imaging derived from breath hold CT exhibits good regional accuracy with galligas PET. Radiotherapy and Oncology, 127(2):267-273, 2018.
[12] E. M. Eslick, J. Kipritidis, D. Gradinscak, M. J. Stevens, D. L. Bailey, B. Harris, J. T. Booth, and P. J. Keall. CT Ventilation as a functional imaging modality for lung cancer radiotherapy (CTvs-PET-Ventilation-Imaging), 2022. URL https://wiki.cancerimagingarchive.net/x/YIF8Bw.
[58] Y. Xiao, H. Rivaz, M. Chabanas, M. Fortin, I. Machado, Y. Ou, M. P. Heinrich, J. A. Schnabel, X. Zhong, A. Maier, et al. Evaluation of MRI to ultrasound registration methods for brain shift correction: the CuRIOUS2018 challenge. IEEE transactions on medical imaging, 39(3): 777-786, 2019.
[59] Z. Xu, C. P. Lee, M. P. Heinrich, M. Modat, D. Rueckert, S. Ourselin, R. G. Abramson, and B. A. Landman. Evaluation of six registration methods for the human abdomen on clinically acquired CT. IEEE Transactions on Biomedical Engineering, 63(8):1563-1572, 2016.
[60] B. T. Yeo, M. R. Sabuncu, T. Vercauteren, N. Ayache, B. Fischl, and P. Golland. Spherical demons: fast diffeomorphic landmark-free surface registration. IEEE transactions on medical imaging, 29(3):650-668, 2009.
[61] G. Zhao, K. Liang, C. Pan, F. Zhang, X. Wu, X. Hu, and Y. Yu. Graph convolution based cross-network multi-scale feature fusion for deep vessel segmentation. IEEE Transactions on Medical Imaging, 2022.