SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fenja Falta
Institute of Medical Informatics
University of Lubeck
fenja.falta@student.uni-luebeck.de
ChristophGroßbröhmer
c.grossbroehmer@uni-luebeck.de
Alessa Hering
Departments of Imaging
Radboud University Medical Center, Nijmegen
Alexander Bigalke
Mattias P. Heinrich
University of Lübeck
A popular benchmark for intra-patient lung registration is provided by the DIR-LAB COPDgene dataset consisting of large-motion in- and expiratory breathhold CT pairs. This dataset alone, however, does not provide enough samples to properly train state-of-the-art deep learning methods. Other public datasets often also provide only small sample sizes or include primarily small motions between scans that do not translate well to larger deformations. For point-based geometric registration, the PVT1010 dataset provides a large number of vessel point clouds without any correspondences and a labeled test set corresponding to the COPDgene cases. However, the absence of correspondences for supervision complicates training, and a fair comparison with image-based algorithms is infeasible, since CT scans for the training data are not publicly available. We here provide a combined benchmark for image- and point-based registration approaches. We curated a total of 248 public multi-centric in- and expiratory lung CT scans from 124 patients, which show large motion between scans, processed them to ensure sufficient homogeneity between the data and generated vessel point clouds that are well distributed even deeper inside the lungs. For supervised training, we provide vein and artery segmentations of the vessels and thousands of image-derived keypoint correspondences for each pair. For validation, we provide multiple scan pairs with manual landmark annotations. Finally, as baselines on our new benchmark, we evaluate several image and point cloud registration methods on the dataset.
Image registration continues to be an important and challenging task in medical computer vision. A large part of recently developed methods for registration incorporates deep learning [13]. With advances in geometric deep learning, point cloud-based registration approaches have become increasingly more prevalent and compete well with image-based approaches, especially in terms of efficiency and robustness [39]. However, as deep learning is data-driven, its development is highly dependent on the existence of appropriate datasets and benchmarks.
As the results of the Learn2Reg benchmark [21] indicate, 3D lung registration remains challenging for learning-based approaches, and contrary to other image analysis tasks, conventional optimisation-based approaches remain state-of-the-art (SOTA). We believe this is in part due to the limitation of available lung datasets, which typically lack both sample size and range of motion, thereby restricting further advances through deep learning methods.
With Lung250M-4B we propose a combined dataset consisting of Lung4B, 248 paired CT scans from 124 patients (with approximately 4 billion voxels in total), and Lung250M, with corresponding lung vessel point clouds with approximately 250 million 3D points and features. This dataset comprises multi-centric data from different examination types with more cases than comparable datasets and furthermore enables a fair comparison between image- and point cloud-based registration approaches. The dataset is available at https://cloud.imi.uni-luebeck.de/s/s64fqbPpXNxBPP.
Clinical Motivation for Lung Registration 3D registration in general often deals with the alignment of scenes and (deformable) objects therein, e.g. for shape completion and scene flow estimation. 3D medical registration, in particular, focuses on finding detailed anatomical or functional correspondences. This could be either for different time points of the same patient or across a population for shape analysis. While subtle small deformations can be reliably estimated with a range of voxel- and intensity-based approaches [2, 29], larger lung motion is more difficult to capture and hence graph- or point-cloud based alignment has become more prominent [48, 47].
Finding correspondences between different respiratory levels of lung CT within the same patient is vital for the diagnosis and characterisation of chronic obstructive pulmonary disease (COPD) [14]. In COPD local airflow is limited due to small airway disease or emphysema, and accurate nonlinear displacement fields provide an accurate means for calculating local volume change (and hence areas of limited airflow) based on the Jacobian determinant [45]. Two other important areas of lung registration are: estimation of tumour motion from 4D-CT in radiotherapy planning [54] and identification and tracking of lung nodules for screening and diagnostics of lung cancer [22, 40].
Benefits of Learning on Point Clouds In recent years there has been an increase in research regarding point cloud learning, which can also be seen in medical research (e.g. [52, 61]). The main difference between point cloud and image data lies in discarding image (intensity) information that is either redundant or not necessary for the task, which can be advantageous. First, focusing on only relevant sparse information makes algorithms more efficient and reduces the risk of overfitting to subtle irrelevant appearance details. Second, graph-based networks excel at learning representations that are more closely aligned to underlying anatomical concepts (e.g. vessel tree topology) and invariant to certain operations (e.g. local rotations). Third, for sensitive data, in particular in the medical domain, the public sharing of images is often not possible due to patient privacy. Datasets thus often remain private. Point clouds generated from CT scans, however, hold significantly less information that makes the patient identifiable than CT scans themselves. Hence, point cloud-based computer vision methods preserve anonymity and enable access to more data.
Medical registration models. The majority of methods address 3D image registration on a dense grid of control points (e.g. B-splines [37]) and minimise a joint cost function of image dissimilarity (e.g. normalised cross-correlation [1]) using a discrete [47], continuous or stochastic optimiser [29]. Deep learning-based approaches follow either a multi-scale architecture [38], use correlation windows for a set of discretised displacements [50, 17] or focus on more subtle diffeomorphic deformations [2]. While there are certain approaches, including spherical demons [60] or graph- and learning-based methods [49] that address cortical matching on the surface of brains [28], only few works addressed medical registration on the basis of purely geometric point clouds. Standard classical approaches like Iterative Closest Point [5] or Coherent Point Drift [43] do naturally apply to the problem but were shown insufficient for registering highly complex structures in [48]. [27] extended the Bayesian Coherent Point Drift to the registration of multiple organ surface point clouds. As a learning-based method, [4] proposed a PointNet-based model for prostate surface registration, trained in an unsupervised manner with the Chamfer distance. [15] performed keypoint-based lung registration by combining a graph network for geometric feature learning with loopy belief propagation for differentiable optimisation, which, however, required keypoint correspondences for
supervision. Finally, [48] combined the deep learning-based PointPWC-Net [56] with modules from optimal transport theory for pre-alignment and fine-tuning to register high-resolution lung vessel trees. Training was performed on synthetic deformations.
Medical registration datasets. Starting from the retrospective intermodality brain registration evaluation [55] that considered only rigid alignment, numerous challenges and public datasets have been proposed for medical image registration, including lung registration [42] and multi-modality MRI to ultrasound registration [58]. A comprehensive challenge that also considered certain aspects of learning-based method is Learn2Reg [21], which comprises several complementary tasks and has compared dozens of SOTA approaches. Nevertheless, apart from OASIS - an inter-subject brain mapping tasks - all mentioned datasets comprise 50 or fewer scans and are thus not ideally suited for deep learning approaches. Public benchmarks for point cloud-based medical image registration are rare since most above point cloud registration methods were evaluated on in-house datasets [4] or the keypoints were derived with private tools and not made publicly available [27]. The only public medical point cloud dataset is the PVT dataset [48], providing 1000 unlabeled inhale-exhale pairs of lung vessel trees along with 10 labeled test cases, each annotated with 300 landmark correspondences.
3D point cloud registration benchmarks. Beyond medical applications, 3D scene flow and point cloud registration are active research fields in computer vision and graphics, where diverse public benchmarks are available. The Flying Things 3D dataset [34] features rigid motions of synthetic 3D object models. Similarly, synthetic 3D models from the ModelNet40 [57] dataset are rigidly transformed and subsampled for partial-to-partial registration. Piecewise rigid motions on real point cloud data are particularly present in datasets from autonomous driving, including the KITTI Scene Flow [36, 35] and Waymo [51] datasets. Unlike (piecewise) rigid motions, the Faust [7] and Dynamic Faust [8] datasets feature deformable human motions. Moreover, the DeformingThings4D [32] and derived 4DMatch datasets [31] provide frames from animated motion sequences of humans and diverse animals. Beyond the different (medical) scope, our dataset differs from all above benchmarks in two decisive technical aspects. 1) Our point clouds do not only represent surfaces but complex interior structures, making the registration particularly challenging and impeding unsupervised learning with similarity metrics. 2) We jointly provide image and point cloud data, enabling a fair comparison of image and point-based methods and facilitating the development of methods that incorporate both types of data.
Contribution Lung250M-4B substantially extends prior lung registration datasets by a factor of  $3 - 4\times$  and provides paired publicly available CT scans and high-resolution point clouds with the aim to bring together research from 3D vision and medical imaging. We enhance the raw data substantially by adding a large amount of automatic high-quality correspondences on both images and point clouds to provide stronger supervision with the potential to ease and/or improve training. We also provide initial benchmark evaluation for several baselines: image- and point cloud-based, learning- and optimisation-based, unsupervised/self-supervised and supervised.
We curated 248 publicly available 3D CT scans of 124 patients from various medical centres, with image acquisition ranging from in- and expiratory breath-hold CT over maximum and minimum volume scans of 4DCT scans to base and follow-up scan pairs at different time points. Data are acquired from various scanners with various resolutions and slice thicknesses, and show a wide variety regarding occurring pathologies. An overview of source datasets used for curation can be found in Tab. 1, while detailed information regarding acquisition centres, subject pathologies and image metadata is provided in the supplementary material.
The DIR-LAB COPDgene dataset[9], published in 2013, includes ten randomly picked pairs of CT-breath hold examinations obtained as part of the COPDGene study (NCT00608764) to evaluate genetic factors for the development of COPD in smokers. Each scan pair is associated with manually annotated landmarks  $(n = 300)$  and an inter-observer landmark localisation error, rendering the dataset well-suited for evaluation of deformable lung registration algorithms. The data are not released under
Table 1: Overview of the data set curation. We include a total of 248 lung scans from different examination types (BH: Breath-Hold, LS: Longitudinal Scans, 4DCT) from a total of 6 public and freely available data collections in our dataset. The last column shows the mean change (and its standard deviation) of the lung volume between the two images of a scan pair.
Source Dataset	# Included Scan Pairs	# Annotated Cases (# Landmarks/case)	Examination Type	Mean Lung Volume Change in ml (±std)
DIR-LAB COPDgene	10	10 (300)	BH	1770 ± 708
Empire10	12	2 (100)	BH,4DCT	1950 ± 1274
L2R-LungCT	30	3 (100)	BH	2155 ± 428
TCIA-NSCLC	20	-	4DCT	426 ± 201
TCIA-Ventilation	20	2 (100)	BH	1137 ± 559
TCIA-NLST	32	10 (100)	LS	766 ± 505
a standard licence, but can be obtained after filing an online request form, with the restriction that the source be acknowledged in case of publication<sup>1</sup>.
The EMPIRE10 dataset was released as part of a MICCAI 2010 workshop for a representative benchmarking of lung registration algorithms [42]. It consists of a total of 30 scan pairs intended to represent a wide range of clinical applications. From this set, we choose the subsets "Insp-Exp" (8 cases) and "4D" (4 cases) to ensure sufficient lung volume change between scans. The dataset is not released under a standard licence, but can be obtained from the EMPIRE10 challenge website<sup>2</sup>.
L2R-LungCT is one of various medical registration tasks included in the Learn2Reg challenge [21] with the goal of registration of in- and expiratory lung CT scans. In total, 30 scan pairs from this task are publicly available<sup>3</sup>, published under CC-BY-4.0. Only scan pairs that a) were breath-hold examinations, b) had sufficient lung coverage in both images, and c) had more than 300 slices were considered. All identifying patient data were anonymized. To determine registration accuracy, manual annotations for 3 validation cases have been released, containing 100 landmarks generally located at vessel, airways and parenchyma bifurcations and surfaces.
The National Lung Screening Trial (NLST, NCT00047385) [44, 53, 10] compared the application of low-dose CT to chest radiography in high-risk lung cancer patients in the USA. More than 26,000 of the 75,000 low-dose CTs are freely available through the Cancer Imaging Archive under a CC-BY 4.0 licence, making it one of the largest medical imaging resources available<sup>4</sup>. The imaging protocol did not provide for a defined breathing state, so the baseline and follow-up scans may have been acquired at different times of the respiratory cycle. Our goal is to include only scan pairs with a sufficiently large lung motion in our dataset. Therefore, a larger set of scan pairs including 281 subjects was initially considered but narrowed down to 22 cases with a minimum lung volume change of  $380\mathrm{ml}$ . In addition, we included additional 10 scan pairs used in the Learn2Reg challenge, which were released with manual landmark annotations. Because we have no influence on the selection of validation cases, the mean change in lung volume for validation scan pairs is  $247\mathrm{ml}$ .
The TCIA-Ventilation dataset[12, 10] contains image data from a study comparing scans during breath-hold examinations, free-breathing examinations, and lung ventilation examinations using Galigas PET with a total of 20 participants [11]. The study was approved by the local ethics committee and registered with the Australian New Zealand Clinical Trials Registry (ACTRN12612000775819). All data have been licenced under CC-BY-4.0 and can be accessed via  $\mathrm{TCIA}^5$ . We retrieve all (20) breath-hold inhale/exhale scan pairs from this dataset.
<sup>1</sup>https://med.emory.edu/departments/radiation-oncology/research-laboratories/deformable-image-registration/downloads-and-reference-data/copdgene.html
$^{2}$ https://empire10.grand-challenge.org/Download/
3https://zenodo.org/record/4279348
<sup>4</sup>https://wiki.cancerimagingarchive.net/display/NLST/National+Lung+Screening+Trial
<sup>5</sup>https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=125600096
Figure 1: Visualization of three sample cases from our dataset. For each case, we show an overlay of fixed (blue) and moving (orange) CT slices (top) and skeletonised 3D lung vessel trees (bottom).
The TCIA-NSCLC data collection[23, 24, 3, 46, 10] includes CT scans of 20 patients who received radiation for the treatment of non-small cell lung cancer, released under CC-BY-3.0 that can be obtained via  $\mathrm{TCIA^6}$ . Prior to image-guided radiation therapy, a 4DCT was acquired from each patient. We include all scans with maximal inspiration or expiration phases, resulting in 20 image pairs.
Additional Validation Cases The proposed dataset consists of 6 subsets, of which only NLST, Learn2Reg, and COPDgene already have scan pairs with manual annotations for in-dataset registration validation. Students with a background in medicine further annotated two additional cases each for the TCIA-Ventilation and EMPIRE10 subsets, following the procedure described in [42] using the freely available software isimatch<sup>7</sup> [41] with 100 distinct landmarks on each scan.
Ethical Discussion The presented Lung250M-4B dataset is a composite of previously published de-identified data collections. Hence, the ethical risk assessment of re-releasing the data is particularly linked to their original publication. We chose to use only data whose associated studies had been published in reputable peer-reviewed journals. In addition, the benchmark datasets EMPIRE10 and L2R-LungCT were reviewed and accepted by the MICCAI conference organisers and are widely used in the medical image registration community. At TCIA, providing medical data can only be done when abiding by the ethical code of conduct, which explicitly requires permission from a local institutional review board if needed. Therefore, we do not see any unassessed ethical risk in publishing the images and information extracted from the dataset. We remark, however, that the occurring pathologies do not reflect the general population (cf. Suppl. Tab. 1) and our dataset is only intended for research purposes and not for clinical usage. Because the dataset we present aims to establish a fair basis for evaluation between image- and point-cloud-based lung registration, we further consider the publication as an important long-term contribution to anonymity-preserving point cloud research in the medical domain.
Our methods for dataset generation comprise steps of image analysis, point cloud and graph processing as well as sparse deformable keypoint matching.
Image Processing To ensure sufficient similarity between the curated data from different datasets, we employ joint preprocessing steps. For each image, a lung segmentation has been generated via a
$^{6}$ https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=21267414
<sup>7</sup>https://www.isi.uu.nl/research/software/isimatch/
Figure 2: Image preprocessing and subsequent generation of point clouds. First, the original scan gets resampled and cropped around the lung mask. Then, arteries and veins are segmented via a trained nnUNet. Based on these, vessel point clouds are generated, which are subsequently skeletonised and reduced to obtain a sparse yet representative point cloud.
trained nnUNet [25] prediction. The network has been trained on 500 publicly available lung CT scans from a variety of different centres and scanners with  $2\mathrm{mm}$  resolution. Since resolution between the original images differ greatly, images and predicted lung masks are resampled to isotropic  $1\mathrm{mm}$  resolution using the public c3d toolbox and, if necessary, reoriented to RAI orientation. The scans are then cropped around the lung masks with a distance of  $10\mathrm{mm}$  to the lung segmentation.
To enable a reproducible pre-processing on further data, e.g. from a private dataset, all source code, tools, scripts and models to perform these steps are provided (cf. Supplementary Material).
Vessel and Point Cloud Extraction We trained another nnUNet [25] to predict vessel segmentations of CT scans, divided into vein and artery labels. To train the network, we used the Parse22 challenge [33] dataset, consisting of 100 training CT scans with ground truth artery segmentations. This dataset is resampled and cropped as described above to ensure applicability of the segmentation network on the registration data. To obtain full vessel segmentations for the training data, we employed a self-supervised vessel enhancement method as described in [26]. We used this vessel segmentation as a base and then marked voxels of the segmentation as either vein or artery voxel. This is done by overlaying the vessel segmentation with the ground truth artery segmentations, however dilated by  $1\mathrm{mm}$  in each direction to ensure consistent labeling of the whole vessel. Mirroring has been disabled as data augmentation to provide the network with more geometric information to distinguish arteries and veins. The nnUNet achieves a validation Dice on the Parse22 dataset (with ground truth semantic segmentations generated as described) of  $81.4\%$  for the vein and  $84.9\%$  for the artery. After inference on the registration data, segmentation predictions have been masked to the inside of the lung again to eliminate potential artefacts.
Based on the predicted vessel segmentations, we generated point clouds covering the full segmentation information by sampling each labelled voxel coordinate after interpolation by a scale factor of 2. This results in point clouds with  $.5\mathrm{mm}$  spacing and over 300k 3D points (1.1M on average) for each case. With these, different sampling strategies can be evaluated. To obtain points that are sampled with a similar density in both shallower and deeper parts inside of the lung, we employ a skeletonisation of the vessel segmentations following the method described in [30], resulting in 30'000 points on average per case. Since many graph networks require a fixed number of points in every cloud, we also
<sup>8</sup>http://www.itksnap.org/pmwiki/pmwiki.php?p=n=Convert3D.Convert3D
Figure 3: Generation of keypoint correspondences: a) geometric interest points on slice of fixed CT scan, b) 3D motion vectors of corrField, c) distribution of initial keypoint errors across datasets.
provide a sparse subset of these skeleton point clouds. For this purpose, we create a kNN-graph from the skeleton point clouds and employ a Gaussian density estimation followed by non-local maximum suppression to sample either exactly 8'192 or 16'384 voxel coordinates for each cloud. For all points, we provide the label of the voxel (artery resp. vein) and the vessel radius as features. A visualisation of the preprocessing and point cloud extraction pipeline can be seen in Fig. 2.
Automatic Correspondences To aid learning-based methods, we computed a large number of highly accurate one-to-one correspondences across matching anatomical details of each scan pair using the public corrField method[20]. Based on image-derived features and geometric interest points found within the lungs, it builds a graph for the inspiration scan and evaluates a combinatorial cost function for a large discretised set of potential displacements to match the most likely points in the expiration scan. At the same time, the correspondence field should be spatially smooth, which is achieved by a Markov random field optimisation. Our training set comprises 700'861 of these correspondence labels, roughly 6'750 per case. Visual details on the distribution of keypoints and correspondences for a particular patient as well as statistics on the initial alignment error for each subset of our data is shown in Fig. 3.
Dataset Composition For each of the 248 scans we provide the following data: 1) Preprocessed CT scans (cropped and resampled), 2) corresponding lung masks and vein/artery-segmentations as .nii.gz-Files, 3) corrField keypoint correspondences as .csv-Files, 4) point clouds of skeletonised vessels (full and sparsely sampled) with corresponding features (label and vessel radius) as .pth-Files and 5) validation landmarks for the annotated scans as .pth-Files (only for a subset).