Deep Learning in Medical Image Registration: Magic or Mirage?
Authors: Rohit Jena
Deeksha Sethi
Pratik Chaudhari
James Gee
Venue: NeurIPS 2024 poster (2024)
Keywords: image registration
image alignment
medical image registration
T1-weighed MRI
image alignment
deformable image registration
diffeomorphism
optimization
fairness
evaluation
Primary area: machine_learning_for_healthcare
TL;DR: This paper establishes the assumptions and conditions under which either classical and deep-learning image registration algorithms surpass each other

Abstract: Classical optimization and learning-based methods are the two reigning paradigms in deformable image registration. While optimization-based methods boast generalizability across modalities and robust performance, learning-based methods promise peak performance, incorporating weak supervision and amortized optimization. However, the exact conditions for either paradigm to perform well over the other are shrouded and not explicitly outlined in the existing literature. In this paper, we make an explicit correspondence between the mutual information of the distribution of per-pixel intensity and labels, and the performance of classical registration methods. This strong correlation hints to the fact that architectural designs in learning-based methods is unlikely to affect this correlation, and therefore, the performance of learning-based methods. This hypothesis is thoroughly validated with state-of-the-art classical and learning-based methods. However, learning-based methods with weak supervision can perform high-fidelity intensity and label registration, which is not possible with classical methods. Next, we show that this high-fidelity feature learning does not translate to invariance to domain shift, and learning-based methods are sensitive to such changes in the data distribution. We reassess and recalibrate performance expectations from classical and DLIR methods under access to label supervision, training time, and its generalization capabilities under minor domain shifts.

Deep Learning in Medical Image Registration: Magic or Mirage?
Rohit Jena
Deeksha Sethi Pratik Chaudhari James C. Gee Computer and Information Science "Electrical and Systems Engineering RRadiology Penn Image Computing and Science Laboratory frjena, deesethi, pratikac]@seas .upenn....pnn..mm www.uueeenn...m edu, geedupenn. edu
Abstract
Classical optimization and leamnin-bbased methods the two reigning paradigms in deformable image registration. While optimization-based methods boast gen- eralizability across modalities and robust performance, bearning-basee methods promise peak performance, incorporating weak supervision and amortized opti- mization. However, the exact conditions for either paradigm perform well over the other are shrouded and not explicitly outlined the existing literature. In this paper, make explicit correspondence between the mutual information of the distribution of per-pixel intensity and labels, and the performance of classical regis- tration methods. This strong correlation hints the fact that architectural designs in learning-baat based methods is unlikely to affect this correlation, and therefore, the performance of learning- based methods. This hypothesis is thoroughly validated with soate-of-the-eee art classical and learning baaning-basedd methods. However, learning- based methods with weak supervision can perform high-fidelity intensity and label registration, which is not possible with classical methods. Next, we show that this high-fidelity feature lcarning does not translate invariance to domain shift, and lcarning-bass based methods are sensitive to such changes the data distribution. We reassess and recalibrate performance expectations from classical and DLIR methods under access to label supervision, training time, and its generalization capabilities under minor domain shifts.
Introduction
Deformable Image Registration (DIR) refers to the local, non-linear (hence deformable) alignment of images by estimating a dense displacement field. Many workflows in medical image analysis require images to be in a standard coordinate system for comparison, analysis, and visualization. In neuroimaging, communicating and comparing data between subjects requires the images to lie 85). This assumption universally does not apply when brain image data are compared across individuals or for the same individual at different time points. Anatomical correspondences between diseased patients and normative brain templates help identify and localize abnormalities like tumors, lesions, or atrophy. Failed or anomalous correspondences impact diagnosis. treatment planning. and disease progression monitoring. DIR is also used to capture and quantify biomechanics and dynamics of difffrrnnt anatomical structures including myocardial motion tracking 774 73 77, improved monitoring of airflow and pulmonary function in lung imaging [66] 27 97]. and tracking of organ motion in radiation therapy 45 [4] Latest breakthrough advances in imaging techniques like fluorescence and light-sheet microscopy [36, 69) 29 98], in-situ hybridization, and multiplexing [65] [02] have led to image
in a standard coordinate system 96
gy
68 7
registration being imperative in advancing life sciences research. Relevant research includes brain- wide mesoscale connectome of the mouse brain 67]. uncovering behavior of individual neurons in C.
Equual advising
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

elegans 1911 building cellular-Ievel atlases of C. elegans, Drosophila melanogaster, and the mouse
brain [05 90, 96
76 [3).
Classical optimization- based and learning-basee methods are the two reigning paradigms in DIR. Classical DIR methods are based on solving variational optimization problem, where similarity metric is optimized to find the best transformation that aligns the images. Most classical methods are formulated without any particular domain knowledge encoded in the optimization problem, and are therefore general and applicable to a wide range of problems. For instance, the popularly known registration toolkit ANTs (5) has been successfully applied to structural and functional neuroimaging data [48] 104 43], CT lung imaging [66], cardiac motion modeling [53], developmental mouse brain atlases utilizing MRI and light sheet fluorescence microscopy (50] with virtually no change in the optimization algorithm. However, classical iterative methods have slow convergence, their performance is limited by the fidelity of image intensities, and they cannot incorporate learning to leverage training set containing weak supervision such as anatomical landmarks, label maps or expert annotations. Deep Learning for Image Registration (DLIR) is an interesting paradigm to overcome these challenges. DLIR methods take pair of images as input to a neural network and outputs warp field that aligns the images, and their associated anatomical landmarks. The neural network parameters are trained to minimize the alignment loss over image pairs and landmarks in a training set. During inference, an image pair is provided and the network regresses a warp field. A primary benefit of this method is the ability to incorporate weak supervision like anatomical landmarks or expert annotations during training, which performs better landmark alignment without access to landmarks at inference time.
Motivation However, the benefits of using DLIR methods over classical DIR methods in terms of accuracy or robustness to domain shift are still topics with no clear consensus. Several DLIR methods claim that architectural choices and loss function design combined with amortized optimization of neural network parameters significantly outperform classical methods [63] 61 [7] On the contrary, classical iterative methods that leverage implicit or explicit conventional priors have shown to outperform most deep learning methods on other challenging datasets [100] 79]. For example, in the context of lung registration, an implicit neural optimization method surpasses every deep learning baseline on the DIR-lab dataset [TOO]. In EMPIREIO challenge without access to labeled data (66). classical methods are highly performant compared to deep learning methods. In the ANHIR histology registration challenge [2], the best performing algorithms were classical methods, and the deep learning method was fast and performed well, but did not have good generalization capabilities. Mok el al. [62] also mention that deep baselines typically fail 'spectacularly" on out of distribution data, and classical methods like Elastix and ANT's come out on top. However, these observations are relatively unstructured and not studied directly. The confounding variable of using labelmap supervision has urged the Learn2Reg 2024 LUMIR challenge (35) to be performed on fully unsupervised data, our own empirical evaluations, we found that classical methods typically outperform deep methods under certain conditions and assumptions. Image registration is Nard being a non-convex optimization problem, and approximating the solution of NP-hard problems with deep learning methods is not guaranteed to be optimal, or even a minima of the registration loss at time. Deep learning methods also claim to provide amortized optimization since classical methods are extremely slow to run, however, modern GPU implementations [55] [9 41 have patched this shortcoming of classical methods while providing state-of-te of-te-of-thh-arr the--rtrttt performance.
Contributions. The conditions needed for either paradigm to perform well over the other clouded and not explicitly outlined in the existing literature. This has prolonged the war-of-war between classical and deep learning methods. We perform more structured problem setup and empirical evaluation to determine consensus on the benefits and limitations of each paradigm. First, observe a strong correlation between the mutual information between per-pixel intensity and label maps, and the performance of classical registration methods. This strong correlation hints to the fact that the Jacobian projection in DLIR methods is unlikely to affect this correlation, and therefore, the performance of DLIR methods in the unsupervised setting. We empirically verify this hypothesis on variety of oate-of-theearte classical and DLIR methods, and address instrumentation bias in the existing literature. Secondly, since the label map is deterministic function of the intensity image, DLIR methods can learn to perform better label matching when this constraint is enforced during training, by implicitly discovering the label map within the network features and predicting a warp field that minimizes the aligmment error between label maps. This is a key strength of DLIR methods, that classical methods cannot leverage. Third, we show that even though learning methods implicit capture semantic information from the image which is not explicitly captured by classical methods,

---

this additional feature learning does not translate to invariance to domain shift, and DLIR methods are brittle these changes. These empirical findings allow us to reassess and recalibrate performance expectations from classical and DLIR methods, using a systematic, unbiased and fair evaluation.
2 Related Work
2.1 Classical Optimization-Baser Methods
Classical image registration algorithms employ iterative optimization on variational objective to estimate the dense displacement field between two images. Some of the earliest approaches to deformable registration considered models for small deformations using elastic deformation
[21]. conceptualizing the moving image volume as elastic continuum that undergoes deformation to align with the appearance of the fixed image. This was in conjunction with alternate formulations based on fluid-dynamical Navier-Stokes [22 21 and Euler-Lagrange equations
assumptions [51 23 g 31
30 20
and their subsequent optimization strategies. The seminal work of Beg.er al. 11 introduces an explicit Euler- Langrange formulation and metric distance on the images as measured by the geodesic shortest paths in the space of diffeomorphisms used to transform the moving image to the fixed image. However, storing the explicit velocity fields expensive in terms of compute and memory. This limitation motivated semi- Langrangian to avoid storing velocity fields explicitly, and only storing the final diffeomorphism. ANTs [ widely used toolkit that employs the Euler- Langraage formulation with symmetric objective function [2). Yet another approach is to interpret deformable registration as an optical flow
[2 11
56 58
formulations 4 3
problem 70 [03 leading the famous Demons algorithm and its diffcomorphic and symmetric variants [106 93 92, 95 implemented as part of the Insight Toolkit (ITK) 40 25). However, most
of these methods are still computationally expensive to run owing to their CPU implementations. Recently, modern implementations leverage the massively parallelizable nature of the registration problem to run on GPUs, leading to orders magnitude of speedups while retaining the robustness
and accuracy of the classical methods [55] 59 41]. However, as we show in Sectionn 4 the registration performance of classical methods limited by the fidelity of image intensities.
2.2
Deep Learning for Image Registration
In contrast to most classical methods, earliest Deep Learning for Image Registration (DLIR) methods employed supervised learning for reg- istration tasks [15]
OASIS
IBSR18
0.8
MGH10
where the de- formation field is obtained either manually from a classical method. Voxelmorph [9] was one of the first approaches that introduced un- supervised learning for registration of in-vivo brain MRI images. Subsequent research ex- panded upon this paradigm, exploring diverse architectural designs [8
777 80
LPBA40
Dice Score
0.6
52
42
62
loss func-
tions [099 108
60
107
75
[6], and
0.5
formulations based incorporating inverse- consistency or symmetric transforms [6]] (46]
47 83
[09]. However, hyperparameter tuning became a challenge for DLIR methods since the methods had to be retrained for every new value of the regularization parameter. This mo- tivated techniques such conditional hyper- parameter injection which addressed hyperpa- rameter tuning [64 38). while domain random-
0.4
0.1 0.2 0.3
0.4
0.5
0.6
0.7
0.8
Mutual Information MI(I: S)
Figure 1: Correlation between Dice Score and Mu- tual Information. Classical registration methods like ANTs show strong correlation between the Dice Score of registered pairs, and the mutual information be- tween the corresponding image and label across 4 brain datasets.
72 28 aimed to addressed generalizability of DLIR methods across domains. Recently, pretrained or foun- dation models are also proposed to address the generalizability of DLIR methods across differ- However, these methods perform monolithic prediction of the warp field from the input images, losing feedback from the intermediate stages of the registration process done in classical methods. To refine the warp fields, recurrent cascade-based based archi-
ization and fine-tuning [37 88
ent imaging and anatomy [54 84
3

---

0.85
0.80
Dice score
0.75
0.70
0.65
Split
0.60
trainval
0.55
Morphh
Gready
Log Demons
FreANTss
SymthMorph NiftyReg
SymMet
LKU-Net (NCC)
Method
Comparison classical and methods on OASIS trai set.
Comparison
methods
1.00
on
1.00
NittYeg
NiftyReg
ANTs
0:34
ANTs
9.921
<0.01
0.300
Demons-
<0.01
+0.01
+0.01
<0.01
<0.01
+2.01
0.00
Log Demons-
3.020
x0.01
+0.01
+0.01
<0.01
+0.01
-0.25
-0.25
Greedy-
+0.01
<0.01
00.01
0.161
K0.01
<0.01
Greedy
<0.01
<0.01
00.01
0.73
+0.01
+0.01
-0.50
-0
RreANT
6.17
40.01
<0.01
<0.01
0.75
RreANTs
3.000
+0.01
<0.01
<0.01
0.231
0.75
LaoIR
LKL-Net
SymWet
Trnstforph
Synthterph
Symbhfoorph
-1.00
LADRN
LKt-Net
SynWeet
Manstforph
Synth
Synthttorph
(NCC)
(NCC)
(Dice)
Figure 2: Performance of classical and unsupervised DLIR methods on OASIS data. Boxplots (top) show that classical methods average ranked higher than DLIR methods, both the trainval and val splits. Interestingly, the performance of unsupervised DLIR methods does not improve on the trainval split compared val split - showing that deep learning does not have an intrinsic advantage in label alignment. Tables (bottom) p-values show the results of pairwise two-sided t-test between the performance of classical and DLIR methods on the trainval and splits. denotes cell where the classical method significantly better than DLIR
method (p < 0.01), denotes the opposite, denotes no significant difference. Most the cells are indicating that classical methods are significantly better than DLIR methods.
tectures were proposed [108
109
16 However, cascade-based methods create a substantial
memory overhead due to backpropagation through cascades and storage of intermediate volumes [ Another promising avenue is to leverage deep implicit priors [87] within optimization frameworks to
improve performance of optimization methods or incorporate implicit constraints of the optimized
warp field [101] [99 [4, 39) We refer the reader to [26 33 for a comprehensive review of image
registration techniques.
Despite the plethora of architectural formulations, loss functions, and output representations proposed in Deep Learning for Image Registration methods, we identify that these methods highly sensitive to the domain gap between the distributions of training and test data, and in the unsupervised casc, do not provide any benefit in terms of performance over classical methods. Their primary benefit is their ability to incorporate weak supervision like anatomical landmarks or expert annotations during training, which performs better landmark alignment unseen image pairs (from the same distribution) without access to landmarks at inference time.
3 Preliminaries
We rehash the image registration problem statement unify both classical and deep learning methods. Consider a dataset of image pairs = {(T(m) r(71) n N, 5 n < N where r4m) and 7(0) the fixed and moving images defined over spatial domain Q E gd. We drop the superscript

---

for simplicity. Also consider segmentation maps Sy and Sm for the fixed and moving images, respectively, defined over f,. Given a family of transformations T(2). the goal of image registration is to estimate transformations so ()))m) E T(?)) parameterized by 0 that minimize the following objective:
arg min f.m
Σ C(T Im o (o((.m)) + R((o(f,
where L is a dissimilarity function such as mean squared error, or negative local cross correlation, and R regularization term that encourages desirable properties of the transformation, such as smoothness or elasticity. We call Eq. (1) the image matching objective, since the transformations only need align the intensity images. We can also call this the unsupervised objective, since does not require any labeled data. If a suitably chosen label alignment loss 'D is added as well, the optimization problem becomes:
arg min
Σ Cllg, Im o vo(s, m)) + D(Sy, Sm o volf, m)) + R(po(f, m)) f.
We call Eq.. (2) the label matching objective, or a weakly-superrviee objective. The image matching objective can subsume both DLIR and classical methods by choosing
So(ly:Am)) for deep networks, (P(mm))>
(o(s. m)
for classical methods.
where fo is deep network parameterized by 0 and (ff..) are optimizable free parameters that are indexed by the 2-tuple (f, m), i.e. e = y.m (PPL. In this paper, we consider methods that solve Eq. (1) using gradient-based methods. The gradient of Eq. (1) with respect to 0 is given by (we remove the R term for simplicity):
ac a0
OC
@po(f.m) m)
Σ f.m
@.po (f, m)
00
8c is the training signal from the dissimilarity function which does not depend The first term Tpp[f.]]
on the parameters 0 for a given value of (o(s,, m) and choice of C. The second term @pa(f.m) is the Jacobian of the transformation with respect to the parameters, which is a projection of the gradient from the space of warp fields to the space of arbitrary parameters. For classical methods, the Jacobian is the identity matrix, for deep networks is determined by the functional relationship of the output with respect to network parameters. Therefore, the difference in training dynamics and overall performance gap between classical and deep learning methods is likely to be attributed to the choice of @.o(f.m) 80
4 Unsupervised DLIR does not improve label matching performance
A speculated claim of deep learning methods is that they can provide better label matching per- formance by simply training a network to minimize [Eq. (1) in an unsupervised setting. Such
improvements are claimed to come from architectural designs, which correspond choice of Jaco-
bian @pa(f.m) A variety of architectures and parameterizations [17
101 have
30
been proposed to this effect. However, we show that this is not the case.
Image matching objectives ensure that intensities from the moving image displaced to locations in the fixed image where they are most similar, without regard for alignment for any higher order structures. Intuitively, this will ensure label matching only to the extent that the intensity is predictive of the label. If an intensity value strongly corresponds to particular label, then image matching will lead to label matching. Similarly, if given intensity value corresponds multiple possible labels, then image matching does not tell us which labels are matched via the image matching objective. More formally, considering the per-pixel intensity and labels random variables, one can compute the mutual information between the intensity and label maps, denoted as MI; to determine the predictability of one from the other. We now show that the label matching performance of classical methods is highly correlated with s)) We consider a widely used classical method, ANTs 22 5), to eliminate the effect of any Jacobian term. We consider four brain datasets OASIS, LPBA40, MGHIO, and IBSR18, which are acquired under different scanners, under different resolutions, and have different preprocessing, labelling and postprocessing protocols [57] 48). For
5

---

each dataset, we use ANTs for registering all pairs within the dataset and then evaluate the Dice score as an indicator of label matching performance. For each image and its corresponding label map S. we compute the probability maps p(i), p()s.((ss, s) using histogram binning, followed by the mutual information (i: s) = (()) - H())).. A Pearson's correlation coefficient between the Dice scores and the mutual information of the image and label (Fig. T) reveals a strong linear (r = 0. 0.886) and logarithmic (r 0.933) relationship between the two quantities, shown by the gray and black lines respectively. Image matching improves label matching performance only to the extent of the information about the label obtained from the image (i.e. (i)s))) At first glance, the Jacobian term f.p(f.m) seemingly does not have a role improving this mutual information further. 89
Empirical Validation. We verify this claim empirically on the OASIS dataset, by minimizing Eq. (I) in both DLIR and classical methods. We split the OASIS dataset into a training set of 364 images and validation set of 50 images. We choose 50 instead of 20 images as in the original
split (35] to compute statistical significance. Dice score over 35 subcortical structures is used as the
label matching metric. We choose SynthMorph (37, LapIRN [3], SymNet [6]] LKU-Net (42
and Trrph [9] as state-of-the-art DLIR baselines and ANT's 5 NiftyReg (59] Symmetric Log Demons (94), Greedy [106). FireANTss [1 as of-te-oftthe-ar classical baselines. For all DLIR methods, we use pretrained models if they are trained with Eq. (1) or train them with the architecture and hyperparameters provided in their original source code. The only exception is SynthMorph, which is trained on synthetically generated data and Dice loss of its corresponding synthetic labels (shapes- sm model). To compare SynthMorph's 's domain generalization capabilities with only the image matching objective, we add another model, dubbed Shappss-m-nnee synthetically generated data in the original pretrained model, but with the normalized cross- correlation of the aligned synthetic images. For all classical methods, we follow their recommended hyperparameters and run till convergence. All experiments are run cluster with 2 AMD EPYC 7713 CPUs and 8 NVIDIA A6000 GPUs.
that is trained on
Results. For all methods, we compute the Dice score of all 35 subcortical regions on images in the validation set (denoted as val), and all images (denoted as trainval). These Dice scores are sorted by median validation performance in Fig.2 top)) Moreover, we perform two-sided t-test for each (classical, DLIR) pair, both on the trainval and validation sets, shown in Fig. 2/bottom). Fig. 2 shows the following conclusions: (a) the top performing classical method (Greedy) and the top performing DLIR method (TransMorph) achieve similar label matching performance on the val and the trainval set, i.e. the differences not statistically significant (p 0.161), (b) classical methods almost always perform better than DLIR methods, even on the training set showing that the Jacobian term does not improve label matching more than the mutual information between the image and label, and (c) for unsupervised DLIR methods, there is no improvement label matching performance in the training set compared to val set. The only role of the Jacobian term is to perform amortized learning, but without supervised objectives, this does not guarantee any additional boost in label matching.
The astute reader may observe that this result is in contrast 101 19 [O]. We note that this is due to instrumentation bias [86). where the baselines' performance may be misrepresented due to changes in hyperparameters, early stopping, or different preprocessing protocols. For instance, [10] mention that the default parameters of ANTs are not optimal, and choose very different set of parameters (a Gaussian smoothing of 9 pixels, followed by an extremely small 0.4 pixels at the next scale). By stark contrast, we found the recommended parameters work extremely well for all datasets considered in this paper. We speculate that these changes are done to tradeoff accuracy for speed, since classical methods converge slowly. However, this leads to mistepresentation of the performance of classical baselines. We found much better results (Fi.. 2} for classical baselines simply by using their recommended scripts. We compare the discrepancy in performance between the baselines reported in the literature and the ones we obtained in Fig. 3 We follow the guidelines in [86] to evaluate all methods. To ensure our work does not introduce its own instrumentation bias for DLIR basclines,
The effect of instrumentation bias. to results shown in prior literature [1 63
compare the performance of our trained/pretrained models to the ones reported in the literature (Fig. 3). We make all evaluation scripts and trained models public] to encourage fairness and transparency in evaluations.

---

Evaluation of classical methods reported by baselines
Method
Evaluated Baseline
Statistic
Reported value
Our eval
Difference
SymNet
ANTs
Mean
0.680
0.787
0.107
PIRATE
ANTs
Mean
0.699
0.787
0.088
LaplRN
Demons
Mean
0.715
0,802
0.087
LaplRN
ANTs
Mean
0.723
0.787
0.064
NODEO
Demons
Mcan
0.764
0,802
0.038
NODEO
ANTs
Mcan
0.729
0.787
0.058
Voxelmorph
ANTs
Mean
0.749
0.787
0.038
Voxelmorph
NiftyReg
Mean
0.755
0.776
0.021
MynthMorph
ANTs
Median
0.770
0.797
0.027
Evaluation of DLIR baselines reported by us
Method
Dice supervision
Statistic
Reported value
Our eval
Difference
Synth
Median
0.780
0.785
0.005
TransMorph-Regul
Mean
0.858
0.855
-0.003
LKU-Net
Mean
0.886
0,904
0.018
LapIRN
X
Mean
0.808
0.788
-0.020
SymNet
x
Mean
0.743
0.748
0.005
Figure 3: Instrumentation bias in evaluation of image registration algorithms. We highlight a significant difference in evaluation metrics reported by baselines and our evaluation on the OASIS validation dataset. This difference can be attributed to deviation in hyperparameters from the recommended parameters or early stopping to save time. In either case, this misrepresentation leads to incorrect conclusions about the performance of the algorithm. The reported dice scores are anywhere from 2 to 10 Dice points lower than our evaluation, showing non-trivial instrumentation bias. We report our own evaluation of DLIR algorithms and compare them with reported values to avoid introducing instrumentation bias in our evaluation.
5 Supervised DLIR methods demonstrate enhanced label matching
When label matching is introduced as an objective in Eq. (2) DLIR methods show superior perfor- mance than classical methods. Unlike the previous discussion, where only pixelwise definition of MT(i; s) was used to quantify the coaction of image intensities and label maps, we consider the entire image I and label volume S as high-dimensional random variables. Label maps now deterministic function of the image, i.e. 5 = fJ), where f is the labelling protocol. In addition to image intensity, label maps a function of morphological features, location, contrast, and the labelling protocol itself. When trained with the label maps as extra supervision, the network can infer these deterministic relationships output a warp field that maximize both image similarity and label overlap. Classical intensity- based methods, on the other hand, do not have any mechanism to encode this additional relationship. Aligning intensities intensity patches discards any functional relation- ship between high-level image features and labels. To show this, we repeat the same experiment setup in Seccionn 4 on the same splits, but with the label matching objective added well.
Results. Fig. 4(top) shows the Dice scores for supervised classical and DLIR methods trained on the OASIS dataset, sorted by median validation performance. In this case, state-of-the-art DLIR methods outperform classical methods by large margin, with notably higher Dice score on the trainval set than the val set, due to overfitting to the label matching for the training set. This is unlike unsupervised DLIR, where there was no improvement in label matching performance on the training set, emphasizing the fact that performing amortized training does not improve label matching performance by itself. These differences are statistically significant, with the exception of SymNet, which diverged under many training settings with the Dice loss, and only works marginally better than its unsupervised counterpart. SynthMorph is not trained on real data, and is added only as reference for fomainnagnosiin performance.
This is an unsurprising result the label matching objective provides additional training signal to the registration task, which is highly iol-posed problem. Classical methods cannot incorporate this additional signal from a training dataset, and sorming-based methods exploit this to achieve better registration unseen data. Classical methods are, however, agnostic to modalties, intensity distributions, voxel resolutions, and anisotropy. The same registration algorithm (with possibly modified parameters) is applied to datasets with different characteristics, and they still retain their state-of- the-art performance. A related question arises for DLIR methods trained with label matching does label matching performance transfer to other datasets?
7

---

0.95
Split
trainval
0.90
val
Dice score
0.85
0.80
0.75
0.70
LKU-Net
LapiRN
TransMorph
Gready
Log Demans
FireANTs
Synth
NiftyReg
Method
Comparison classical and methods on OASIS
Comparison
methods
set.
-1.00
on
1.00
Nittyegg
NiftyReg
ANTs
ANTs
0.300
Demons-
<0.01
x0.01
<0.01
<0.01
+0.01
0.00
Log Demons-
+0.01
+0.01
<0.01
<0.01
+0.01
-0.25
-0.25
Greedy
<0.02
x0.01
+0.01
x0.01
+0.01
Greedy-
<0.01
+0.01
x0.01
+0.01
<0.01
30
-..00
RreANT
<3.02
+0.01
+0.01
<2.01
<0.01
0.75
RreANTs
<0.01
-0.01
0.012
0.21
0.75
Lapony
LRL-Net
Panstooph
SymWee
Synchiffrpph
-1.00
LApION
LKU-Net
Ranstorm
Syndket
Symthherph
(Dice)
(Dice)
Figure 4: Performance of classical and supervised DLIR methods on OASIS data. Boxplots (top) show that DLIR methods show superior performance compared to classical methods. Unlike the unsupervised case, the effect of overfitting elearly visible the gap between the trainval and val splits. Tables (bottom) of p-values show the results of pairwise two-sided i-test between performance classical and DLIR methods trainval and val splits. denotes cell where the classical method significantly better than the DI. method
(p < 0.01). a
denotes the opposite, denotes no significant difference. State-of-the-ar DLIR methods show significantly better performance than classical methods when label supervision is added.
DLIR methods do not generalize across datasets
key strength classical optimization registration algorithms their agnostic nature to the image modality, physical resolution, voxel sizes, and preprocessing protocols. Most DLIR methods, on the contrary, have been evaluated extensively on the same distribution of validation datasets as the training data, it is unclear if the performance improvements transfer to other datasets of the same anatomy. To this end, we evaluate the performance of both the classical and DLIR methods on four brain datasets brain
CUMC12, LPBA40, MGH10, and IBSR18. These datasets represent community-standee mapping challenge data 48 for a comprehensive evaluation of 14 nonlinear classical registration methods, across various acquisition, preprocessing and labelling protocols. For all datasets, we follow the preprocessing steps followed by 488
Each dataset contains a different set of labeled regions acquired manually using different labeling protocols. For each dataset, all previously considered registration algorithms pairs, and the mean Dice score over all labeled regions is computed. The methods are then sorted by median validation performance in Fig. 5] For DLIR methods, plot the performance with models trained with and without the label matching loss in the OASIS dataset, shown as blue and green boxplots respectively. Across all datasets, FireANTs, Greedy, ANTs and NiftyReg consistently perform better than DLIR methods. Among the DLIR methods, SynthMorph performs consistently better due to its domain- agnostic training paradigm. Remarkably, even though DLIR methods outperform classical methods on the OASIS dataset with label matching objective, the performance does not transfer to other datasets, even compared to its unsupervised variant. This is negative
all image

---

0.7
0.6
0.3
0.2
Supervision
Unsupervised
Supervised
FireANT
Greedy
ANTS
NiftyReg
SynthMorph TransMorph SymNet Method
LKU-Net
LapiRN
Log Demons
0.75
0.70
Dice score
0.60
0.55
Supervision
Unsupervised
0.50
Supervised
FireANTB
Greedy
ANT
NiflyReg
Demons Symthn Myrth TransMorph SymNet Method
LKU-Net
LapiRN
0.6
DiGe
0.3
Supervision
0.2
Unsuperriiss
Supervised
FireANTs
Greedy
ANT:
NiftyReg
Lag Demons TransMorph SynthMorph SymNet
LKU-Nel
LapiRN
Method
0.6
0.5
Dice score
0.4
0.3
Supervision
0.2
Unsuperviss
Supervised
FireANTs
ANTS
TransMorph SynthMorph
LapiRN
SymNet
Log Demans
NiftyReg
LKU-NNe Net
Greedy
Method
Figure 5: Classical methods retain robustness across different datasets. Boxplots show the performance classical DLIR methods trained on OASIS dataset, on four TI-brain datasets. DLIR methods. we plot the performance of the supervised and unsupervised models. Across all datasets, FireANTs and ANTs consistently outperform DLIR methods, showing robustness to domain shift. Among DLIR methods, Morth and TransMorph show robust performance, and training with label matching objective does not lead to significant improvement.

---

result- implying that to improve performance on new dataset, one must collect label maps from that dataset and retrain the model existing collections of label maps are not sufficient improve performance on new datasets. Unlike in tasks like segmentation, deep methods do not transfer their performance to out-of-distribution datasets, even with the same resolution, and the expected performance hierarchy does not hold
Expected: Supervised DLIR ID Supervised DLIR OOD > Classical
Observed: Supervised DLIR ID Classical Supervised DLIR OOD
Practitioners should therefore be cautious when using prediction-based DLIR methods, especially when the training data is not representative of the test data, regardless of the presence of label maps.
7 Discussion
This study aims to provide systematic and unbiased investigation of the performance of classical and DLIR methods under access to label supervision, and their generalization capabilities under small domain shifts. Preceding experiments show that classical methods provide an unprecedented level of robustness and generalizability across datasets, but are limited by the fidelity of the image matching objective. Supervised DLIR methods provide promising step towards improving regis- tration performance of anatomical regions by implicitly discovering these structures and predicting appropriate warp fields within the network architecture. However, this anatomical-awarrnes. on the training dataset does not help in generalizing to other datasets, limiting the practical utility of these methods. The usability of anatomical landmarks and labelmaps obtain registration performance still remains an open research problem. These results also have profound implications for annotated data collection and challenges the notion that large labeled datasets ensure robust generalization.
Although our study is performed on inter-subject registration with in-vivo neuroimaging datasets, none of our analysis, baselines, and evaluation make any domain or subject-specific modeling assumptions, and the datasets being community-standard benchmarks, the results are valuable and general. both within the neuroimaging and the biomedical communities large. At the current state, a practitioner should choose predictive DLIR methods only if they have access to a large labeled dataset, and their application is limited to the same dataset distribution. In all other cases, classical optimization-basee methods are the more accurate and reliable choice, even if labeled data exists but representative of the test data.
7.1 Limitations
Our work performs a comprehensive evaluation of state-of-the-an aftt----tttt-tttee registration algorithms on a variety of neuroimaging datasets. However, our work does not consider hybrid methods, or representations that use optimal matching criteria based on correlation volumes or sparse correspondence features. Although our work considers large-scale community community-standard neuroimaging datasets, the performance of these algorithms may differ on other anatomy or modalities. Our study also considers inter-subject registration only, although no method or evaluation incorporates any subject-specific assumptions. The effects on multimodal registration are not considered in this work. However, our work serves as a foundational step toward a more nuanced discussion on the longstanding technical challenges in image registration, and representations that are effective mitigating these problems.
Acknowledgements
This work was supported by the National Institutes of Health (NIH) under grants RFI-MH124605 HHH-LL33388,, R01-EB031722, U24-NS135568.
10

---

References
[1] ANTsX. Antsx:
Advanced normalization tools (ants). GitHub repository.
[2] B. Avants, C. Epstein, M. Grossman, and C. Gee. Symmetric diffeomorphic image registration with cross- cross-correlation: evaluating automated labeling of elderly and neurodegener- ative brain. Medical Image Analysis, 12(1):26 41, February 2008.
[3] Brian Avants and James C. Gee. Geodesic estimation for large deformation anatomical shape averaging and interpolation. Neurolmage, 23:5 19S--15- January 2004.
[4] Brian B. Avants, Thomas Schoenemann, and James C. Gee. Lagrangian frame diffeomorphic image registration: Morphometric comparison of human and chimpanzee cortex. Medical Image Analysis, 10(3):397-412, June 2006.
[5] Brian B Avants, Nick Tustison, Gang Song, et al. Advanced normalization tools (ants). Insight 2(365):1-335 2009.
[6] Shaojie Bai, Zhengyang Geng, Yash Savani, and J. Zico Kolter. Deep Equilibrium Optical Flow Estimation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 610-620, New Orleans, LA, USA, June 2022. IEEE.
[7] Wenjia Bai, Hideaki Suzuki, Jian Huang, Catherine Francis, Shuo Wang, Giacomo Tarroni, Florian Guitton, Nay Aung, Kenneth Fung. Steffen E Petersen, et al. population-ban -based wide association study of cardiac and aortic structure and function. Nature medicine, 16110):665-1166, 2020.
[8] Ruzena Bajesy, Robert Lieberson, and Martin Reivich. A computerized system for the elastic matching of deformed radiographic images to idcalized atlas images. Journal of computer assisted tomography. 7(4):618 625, 1983.
[9] Guha Balakrishnan, Amy Zhao, Mert R. Sabuneu, John Guttag, and Adrian V. Dalca. Voxel- Morph: A Learning Framework for Deformable Medical Image Registration. IEEE Transac- tions on Medical Imaging, 1788-188 August 2019. arXiv: 1e09.0533) [es].
[10] Guha Balakrishnan, Amy Zhao, Mert R Sabuneu, John Guttag, and Adrian V Dalca. Voxel- morph: a learning framework for deformable medical image registration. IEEE transactions medical imaging, 1888))1788--888 2019.
[11] M Faisal Beg, Michael I Miller, Alain Trouvé, and Laurent Younes. Computing large de- formation metric mappings via geodesic flows of diffeomorphisms. International journal of computer vision, 61:139-157, 2005.
[12] Jiří Borovec, Jan Kybic, Ignacio Arganda-Carreras, Dmitry V Sorokin, Gloria Bueno, Alexan- der Khvostikov, Spyridon Bakas, Eric, Chao Chang, Stefan Heldmann, Anhir: automatic non-rigid histological image registration challenge. IEEE transactions medical imaging, 39( 30053333-3305 2020.
[13] Bella E Brezovec, Andrew B Berger, Yukun A Hao, Feng Chen, Shaul Druckmann, and Thomas R Clandinin. Mapping the neural dynamics of locomotion across the drosophila brain. Current Biology, 34(4):710-72 2024.
[14] Kristy Brock, Sasa Mutic, Todd R McNutt, Hua Li, and Marc Kessler. Use of image registration and fusion algorithms and techniques in radiotherapy: Report of the aapm radiation therapy committee task group no. 132. Medical physics, 44(7))e43-476, 2017.
[15) Xiaohuan Cao, Jianhua Yang. Jun Zhang, Dong Nie, Minjeong Kim, Qian Wang, and Dinggang Shen. Deformable image registration based on similarity-steeree cnn regression. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September I 11-13, 2017, Proceedings, Part I 20, pages 500-308. Springer, 2017.
[16] Junyu Chen, Eric C Frey, and Yong Du. Unsupervised learning of diffeomorphic image registration via transmorph. In International Workshop on Biomedical Image Registration. pages 96-102. Springer, 2022.

---

[17] Junyu Chen, Eric C Frey, Yufan He, William P Segars, Ye Li, and Yong Du Transmorph: Transformer for unsupervised medical image registration. Medical image analysis, 82:102615, 2022.
[18] Junyu Chen, Eric C. Frey, Yufan He, William P. Segars, Ye Li, and Yong Du. TraphMorph: Transformer for unsupervised medical image registration. Medical Image Analysis, 82:102615, November 2022.
[19] Junyu Chen, Eric C. Frey, Yufan He, William P. Segars, Ye Li, and Yong Du TrapsMorph: Transformer for unsupervised medical image registration. Medical Image Analysis, 82:102615 November 2022. arXiv:2111, 10448 [es, eess].
[20] Gary E Christensen and Hans J Johnson. Consistent image registration. IEEE transactions on medical imaging, 20(7):568-555) 582, 2001.
[21] Gary Christensen, Richard D Rabbitt, and Michael I Miller. Deformable templates using large deformation kinematics. IEEE transactions image processing, 5(10):1435-144) 1447, 1996.
[22] G.E. Christensen, S.C. Joshi, and M.I. Miller. Volumetric transformation of brain anatomy. IEEE Transactions on Medical Imaging, 16(6):844-877, December 1997. Conference Name: IEEE Transactions Medical Imaging.
[23] Christos Davatzikos. Spatial transformation and registration of brain images using elastically deformable models. Computer Vision and Image Understanding, 66(2):207-222, 1997.
[24] Bob D De Vos, Floris F Berendsen, Max A Viergever, Hessam Sokooti, Marius Staring, and Ivana lggum. A deep learning framework for unsupervised affine and deformable image registration. Medical image analysis, 52: 143, 2019.
[25] Florence Dru, Pierre Fillard, and Tom Vercauteren. An ITK Implementation of the Symmetric Log- Log-Domain Diffeomorphic Demons Algorithm. The Insight Journal, September 2010.
[26] Yabo Fu, Yang Lei, Tonghe Wang, Walter Curran, Tian Liu, and Xiaofeng Yang. Deep learn- ing in medical image registration: a review. Physics in Medicine & Biology, 65(20)220TR01, October 2020.
[27] Yabo Fu, Yang Lei, Tonghe Wang. Kristin Higgins, Jeffrey D. Bradley, Walter J. Curran, Tian Liu, and Xiaofeng Yang. LungRegNet: unsupervised deformable image registration method for 4D-CT lung. Medical physics, 1763) April 2020.
[28] Yabo Fu, Yang Lei, Jun Zhou, Tonghe Wang. S Yu David, Jonathan J Beitler, Walter J Curran, Tian Liu, and Xiaofeng Yang. Synthetic ct-aided mri-ct image registration for head and neck radiotherapy. In Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging, volume 11317, pages 572-578. SPIE, 2020.
[29] Davide Gambarotto, Fabian U Zwettler, Maeva Le Guennec, Marketa Saamiddt-eerohooska. Denis Fortun, Susanne Borgers, Jöm Heine, Jan-Gero Schloetel, Matthias Reuss, Michael Unser, et al. Imaging cellular ultrastructures using expansion microscopy (u-exm). Nature methods, 16(1):71-74, 2019.
[30] James Gee and Ruzena K Bajesy. Elastic matching: Continuum mechanical and probabilistic analysis. Brain warping, 2: 197, 1998.
[31] James C Gee, Martin Reivich, and Ruzena Bajesy. Elastically deforming a chree-dimensional atlas to match anatomical brain images. 1993.
[32] Maged Goubran, Cathie Crukley, Sandrine De Ribaupierre, Terence M Peters, and Ali R Khan. Image registration of ex-vivo mri to sparsely sectioned histology of hippocampal and neocortical temporal lobe specimens. Neuroimage, 83:770-781, 2013.
[33] Grant Haskins, Uwe Kruger, and Pingkun Yan. Deep learning in medical image registration: a survey. Machine Vision and Applications, 31());8, January 2020.
12

---

[34] Mattias P. Heinrich, Heinz Handels, and Ivor J. A. Simpson. Estimating Large Lung Motion in COPD Patients by Symmetric Regularised Correspondence Fields. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro Frangi, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, Lecture Notes Computer Science, pages 338-345, Cham, 2015. Springer International Publishing.
[35] Alessa Hering, Lasse Hansen, Tony CW Mok, Albert CS Chung, Hanna Siebert, Stephanie Häger, Annkristin Lange, Sven Kuckertz, Stefan Heldmann, Wei Shao, et al. Learn2reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning, IEEE Transactions on Medical Imaging, 42(3):697-712, 2022.
[36] Elizabeth MC Hillman, Venkatakaushik Voleti, Wenze Li, and Hang Yu. Light-sheet mi- croscopy in neuroscience. Annual review of neuroscience, 42:295-313, 2019.
[37] Malte Hoffmann, Benjamin Billot, Douglas N Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian V Dalca. Synthmorph: learning contrast-invarian registration without acquired images. IEEE transactions on medical imaging. 41(3):5433-55
2021.
[38] Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, and Adrian V Dalca. Hyper- morph: Amortized hyperparameter learning for image registration. In Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28-June 30, 2021, Proceedings 27, pages 3-17. Springer, 2021.
[39] Junhao Hu, Weijie Gan, Zhixin Sun, Hongyu An, and Ulugbek S. Kamilov. A Plug-and-Play Image Registration Network, March 2024. arXiv:2310 10..0499 [eess].
[40] Luis Ibanez, Will Schroeder, Lydia Ng, Josh Cates, et al. The itk software guide, 2003.
[41] Rohit Jena, Pratik Chaudhari, and James C Gee. Fireants: Adaptive riemannian optimization for multi-scale diffeomorphic registration. arXiv preprint arXiv:2404.01144 2024.
[42] Xi Jia, Joseph Bartlett, Tianyang Zhang, Wenqi Lu, Zhaowen Qiu, and Jinming Duan. U-net vs transformer: u-net outdated medical image registration? arXiv preprint arXiv:2208.0499 04939, 2022.
[43] Di Jiang, Yuhui Du Hewei Cheng, Tianzi Jiang, and Yong Fan. Groupwise spatial nor- malization of fmri data based on multi-range functional connectivity patters. Neuroimage, 82:355 372, 2013.
[44] Ankita Joshi and Yi Hong. Diffeomorphic Image Registration using Lipschitz Continuous Residual Networks. page 13.
[45] Mare Kessler. Image registration and data fusion in radiation therapy. The British journal of radiology, Dist:iiaaaassseess,9-1Ss-- 2006.
[46] Boah Kim, Dong Hwan Kim, Seong Ho Park, Jieun Kim, June-Goo Lee, and Jong Chul Ye. Cyclemorph: cycle consistent unsupervised deformable image registration. Medical image analysis, 71:102036, 2021.
[47] Boah Kim, Jieun Kim, June-Goo Lee, Dong Hwan Kim, Seong Ho Park, and Jong Chul Ye. Unsupervised deformable image registration using cycle-consistent cnn. In Medical Image Computing and Computer Assisted Intervention 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part VI 22, pages 166-174. Springer, 2019.
[48] Arno Klein, Jesper Andersson, Babak A. Ardekani, John Ashburner, Brian Avants, Ming- Chang Chiang, Gary F. Christensen, D. Louis Collins, James Gee, Pierre Hellier, Joo Hyun Song, Mark Jenkinson, Claude Lepage, Daniel Rueckert, Paul Thompson, Tom Vercauteren, Roger P. Woods, J. John Mann, and Ramin V. Parsey. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neurolmage, 46(3):786 802, July 2009.
[49] Julian Krebs, Tommaso Mansi, Hervé Delingette, Li Zhang, Florin C Ghesu, Shun Miao, Andreas K Maier, Nicholas Ayache, Rui Liao, and Ali Kamen. Robust non-rigid registration through agent-based action learning. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part I 20, pages 344-352. Springer, 2017.
13

---

[50] Fae A Kronman. Josephine K Liwang, Rebecca Betty, Daniel Vanselow, Yuan-Timg Wu, Nicholas J Tustison, Ashwin Bhandiwad, Steffy Manjila, Jennifer A Minteer, Donghui Shin, et al. Developmental mouse brain common coordinate framework. bioRxiv, 2023.
[51] Jan Kybic and Michael Unser. Fast parametric elastic image registration. IEEE transactions on image processing. ((11)):4427-4444 2003.
[52] Leo Lebrat, Rodrigo Santa Cruz, Frederic de Gournay, Darren Fu, Pierrick Bourgeat, Jurgen Fripp. Clinton Fookes, and Olivier Salvado. CorticalFlow: A Diffeomorphic Mesh Transformer Network for Cortical Surface Reconstruction. In Advances in Neural Information Processing Systems, volume 34, pages
Curran Associates, Inc., 2021.
[53] Devavrat Likhite, Ganesh Adluru, and Edward DiBella. Deformable and rigid model-based image registration for quantitative cardiac perfusion. In Statistical Atlases and Computational Models of the Heart-Imaging and Modelling Challenges: 5th International Workshop, STA- COM 2014, Held in Conjunction with MICCAI 2014, Boston, MA, USA, September 18, 2014, Revised Selected Papers 5, pages 41-50. Springer, 2015.
[54] Fengze Liu, Ke Yan, Adam P. Harrison, Dazhou Guo, Le Lu, Alan L. Yuille, Lingyun Huang, Guotong Xie, Jing Xiao, Xianghua Ye, and Dakai Jin. SAME: Deformable Image Registration Based on Self-supervised Anatomical Embeddings. In Marleen de Bruijne, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert, editors, Medical Image Computing and Computer Assisted Intervention - MICCAI 2021, Lecture Notes in Computer Science, pages 87-97, Cham, 2021. Springer International Publishing.
[55] Andreas Mang, Amir Gholami, Christos Davatzikos, and George Biros. CLAIRE: distributed- memory solver for constrained large deformation diffeomorphic image registration. SIAM Journal on Scientific Computing, ((5):C548-C58) January 2019. arXiv: 10Xi:1100800488 [es,
math].
[56] Andreas Mang and Lars Ruthotto. A lagrangian gauss-newton-krybbo solver for mass-and intensity-preserving diffeomorphic image registration. SIAM Journal on Scientific Computing, (9(5):860-B8855 2017.
[57] Daniel S Mareus, Tracy Wang, Jamie Parker, John G Csernansky, John Morris, and Randy L Buckner. Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience, 19(9): 1498::498-1507, 2007.
[58] Michael I. Miller, Alain Trouvé, and Laurent Younes. On the Metries and Euler-Lagrang, Equations of Computational Anatomy. Annual Review of Biomedical Engineering, 4(1):375- 405, 2002. eprint: https://doi.org/IIIII 10.1
[59] Mare Modat, Gerard R Ridgway, Zeike A Taylor, Manja Lehmann, Josephine Barnes, David Hawkes, Nick C Fox, and Sébastien Ourselin. Fast free-form deformation using graphics processing units. Computer methods and programs in biomedicine, (8(3))278-284) 2010.
[60] Tony C. W. Mok and Albert C. S. Chung. Large Deformation Diffeomorphic Image Registration with Laplacian Pyramid Networks, June 2020. arXiv:2006.166 16148 [cs, eess].
[61] Tony CW Mok and Albert Chung. Fast symmetric diffeomorphic image registration with convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4644-4653, 2020.
[62] Tony CW Mok and Albert Chung. Affine medical image registration with coarse-to-fine vision transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20835-20844, 2022.
[63] Tony CW Mok and Albert CS Chung. Large deformation diffcomorphic image registration with laplacian pyramid networks. pages 21 211-221, 2020.
[64] Tony CW Mok and Albert CS Chung. Conditional deformable image registration with convolutional neural network. pages 35-45, 2021.
14

---

[65] Annette Moter and B Göbel. Fluorescence in situ hybridization (fish) for direct visualization of microorganisms. Journal of microbiological methods, 41(2):85-112, 2000.
[66] Keelin Murphy, Bram Van Ginneken, Joseph M Reinhardt, Sven Kabus, Kai Ding, Xiang Deng, Kunlin Cao, Kaifang Du, Gary E Christensen, Vincent Garcia, al. Evaluation of registration methods on thoracic ct: the empire 10 challenge. IEEE transactions medical imaging. 30(1 (0911):19011999
2011.
[67] Seung Wook Oh, Julie A Harris, Lydia Ng, Brent Winslow, Nicholas Cain, Stefan Mihalas, Quanxin Wang, Chris Lau, Leonard Kuan, Alex M Henry, et al. A mesoscale connectome of the mouse brain. Nature, 508(7495) 2014.
[68] Seungjong Oh and Siyong Kim. Deformable image registration in radiation therapy. Radiation oncology journal, 35(2):101, 2017.
[69] Omar E Olarte, Jordi Andilla, Emilio Gualda, and Pablo Loza-Alvarez. Light-sheet mi- croscopy: a tutorial. Advances in Optics and Photonics, 10(1):111-17,, 2018.
[70] Karsten Østergaard Noe, Baudouin Denis De Senneville, Ulrik Vindelev Elstrøm, Kari Tanderup, and Thomas Sangild Sørensen. Acceleration and validation of optical flow based deformable registration for image-guided radiotherapy. Acta Oncologica, 47(7):1286-1293, 2008.
[71] Hanchuan Peng, Phuong Chung. Fuhui Long, Lei Qu, Arnim Jenett, Andrew M Seeds, Eugene W Myers, and Julie Simpson. Brainaligner: 3d registration atlases of drosophila brains. Nature methods, (66):493498,, 2011.
[72] Javier Pérez de Frutos, André Pedersen, Egidijus Pelanis, David Bouget, Shanmugapriya Survarachakan, Thomas Langø, Ole-Jakob Elle, and Frank Lindseth. Learning deep abdominal et registration through adaptive loss weighting and synthetic data generation. Plos one, 18(2):c02821 10, 2023.
[73] Chen Qin, Shuo Wang, Chen Chen, Wenjia Bai, and Daniel Rueckert. Generative Myocardial Motion Tracking via Latent Space Exploration with Chomechanics-infomee Prior, June 2022. arXiv:2206.038300 [cs, eess].
[74] Chen Qin, Shuo Wang, Chen Chen, Huaqi Qiu, Wenjia Bai, and Daniel Rueckent. Biomechanics-infores Neural Networks for Myocardial Motion Tracking in MRI, July
2020. arXiv:2006.0477 04725 [cs, eess].
[75] Huaqi Qiu, Chen Qin, Andreas Schuh, Kenstin Hammemnik, and Daniel Rueckert. Learning diffeomorphic and modality- registration using b-splines. 2021.
[76] Lei Qu, Fuhui Long, and Hanchuan Peng. 3-d registration of biological images and models: registration of microscopic images and its uses in segmentation and annotation. IEEE Signal Processing Magazine, 32(1):70-77, 2014.
[77] Marc-Michel Rohé. Manasi Datar, Tobias Heimann, Maxime Sermesant, and Xavier Pennee. Svf-net: learning deformable image registration using shape matching. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part I 20, pages 266-274. Springer, 2017.
[78] Julian G Rosenman, Elizabeth P Miller, and Tim J Cullip. Image registration: an essential part of radiation therapy treatment planning. International Journal of Radiation Oncology* Biology* Physics, 197 1998.
[79] Hanna Siebert, Lasse Hansen, and Mattias P Heinrich. Fast 3d registration with accurate optimisation and little learning for learn2reg 2021. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 174-179. Springer, 2021.
[80] Hessam Sokooti, Bob De Vos, Floris Berendsen, Boudewijn PF Lelieveldt, Ivana Isgum, and Marius Staring. Nonrigid image registration using multi-scule 3d convolutional neural networks. In Medical image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings. Part I 20, pages 232-239. Springer, 2017.
15

---

[81] Joo Hyun Song, Gary E Christensen, Jeffrey A Hawley, Ying Wei, and Jon G Kuhl. Evaluating image registration using nirep. In Biomedical Image Registration: 4th International Workshop, WBIR 2010, Lübeck, Germany, July 11-13, 2010. Proceedings 4. pages 140-150. Springer, 2010.
[82] Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Ap- proximators. In Advances in Neural Information Processing Systems, volume 33, pages 3362-3373. Curran Associates, Inc., 2020.
[83] Lin Tian, Hastings Greer, François-Xatie Xavier Vialard, Roland Kwitt, Raúl San José Estépar, Richard Jarrett Rushmore, Nikolaos Makris, Sylvain Bouix, and Marc Niethammer. Gradi- con: Approximate diffeomorphisms via gradient inverse consistency. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18084-18094, 2023.
[84] Lin Tian, Zi Li, Fengze Liu, Xiaoyu Bai, Jia Ge, Le Lu, Marc Niethammer, Xianghua Ye, Ke Yan, and Daikai Jin. SAME++: A Self-supervised Anatomical eMbeddings Enhanced medical image registration framework using stable sampling and regularized transformation, November 2023, 12Xiv22311.14988 [cs].
[85] Arthur W Toga and Paul M Thompson. The role of image registration in brain mapping. Image and vision computing, 19(1-2):3-24, 2001.
[86] Nicholas Tustison, Hans J Johnson, Torsten Rohlfing, Arno Klein, Satrajit Ghosh, Luis Ibanez, and Brian B Avants. Instrumentation bias in the use and evaluation of scientific software: recommendations for reproducible practices in the computational sciences, 2013.
[87] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep Image Prior. International Journal of Computer Vision, 128(7):1867-188) July 2020. arXiv: 771 17771111199 10925 [cs, stat].
[88] Hristina Uzunova, Matthias Wilms, Heinz Handels, and Jan Ehrhardt. Training cnns for image registration from few samples with model-based data augmentation. In Medical Image Computing and Computer Assisted Intervention- MICCAI 20/7: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part I 20, pages 223-231 Springer, 2017.
[89] David C Van Essen, Heather A Drury, Sarang Joshi, and Michael I Miller. Functional and structural mapping of human cerebral cortex: solutions are in the surfaces. Proceedings the National Academy of Sciences, 95(3):788-795 1998.
[90] Erdem Varol, Amin Nejatbakhsh, Ruoxi Sun, Gonzalo Mena, Eviatar Yemini, Oliver Hobert, and Liam Paninski. Statistical atlas of c. elegans neurons. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part V 23, pages 119-129. Springer, 2020.
[91] Vivek Venkatachalam, Ni Ji, Xian Wang, Christopher Clark, James Kameron Mitchell, Mason Klein, Christopher J Tabone, Jeremy Florman, Hongfei Ji, Joel Greenwood, et al. Pen-nnuronal imaging in roaming caenorhabditis elegans. Proceedings of the National Academy of Sciences, 113(8):E1082 -11082---188,, 2016.
[92] Tom Vercauteren, Xavier Pennec, Aymeric Perchant, and Nicholas Ayache. Son-parametric diffeomorphic image registration with the demons algorithm. In International conference on medical image computing and computer-assisted intervention, pages 319-326. Springer, 2007.
[93] Tom Vercauteren, Xavier Pennec, Aymeric Perchant, and Nicholas Ayache. Symmetric Log- Domain Diffeomorphic Registration: A Demons-Basse Approach. In Dimitris Metaxas, Leon Axel, Gabor Fichtinger, and Gábor Székely, editors, Medical Image Computing and Computer- Assisted Intervention - MICCAI 2008, Lecture Notes in Computer Science pages 754-761, Berlin, Heidelberg, 2008. Springer.
[94] Tom Vercauteren, Xavier Pennec, Aymeric Perchant, and Nicholas Ayache. Diffcomorphic demons: Efficient non non-parametric image registration. Neurolmage, 45(1):561-S7) March 2009.
16

---

[95] Tom Vercauteren, Xavier Pennec, Aymeric Perchant, Nicholas Ayache, et al. Diffeomorphic demons using itk's finite difference solver hierarchy, The Insight Journal, 1, 2007.
[96] Quanxin Wang. Song-Lin Ding, Yang Li, Josh Royall, David Feng, Phil Lesnar, Nile Graddis, Maitham Naeemi, Benjamin Facer, Anh Ho, Tim Dolbeare, Brandon Blanchard, Nick Dee, Wayne Wakeman, Karla E. Hirokawa, Aaron Szafer, Susan M. Sunkin, Seung Wook Oh, Amy Bernard, John W. Phillips, Michael Hawrylycz, Christof Koch, Hongkui Zeng, Julie A. Harris, and Lydia Ng. The Allen Mouse Brain Common Coordinate Framework: A 3D Reference Atlas. Cell, (81:4))33 (43:4)))6-953.220) May 2020.
[97] Yan Wang, Xu Wei, Fengze Liu, Jieneng Chen, Yuyin Zhou, Wei Shen, Elliot K. Fishman, and Alan L. Yuille. Deep Distance Transform for Tubular Structure Segmentation in CT Scans. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1832-3841. Seattle, WA. USA, June 2020. IEEE.
[98] Asmamaw T Wassie, Yongxin Zhao, and Edward S Boyden. Expansion microscopy: principles and uses in biological research. Nature methods, 16(1):33-41, 2019.
[99] Jelmer M Wolterink, Jesse C Zwienenberg, and Christoph Brune. Implicit Neural Representa- tions for Deformable Image Registration. page 11.
[100] Jelmer M Wolterink, Jesse C Zwienenberg, and Christoph Brune. Implicit neural representa- tions for deformable image registration. In International Conference on Medical Imaging with Deep Learning, pages 1349-1359. PMLR, 2022.
[101] Yifan Wu, Tom Z. Jiahao, Jiancong Wang, Paul A. Yushkevich, M. Ani Hsieh, and James C. Gee. NODEO: A Neural Ordinary Differential Equation Based Optimization Framework for Deformable Image Registration. arXiv:2108 03443 [cs], February 2022. arXiv: 2108.0344 03443.
[102] Chenglong Xia, Jean Fan, George Emanuel, Junjie Hao, and Xiaowei Zhuang. Spatial transcriptome profiling by merfish reveals subcellular ma compartmentallzzaioon and cell cycle- dependent gene expression. Proceedings of the National Academy of Sciences, 116(39):19490- 19499, 2019.
[103] Deshan Yang, Hua Li, Daniel A Low, Joseph O Deasy, and Issam El Naqa. A fast inverse consistent deformable image registration method based on symmetric optical flow computation. Physics in Medicine & Biology. 53(21):6143, 2008
[104) Michael A Yassa, Shauna M Stark, Arnold Bakker, Marilyn S Albert, Michela Gallagher, and Craig EL Stark. High-resolutinn structural and functional mri of hippocampal ca3 and dentate gyrus in patients with amnestic mild cognitive impairment. Neuroimage, 51()):1242 -22-1222,, 2010.
[105] Inwan Yoo, David GC Hildebrand, Willie F Tobin, Wei-Chung Allen Lee, and Won- Ki Jeong. ssemnet: Serial-section electron microscopy image registration using a spatial transformer network with learned features. pages 249-257, 2017.
[106] Paul A Yushkevich, John Pluta, Hongzhi Wang, Laura EM Wisse, Sandhitsu Das, and David Wolk. Ic-p-174: fast automatic segmentation of hippocampal subfields and medial temporal lobe subregions in 3 tesla and 7 tesla (2-weighted mri. Alzheimer's 's & Dementia, 12:P126 12:P12--PI27) P127, 2016.
[107] Liutong Zhang, Lei Zhou, Ruiyang Li, Xianyu Wang. Boxuan Han, and Hongen Liao. Cas- caded feature warping network for unsupervised medical image registration. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 913-916. IEEE, 2021.
[108] Shengyu Zhao, Yue Dong, Eric I-Chao Chang, and Yan Xu. Recursive cascaded networks for unsupervised medical image registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[109] Shengyu Zhao, Tingfung Lau, Ji Luo, Eric, Chao Chang, and Yan Xu. Unsupervised 3d end- to-end medical image registration with volume tweening network. IEEE journal of biomedical and health informatics, 24(5):1394 1404, 2019.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are shown empirically in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The families of registration algorithms is limited to gradient- based methods isolate dynamics of gradient- rraseent-based methods. Our paper only uses brain datasets, since neuroimaging is one of the most (if not the most) popular modalities for studying registration algorithms. This is discussed in the limitations subsection.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specine specs-iefiii,,nnn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Supplemental material contains scripts to reproduce all experiments of the paper. Code and pretrained models will be published to Github upon acceptance, with additional documentation, tutorials and instructions. Data is publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
19

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is provided in the supplemental material. Data is publicly available and instructions to reproduce the results are provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Stteee/guidee//oSSuteeeeee stt.subbmssiooooccoo...... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https w/tpps.cc/pubicc/// wwwpbiicgguideee//eeeeee Soa.eeee/e/oSSSemmisseeeeeeee asionnlicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All recommended parameters of the existing methods used (and mentioned in the paper).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: p-values reported for t-tests comparing classical and deep learning methods. Boxplots with interquartile ranges are reported for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Each baseline used in the paper has its own requirement specified in their work. We use single machine for all experiments mentioned (specs are specified in the paper).
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: No research is performed involving new human subjects, animals, or environ- mental impact. Existing datasets comply with Code of Ethics. The proposed research is entirely computational. The proposed research has no immediate negative societal impact.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Medical image registration has no immediate negative socictal impact necessi- tating dedicated discussion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appropriate citations are provided for existing code and data.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcode com com/datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Instructions to reproduce the results are provided in the paper and supplemental material. Only instructions to run the existing baselines are provided, no new method is proposed.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "xRhNpfL1O0", "reviewer": "Reviewer_tuv2", "summary": "The manuscript investigates the characteristics of two types of registration approach based on traditional variational optimization and deep learning. Experiments revealed a correlation between the mutual information of the distribution of per-pixel intensity and labels, and the performance of classical registration methods. Then the manuscript argues unsupervised deep learning does not improve label matching performance compared to traditional methods, whereas supervised learning methods show improved label matching. Lastly they show learning methods do not generalize well.", "questions": "In Fig. 1, shouldn't there also be a correlation within each individual dataset?", "limitations": "While the intro and abstract suggest a study on general registration problems, all experiments are based on brain MRI. It is not clear whether the problem/hypothesis/conclusion is specific to brain MRI.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This study reflects original thinking, trying to draw insights on a challenging topic.\n2. Experiments design is well motivated to study the hypothesis.\n3. The results challenge a claim made by a number of existing studies \"learning methods can provide improve label matching when optimized in an unsupervised fashion\"", "weaknesses": "1. Two of the three claims discussed herein seem trivial. It is not surprised that \"Supervised DLIR methods demonstrate enhanced label matching\" and \"DLIR methods do not generalize across datasets\".\n2. I feel that the abstract and intro set up a high expectation by saying \"we propose a general recipe to choose the best paradigm for a given registration problem, based on these observations.\" but then again it is a one-sentence recipe in the end that is not surprising to readers \"a practitioner should choose DLIR methods only if they have access to a large labeled dataset, and their application is limited to the same dataset distribution. In all other cases, classical optimization-based methods are the more accurate and reliable choice\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful feedback, and are pleased to note the recognition of the originality of our work, the well-motivated design of our experiments, and the challenge posed to existing claims in the DLIR literature. We address some of their concerns below, and hope that their review and discussion can help convince all reviewers of the importance of our work, and champion our paper.\n\n**Two of the three claims seem trivial.**\n\nWe agree that the claims seem straightforward in hindsight. However, we contextualize this conclusion from the claims made by existing DLIR literature, i.e. a carefully selected intensity loss function, network architecture, pretraining schema and large datasets lead to SOTA performance on alignment of anatomical regions. A variety of studies cited in our paper report results with and without label supervision, which we show is the confounding variable that leads to a significant improvement in performance. Therefore, performance of DLIR methods with and without Dice score must be compared to justify the factors involved in improving performance of DLIR methods. \nOur analysis in Section 3 elucidates that unsupervised methods do not benefit from advancements in network architecture, a point further verified in Section 4. \nSection 5 is included to underscore the fact that DLIR methods’ superior performance indeed comes from incorporating label supervision during training, a capability absent in classical methods.\n\nSection 6 seems trivial in the context of our paper, but for the biomedical community at large, pretrained models and large datasets are built to train models that generalize outside the training domain (see common rebuttal). Our findings in Section 6 suggest a need to reassess huge labeled data collection efforts and to formulate DLIR advancements that exhibit robustness against minor domain shifts instead. This reevaluation is crucial for leveraging large amounts of annotated data effectively. We have added this both in the Motivation and Discussion sections.\n\n**“I feel that the abstract and intro set up a high expectation..”**\n\nWe acknowledge that the phrasing of our contribution may have set an overly ambitious expectation. We have revised the statement in the paper to: “We reassess and recalibrate performance expectations from classical and DLIR methods under access to label supervision and training time, and its generalization capabilities under minor domain shifts.” This modification more accurately reflects the scope and implications of our study.\n\n**“shouldn't there also be a correlation within each individual dataset”**\n\nWe find that since the labeled regions are fixed within a dataset, the variation between MI and registration score is dominated by other intra-dataset factors. The correlation between MI and registration performance occurs at the granularity of the dataset, which is shown in Fig.3. This granularity highlights the broader applicability of our findings across different datasets rather than within a single dataset.\n\n**“are the hypothesis/conclusion specific to brain MRI”**: \n\nWe assert that the methods and evaluation metrics employed in our study do not incorporate domain-specific information, making our observations broadly applicable. Brain MRI was chosen due to the extensive availability of solutions and open-access resources provided by the neuroimaging and machine learning communities. This choice ensures that our analysis is unhindered by fairness or reproducibility concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Y7qDz51oj", "reviewer": "Reviewer_23gn", "summary": "This paper discussed about an explicit correspondence between mutual information of the distribution and performance of classification registration methods. The authors argued that this correlation will not be affected by the learning-based methods. They validated this hypothesis on both classical and learning-based registration models and found that the weakly supervised learning-based methods performed high-fidelity intensity and label registration. Besides, they showed that these high-fidelity feature learning methods did not translate to invariance to domain shift, ending with proposing a general approach to select the best paradigm for a registration task.", "questions": "Please see the weaknesses section. I tried to summarize all the findings, concerns, and questions there.", "limitations": "Limitations have been discussed in the paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**S1.** The authors experimented on the feature learning approach and their translation to invariancy to domain shift, which is an important aspect in deformable image registration. Also, they covered both classical and learning-based image registration to validate their hypothesis.   \n\n**S2.** The authors considered out-of-distribution datasets for testing and evaluating performances, which showed promising results regarding generalization.", "weaknesses": "**W1. Experimental findings.** \n\n1. I appreciate the authors' finding on getting improved performance for classical methods over learning-based methods. However, I would argue that this is typically not always true considering the reported findings in the learning-based papers, e.g., VoxelMorph (VM), TransMorph (TM), etc. This registration results highly depending on the image pairs that are being considered. For example, image pairs with large age variations won't perform well in the learning-based registration tasks (please check VM, Deepflash, and related papers where they restricted the subject ages to deal with large deformations). Besides, inter-patient registration has a higher probability of getting similar results compared to the atlas-based (or pre-selected template) registration, thoroughly discussed in VM and TM papers. I found this important information missing from their experimental evaluation to support their first hypothesis (Sec. 4). I would encourage the authors to consider experimenting and reporting performances for within-patient, inter-patient, and atlas-to-patient registration results. Also, comparing the anatomical dice scores further justifies the registration models as in the existing SOTA it has been discussed about getting larger variations for critical anatomical structures such as ventricles, pallidum, cerebellum, etc.  \n\n2. I found some of the mode's performance very inconsistent with the reported version in the original papers. For example - LKU-Net is achieving more than $~0.925\\pm0.025$ dice score where the authors' reported best within subject accuracy is $0.8861\\pm0.01$ with an average dice score of $0.7758\\pm0.0390$. The same goes for the LapIRN model's performance. There's a large difference between the reported dice in this paper vs the original paper's dice. I understand there might be different biases that might be there in terms of implementation. However, the current reported performance of these models raises questions about the credibility of the adapted implementations, preparation of the pairwise images, hyperparameter setup, etc. I would suggest the authors kindly shed some light on this part.\n\n3. I found the supporting experiments to validate the hypothesis presented in Sec. 6 missing some important information. Did the authors perform an affine transformation on all the datasets and what are the pre-processing steps? Are they considering all 3D volumes for their experiments?\n\n\n**W2. Missing large deformation diffeomorphic and related baselines.** I appreciate the authors for trying to carry out thorough experiments on different classical and learning-based registrations. However, I believe the current hypothesis would be more understandable and justifiable if the authors could initiate some experiments considering large deformation diffeomorphic-based registration methods such as LDDMM [1,2,3], where this kind of method is structured upon time-varying velocity fields, that is proven to be more robust in various deformation-based image analysis tasks.\n\n\n**W3. Overclaimed hypothesis.** Statements in L332-L335 seem overstated. The authors need to discuss and experiment on within-subject/class, inter-subject/class, and atlas/template-based registration to validate the stated hypothesis in that line, which seems to be missing in the current version. Besides, the selection of the DLIR/classical registration method depends on the image analysis tasks that are being performed over image registration. Without verifying the studied registration on different image analysis tasks, it is inappropriate to come to a conclusion stated in L332.\n\n\n**W4. (Minor) Technical writing.** The authors tried to aggregate all their potential findings in a structured way. But reading the whole paper kind of messed me up in understanding what are the actual contribution of this paper compared to the other survey/review papers in this domain other than performing experiments on OOD data. Overall, the presentation is kind of above the borderline but I guess if the authors tried to focus on their storyline and make their findings more clearer that would be great for the readers. For example, I found implementation details in most sections which is kind of redundant.  \n\n\n\nOverall, I appreciate the authors for working on this paper which is very relevant as well as important in the medical imaging domain, specifically in medical image registration. However, the current version of the manuscript lacks some important experimental justification and further experiments. With that being said, the current version of the manuscript is under the threshold of acceptance. However, I am open to reconsidering the initial rating if the above concerns are adequately justified.\n\n\nReferences\n----------------\n[1] Yang, Xiao, Roland Kwitt, and Marc Niethammer. \"Fast predictive image registration.\" Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings 1. Springer International Publishing, 2016.\n\n[2] Shen, Zhengyang, François-Xavier Vialard, and Marc Niethammer. \"Region-specific diffeomorphic metric mapping.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[3] Niethammer, Marc, Roland Kwitt, and Francois-Xavier Vialard. \"Metric learning for image registration.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] Wang, Jian, and Miaomiao Zhang. \"Deepflash: An efficient network for learning-based medical image registration.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their highly detailed and insightful feedback, this has immensely helped in improving the quality of our work. We believe we have addressed all major concerns and incorporated changes in the paper.\n\n**“I would argue that this is typically not always true considering the reported findings in the learning-based papers, e.g., VoxelMorph (VM), TransMorph (TM), etc”**\n\nWe appreciate this perspective and have addressed this concern in Sec4 (Instrumentation Bias) and Fig3. VM reports a Dice score of 0.749 for ANTs using suboptimal parameters ($\\sigma$ of 9px followed by 0.4px), while we observe a Dice score of 0.787 with default parameters, highlighting the need for unbiased evaluation. Our evaluation of VM uses their pretrained model and scripts, yielding consistent results w/ their paper. Other studies on lung [5] and histology [6] registration also show that classical DIR methods far outperform DLIR methods. An unbiased and unified evaluation of algorithms is necessary to fairly compare the performance gaps of said algorithms. \nOur paper finds that TM is the only DL method beating classical methods (p<0.01) on the OASIS dataset with and without Dice loss (Fig 2 and 4). TM also demonstrates good generalization among DLIR methods competing with SynthMorph, although still underperforming compared to classical methods. We urge the reviewer to review these figures and results. We are happy to clarify any remaining discrepancies in the discussion.\n\n**“Performance is different for inter-subject and atlas-based registration, image pairs with large age variations won't perform well in the learning-based registration tasks”**\n\nIn general, a vast majority of methods focus on inter-subject registration, and atlas-building and atlas-based registration are considered separate problems altogether. Therefore, we keep inter-subject registration datasets a consistent theme in the paper to corroborate existing findings and avoid mixing up results from different problem statements.\n\nRe large age variations: A good registration model in general should perform well across large variations in age, subject demographics, and acquisition configurations. Classical methods performing well for age variations further strengthen our findings in Sec.4 and Sec.6, i.e. labeled data becomes necessary in these settings for DLIR methods to perform better. We have added these observations into the paper further motivating our study. \n\n**“consider experimenting and reporting performances for within-patient, inter-patient, and atlas-to-patient registration results”**\n\nThese are interesting experiments indeed and form the scope for future work, along with multimodal registration, and registration of different anatomy like chest CT or abdomen. Due to space constraints, fairness and reproducibility considerations, we choose OASIS dataset for training since a large number of SOTA DLIR methods provide training recipes for this dataset, and IBSR/CUMC/MGH/LPBA datasets that have been extensively used in unbiased community-standard neuroimaging evaluations [7,9]. \n\n**“model performance is very inconsistent with the reported version in the original papers, LKU-Net is achieving 0.77 average Dice… same with LapIRN model's performance”**\n\nWe’re not sure where these numbers come from. In Fig.3 we show that the LKU paper reports a 0.88 Dice score, and our evaluation shows a 0.904 Dice score – Fig.4. (top) shows that LKU-Net is the top performer with Dice supervision. Similarly, we show in Fig2. that LapIRN reports 0.808 Dice but we report 0.788 – a reasonable deviation. In fact, LapIRN performance improves significantly when Dice supervision is added (Fig2 v/s Fig4), the latter of which the original paper did not implement or report. \n\nOur work takes great care to accurately reproduce and not misrepresent the performance of DLIR and classical methods. We ask the reviewer to review the subsection on Instrumentation bias (Sec 4) – we’re confident they will find DLIR methods represented accurately. DLIR methods may misrepresent classical baselines by choosing suboptimal hyperparameters; we work towards a more fair, unbiased and unified evaluation of classical and DLIR methods under the same umbrella.\n\n**“hypothesis presented in Sec 6 missing some important information”**\n\nFor all datasets, we follow the preprocessing steps followed by Klein et.al. [7]. We have added this detail in the paper.\n\n**“W2. Missing large deformation diffeomorphic and related baselines”**\n\nClassical methods like ANTs, Greedy, FireANTs build on the LDDMM framework to avoid explicitly storing the full 4D velocity fields by integrating the infinitesimal velocities using gradient descent, & are shown to model *very* large deformations by independent evaluations [7,8]. Moreover, these are widely used classical baselines used by the broader biomedical community, and should not be seen as a reason to invalidate the results in the paper.\n\n**“W3. Overclaimed hypothesis”**\n\nWe have changed L332-335 to reflect the conclusions for inter-subject registration, which is a defacto standard for registration. Since none of the analysis, baselines, and evaluation make any domain/subject-specific modeling assumptions, and the datasets are community-standard benchmarks, the results are valuable, both within the neuroimaging and the biomedical communities at large.\n\n**“W4. (Minor) Technical writing”**\n\nThe contribution of the paper is to recalibrate claims about DLIR methods' performance by isolating label supervision as the primary contributor to their superiority. This motivates the question if label supervision can be robust to domain shift and finds it is not, prompting a reconsideration of data collection pipelines to improve registration performance, and rethinking DLIR design decisions to be robust to domain shift. We have made these changes in the contributions and discussion sections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sd5urmBC6B", "reviewer": "Reviewer_Xf1e", "summary": "The work benchmarks the traditional methods and deep learning-based methods for medical image registration and gives a general recipe to choose registration methods.", "questions": "- Please explicitly clarify the advantages compared to the well-known lean2reg challenge.\n- Does the conclusion still hold on large datasets? AMOS dataset provides lots of CT and MR images can be used for model training. \n- Why not submit this work to the dataset and benchmark track?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Comprehensive experiments: The authors implement several classical variational and deep learning-based registration models on four public datasets of brain CT/MR images.", "weaknesses": "- The training set of deep learning models may not be sufficient because the number of image pairs is obviously not enough, which potentially makes the comparison with classical models unfair. \n- Medical image registration contains a lot of content, it cannot and should not be simply represented by monomodal registration on four brain image datasets. The authors need to implement more experiments on images of more organs and tissues with monomodal and multimodal registration tasks.\n- The subtitle “DLIR methods do not generalize across datasets” is almost common sense that networks trained on specific datasets cannot easily generalize to other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and helpful feedback, which has significantly contributed to improving the quality of our paper. We believe we have addressed all the remaining concerns and hope the reviewer will consider increasing their score and advocating for the acceptance of our paper.\n\n**The training set of deep learning models may not be sufficient**\n\nAll methods are trained on the modified OASIS training split of 364 images, yielding approximately 132k inter-subject training pairs (364 * 363). This is a substantial number of image pairs, and in practice, most DLIR methods converge within 2-3 epochs. Furthermore, most baselines like LapIRN train on an even smaller subset of OASIS, approximately 62k training pairs (250 * 249)  (although we train on 364 images for consistency). Thus, the dataset size used is sufficient for a fair comparison with classical models. \n\n**The authors need to implement more experiments on images of more organs and tissues with monomodal and multimodal registration tasks**\n\nWe have already acknowledged this in the Limitations section in our paper, which forms the basis for future research. We chose the inter-subject MRI brain registration problem due to its widespread adoption by the biomedical ML community, ensuring adequate reproducibility and unbiased evaluation. While we recognize the importance of exploring other organs and tissues, none of our analysis, baselines, or evaluation protocols use domain-specific knowledge, suggesting that our results are not strictly limited to brain MRI. Expanding to more variations is a priority for future work.\n\n**The subtitle “DLIR methods do not generalize across datasets” is almost common sense**\n\nWe agree, this is almost trivial in hindsight! However, seeing the results in Sec.4. and Sec.5., one might expect the domain-shift performance to fall between the results of in-distribution unsupervised and supervised training, with supervised DLIR methods outperforming classical methods to some extent. However, this is not the case (please see common rebuttal for more details), showing that large amounts of labeled data would not necessarily help on datasets with even the same modality and resolution but slightly different intensity statistics. This raises a question for large-scale, annotated data collection efforts, and addressing current limitations of DLIR methods to be robust to domain shift.\n\n**Please explicitly clarify the advantages compared to the well-known lean2reg challenge.**\n\nLearn2reg challenge which focuses on methods to maximize registration performance (measured by label overlap, landmark distance, determinant of Jacobian) on a particular dataset with data/modality specific challenges. In contrast, we perform a meta-analysis and unbiased evaluation on the factors that lead to better performance of aforementioned registration algorithms. We find that DLIR methods attribute the performance improvement to architecture, loss functions, and training specifications. However, we show that most of the improvement is attributed to being able to learn from supervised label map objectives during training. Moreover, we show that training with label maps (which are intrinsic to modality and therefore invariant to the downstream modality – see more in common rebuttal) does not necessarily imply robustness in performance to domain shift. We have modified the “Contributions” subsection to emphasize these distinctions.\n\n**Does the conclusion still hold on large datasets? AMOS dataset provides lots of CT and MR images can be used for model training**\n\nThe AMOS dataset, with 500 CT and 100 MR images, is comparable to the OASIS dataset in size (414 images). We selected the OASIS dataset for reproducibility and fairness considerations, as numerous DLIR methods provide pretrained models and training recipes for OASIS. The active community around OASIS and its long-standing presence has led to detailed analyses of dataset-specific challenges (for example, registration of smaller subcortical structures). Although the AMOS dataset is relatively new, it holds significant potential for future work, and we plan to explore it in subsequent studies.\n\n**Why not submit this work to the dataset and benchmark track?**\n\nOur work goes beyond merely reproducing or benchmarking existing methods. We provide a theoretical and empirical analysis of the factors responsible for improved performance in registration tasks, particularly the access to label maps during training, and their implications for robustness across similar datasets. This paper is a meta-analysis and unbiased evaluation, similar in spirit to [7].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tcboDXPYsw", "reviewer": "Reviewer_Aeaf", "summary": "The paper delves into the comparison between classical optimization and learning-based medical image registration methods. Some valuable insights are proposed and the authors propose a general recipe to choose the best paradigm for a given registration problem.", "questions": "What are the potential directions of future work based on this paper?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation behind the work is clear. The observation is in detail.\n2. Rich experiments discover the regular and make people rethink the development of learning-based registration methods.\n3. The illustrations are clear and easy to understand.\n4. Clear conclusions are obtained for registration paradigms in different conditions.", "weaknesses": "1. New solutions or deeper insights are missed.\n2. Findings in Section 4,5,6 are somewhat valuable, but the relevancy between different phenomena is not very clear. The author failed to point out what should we do or what we may further research for unsupervised or supervised DLIR methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive and helpful feedback, we are glad to find they found the paper having a clear motivation, detailed observations, clear illustrations and easy-to-read paper. We believe we have addressed all the remaining concerns and hope the reviewer increases their score and advocates for the acceptance of our paper.\n\n**”New solutions or deeper insights are missed.”, “Findings in Section 4,5,6 are somewhat valuable, but the relevancy between different phenomena is not very clear”**\n\nOur work serves as a bedrock for contextualizing the performance of current state-of-the-art deep learning and classical registration methods, and identifying the critical confounding variables responsible for the performance gaps observed in DLIR and classical methods. We show empirically the two confounding variables – instrumentation bias of classical methods, and label supervision are majorly responsible for the superior DLIR performance of in-distribution data. \n\nOur work provides an independent, fair and unbiased evaluation of both DLIR and classical methods, and show that in the unsupervised case, classical baselines outperform DLIR methods without having any in-distribution data – making it useful in the low-data regime. Furthermore, since anatomical labels are an intrinsic property of the anatomy (hence being domain-invariant), one may expect label supervision to add some robustness to domain shift. However, this is not the case (Sec 6).\n\nThese experiments naturally imply a recipe to use for a given instance of registration (mentioned in Sec 7). Moreover, these results also allow us to reconsider deploying expensive data collection pipelines, especially when domain shift may be expected at test time (as a consequence of the finding in Sec 6). Moreover, observations in Sec.4. and Sec.6. serve as a problem statement for DLIR-based methods to alleviate these limitations. We urge the reviewer to also read more details about this in the common rebuttal.\n\n**What are the potential directions of future work based on this paper?**\n\nWe appreciate the reviewer's interest in future research directions. Our limitations section outlines several avenues for further investigation. This study can serve as a foundational basis for examining and disentangling the variables responsible for robust performance in various contexts, such as:\nMultimodal Registration: Extending our analysis to multimodal registration scenarios to explore the applicability of our findings across different imaging modalities.\nNon-Brain Anatomy: Investigating the generalizability of our results to other anatomical regions beyond brain MRI.\nDiverse Registration Settings: As suggested by Reviewer 23gn, conducting experiments on inter-subject, intra-subject, and atlas-based registration to provide a comprehensive understanding of performance across different registration paradigms.\nWe have also incorporated these future work suggestions more explicitly in the revised paper to guide readers on how our findings can be extended and applied in broader contexts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful feedback and for taking time to improve the quality of our work. We are glad that reviewers found our work reflecting original thinking and drawing useful insights [tuv2], well motivated problem [Aeaf], consideration of both classical and DLIR methods [23gn], well motivated and well designed experiments [tuv2, Xf1e, Aeaf], clear writing [Aeaf]. We have addressed all questions in the individual comments. We summarize and clarify some common concerns: 

**Some of the claims are trivial**

We agree that these claims look trivial in hindsight. However, these claims must be contextualized from the perspective of claims made by existing DLIR literature. Specifically, most DLIR methods propose either a new architecture, training recipe (multi-scale, cascaded, or pre-training), or loss functions, and claim that these methodologies lead to superior performance over classical registration methods. However, the argument in Section 3 / Fig2. indicates that these contributions do not add any information on the existing correlation between mutual information and dice score. Hence, the primary driver of performance improvement is anatomical label supervision.

Our work provides a unified and fair evaluation of classical and DLIR methods without label supervision (Sec 4), confirming that the conclusions from Section 3/Figure 2 hold true. An ablation study over label supervision (Sec 5) further verifies that label supervision is indeed the confounding variable leading to superior performance. Thus, Sections 4 and 5 are pivotal in isolating the real confounding variable, both theoretically and empirically.

Importantly, anatomical labels are intrinsic to the anatomy and not inherently modality-dependent, making them domain-invariant. This motivation for using synthetic data with consistent label maps is often used to train domain-invariant models [1][2]. Our key non-trivial finding, presented in Section 6, reveals that while DLIR models trained with label supervision may perform worse under domain shift (DS) compared to in-distribution (ID) scenarios, the expected performance hierarchy does not hold:

Expected: (DLIR Sup ID) > (DLIR Sup DS) > (Classical)

Observed: (DLIR Sup ID) > (Classical) > (DLIR Sup DS)

This significant finding highlights that DLIR methods do not maintain their performance advantage over classical methods under domain shift. This result has profound implications for annotated data collection and challenges the notion that large labeled datasets ensure robust generalization. For example, Learn2Reg 2023 [3] mentions “By providing easy-to-use solutions applicable to a variety of medical registration problems, we hope to strengthen both generalization and comparability between them.”. However, our findings underscore that the generalization of DLIR methods to domain shifts is heavily understudied and generally negative. This necessitates a reassessment of expensive labeled data collection efforts and emphasizes the need for developing DLIR methods, like those proposed in [1], that are more robust to domain shifts.
We have added a subsection in Section 3 summarizing these motivations and the broader implications of our findings. We hope this better contextualizes and emphasizes the significance of our results.

**Are the findings applicable to brain MRI only?**

We assert that the methods and evaluation metrics employed in our study do not incorporate domain-specific information, making our observations broadly applicable. Brain MRI was chosen due to the extensive availability of solutions and open-access resources provided by the neuroimaging and machine learning communities, also somewhat inspired by [7]. This choice ensures that our analysis is unhindered by fairness or reproducibility concerns. 

**What about other modalities / anatomy / intra-subject / atlas-based settings?**

We have already acknowledged this in the Limitations section in our paper, forming the basis for future research. We chose the inter-subject MRI brain registration setup due to its widespread adoption by the biomedical ML community, ensuring adequate reproducibility with unbiased and fair evaluation. Registration literature in other anatomy and modalities – hippocampus - Table 1 in [4], lung images [5], and histology [6] also seem to suggest similar results (classical methods outperform DLIR in unsupervised settings), but a more comprehensive study with unified training and inference setup is necessary and forms the motivation for future work.

[1] Hoffmann, Malte, et al. "SynthMorph: learning contrast-invariant registration without acquired images." IEEE transactions on medical imaging 41.3 (2021): 543-558.

[2] Dey, Neel, et al. "AnyStar: Domain randomized universal star-convex 3D instance segmentation." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[3]  https://learn2reg.grand-challenge.org/learn2reg-2023/

[4] W. Zhu, Y. Huang, D. Xu, Z. Qian, W. Fan and X. Xie, "Test-Time Training for Deformable Multi-Scale Image Registration," 2021 IEEE International Conference on Robotics and Automation (ICRA)

[5] Fu, Yabo, et al. "LungRegNet: an unsupervised deformable image registration method for 4D‐CT lung." Medical physics 47.4 (2020): 1763-1774.

[6] Borovec, Jiří, et al. "ANHIR: automatic non-rigid histological image registration challenge." IEEE transactions on medical imaging 39.10 (2020): 3042-3052.

[7] A. Klein, et al. Evaluation of nonlinear deformation algorithms applied to human brain MRI registration. NeuroImage, 46(3):786–802, July 2009.

[8] Murphy, Keelin, et al. "Evaluation of registration methods on thoracic CT: the EMPIRE10 challenge." IEEE transactions on medical imaging 30.11 (2011): 1901-1920.

[9] Mok, Tony CW, and Albert Chung. "Affine medical image registration with coarse-to-fine vision transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Meta-review: This work investigates the classical optimisation and learning-based methods in deformable image registration, with the aim to gain more understanding and insights into the performance, generalisation, and the working conditions of the two typical approaches to this task. Reviewers recognise that this work has merits of original thinking, well motivated experimental design, results that challenge existing understanding, considering out-of-distribution test case, and rich experiments making people rethink. At the same time, the reviewers raise the following issues, including 1) some claims that seem trivial; 2) the need to consider more registration settings; 3) the inconsistency of performance reported; 4) missing baselines and overclaimed hypothesis; 5) the insufficiency of the training set; 6) more experiments on organs, tissues, and modalities; 7) missing new solutions or deeper insights; and 8) not pointing out the potential future direction. The authors have made efforts to provide response to each of the raised issues. Among the four reviewers, after reading the rebuttal, one reviewer is still positive (with rating 6); one reviewer thinks this work is still on the borderline (with rating 4); one reviewer is not fully convinced and slightly raises the rating to 4; and the last reviewer has no further questions and maintains the original rating of 5.

By reading the reviews, rebuttal and the submission, AC thinks that the rebuttal is overall clear, sound and effective. Furthermore, as indicated by two reviewers, this work reflects original thinking and makes people rethink the development of learning-based registration methods. Although some claims do not seem to be surprising enough, validating them via systematic investigation has its value and significance to the relevant research community. This kind of research shall be encouraged in order to better understand the many existing methods. Considering all the factors, AC recommends this submission for acceptance. Meanwhile, the authors shall adequately incorporate the key points in the rebuttal to further strengthen this work, including the presentation.
Decision: Accept (poster)