Deep Learning in Medical Image Registration: Magic or Mirage?

Authors: rohit  jena
deeksha  sethi
pratik  chaudhari
james c gee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 14:46:26

Abstract: Classical optimization and learning-based methods are the two reigning paradigms in deformable image registration. While optimization-based methods boast generalizability across modalities and robust performance, learning-based methods promise peak performance, incorporating weak supervision and amortized optimization. However, the exact conditions for either paradigm to perform well over the other are shrouded and not explicitly outlined in the existing literature. In this paper, we make an explicit correspondence between the mutual information of the distribution of per-pixel intensity and labels, and the performance of classical registration methods. This strong correlation hints to the fact that architectural designs in learning-based methods is unlikely to affect this correlation, and therefore, the performance of learning-based methods. This hypothesis is thoroughly validated with state-of-the-art classical and learning-based methods. However, learning-based methods with weak supervision can perform high-fidelity intensity and label registration, which is not possible with classical methods. Next, we show that this high-fidelity feature learning does not translate to invariance to domain shift, and learning-based methods are sensitive to such changes in the data distribution. We reassess and recalibrate performance expectations from classical and DLIR methods under access to label supervision, training time, and its generalization capabilities under minor domain shifts.

Main paper: 
 introduction: 
 Deformable Image Registration (DIR) refers to the local, non-linear (hence deformable) alignment of images by estimating a dense displacement field. Many workflows in medical image analysis require images to be in a standard coordinate system for comparison, analysis, and visualization. In neuroimaging, communicating and comparing data between subjects requires the images to lie in a standard coordinate system [48, 96, 89, 32, 81, 85] . This assumption universally does not apply when brain image data are compared across individuals or for the same individual at different time points. Anatomical correspondences between diseased patients and normative brain templates help identify and localize abnormalities like tumors, lesions, or atrophy. Failed or anomalous correspondences impact diagnosis, treatment planning, and disease progression monitoring. DIR is also used to capture and quantify biomechanics and dynamics of different anatomical structures including myocardial motion tracking [74, 73, 7] , improved monitoring of airflow and pulmonary function in lung imaging [66, 27, 97] , and tracking of organ motion in radiation therapy [45, 14, 68, 78] . Latest breakthrough advances in imaging techniques like fluorescence and light-sheet microscopy [36, 69, 29, 98] , in-situ hybridization, and multiplexing [65, 102] have led to image registration being imperative in advancing life sciences research. Relevant research includes a brainwide mesoscale connectome of the mouse brain [67] , uncovering behavior of individual neurons in C. elegans [91] , building cellular-level atlases of C. elegans, Drosophila melanogaster, and the mouse brain [105, 90, 96, 76, 71, 13] .Classical optimization-based and learning-based methods are the two reigning paradigms in DIR. Classical DIR methods are based on solving a variational optimization problem, where a similarity metric is optimized to find the best transformation that aligns the images. Most classical methods are formulated without any particular domain knowledge encoded in the optimization problem, and are therefore general and applicable to a wide range of problems. For instance, the popularly known registration toolkit ANTs [5] has been successfully applied to structural and functional neuroimaging data [48, 104, 43] , CT lung imaging [66] , cardiac motion modeling [53] , developmental mouse brain atlases utilizing MRI and light sheet fluorescence microscopy [50] with virtually no change in the optimization algorithm. However, classical iterative methods have slow convergence, their performance is limited by the fidelity of image intensities, and they cannot incorporate learning to leverage a training set containing weak supervision such as anatomical landmarks, label maps or expert annotations. Deep Learning for Image Registration (DLIR) is an interesting paradigm to overcome these challenges. DLIR methods take a pair of images as input to a neural network and outputs a warp field that aligns the images, and their associated anatomical landmarks. The neural network parameters are trained to minimize the alignment loss over image pairs and landmarks in a training set. During inference, an image pair is provided and the network regresses a warp field. A primary benefit of this method is the ability to incorporate weak supervision like anatomical landmarks or expert annotations during training, which performs better landmark alignment without access to landmarks at inference time.Motivation However, the benefits of using DLIR methods over classical DIR methods in terms of accuracy or robustness to domain shift are still topics with no clear consensus. Several DLIR methods claim that architectural choices and loss function design combined with amortized optimization of neural network parameters significantly outperform classical methods [63, 61, 17] . On the contrary, classical iterative methods that leverage implicit or explicit conventional priors have shown to outperform most deep learning methods on other challenging datasets [100, 79] . For example, in the context of lung registration, an implicit neural optimization method surpasses every deep learning baseline on the DIR-lab dataset [100] . In EMPIRE10 challenge without access to labeled data [66] , classical methods are highly performant compared to deep learning methods. In the ANHIR histology registration challenge [12] , the best performing algorithms were classical methods, and the deep learning method was fast and performed well, but did not have good generalization capabilities. Mok et al. [62] also mention that deep baselines typically fail 'spectacularly' on out of distribution data, and classical methods like Elastix and ANTs come out on top. However, these observations are relatively unstructured and not studied directly. The confounding variable of using labelmap supervision has urged the Learn2Reg 2024 LUMIR challenge [35] to be performed on fully unsupervised data. In our own empirical evaluations, we found that classical methods typically outperform deep methods under certain conditions and assumptions. Image registration is NP-hard being a non-convex optimization problem, and approximating the solution of NP-hard problems with deep learning methods is not guaranteed to be optimal, or even a minima of the registration loss at test-time. Deep learning methods also claim to provide amortized optimization since classical methods are extremely slow to run, however, modern GPU implementations [55, 59, 41] have patched this shortcoming of classical methods while providing state-of-the-art performance.Contributions. The conditions needed for either paradigm to perform well over the other are clouded and not explicitly outlined in the existing literature. This has prolonged the tug-of-war between classical and deep learning methods. We perform a more structured problem setup and empirical evaluation to determine consensus on the benefits and limitations of each paradigm. First, we observe a strong correlation between the mutual information between per-pixel intensity and label maps, and the performance of classical registration methods. This strong correlation hints to the fact that the Jacobian projection in DLIR methods is unlikely to affect this correlation, and therefore, the performance of DLIR methods in the unsupervised setting. We empirically verify this hypothesis on a variety of state-of-the-art classical and DLIR methods, and address instrumentation bias in the existing literature. Secondly, since the label map is a deterministic function of the intensity image, DLIR methods can learn to perform better label matching when this constraint is enforced during training, by implicitly discovering the label map within the network features and predicting a warp field that minimizes the alignment error between label maps. This is a key strength of DLIR methods, that classical methods cannot leverage. Third, we show that even though learning methods implicit capture semantic information from the image which is not explicitly captured by classical methods, this additional feature learning does not translate to invariance to domain shift, and DLIR methods are brittle to these changes. These empirical findings allow us to reassess and recalibrate performance expectations from classical and DLIR methods, using a systematic, unbiased and fair evaluation.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: All claims are shown empirically in the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
classical optimization-based methods: 
 Classical image registration algorithms employ iterative optimization on a variational objective to estimate the dense displacement field between two images. Some of the earliest approaches to deformable registration considered models for small deformations using elastic deformation assumptions [51, 23, 8, 31, 30, 20, 21] , conceptualizing the moving image volume as an elastic continuum that undergoes deformation to align with the appearance of the fixed image. This was in conjunction with alternate formulations based on fluid-dynamical Navier-Stokes [22, 21] and Euler-Lagrange equations [2, 11, 4, 56, 58] and their subsequent optimization strategies. The seminal work of Beg.et al. [11] introduces an explicit Euler-Langrange formulation and a metric distance on the images as measured by the geodesic shortest paths in the space of diffeomorphisms used to transform the moving image to the fixed image. However, storing the explicit velocity fields is expensive in terms of compute and memory. This limitation motivated semi-Langrangian formulations [4, 3] to avoid storing velocity fields explicitly, and only storing the final diffeomorphism. ANTs [5, 1] is a widely used toolkit that employs the Euler-Langrange formulation with a symmetric objective function [2] . Yet another approach is to interpret deformable registration as an optical flow problem [70, 103] , leading to the famous Demons algorithm and its diffeomorphic and symmetric variants [106, 93, 92, 95] implemented as part of the Insight Toolkit (ITK) [40, 25] . However, most of these methods are still computationally expensive to run owing to their CPU implementations. Recently, modern implementations leverage the massively parallelizable nature of the registration problem to run on GPUs, leading to orders of magnitude of speedups while retaining the robustness and accuracy of the classical methods [55, 59, 41] . However, as we show in Section 4, the registration performance of classical methods is limited by the fidelity of image intensities. In contrast to most classical methods, earliest Deep Learning for Image Registration (DLIR) methods employed supervised learning for registration tasks [15, 49, 77, 80] where the deformation field is obtained either manually or from a classical method. Voxelmorph [9] was one of the first approaches that introduced unsupervised learning for registration of in-vivo brain MRI images. Subsequent research expanded upon this paradigm, exploring diverse architectural designs [18, 52, 42, 62] , loss functions [109, 108, 44, 24, 60, 107, 75, 16] , and formulations based on incorporating inverseconsistency or symmetric transforms [61, 46, 47, 83, 109] . However, hyperparameter tuning became a challenge for DLIR methods since the methods had to be retrained for every new value of the regularization parameter. This motivated techniques such as conditional hyperparameter injection which addressed hyperparameter tuning [64, 38] , while domain randomization and fine-tuning [37, 88, 72, 28] aimed to addressed generalizability of DLIR methods across domains. Recently, pretrained or foundation models are also proposed to address the generalizability of DLIR methods across different imaging and anatomy [54, 84] . However, these methods perform a monolithic prediction of the warp field from the input images, losing feedback from the intermediate stages of the registration process as done in classical methods. To refine the warp fields, recurrent or cascade-based archi- tectures were proposed [108, 109, 107, 16] . However, cascade-based methods create a substantial memory overhead due to backpropagation through cascades and storage of intermediate volumes [6] . Another promising avenue is to leverage deep implicit priors [87] within optimization frameworks to improve the performance of optimization methods or incorporate implicit constraints of the optimized warp field [101, 99, 44, 39] . We refer the reader to [26, 33] for a comprehensive review of image registration techniques.Despite the plethora of architectural formulations, loss functions, and output representations proposed in Deep Learning for Image Registration methods, we identify that these methods are highly sensitive to the domain gap between the distributions of training and test data, and in the unsupervised case, do not provide any benefit in terms of performance over classical methods. Their primary benefit is their ability to incorporate weak supervision like anatomical landmarks or expert annotations during training, which performs better landmark alignment on unseen image pairs (from the same distribution) without access to landmarks at inference time.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The families of registration algorithms is limited to gradient-based methods to isolate dynamics of gradient-based methods. Our paper only uses brain datasets, since neuroimaging is one of the most (if not the most) popular modalities for studying registration algorithms. This is discussed in the limitations subsection. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 We rehash the image registration problem statement to unify both classical and deep learning methods.D = {(I (n) f , I (n) m ) | n ∈ N, 1 ≤ n ≤ N }, where I (n) f and I (n) mare the fixed and moving images defined over a spatial domain Ω ∈ R d . We drop the superscript n for simplicity. Also consider segmentation maps S f and Sm for the fixed and moving images, respectively, defined over Ω. Given a family of transformations T (Ω), the goal of image registration is to estimate transformations φ θ (f, m) ∈ T (Ω) parameterized by θ that minimize the following objective:EQUATIONwhere L is a dissimilarity function such as mean squared error, or negative local cross correlation, and R is a regularization term that encourages desirable properties of the transformation, such as smoothness or elasticity. We call Eq. ( 1) the image matching objective, since the transformations only need to align the intensity images. We can also call this the unsupervised objective, since it does not require any labeled data. If a suitably chosen label alignment loss D is added as well, the optimization problem becomes:EQUATIONWe call Eq. ( 2) the label matching objective, or a weakly-supervised objective. The image matching objective can subsume both DLIR and classical methods by choosingφ θ (f, m) = f θ (I f , Im), for deep networks, φ (f,m) ,for classical methods.(EQUATIONwhere f θ is a deep network parameterized by θ and φ (f,m) are optimizable free parameters that are indexed by the 2-tuple (f, m), i.e. θ = f,m {φ (f,m) }. In this paper, we consider methods that solve Eq. ( 1) using gradient-based methods. The gradient of Eq. ( 1) with respect to θ is given by (we remove the R term for simplicity):EQUATIONThe first term ∂L ∂φ θ (f,m) is the training signal from the dissimilarity function which does not depend on the parameters θ for a given value of φ θ (f, m) and choice of L. The second term ∂φ θ (f,m) ∂θ is the Jacobian of the transformation with respect to the parameters, which is a projection of the gradient from the space of warp fields to the space of arbitrary parameters. For classical methods, the Jacobian is the identity matrix, for deep networks it is determined by the functional relationship of the output with respect to network parameters. Therefore, the difference in training dynamics and overall performance gap between classical and deep learning methods is likely to be attributed to the choice of ∂φ θ (f,m) ∂θ .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Code is provided in the supplemental material. Data is publicly available and instructions to reproduce the results are provided in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
unsupervised dlir does not improve label matching performance: 
 A speculated claim of deep learning methods is that they can provide better label matching performance by simply training a network to minimize Eq. ( 1) in an unsupervised setting. Such improvements are claimed to come from architectural designs, which correspond to choice of Jacobian ∂φ θ (f,m) ∂θ . A variety of architectures and parameterizations [17, 63, 64, 62, 34, 82, 101] have been proposed to this effect. However, we show that this is not the case.Image matching objectives ensure that intensities from the moving image are displaced to locations in the fixed image where they are most similar, without regard for alignment for any higher order structures. Intuitively, this will ensure label matching only to the extent that the intensity is predictive of the label. If an intensity value strongly corresponds to a particular label, then image matching will lead to label matching. Similarly, if a given intensity value corresponds to multiple possible labels, then image matching does not tell us which labels are matched via the image matching objective. More formally, considering the per-pixel intensity i and labels s as random variables, one can compute the mutual information between the intensity and label maps, denoted as M I(i; s) to determine the predictability of one from the other. We now show that the label matching performance of classical methods is highly correlated with M I(i; s). We consider a widely used classical method, ANTs [2, 5] , to eliminate the effect of any Jacobian term. We consider four brain datasets -OASIS, LPBA40, MGH10, and IBSR18, which are acquired under different scanners, under different resolutions, and have different preprocessing, labelling and postprocessing protocols [57, 48] . For each dataset, we use ANTs for registering all pairs within the dataset and then evaluate the Dice score as an indicator of label matching performance. For each image I and its corresponding label map S, we compute the probability maps p(i), p(s), p(i, s) using histogram binning, followed by the mutual information M I(i; s) = H(s) -H(s|i). A Pearson's correlation coefficient between the Dice scores and the mutual information of the image and label (Fig. 1 ) reveals a strong linear (r = 0.886) and logarithmic (r = 0.933) relationship between the two quantities, shown by the gray and black lines respectively. Image matching improves label matching performance only to the extent of the information about the label obtained from the image (i.e. M I(i; s)). At a first glance, the Jacobian term ∂φ θ (f,m) ∂θ seemingly does not have a role in improving this mutual information further.Empirical Validation. We verify this claim empirically on the OASIS dataset, by minimizing Eq. ( 1) in both DLIR and classical methods. We split the OASIS dataset into a training set of 364 images and a validation set of 50 images. We choose 50 instead of 20 images as in the original split [35] to compute statistical significance. Dice score over 35 subcortical structures is used as the label matching metric. We choose SynthMorph [37] , LapIRN [63] , SymNet [61] , LKU-Net [42] and TransMorph [19] as state-of-the-art DLIR baselines and ANTs [5] , NiftyReg [59] , Symmetric Log Demons [94] , Greedy [106] , FireANTs [41] as state-of-the-art classical baselines. For all DLIR methods, we use pretrained models if they are trained with Eq. ( 1), or train them with the architecture and hyperparameters provided in their original source code. The only exception is SynthMorph, which is trained on synthetically generated data and Dice loss of its corresponding synthetic labels (shapes-sm model). To compare SynthMorph's domain generalization capabilities with only the image matching objective, we add another model, dubbed 'shapes-sm-ncc' that is trained on synthetically generated data as in the original pretrained model, but with the normalized crosscorrelation of the aligned synthetic images. For all classical methods, we follow their recommended hyperparameters and run till convergence. All experiments are run on a cluster with 2 AMD EPYC 7713 CPUs and 8 NVIDIA A6000 GPUs.Results. For all methods, we compute the Dice score of all 35 subcortical regions on images in the validation set (denoted as val), and all images (denoted as trainval). These Dice scores are sorted by median validation performance in Fig. 2 (top). Moreover, we perform a two-sided t-test for each (classical, DLIR) pair, both on the trainval and validation sets, shown in Fig. 2 (bottom). Fig. 2 shows the following conclusions: (a) the top performing classical method (Greedy) and the top performing DLIR method (TransMorph) achieve similar label matching performance on the val and the trainval set, i.e. the differences are not statistically significant (p = 0.161), (b) classical methods almost always perform better than DLIR methods, even on the training set showing that the Jacobian term does not improve label matching more than the mutual information between the image and label, and (c) for unsupervised DLIR methods, there is no improvement label matching performance in the training set compared to val set. The only role of the Jacobian term is to perform amortized learning, but without supervised objectives, this does not guarantee any additional boost in label matching.The effect of instrumentation bias. The astute reader may observe that this result is in contrast to results shown in prior literature [61, 63, 101, 19, 10] . We note that this is due to instrumentation bias [86] , where the baselines' performance may be misrepresented due to changes in hyperparameters, early stopping, or different preprocessing protocols. For instance, [10] mention that the default parameters of ANTs are not optimal, and choose a very different set of parameters (a Gaussian smoothing of 9 pixels, followed by an extremely small 0.4 pixels at the next scale). By stark contrast, we found the recommended parameters to work extremely well for all datasets considered in this paper. We speculate that these changes are done to tradeoff accuracy for speed, since classical methods converge slowly. However, this leads to misrepresentation of the performance of classical baselines. We found much better results (Fig. 2 ) for classical baselines simply by using their recommended scripts. We compare the discrepancy in performance between the baselines reported in the literature and the ones we obtained in Fig. 3 . We follow the guidelines in [86] to evaluate all methods. To ensure our work does not introduce its own instrumentation bias for DLIR baselines, we compare the performance of our trained/pretrained models to the ones reported in the literature (Fig. 3 ). We make all evaluation scripts and trained models public2 to encourage fairness and transparency in evaluations. Figure 3 : Instrumentation bias in evaluation of image registration algorithms. We highlight a significant difference in evaluation metrics reported by baselines and our evaluation on the OASIS validation dataset. This difference can be attributed to deviation in hyperparameters from the recommended parameters or early stopping to save time. In either case, this misrepresentation leads to incorrect conclusions about the performance of the algorithm. The reported dice scores are anywhere from 2 to 10 Dice points lower than our evaluation, showing a non-trivial instrumentation bias. We report our own evaluation of DLIR algorithms and compare them with reported values to avoid introducing instrumentation bias in our evaluation. 
supervised dlir methods demonstrate enhanced label matching: 
 When label matching is introduced as an objective in Eq. ( 2), DLIR methods show superior performance than classical methods. Unlike the previous discussion, where only a pixelwise definition of M I(i; s) was used to quantify the coaction of image intensities and label maps, we consider the entire image I and label volume S as high-dimensional random variables. Label maps are now a deterministic function of the image, i.e. S = f (I), where f is the labelling protocol. In addition to image intensity, label maps are a function of morphological features, location, contrast, and the labelling protocol itself. When trained with the label maps as extra supervision, the network can infer these deterministic relationships to output a warp field that maximize both image similarity and label overlap. Classical intensity-based methods, on the other hand, do not have any mechanism to encode this additional relationship. Aligning intensities or intensity patches discards any functional relationship between high-level image features and labels. To show this, we repeat the same experiment setup as in Section 4 on the same splits, but with the label matching objective added as well.Results. Fig. 4 (top) shows the Dice scores for supervised classical and DLIR methods trained on the OASIS dataset, sorted by median validation performance. In this case, state-of-the-art DLIR methods outperform classical methods by a large margin, with notably higher Dice score on the trainval set than the val set, due to overfitting to the label matching for the training set. This is unlike unsupervised DLIR, where there was no improvement in label matching performance on the training set, emphasizing the fact that performing amortized training does not improve label matching performance by itself. These differences are statistically significant, with the exception of SymNet, which diverged under many training settings with the Dice loss, and only works marginally better than its unsupervised counterpart. SynthMorph is not trained on real data, and is added only as a reference for domain-agnostic performance. This is an unsurprising result -the label matching objective provides additional training signal to the registration task, which is a highly ill-posed problem. Classical methods cannot incorporate this additional signal from a training dataset, and learning-based methods exploit this to achieve better registration on unseen data. Classical methods are, however, agnostic to modalties, intensity distributions, voxel resolutions, and anisotropy. The same registration algorithm (with possibly modified parameters) is applied to datasets with different characteristics, and they still retain their state-of-the-art performance. A related question arises for DLIR methods trained with label matching -does label matching performance transfer to other datasets? denotes a cell where the classical method is significantly better than the DLIR method (p < 0.01), a denotes the opposite, denotes no significant difference. State-of-the-art DLIR methods show significantly better performance than classical methods when label supervision is added. 
dlir methods do not generalize across datasets: 
 A key strength of classical optimization registration algorithms is their agnostic nature to the image modality, physical resolution, voxel sizes, and preprocessing protocols. Most DLIR methods, on the contrary, have been evaluated extensively on the same distribution of validation datasets as the training data, it is unclear if the performance improvements transfer to other datasets of the same anatomy. To this end, we evaluate the performance of both the classical and DLIR methods on four brain datasets -CUMC12, LPBA40, MGH10, and IBSR18. These datasets represent community-standard brain mapping challenge data [48] for a comprehensive evaluation of 14 nonlinear classical registration methods, across various acquisition, preprocessing and labelling protocols. For all datasets, we follow the preprocessing steps followed by [48] .Each dataset contains a different set of labeled regions acquired manually using different labeling protocols. For each dataset, all previously considered registration algorithms are run on all image pairs, and the mean Dice score over all labeled regions is computed. The methods are then sorted by median validation performance in Fig. 5 . For DLIR methods, we plot the performance with models trained with and without the label matching loss in the OASIS dataset, shown as blue and green boxplots respectively. Across all datasets, FireANTs, Greedy, ANTs and NiftyReg consistently perform better than DLIR methods. Among the DLIR methods, SynthMorph performs consistently better due to its domain-agnostic training paradigm. Remarkably, even though DLIR methods outperform classical methods on the OASIS dataset with label matching objective, the performance does not transfer to other datasets, even compared to its own unsupervised variant. This is a negative result -implying that to improve performance on a new dataset, one must collect label maps from that dataset and retrain the model -existing collections of label maps are not sufficient to improve performance on new datasets. Unlike in tasks like segmentation, deep methods do not transfer their performance to out-of-distribution datasets, even with the same resolution, and the expected performance hierarchy does not hold• Expected: Supervised DLIR ID > Supervised DLIR OOD > Classical • Observed: Supervised DLIR ID > Classical > Supervised DLIR OOD Practitioners should therefore be cautious when using prediction-based DLIR methods, especially when the training data is not representative of the test data, regardless of the presence of label maps. 
discussion: 
 This study aims to provide a systematic and unbiased investigation of the performance of classical and DLIR methods under access to label supervision, and their generalization capabilities under small domain shifts. Preceding experiments show that classical methods provide an unprecedented level of robustness and generalizability across datasets, but are limited by the fidelity of the image matching objective. Supervised DLIR methods provide a promising step towards improving registration performance of anatomical regions by implicitly discovering these structures and predicting appropriate warp fields within the network architecture. However, this anatomical-awareness on the training dataset does not help in generalizing to other datasets, limiting the practical utility of these methods. The usability of anatomical landmarks and labelmaps to obtain domain-invariant registration performance still remains an open research problem. These results also have profound implications for annotated data collection and challenges the notion that large labeled datasets ensure robust generalization.Although our study is performed on inter-subject registration with in-vivo neuroimaging datasets, none of our analysis, baselines, and evaluation make any domain or subject-specific modeling assumptions, and the datasets being community-standard benchmarks, the results are valuable and general, both within the neuroimaging and the biomedical communities at large. At the current state, a practitioner should choose predictive DLIR methods only if they have access to a large labeled dataset, and their application is limited to the same dataset distribution. In all other cases, classical optimization-based methods are the more accurate and reliable choice, even if labeled data exists but is not representative of the test data.Our work performs a comprehensive evaluation of state-of-the-art registration algorithms on a variety of neuroimaging datasets. However, our work does not consider hybrid methods, or representations that use optimal matching criteria based on correlation volumes or sparse correspondence features. Although our work considers large-scale community-standard neuroimaging datasets, the performance of these algorithms may differ on other anatomy or modalities. Our study also considers inter-subject registration only, although no method or evaluation incorporates any subject-specific assumptions. The effects on multimodal registration are not considered in this work. However, our work serves as a foundational step toward a more nuanced discussion on the longstanding technical challenges in image registration, and representations that are effective in mitigating these problems. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Appropriate citations are provided for existing code and data. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: Instructions to reproduce the results are provided in the paper and supplemental material. Only instructions to run the existing baselines are provided, no new method is proposed. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://github.com/rohitrango/Magic-or-Mirage/ 
 
 Appendix:  


Decision: This work investigates the classical optimisation and learning-based methods in deformable image registration, with the aim to gain more understanding and insights into the performance, generalisation, and the working conditions of the two typical approaches to this task. Reviewers recognise that this work has merits of original thinking, well motivated experimental design, results that challenge existing understanding, considering out-of-distribution test case, and rich experiments making people rethink. At the same time, the reviewers raise the following issues, including 1) some claims that seem trivial; 2) the need to consider more registration settings; 3) the inconsistency of performance reported; 4) missing baselines and overclaimed hypothesis; 5) the insufficiency of the training set; 6) more experiments on organs, tissues, and modalities; 7) missing new solutions or deeper insights; and 8) not pointing out the potential future direction. The authors have made efforts to provide response to each of the raised issues. Among the four reviewers, after reading the rebuttal, one reviewer is still positive (with rating 6); one reviewer thinks this work is still on the borderline (with rating 4); one reviewer is not fully convinced and slightly raises the rating to 4; and the last reviewer has no further questions and maintains the original rating of 5.

By reading the reviews, rebuttal and the submission, AC thinks that the rebuttal is overall clear, sound and effective. Furthermore, as indicated by two reviewers, this work reflects original thinking and makes people rethink the development of learning-based registration methods. Although some claims do not seem to be surprising enough, validating them via systematic investigation has its value and significance to the relevant research community. This kind of research shall be encouraged in order to better understand the many existing methods. Considering all the factors, AC recommends this submission for acceptance. Meanwhile, the authors shall adequately incorporate the key points in the rebuttal to further strengthen this work, including the presentation.

Reviews: review: paper_summary: paper_summary: The manuscript investigates the characteristics of two types of registration approach based on traditional variational optimization and deep learning. Experiments revealed a correlation between the mutual information of the distribution of per-pixel intensity and labels, and the performance of classical registration methods. Then the manuscript argues unsupervised deep learning does not improve label matching performance compared to traditional methods, whereas supervised learning methods show improved label matching. Lastly they show learning methods do not generalize well.
strength_weakness: strength_weakness: 1. Two of the three claims discussed herein seem trivial. It is not surprised that "Supervised DLIR methods demonstrate enhanced label matching" and "DLIR methods do not generalize across datasets".
2. I feel that the abstract and intro set up a high expectation by saying "we propose a general recipe to choose the best paradigm for a given registration problem, based on these observations." but then again it is a one-sentence recipe in the end that is not surprising to readers "a practitioner should choose DLIR methods only if they have access to a large labeled dataset, and their application is limited to the same dataset distribution. In all other cases, classical optimization-based methods are the more accurate and reliable choice"
questions: questions: In Fig. 1, shouldn't there also be a correlation within each individual dataset?
limitations: limitations: While the intro and abstract suggest a study on general registration problems, all experiments are based on brain MRI. It is not clear whether the problem/hypothesis/conclusion is specific to brain MRI.
paper_summary: paper_summary: This paper discussed about an explicit correspondence between mutual information of the distribution and performance of classification registration methods. The authors argued that this correlation will not be affected by the learning-based methods. They validated this hypothesis on both classical and learning-based registration models and found that the weakly supervised learning-based methods performed high-fidelity intensity and label registration. Besides, they showed that these high-fidelity feature learning methods did not translate to invariance to domain shift, ending with proposing a general approach to select the best paradigm for a registration task.
strength_weakness: strength_weakness: **W1. Experimental findings.** 

1. I appreciate the authors' finding on getting improved performance for classical methods over learning-based methods. However, I would argue that this is typically not always true considering the reported findings in the learning-based papers, e.g., VoxelMorph (VM), TransMorph (TM), etc. This registration results highly depending on the image pairs that are being considered. For example, image pairs with large age variations won't perform well in the learning-based registration tasks (please check VM, Deepflash, and related papers where they restricted the subject ages to deal with large deformations). Besides, inter-patient registration has a higher probability of getting similar results compared to the atlas-based (or pre-selected template) registration, thoroughly discussed in VM and TM papers. I found this important information missing from their experimental evaluation to support their first hypothesis (Sec. 4). I would encourage the authors to consider experimenting and reporting performances for within-patient, inter-patient, and atlas-to-patient registration results. Also, comparing the anatomical dice scores further justifies the registration models as in the existing SOTA it has been discussed about getting larger variations for critical anatomical structures such as ventricles, pallidum, cerebellum, etc.  

2. I found some of the mode's performance very inconsistent with the reported version in the original papers. For example - LKU-Net is achieving more than $~0.925\pm0.025$ dice score where the authors' reported best within subject accuracy is $0.8861\pm0.01$ with an average dice score of $0.7758\pm0.0390$. The same goes for the LapIRN model's performance. There's a large difference between the reported dice in this paper vs the original paper's dice. I understand there might be different biases that might be there in terms of implementation. However, the current reported performance of these models raises questions about the credibility of the adapted implementations, preparation of the pairwise images, hyperparameter setup, etc. I would suggest the authors kindly shed some light on this part.

3. I found the supporting experiments to validate the hypothesis presented in Sec. 6 missing some important information. Did the authors perform an affine transformation on all the datasets and what are the pre-processing steps? Are they considering all 3D volumes for their experiments?


**W2. Missing large deformation diffeomorphic and related baselines.** I appreciate the authors for trying to carry out thorough experiments on different classical and learning-based registrations. However, I believe the current hypothesis would be more understandable and justifiable if the authors could initiate some experiments considering large deformation diffeomorphic-based registration methods such as LDDMM [1,2,3], where this kind of method is structured upon time-varying velocity fields, that is proven to be more robust in various deformation-based image analysis tasks.


**W3. Overclaimed hypothesis.** Statements in L332-L335 seem overstated. The authors need to discuss and experiment on within-subject/class, inter-subject/class, and atlas/template-based registration to validate the stated hypothesis in that line, which seems to be missing in the current version. Besides, the selection of the DLIR/classical registration method depends on the image analysis tasks that are being performed over image registration. Without verifying the studied registration on different image analysis tasks, it is inappropriate to come to a conclusion stated in L332.


**W4. (Minor) Technical writing.** The authors tried to aggregate all their potential findings in a structured way. But reading the whole paper kind of messed me up in understanding what are the actual contribution of this paper compared to the other survey/review papers in this domain other than performing experiments on OOD data. Overall, the presentation is kind of above the borderline but I guess if the authors tried to focus on their storyline and make their findings more clearer that would be great for the readers. For example, I found implementation details in most sections which is kind of redundant.  



Overall, I appreciate the authors for working on this paper which is very relevant as well as important in the medical imaging domain, specifically in medical image registration. However, the current version of the manuscript lacks some important experimental justification and further experiments. With that being said, the current version of the manuscript is under the threshold of acceptance. However, I am open to reconsidering the initial rating if the above concerns are adequately justified.


References
----------------
[1] Yang, Xiao, Roland Kwitt, and Marc Niethammer. "Fast predictive image registration." Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings 1. Springer International Publishing, 2016.

[2] Shen, Zhengyang, François-Xavier Vialard, and Marc Niethammer. "Region-specific diffeomorphic metric mapping." Advances in Neural Information Processing Systems 32 (2019).

[3] Niethammer, Marc, Roland Kwitt, and Francois-Xavier Vialard. "Metric learning for image registration." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[4] Wang, Jian, and Miaomiao Zhang. "Deepflash: An efficient network for learning-based medical image registration." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
questions: questions: Please see the weaknesses section. I tried to summarize all the findings, concerns, and questions there.
limitations: limitations: Limitations have been discussed in the paper.
paper_summary: paper_summary: The work benchmarks the traditional methods and deep learning-based methods for medical image registration and gives a general recipe to choose registration methods.
strength_weakness: strength_weakness: - The training set of deep learning models may not be sufficient because the number of image pairs is obviously not enough, which potentially makes the comparison with classical models unfair. 
- Medical image registration contains a lot of content, it cannot and should not be simply represented by monomodal registration on four brain image datasets. The authors need to implement more experiments on images of more organs and tissues with monomodal and multimodal registration tasks.
- The subtitle “DLIR methods do not generalize across datasets” is almost common sense that networks trained on specific datasets cannot easily generalize to other datasets.
questions: questions: - Please explicitly clarify the advantages compared to the well-known lean2reg challenge.
- Does the conclusion still hold on large datasets? AMOS dataset provides lots of CT and MR images can be used for model training. 
- Why not submit this work to the dataset and benchmark track?
limitations: limitations: yes
paper_summary: paper_summary: The paper delves into the comparison between classical optimization and learning-based medical image registration methods. Some valuable insights are proposed and the authors propose a general recipe to choose the best paradigm for a given registration problem.
strength_weakness: strength_weakness: 1. New solutions or deeper insights are missed.
2. Findings in Section 4,5,6 are somewhat valuable, but the relevancy between different phenomena is not very clear. The author failed to point out what should we do or what we may further research for unsupervised or supervised DLIR methods.
questions: questions: What are the potential directions of future work based on this paper?
limitations: limitations: N/A

Comments: title: Official Comment by Reviewer 23gn
Revising rating
comment: Thanks for the response. The claimed new advantages compared to learn2reg are not convincing to me because these conclusions can also be derived from participants' algorithms. Accordingly, I slightly raise my score.
I thank the reviewers for their response. After reading the rebuttal, I have the following standings — 

- Some of my concerns regarding LDDMM and the performances of different existing models have been adequately addressed. 
- The paper's contribution is limited considering the hypothesis that the authors evaluated in this paper. 
- I agree with my other reviewer that this paper is more aligned with the Dataset and Benchmarking track. Besides, I agree that experiments on datasets other than Brain could reinforce the contribution of their standings. More experiments related to intra-subject, atlas-based registration could further rectify their claims. 

*After reading all the reviewers' comments and the rebuttal, I think the paper is still on the Borderline (keeping my score as it is), considering the technical contribution, carried out experimentations, overall writing, and the submission track.*

I suggest that the authors to address the findings from all reviewers in their revised version.
Thanks for clarification. I'm still positive about the paper after reading all responses and therefore remain my rating.
We thank you for reading the rebuttal. We would request you to increase your score if you think our responses are satisfactory. If not, we would love an opportunity to discuss your concerns.
Thank you for addressing my concerns. I have no further questions and currently maintain my original rating.