SINR: Spline-enhanced implicit neural representation for multi-modal registration

Authors: vasiliki  sideri-lampretsa
julian  mcginnis
huaqi  qiu
walter  simson
daniel  rueckert
m  paschali
Venue: MIDL_io_2024_Conference
Published: 2024-01-06 15:06:33

Abstract: Deformable image registration has undergone a transformative shift with the advent of deep learning. While convolutional neural networks (CNNs) allow for accelerated registration, they exhibit reduced accuracy compared to iterative pairwise optimization methods and require extensive training cohorts. Based on the advances in representing signals with neural networks, implicit neural representations (INRs) have emerged in the registration community to model dense displacement fields continuously. Using a pairwise registration setup, INRs mitigate the bias learned over a cohort of patients while leveraging advanced methodology and gradient-based optimization. However, the coordinate sampling scheme makes dense transformation parametrization with an INR prone to generating physiologically implausible configurations resulting in spatial folding. In this paper, we introduce SINR - a method to parameterize the continuous deformable transformation represented by an INR using Free Form Deformations (FFD). SINR allows for multi-modal deformable registration while mitigating folding issues found in current INR-based registration methods. SINR outperforms existing state-of-the-art methods on both 3D mono- and multi-modal brain registration on the CamCAN dataset, demonstrating its capabilities for pairwise mono- and multi-modal image registration.

Main paper: 
 introduction: 
 Image registration involves aligning corresponding semantic regions in two or more images acquired with di↵erent imaging modalities or at separate points in time (Sotiras et al., 2013) . In medical imaging, registration is vital for the quantitative interpretation of multiple images of the same patient, e.g., multi-modal image fusion, motion correction, and disease progression tracking. Conventional registration methods rely on pairwise instance optimization to learn gridded displacement fields, where a dissimilarity measure over a space of transformations is iteratively minimized (Ashburner, 2007; B. et al., 2005; Rueckert et al., 1999) . Data-driven registration-learning methods (Haskins et al., 2020) , commonly implemented with CNNs, learn image correspondence over a dataset of image pairs and predict dense displacement fields (Balakrishnan et al., 2019) , di↵eomorphisms (Dalca et al., 2018; Mok and Chung, 2020) or parameters of the transformation model (Qiu et al., 2021) . Due to training bias, these approaches are often resolution-dependent and can fail to generalize to other modalities. This registration-learning paradigm can o↵er fast inference time at the cost of accuracy (Hansen and Heinrich, 2021) .Recently, coordinate-based implicit neural representations (INRs) have been proposed to encode signals such as images or transformations as a function stored in the weights of a multi-layer perceptron (MLP) (Sitzmann et al., 2020; Tancik et al., 2020; Mildenhall et al., 2021) . These approaches allow for a continuous representation of the underlying signal with potentially lower storage requirements than gridded representations (Dupont et al., 2021) . Activation functions, such as ReLUs, sinusoids (SIRENs), and Gaussian activations, have been proposed (Rahaman et al., 2019; Sitzmann et al., 2020) , with benefits to fidelity and training speed. INRs have also been used for registration; (Wolterink et al., 2022) proposed representing a dense deformable transformation between lung Computer Tomography (CT) images using INRs, while (Han et al., 2023) introduces mono-modal di↵eomorphic registration with INRs. Finally, (Byra et al., 2023) examines the e cacy of INRs in improving the registration of mono-modal brain images in MRI. In these works, INRs are fitted only with the normalized coordinates of a single image pair and predict the dense displacement field, minimizing a conventional intensity-based dissimilarity measure. Additionally, unlike CNNs, INRs do not require a large training dataset.INR-based registration methods commonly leverage sinusoidal activations, which can better represent higher frequency signal components (Sitzmann et al., 2020) . However, the expressiveness of SIRENs is explicitly controlled by the sinusoids' frequency term !. Large ! values can lead to spatial folding and require higher explicit regularization to enforce smoothness (Byra et al., 2023) . These explicit regularization terms could negatively influence registration's accuracy.The coordinate sampling might also a↵ect the convergence and registration performance in the context of INR registration. To mitigate this negative e↵ect due to random sampling, (Wolterink et al., 2022) suggested using a mask to ensure that regions of interest are more frequently sampled. This issue of sample prioritization becomes more pronounced in multi-modal registration where information-based metrics such as (normalized) mutual information (NMI) (Studholme et al., 1999; Wells et al., 1996) are employed. NMI is computationally expensive because it uses histograms to approximate the joint intensity distribution and, therefore, requires a large batch of coordinates for a representative distribution of the image content for successful registration. Consequently, the computational resources scale quickly with image size and dimensionality, reducing tractability with scale. Conversely, smaller batch sizes have been shown to lead to higher signal modeling accuracy in INRs and increased training stability (McGinnis et al., 2023) , thus posing a challenge to the NMI-based multi-modal registration.To address the limitations of current implicit registration methods, we propose Splineenhanced INR (SINR), which parameterizes the implicit representation of a deformable transformation using Free Form Deformations (FFD) (Rueckert et al., 1999) . FFD, originally proposed for the flexible manipulation of 3D shapes, deforms a control lattice, allowing the implicit regularization of SINR to produce smoother transformations without compromising registration accuracy. Further, the FFD model reduces the sensitivity of SINR to the choice of frequency (!) in the SIREN activation. SINR only parameterizes spatially sparse FFD control points, reducing the computational burden of coordinate sampling. Unlike previous work (Wolterink et al., 2022) , this allows SINR to use NMI e ciently with INRs not only for mono-modal but also for multi-modal registration. Our contributions are the following:• We propose SINR, a registration method that parameterizes deformable transformation by combining implicit neural representation (INR) with free-form deformation (FFD). The e cient spatial sampling and intrinsic smoothness, benefits of the FFD model, lead to improved optimization and state-of-the-art registration performance; • SINR exploits the FFD control point sparsity to e ciently calculate NMI, which enables multi-modal INR-based registration for the first time; • SINR achieves accurate registration with comparable or improved transformation regularity. We evaluate registration performance on mono-modal and multi-modal brain MRIs and compare it with iterative and learning-based methods. 
pairwise image registration: 
 Given two n-dimensional images, a fixed image F and a moving image M with F, M : ⌦ ⇢ n ! (n = 3 for 3D MRIs), image registration aims to find an optimal spatial transformation : n ! n such that the transformed moving image is most similar to the fixed image. Typically, this is formulated as an optimization problem ⇤ = arg max J (F, M, )where the distance between the images is minimized with constraints on the transformation. We denote the objective function J as:EQUATION)where D is an intensity dissimilarity measure and R is the regularization on the transformation field whose e↵ect is controlled by the parameter .Free-form deformations involve the flexible alteration of images by adjusting control points within a parametric space, allowing non-rigid transformations. B-spline-based FFD models parametrize a deformable transformation between two images by defining a mesh of control points in the spatial domain of the image volume (Rueckert et al., 1999) . Giving a uniform spacing , the FFD can be formulated as:EQUATIONwhere (i, j, k) are the indices of the control point which is closest to the origin in the control point cube that encloses (x, y, z), B are the B-spline basis functions as presented in (3), and (u, v, w) are the normalized local coordinates of (x, y, z) in its enclosing control point cube.B 0 (u) = (1 u) 3 6 , B 1 (u) = 3u 3 6u 2 + 4 6 , B 0 (u) = u 3 6 (3)This transformation parametrization has some advantages due to its B-spline formulation. Firstly, B-splines have local support, which makes them a compelling choice for parameterizing deformable transformation. In other words, each control point a↵ects the transformation only in its local neighborhood. Moreover, the resolution of the control point mesh is proportional to the transformation smoothness. Larger spacing between control points facilitates the representation of more global, smoother, nonrigid deformations, while smaller spacing allows for the modeling of highly localized nonrigid deformations. Therefore, the control point spacing can implicitly act as a method constraint, promoting smoothness.While conventional iterative methods estimate the transformation using pairwise optimization and CNN-based methods learn the transformation over a cohort, we employ a coordinate-based INR with the transformation between a pair of images due to its high signal fidelity and fast training speed. Leveraging the compressed representation, we propose to exploit the inherent smoothness of the B-spline FFD we previously described and use an INR f ✓ with trainable parameters ✓ to approximate the transformation (x cp ) = x cp + u(x cp ) between a given pair of images F, M , where x cp 2 ⌦ are the control point coordinates and (x) = f ✓ (x) are the displacements on the control point coordinates. The L-layer network is modelled as where W l denote the weights, b l the bias, h l the hidden feature vector for the l-th layer, and the network's activation function, where we experiment with both, ReLU and SIREN.EQUATION 
experimental setup: 
 Evaluation metrics. Assessing the registration performance can be challenging since the ground truth deformations are unknown. Therefore, registration accuracy and regularity are evaluated with surrogate measures. Accuracy is determined by assessing the overlap between the anatomical segmentation using the Dice score. The regularity of transformation is evaluated based on the Jacobian determinant. The extent of folding in the image due to the transformation is measured by the percentage of points with J = |r | < 0.Datasets. We evaluate our work on the inter-subject brain registration using the Cam-CAN1 dataset (Shafto et al., 2014) , (Taylor et al., 2017) . The dataset consists of 310 T1w and T2w MR 3D volumetric images of size 192 ⇥ 192 ⇥ 192 and 1mm 3 isotropic spatial resolution, which we split into 80% training, 10% validation and 10% test set. We normalize all images to the MNI space (Horn, 2016) using a ne registration, ensuring an isotropic spatial resolution with a voxel size of 1mm 3 . We perform skull-stripping using ROBEX (Iglesias et al., 2011) and bias-field correction with SimpleITK (Lowekamp et al., 2013) . For assessment purposes, we obtained automated segmentation of 138 cortical and subcortical structures, categorized into 5 groups, using MALPEM (Ledig et al., 2015) .Baselines. We first compare the proposed method to a conventional iterative method Medical Image Registration ToolKit (MIRTK) (Schuh et al., 2014) , which is based on the FFD model. We also compare against two CNN-based deep learning methods; the widely used Voxelmorph (VMorph) (Dalca et al., 2018) , which outputs a dense displacement field, and Modality-Invariant Di↵eomorphic Deep Learning Image Registration (MIDIR) (Qiu et al., 2021) , which predicts FFD as transformation. Furthermore, we compare against an INR-based method named Implicit Neural Representations for Deformable Image Registration (IDIR) (Wolterink et al., 2022) , which outputs a dense displacement field instead of a Implementation. We trained all the mono-modal experiments with Normalized Cross Correlation and the multi-modal ones using di↵erentiable NMI (De Vos et al., 2019) as image similarity measure. All baselines and the proposed method incorporate bending energy for regularization as introduced by (Rueckert et al., 1999) . All the INRs were trained using the ADAM optimizer with a 10 4 learning rate for a maximum of 2500 epochs. We considered sampling the coordinates inside the brain mask only for the baseline INRs and not for SINR. The dense mono-modal experiments used a coordinate batch size of 10k samples, while the multi-modal experiments used a batch size of 890k ( 1 8 of the total points) to ensure convergence with NMI. We tune the selection of hyperparameters, namely the regularization weight and !, by evaluating every 50 steps and performing an early stopping if the folding ratio becomes larger than 0.9%. This threshold is chosen empirically by evaluating the registration performance qualitatively. We refer the reader to Appendix A, Figure 5 , where the resulting transformation demonstrates approximately 0.9% folding. We choose the hyperparameters that achieve the highest Dice score for every method while not surpassing this folding ratio threshold. For the dataset, optimal outcomes are observed when the control points are spaced at 2mm 3 . The code is publicly available2 . 
results and discussion: 
 In our experiments, SINR, using sinusoidal activation functions, achieves the highest registration accuracy in Dice, surpassing the INR-based IDIR by 1.8% in the mono-modal and 4.8% in the multi-modal case. SINR performs sparse spatial sampling with its use of FFD control points. This sparse sampling strategy allows for stable computation of NMI via more e cient spatial sampling over the entire domain compared with random sampling of (a) T1w-T1w (b) T1w-T2w Figure 3 demonstrates the accuracy of the SINR compared to the baselines for individual classes and the overall mean, confirming the finding of Table 1 . SINR with SIREN demonstrates a superior mean Dice score and outperforms all baselines in almost all the individual classes in both registration tasks. MIRTK achieves a marginally higher Dice score for Noncortical GM in the mono-modal case and a comparable score in the multi-modal case for White Matter. However, the proposed method achieves substantially higher Dice for other structures in both mono-and multi-modal cases (c.f. Table 3 ). We further refer the reader to the Appendix A, Figure 6 for an enlarged version of Figure 3 .Regarding folding ratio, the proposed SINR with SIREN and ReLU activations manages to mitigate the folding ratio e↵ect, which IDIR is prone to, as shown in Table 1 . SINR equipped with ReLUs demonstrates the lowest folding ratio among all its competitors, making it a suitable candidate for applications in which a smooth transformation is desired, such as inhale-exhale lung registration. For multi-modal registration, our SIREN-based SINRs' folding ratio was marginally higher compared to other baselines but demonstrated an improved Dice score over them. Qualitatively this can also be confirmed by Figure 2 , where it can be seen that the FFD results in a smoother, more accurate transformation suitable for brain registration in comparison to IDIR. Hyperparameter robustness: We examine the influence of ! on registration accuracy while maintaining the folding ratio below 0.9% using the dense displacement INR method IDIR and the proposed SINR, which uses FFD. The results are presented in Figure 4 as the average and standard deviation of Dice scores over ! ranging from 5 to 70. SINR displays consistently higher Dice scores over all ! values, showing robustness to ! hyperparameter selection. Notably, for the T1w-T2w registration setting, both methods achieve the peak Dice score with an ! value 30, as proposed in (Sitzmann et al., 2020) . In the mono-modal setting, SINR achieves the best results with an ! of 30, while IDIR requires an ! value of 20 for maximal performance. Low values of ! reduce Dice scores for both SINR and IDIR. The performance degradation over ! is more pronounced on IDIR and remains constant with larger !, while SINR displays reduced sensitivity to ! and stabilized performance for larger ! values. 
conclusion: 
 In this work, we propose parameterizing the deformable transformation using an INR with Free Form Deformations. Through this combination, we benefit from the lightweight, fastfitting INR and the inherent smoothness of B-spline FFD parametrization to achieve stateof-the-art performance in mono-modal and multi-modal brain registration. Extensive experimentation demonstrates the versatility of our approach, which not only outperforms conventional approaches, CNN methods, and dense INRs but also mitigates the implausible transformation percentage of the latter. We further perform an ablation study showing that the proposed FFD-enhanced INR is more robust against the activation function's frequency choice. Future work will extend this approach to other data modalities and anatomies, such as abdominal CT scans and further loss functions. Finally, an interesting future direction is to design an architecture that combines the registration with the FFD within the INR. 
 
 Appendix: https://cam-can.mrc-cbu.cam.ac.uk/dataset/https://github.com/vasl12/SINR.git 


Decision: This is a very well-written paper which proposes a new method that is general and will likely have impact on registration tasks. The reviewers are unanimous in their appreciation of the paper and find the results interesting.

Reviews: review: paper_summary: paper_summary: This paper presents a new registration method using implicit neural representation for free-form deformations. The motivation of parameterizing the continuous deformable transformation is to reduce foldings. The effectiveness of this method is demonstrated through evaluations on the CamCAN dataset, where it shows promising outcomes.
main_review: main_review: The rebuttal has addressed the open points and the manuscript has been improved. There are no further points for discussion. I have raised my ranking to Strong accept as I would like to see it presented at the MIDL conference. It is a very good and solid paper, but nothing ground-breaking.
strength_weakness: strength_weakness: - The paper could be improved by giving a few more details about the method to make it easier for the reader to understand it. (see detailed comments).
- The authors do not compare the runtimes of the different algorithms. These are of course highly dependent on the available hardware, but give an indication of the speed. 
- The authors do not evaluate their method on a challenge dataset, which would make it even easier to compare the method with state-of-the-art methods.
- The limitations of the proposed method could have been discussed in more detail.
questions: questions: -	Please clarify open questions to make it easier to understand the method.
-	Please add the runtime of the method.
-	If possible (space-wise) add a discussion on the limitations of the proposed method. This could also be done in the experiments. Are there cases where the method doesn’t work well?
paper_summary: paper_summary: This work proposes a registration method that merges traditional free form deformations with INR registration. Rather than parameterizing a deformation or velocity field directly, the INR is used to parameterize the grid of control points for B-splines. The method is evaluated on the Cam-CAN dataset compares favourably with sensible baselines.
main_review: main_review: My assessment remains the same: the paper presents a strong, well explained registration method and the experiments show the method is beneficial compared to sensible baselines. My main concerns have been addressed appropriately, and I look forward to discussing the work with the authors in more detail.
strength_weakness: strength_weakness: The experiments are performed with the same number of iterations, but the batch size of the presented method is much larger in the mono-modal experiments. Hence, that network saw far more samples than the baseline, meaning for some images it may have converged while the baseline had not yet. Conversely, in the multi-modal experiments, the batch size of the baseline is much higher than what is typically used (i.e. 3.5M instead of 10k), which may have had a detrimental effect on the optimisation process.

The main drawback of the baseline INR method is its slow inference compared to other modern registration methods; the paper makes no mentions of runtime, which I feel would be valuable to better understand the potential applications.

It is somewhat unclear what data the method was tested on. The authors specify there was a train/validation/test split of 80/10/10, but unless I've misunderstood, the method does not require training. Was 80% of the dataset used for hyper-parameter tuning? This seems excessive.
questions: questions: If I understand correctly, in the presented method, the authors fully sample their space of control points each iteration. While this is evidently effective, it makes me question what the INR adds to the original FFD formulation in this setting. INRs are useful for latent interpolation, arbitrarily precise derivatives for regularization and compact representation of deformation fields, but none of these properties is being exploited here. I would expect the results to be very similar to the MIRTK results. While the results are reasonably close, the method does clearly outperform MIRTK. The paper should discuss what the advantage of the proposed method is compared to classical FFD; is this merely better hyper-parameter tuning? Or is there another intuitive reason why SINR can consistently work better?

The main drawback of registration with INRs is the inference time, which is generally slower than both learning-based methods and GPU implementations of traditional methods. How does the presented method compare to the INRs by Wolterink et al. in terms of runtime? How do the performance over time graphs look compared to the baseline INRs? It would be nice to see this, possibly in an appendix.

I'm worried the 3.5M batch size for the multi-modal experiments may have had a detrimental effect on the optimisation process, which may have a larger effect than mild instability of the mutual information computation. I would expect 100k would be more than enough to get a representative distribution. Did you try this with multiple batch sizes to confirm 3.5M is the best option?

Please clarify what is meant by the train/validation/test split of 80/10/10. There is no training in the "classical" sense, so which parts of the dataset were used for what, exactly?
paper_summary: paper_summary: The authors introduce a novel multi-modal non-linear volumetric registration framework. It is a spline-enhanced INR (SINR) solution that implements the classic FFD registration framework. This allows for both a uni-modal and multi-modal problem setup (via the use of NMI implementation for the latter). 

The submission is well-written and well-motivated. Use of INR for non-rigid registration in both mono-modal and multimodal registration. The FFD model helps with reducing folding issues compared to INR (but not with other CNN and traditional methods). 

The authors committed to sharing their code on GitHub.
main_review: main_review: The revised additions have answered my (and the other reviewers') questions and the submission improved. Details about the experimental setup and the folding ratio optimization have been provided. I increased my score to strong accept even with the limited experimental data presented.
strength_weakness: strength_weakness: It would have been great if more details are provided on the segmentation labels that were used for the performance evaluation.
It would have been great if the authors were more clear / explained it in more detail, whether their higher folding percentage compared to the other solutions is of a concern.
questions: questions: Can you provide further details on the segmentation labels that were used for the performance evaluation?
Can you explain in more detail whether the higher folding percentage of SINR compared to the other solutions is of a concern?
paper_summary: paper_summary: This work parameterizes the deformable transformation using an INR with Free Form Deformations, benefiting from the lightweight, fast fitting from INR and the inherent smoothness from B-spline FFD parametrization.

The proposed method achieves state-of-art performance in mono-modal and multi-modal brain registration.
main_review: main_review: The rebuttal has addressed concerns about novelty and unclear motivation and the manuscript has been improved. 
I have raised my ranking to Weak Accept.  It is a solid paper, but nothing ground-breaking.
strength_weakness: strength_weakness: - Simply combining existing Free Form Deformations formulation with implicit neural representation is novelty-limited. As mentioned in related work, both are well explored in registration.
- MIND-loss and diffeomorphisms can solve the limitations of current implicit registration methods, such as inefficiency and smoothness. The authors should give more clear motivation for SIRN registration.
- In Table 1, the running time should be demonstrated. Additionally, IDIR [ReLU-MLP] shows higher accuracy than the proposed SINR [ReLU-MLP], indicating the inefficiency of the FFD-enhanced registration. The activation function choices seem to have a more important influence than the FFD-enhanced strategy. Please explain the reasons.
questions: questions: Please address the concerns about novelty, motivation, and experiment verification.

- Simply combining existing Free Form Deformations formulation with implicit neural representation is novelty-limited. As mentioned in related work, both are well explored in registration.
- MIND-loss and diffeomorphisms can solve the limitations of current implicit registration methods, such as inefficiency and smoothness. The authors should give more clear motivation for SIRN registration.
- In Table 1, the running time should be demonstrated. Additionally, IDIR [ReLU-MLP] shows higher accuracy than the proposed SINR [ReLU-MLP], indicating the inefficiency of the FFD-enhanced registration. The activation function choices seem to have a more important influence than the FFD-enhanced strategy. Please explain the reasons.

Comments: title: the effect of large batch sizes
Change my score from 3: Borderline to 4: Weak accept
comment: Thank you for the authors' clarifications. The revised additions have answered my questions and the submission improved.
>we have not detected any detrimental effects [of the large batch size] on the optimization process

While I do not think it is a big enough concern to downgrade my rating of this manuscript, the above point should be carefully investigated in follow-up work. Running the IDIR code on the DIRLAB set with 100k instead of 10k points for 2500 iterations, I do observe some performance degradation (mean error ~1.1mm @10k, ~1.3mm @100k batch size), despite the many more "examples" seen by the network. Hence, while it is possible that this had no impact in your dataset and experimental setting, I would not be surprised if this has a relevant effect after all. 

It _is_ possible I'm just seeing random variations. I only ran this as a quick experiment with a single random seed to check if my intuition made sense. I would say it warrants further investigation, though.
I would like to thank the authors for the clarification. The rebuttal has addressed the open points and the manuscript has been improved. There are no further points for discussion.
Concerns about novelty and unclear motivation have been addressed.
We would like to thank Reviewer 8syk for providing insightful feedback and interesting questions. We address all concerns in the subsequent rebuttal and have revised the manuscript accordingly. 

>The experiments are performed with the same number of iterations, but the batch size of the presented method is much larger in the mono-modal experiments. [...] 

>I'm worried the 3.5M batch size for the multi-modal experiments may have had a detrimental effect on the optimisation process, which may have a larger effect than mild instability of the mutual information computation. I would expect 100k would be more than enough to get a representative distribution. Did you try this with multiple batch sizes to confirm 3.5M is the best option?

We appreciate the reviewer's comment. Firstly, we want to clarify that we do not utilize the same number of iterations for all experiments and we revise the manuscript accordingly. Instead, for each image pair the number of iterations is determined by the stopping criteria. This approach ensures that the baseline has been exposed to no fewer points than the SINR.
Employing a larger number of points in the multi-modal case may result in a slight decrease in optimization speed; however, we have not detected any detrimental effects on the optimization process. This number of points ensures accurate and robust results. Nevertheless, we acknowledge that conducting further experiments on the batch size would be advantageous to investigate its impact on the optimization process.

>The main drawback of the baseline INR method is its slow inference compared to other modern registration methods; the paper makes no mentions of runtime, which I feel would be valuable to better understand the potential applications.

>[...] How does the presented method compare to the INRs by Wolterink et al. in terms of runtime? How do the performance over time graphs look compared to the baseline INRs? It would be nice to see this, possibly in an appendix.

Thank you for this suggestion, please refer to Table 2 in the updated manuscript.

>Please clarify what is meant by the train/validation/test split of 80/10/10. There is no training in the "classical" sense, so which parts of the dataset were used for what, exactly?

Deep learning CNN networks for registration, are trained on a cohort basis, whereas conventional methods like MIRTK and INR-based approaches are fitted on a pair of subjects. We opted to split the dataset into 80% for training, 10% for validation, and 10% for testing. Subsequently, we trained the CNNs on the training set and fine-tuned their hyperparameters using the validation set. The test set evaluated the networks' ability for accurate and robust registration. Unlike CNN networks, pairwise methods do not require a training/validation/test split, and their hyperparameters can be tuned pairwise. Nevertheless, for comparative purposes against CNN methods, we calibrated the hyperparameters on the validation set and evaluated registration performance using the same test set as mentioned earlier. 

>I may have missed it, but I do not think the paper mentions how batches are sampled in the presented method. Looking at the code, the answer seems to be "use everything all the time"; this is a significant difference from the baseline INR method, so it should be mentioned.

>[...] Were coordinates sampled from a foreground mask, or were they sampled uniformly in the images?

We would like to thank the reviewer for bringing this omission to our attention and we revise the manuscript accordingly. For the baseline INR, points are randomly sampled within a brain mask, which we empirically observe enhances the performance and convergence speed. In the case of SINR, we instead sample only a subset of points at intervals of 2mm^3 spacing on a grid (⅛ of the total points). During inference, all points are considered.

>[...] the authors fully sample their space of control points each iteration. While this is evidently effective, it makes me question what the INR adds to the original FFD formulation in this setting. [...] While the results are reasonably close, the method does clearly outperform MIRTK. The paper should discuss what the advantage of the proposed method is compared to classical FFD; [...] 

The proposed SINR method samples 1/8 of all points as control points at each iteration. This approach allows it to retain the advantages of INR, as the optimization occurs exclusively within the INR framework. The FFD is incorporated post-INR fitting, leveraging its demonstrated capability to alleviate significant spatial folding issues, to which baseline INRs are particularly susceptible. In essence, the SINR method combines the best of both worlds: utilizing a neural network for the fitting process and FFD to ensure smooth and realistic transformations. Additionally, SINR outperforms the conventional method MIRTK in terms of DICE score, but at the same time, it also demonstrates a higher folding ratio.
We would like to thank Reviewer n1vA for providing insightful feedback and interesting questions. We address all concerns in the subsequent rebuttal and have revised the manuscript accordingly. 

> How generalizable is the performance of this method on data sets other than the one tested? VMorph was, as far as I know, tested on many different data sets.
> How generalizable is the tool to non-CamCAN data sets? Eval was done on N = 31, if I understand.

The proposed approach was solely assessed using T1w-T1w brain registration and T1w-T2w brain registration within the CamCAN dataset. Nonetheless, given that our methodology isn't specifically tailored to these modalities within this dataset, we are confident in its potential to generalize across diverse datasets, which we will assess, together with other ideas from the reviewing process, in future work. 

> Some details are missing regarding Dice computation (Was an average or a global Dice measure computed? -- Mean over all structures? Sum of Dices?). What is the Dice computed over -- what are the "anatomical structures"? Were there many labels segmented or just the same larger ones as they appear on the performance plots? Is this an average Dice score? Regarding the quantitative experimental evaluation, under what ROIs are the metrics computed? For example, what falls under "Other"?

> Can you provide further details on the segmentation labels that were used for the performance evaluation? Can you explain in more detail whether the higher folding percentage of SINR compared to the other solutions is of a concern?

We would like to thank the reviewer for highlighting the absence of details regarding the segmentation labels. To assess our work, we acquired automated segmentation of 138 cortical and subcortical structures, grouped into 5 categories, utilizing MALPEM [i]. More information about the labels can be found at http://christianledig.com/Material/MALPEM/m100_Report.pdf. We have updated the manuscript to incorporate this previously omitted information. Concerning the DICE score, we employ the standard computation method. If reviewer n1vA deems it necessary, we will include its formula in the Appendix.

(i) Ledig, Christian, et al. "Robust whole-brain segmentation: application to traumatic brain injury." MedIA (2015).

> It would have been great if the limitation of high folding compared to the other methods was discussed. Especially in the T1T1 experiments, where this number is half in the non-IDIR methods. The authors, however, describe it as "comparable".

Balancing the folding ratio and registration accuracy entails a trade-off, especially in the challenging scenarios of inter-subject registration where the folding cannot be avoided. Decreasing the folding ratio can be achieved through heightened regularization, which occasionally compromises accuracy. Employing weak deformation regularization permits meticulous alignment by accommodating anatomically implausible deformation magnitudes and irregularities. Conversely, intense regularization yields diminutive and more centralized deformations but yields suboptimal alignment. We choose early stopping criteria and an optimal regularization weight to maintain an acceptable folding ratio, which is why we referred to it as comparable. However, we recognize that the folding ratio tends to be higher in INR methods. Consequently, we have revised the manuscript to reflect this observation.
We would like to thank Reviewer b4Ub for providing insightful feedback and raising unclear questions. We address all concerns in the subsequent rebuttal and have revised the manuscript accordingly. 

> Simply combining existing Free Form Deformations formulation with implicit neural representation is novelty-limited. As mentioned in related work, both are well explored in registration.

INRs, specifically INRs for registration, is an emerging field in deep learning. For this reason, exploring the combination of well-established registration methods (FFD) with novel deep-learning methods could benefit the field. The INR field as such is not yet thoroughly explored. When registering brain images we were presented by the umnet challenge of spacial folding in the context of INRs. In this work, we wanted to motivate the research on the regularisation of INRs for registration in terms of folding issues and highlight potential future research in this direction. We believe our work may be one of the first to explore this research area from one perspective, while there are likely other meaningful and interesting directions as well.

> MIND-loss and diffeomorphisms can solve the limitations of current implicit registration methods, such as inefficiency and smoothness. The authors should give more clear motivation for SIRN registration.

We appreciate the reviewer's suggestion to investigate the application of MIND-loss in the realm of image registration. We recognize the potential value this extension could bring to our work. However, it's essential to note that each multi-modal similarity or distance measure carries its own set of strengths and limitations. We acknowledge that there isn't a singular perfect measure capable of facilitating seamless multi-modal registration without encountering challenges. This far, Mutual Information (in various forms such as normalized, local, etc.) has been the most commonly utilized and robust measure for multi-modal registration. Consequently, we opted to conduct experiments using MI, but we will happily consider MIND in our future work.
Selecting the optimal transformation to accurately represent the underlying deformation poses a significant challenge, primarily because image registration constitutes an inverse problem with the sought transformation being unknown a priori. Particularly in scenarios like inter-subject brain registration, the diffeomorphic transformation model may not be immediately suitable, as the sought transformation could display non-smooth behavior owing to the intricate nature of brain structures. In this study, we deliberately refrained from making this assumption. Nonetheless, we acknowledge that diffeomorphisms could enhance overall performance in various applications or modalities.

> In Table 1, the running time should be demonstrated. Additionally, IDIR [ReLU-MLP] shows higher accuracy than the proposed SINR [ReLU-MLP], indicating the inefficiency of the FFD-enhanced registration. The activation function choices seem to have a more important influence than the FFD-enhanced strategy. Please explain the reasons.

The selection of activation functions remains an active area of investigation within the realm of INRs [i, ii]. Specifically, ReLUs are known for their inability to capture high-frequency details, resulting in smoother and potentially "blurrier" representations. In this study, we included comparisons with INRs employing ReLUs, both for baseline and SINR cases, to provide a comprehensive analysis. However, we contend that INRs utilizing ReLUs may not be well-suited for inter-subject brain registration, and incorporating FFD cannot fully mitigate the drawbacks associated with INRs. Conversely, INRs employing sinusoidal activation functions emerge as a suitable choice for image registration. In this scenario, it becomes evident that integrating FFD not only enhances INR registration accuracy but also improves registration regularity (folding ratio).
(i) Sitzmann, Vincent, et al. "SIREN" NeurIPS2020
(ii) Tancik, Matthew, et al. "Fourier Features." NeurIPS2020

> The training and test time of learning-based registration and test time of conventional registration should be illustrated.

Thank you for this suggestion, please refer to Table 2 in the updated manuscript.

> It would be better to add results about diffeomorphic registration methods, such as Diffeomorphic VMorph.

As we do not assume diffeomorphism, we feel that the comparison with these methods would not be fair, as we would also need to enforce these conditions on INRs, which is a new direction of research as such. This is very interesting, and we aim to investigate these ideas in the future!

> The GitHub rep should add detailed ReadME.md to explain the usage.

Thank you for the suggestion, we have adapted the Readme.md accordingly.
We would like to thank Reviewer kLTk for providing insightful feedback and highlighting inconsistencies in our manuscript. This constructive critique has prompted us to revise and enhance the clarity of these paragraphs. We address all concerns in the subsequent rebuttal and have revised the manuscript accordingly. 

>“we split into 80% training, 10% validation and 10% test set.” For the training of the DL models or also for the presented method? 

>[...] hyperparameters that achieve the highest Dice score [...] This is performed on the training, validation or test dataset? 

Deep learning CNN networks for registration, such as V-Morph and MIDIR, are trained on a cohort basis, whereas conventional methods like MIRTK and INR-based approaches are fitted on a pair of subjects. We opted to split the CamCAN dataset into 80% for training, 10% for validation, and 10% for testing. Subsequently, we trained the CNNs on the training set and fine-tuned their hyperparameters using the validation set. The test set evaluated the networks' ability for accurate and robust registration. Unlike CNN networks, pairwise methods do not require a training/validation/test split, and their hyperparameters can be tuned pairwise. Nevertheless, for comparative purposes against CNN methods, we calibrated the hyperparameters on the validation set and evaluated registration performance using the same test set as mentioned earlier. 

>[...] For both mono-modal and multi-modal registration? Maybe I am doing the maths wrong but it only corresponce to the 3.5M with a grid of 152x152x152. 

> For the mono-modal registration 10k and the multi-modal registration 3.5M coordinates are used. It could be a nice addition to add to how many grid-points that would corresponde to. 

We would like to thank the reviewer for raising the question regarding the grid size corresponding to 3.5 million points, as it prompted us to revisit our calculations and identify an error. As stated in the manuscript, our sampling entails two points in every direction at intervals of 2 mm³, resulting in a grid of (192/2)^3 = 884736 points (~890k, ⅛ of all points) for both mono and multi-modal registration, not the erroneously reported 3.5M points (½ of all points). We confirm that all experiments were conducted with the correct grid size of 884736 points, and the incorrect number was solely a mistake in reporting. Thank you for pointing this out!
Moreover, since the grid spacing is 1mm^3, the number of sampled coordinates corresponds to the number of points we use.

>[...] early stopping if the folding ratio becomes larger than 0.9%.” Is it then just stopped or rerun with different initialization or different parameter? [...] you actually stop after that number of iteration and use that result. 

Through empirical observation, we noted that the folding ratio initially begins at a low level during the fitting of the INR and significantly increases as image matching improves. Baseline INRs are more susceptible to this phenomenon than the proposed FFD-enhanced SINR. We fine-tune the hyperparameters to achieve a low folding ratio and a high dice score to ensure realistic deformation fields and attain high-quality registration results. During testing, we implement early stopping by setting a threshold of 0.9% for the folding ratio to guarantee the realism of the resulting transformations.

>Please add the runtime of the method.

We have included an indicative runtime table in the Supplementary Material [Table 2], and feel that this brings value to our manuscript. 

>How is different runtime using a batch size of 10k and 3.5M?

In our experiments, the results obtained from small batch size training are generally more favorable, which is why we decided to compute the ReLU baselines for the optimal small batch sizes. The run-time may range from 45s (baseline without NMI) to1min39s (SINR with NMI). 

> If possible (space-wise) add a discussion on the limitations [...] Are there cases where the method doesn’t work well?

Due to space constraints, we would like to clarify these ideas in the following section:
Empirical Stopping Criteria: In the context of reconstruction tasks, INRs are typically trained to faithfully represent signals in the weights of a network, and the loss regarding the INR’s predictions/output only serves as a poor indicator to determine the faithfulness of the reconstruction in interpolation applications. In registration, this is even intensified, as we do not have a ground truth deformation field. 
While SINR allows us to successfully use INRs in the context of multi-modal, deformable image registration, it approaches the oscillation problem from a post-processing perspective and tries to address the folding issues within INRs. 
It would be interesting to see if regularization methods e.g. in the INR weight space, could allow to mitigate the oscillations, as SIREN has been suspected to be the cause of these problems.