REINDIR: Repeated Embedding Infusion for Neural Deformable Image Registration

Authors: louis d van harten
rudolf l m van herten
ivana  išgum
Venue: MIDL_io_2024_Conference
Published: 2024-01-12 13:15:18

Abstract: The use of implicit neural representations (INRs) has been explored for medical image registration in a number of recent works. Using these representations has several advantages over both classic optimization-based methods and deep learning-based methods, but it is hindered by long optimization times during inference. To address this issue, we propose REINDIR: Repeated Embedding Infusion for Neural Deformable Image Registration. REINDIR is a meta-learning framework that uses a combination of an image encoder and template representations, which are infused with image embeddings to specialize them for a pair of test images. This specialization results in a better initialization for the subsequent optimization process. By broadcasting the encodings to fill our modulation weight matrices, we greatly reduce the required size of the encoder compared to approaches that predict the complete weight matrices directly. Additionally, our method retains the flexibility to infuse arbitrarily large encodings. The presented approach greatly improves the efficiency of deformable registration with INRs when applied to (near-)IID data, while remaining robust to severe domain shifts from the distribution the method is trained on.

Main paper: 
 introduction: 
 Deformable image registration seeks to transform pairs of images to a shared coordinate space by identifying a deformation vector field (DVF) that maximizes the local correspondences of image content. Many medical image processing pipelines rely on robust image registration as a precursor to any downstream analyses. With the rise of deep learning, convolutional neural network (CNN) approaches have been proposed for this paradigm (Fu et al., 2020) . The inductive bias inherent to CNNs results in a grid-based DVF approximator: once trained, such methods directly predict a rasterized DVF given a set of new, unseen images (de Vos et al., 2019) . Although providing fast approximations, CNN-based methods are highly sensitive to domain shifts resulting from out-of-distribution data. This may be mitigated by optimizing an image registration similarity metric at test-time (Fechter and Baltas, 2020) , though this is typically slow and undesirable for large models used in learning-based methods.In recent years, implicit neural representations (INR) have been proposed for deformable image registration (Wolterink et al., 2022) . Within this paradigm, a neural network is leveraged to approximate the DVF by mapping input coordinates to their respective deformation vectors. This optimization framework has several advantages over both classic optimizationbased methods and deep learning-based methods, such as precise regularization of spatial derivatives, robustness to domain shift, and the ability to capture small details in the DVF, but it is hindered by long optimization times during inference. This makes its use impractical in clinical workflows, especially in cases where either speed is of the essence, such as for stroke patients or image-guided therapy, or in cases where registration over many image acquisition time points is required, such as intestinal or cardiac motion analysis.Several methods have been proposed to address the issue of test-time optimization of INRs. One such method is meta-learning, which focuses on learning an improved INR parameter initialization from a training set. At test time, the INR is then able to quickly adapt to new similar tasks (Nichol et al., 2018; Tancik et al., 2021) . This principle has recently been applied for medical image registration (Baum et al., 2023) . Another effective strategy for optimizing INRs is to infuse the model parameters with a task-specific prior. This approach is exemplified in models like DeepSDF, where an auto-decoder architecture is used to optimize a latent vector at test time while freezing the remaining INR weights (Park et al., 2019) . An alternative approach is the use of hyper-networks, in which an additional neural network is employed to directly predict the weights or modulate the activations of an INR. This immediately provides the INR with a useful bias, allowing it to rapidly adapt to the task at hand (Ha et al., 2017; Perez et al., 2018; Babu et al., 2023) .Given these recent developments in INR optimization, we introduce an accelerated method for the task of deformable image registration. Specifically, we propose REINDIR: Repeated Embedding Infusion for Neural Deformable Image Registration. In addition to improving the optimization speed via meta-learning, we condition the initialization of INR parameters on a prior in the form of a repeated encoding of the to-be-registered image pair. We demonstrate the effectiveness of our method in the 3D lung CT registration paradigm, as well as on the task of cone-beam CT-guided radiation therapy. 
methods: 
 To accelerate the optimization process of INRs for deformable registration, we propose REINDIR: Repeated Embedding Infusion for Neural Deformable Image Registration. The method generates INRs that are infused with a learned embedding of the input images. We infuse these embeddings into the neural representations by broadcasting them into a modulation matrix, which has the same shape as the weight matrices of each hidden layer. The modulation matrices for each layer are applied to the weight matrices of template INRs, which are trained in conjunction with the encoder that produces the embeddings. These template INRs can be thought of as implicit decoders for the embeddings. An overview of the proposed method is presented in Figure 1 .In the paradigm of implicit neural representations, neural networks are specifically leveraged for their capability as universal function approximators. Rather than operating on image values, the INR processes continuous coordinates which are subsequently mapped to some output value. For image analysis tasks, a loss function is typically included which relates outputs to image information. The image information therefore only enters the network through gradient backpropagation.For the task of image registration, a neural representation in the form of Φ(x) = υ(x)+ x is employed (Wolterink et al., 2022) . Here, υ(x) denotes a sinusoidal representation network (SIREN) mapping input coordinates x from the source domain S to deformation vectors (Sitzmann et al., 2020) . By applying the vectors υ(x) to x, the corresponding coordinates in the target domain T can be inferred. Specifically, we employ cycle-consistent INRs (ccIDIR) as presented in (van Harten et al., 2024) for more accurate and more robust registration results. As such, a set of INRs is trained to perform both the mapping S → T and T → S by minimizing an additional cycle-consistency term in the loss function.Since random initialization of INRs is central to their slow test-time optimization, metalearning strategies such as Reptile (Nichol et al., 2018) have been proposed. In this paradigm, the initial parameters of an INR are optimized by fitting the network to batches of examples in a training dataset. The initial state of the network is subsequently updated by taking a small step into the direction of the fully optimized parameters. As such, the layer l, two modulation vectors γ l and δ l are inferred from e. In order to perform pointwise modulation of weight matrix W ϕ l , the γ l vector is broadcasted to W γ l through repetition (R). Meanwhile, bias β ϕ l is directly modulated by the same-size vector δ l . This allows the model parameters W l and β l to instantly incorporate motion pattern information between source and target images. This is repeated for all L layers of each INR. Note that no modulation matrices or bias vectors are predicted for the first (x, y, z → a 0 ) and final (a L → dx, dy, dz) INR layers.INR learns a generalizable template for function approximation that can rapidly fine-tune to new instances from the same distribution. However, as the template is shared between all examples in the training set, the benefit of this approach is bounded by the dissimilarity of different examples within the target distribution.To address this, we introduce an image encoder that creates an embedding from the concatenation of the source and target images. By infusing the template INRs with this embedding, we can leverage the image encoder as a hyper-network for specializing the initializations to each image pair. As illustrated in Figure 2 , we achieve this by predicting two modulation vectors (γ l and δ l ) for each INR layer, which modulate the weight matrix W ϕ l and the bias vector β ϕ l respectively. Modulation of W ϕ l is achieved by repeating and subsequently reshaping γ l to fit W ϕ l , allowing for pointwise multiplication. Meanwhile, β ϕ l is directly modulated by the same-size vector δ l . The INRs used in this work are SIRENs (Sitzmann et al., 2020) with three hidden layers containing 256 features each. Modulation is performed for the two weight matrices between these hidden layers.During training, an image pair is first embedded by a convolutional encoder based on the LKU-Net architecture (Jia et al., 2022) , which is used to generate modulation vectors. INRs are subsequently optimized for a number of iterations using this embedding, after which the image encoder is updated. This allows us to train both the template INRs and the image encoder end-to-end. The full training procedure is detailed in Algorithm 1. Additional implementation details are included in Appendix A. for iteration = 1, 2, 3, . . . , N inner do ϕ S→T , ϕ T →S ← UpdateINRs(S, T, e, ϕ S→T , ϕ T →S ) end E ← UpdateEncoder(S, T, E, ϕ S→T , ϕ T →S ) δ IN Rs ← δ IN Rs + (ϕ o S→T ,ϕ o T →S )-(ϕ S→T ,ϕ T →S ) Nsets end ϕ o S→T , ϕ o T →S ← (ϕ o S→T , ϕ o T →S ) + δ IN Rs end 
data: 
 In this work, a total of three datasets are used for two different registration tasks. All scans from all sets were resampled to an isotropic 1 mm 3 using trilinear interpolation.The first two datasets focus on the task of 4D lung CT inspiration-to-expiration image registration. We include data from the DIR-LAB (10 patients) and NLST (110 patients) studies (Castillo et al., 2009; Aberle et al., 2011) . The DIR-LAB dataset contains 300 manually annotated lung landmarks per scan, which serve as a ground truth for determining the registration error. For the NLST data, we use the subset made available for the Learn2Reg challenge (Hering et al., 2022) . For 100 out of 110 cases in this set, a varying number of automatically generated matched keypoints are made available.In the third dataset, we focus on registration between pre-therapeutic fan-beam CT (FBCT) and interventional low-dose cone-beam CT (CBCT). The dataset made available for this task is also part of the Learn2Reg challenge, and features a total of 14 patients (Hugo et al., 2016) . For each patient, a planning FBCT as well as two CBCT scans are made available, of which the first CBCT scan was acquired at the beginning of therapy and the second one at the end of therapy. The challenge considers the registration of all visible thoracic organs, and is required to be both fast and accurate to minimize the radiation dose for organs at risk. Automatically generated keypoints for the first 11 patients were provided by the challenge organizers. However, upon visual inspection, correspondence between keypoints in this set proved to be limited. We therefore generate corresponding keypoints in this set by optimizing image pairs using an existing registration method (van Harten et al., 2024) within automatically generated lung masks (Hofmanninger et al., 2020) 1 . Two image pairs were excluded in this process due to high variability in matched keypoints after optimization. Additional details are included in Appendix B.  
experiments and results: 
 We compare our proposed initialization to ccIDIR with random initialization as proposed in (van Harten et al., 2024) , as well as to a Reptile initialization (Nichol et al., 2018) our results for 5-fold cross-validation. Specifically, for each fold the methods are optimized using images from all sets. Methods are compared in terms of the landmark registration error, i.e. the mean error for keypoints between image pairs. A comparison of quantitative results for INR-based deformable image registration methods is presented in Figure 3 . For each image pair, the method is optimized within a nearinstant timeframe (25 iterations ∼ 1 s, Nvidia RTX 2080Ti). For both the DIRLAB and NLST sets, the embedding prior provided by REINDIR produces a lower registration error at initialization, resulting in faster convergence. For the CBCT set, Reptile and REINDIR perform similarly, with both methods converging faster than the ccIDIR baseline. Errors in this set were typically higher due to the more complex registration task of noisy CBCT data. It should further be noted that registration errors for all methods eventually converge to a similar value given a large number of test-time optimization steps.An example image from the NLST set is shown in Figure 4 , which compares the proposed initialization with random initialization and Reptile initialization, which is equivalent to using trained template INRs without embedding infusion. In this example, the proposed method is able to instantly correct the general motion pattern without any INR update steps. We further exemplify a case of the CBCT set for which REINDIR initialization was not beneficial in Figure 5 . Additional examples are included in Appendix D. 
discussion and conclusion: 
 We have presented a method for deformable image registration with implicit neural representations that explicitly incorporates image information from to-be-registered image pairs. We embed image information into the INR by modulating the weights and biases of the network, which can prove beneficial for the initialization of the registration task.As illustrated in Figure 3 , our proposed method provides a useful bias in the registration of both the DIRLAB and the NLST set, for which the keypoint registration error is significantly lower upon initialization of the INR. This trend is however not visible in the results of the CBCT set, for which the optimization process of REINDIR closely follows Reptile. We hypothesize that the low number of available image pairs featured in this set may hamper the ability of the convolutional encoder to generalize to unseen data. This is further complicated by the fact that the CBCT images only provide a limited view of the thorax while being highly variable in terms of appearance and noise level. Despite this, REINDIR never negatively impact registration performance, as illustrated in Figure 5 . Though the initialization fails to capture the registration motion pattern, the network is able to rectify the deformation given a small number of optimization steps, resulting in a performance similar to Reptile.The results further indicate that REINDIR performance is correlated with training set size: the best results were observed for the NLST set, for which in the cross-validation setting 88 image pairs were available during training. In comparison, only 8 and 20 training image pairs were available for the DIRLAB and CBCT sets, respectively. In line with expectations, performance gain for DIRLAB is therefore less pronounced, and is limited in the CBCT set. Future work may therefore include a large set of more diverse training data, or research more expressive encoding architectures. This could further boost zero-shot performance and limit the necessity of test-time optimization to out-of-distribution data.An important distinction with recent work on INR-based methods is that our method does not use hash encoding (Müller et al., 2022) . Hash grids are used to encode the input space, taking advantage of the sparse nature of data in many natural image processing tasks. The aliasing in these hash grids compresses the problem such that subsequent MLPs can be smaller. However, hash grids introduce discontinuities in the spatial encoding (Huang and Alkhalifah, 2024) , which complicates the optimization of image registration INRs, as this requires computation of spatial derivatives throughout the valid region of the input space.It should further be noted that modulation of the weight matrix is not the only option for INR manipulation. Recent works (Dupont et al., 2022; Papa et al., 2023) proposed to modulate INR activations rather than the weight matrices themselves, which allows the network to condense meaningful sample-specific information into a low number of parameters. Future work may investigate such modulation variations. A possible disadvantage of such an approach is that it is not possible to modulate the interactions between different features. Additionally, the amount of information that can be infused into the template network is limited by the number of features in each layer, which is a very small fraction of the number of weights in those layers. In REINDIR, we keep the length of the encodings a free parameter, which allows the amount of information infused into the template representations to be chosen without such constraints. Future work may further explore the impact of different encoding lengths and types of embedding infusion.In conclusion, this study presented a method for accelerated medical image registration by infusing implicit neural representations with embeddings from to-be-registered image pairs. Our experiments show how this prior can help guide the registration process and produce useful initialization for deformable image registration. 
 
 Appendix: The method is trained using the procedure described in Algorithm 1. The training parameters are listed in Table A .2. The optimization procedure for the inner loop closely resembled the procedure described in (van Harten et al., 2024) , with one notable exception: during training, SGD was used as the optimizer for the template INRs. Training with Adam as the optimizer for the inner training loop caused instability in the training procedure. As this does not impact the test-time optimization, Adam is used during inference. With the exception of the embedding length, parameters for the baseline methods were matched to the parameters used for the proposed method. optimized for 2500 iterations, as detailed in (van Harten et al., 2024) . This registration method produces an uncertainty estimate resulting from the cycle-error after optimization.For each image pair, we ranked the registration results by average estimated uncertainty and propagated the markers using the five most confident instances. These instances were combined by calculating the median deformation vector for each marker.For each propagated image pair, we evaluated the mean uncertainty of the five registration instances used for propagation. This revealed two outliers, which had mean uncertainties of more than 5mm. One of these image pairs is shown in 7. For these image pairs, the cone beam CTs contained artifacts that made them unsuitable for registration with the global NCC loss used in this work. Hence, we excluded these images from further analysis. Figure 8 provides an analysis of inference times for ccIDIR (van Harten et al., 2024) , Reptile, REINDIR and a baseline VoxelMorph approach (Balakrishnan et al., 2019) . The additional overhead of the encoder used in this work is minimal, it being equivalent to approximately 1.5 optimization steps when using Jacobian determinant regularization, or 0.23 steps when regularizing with a Bending penalty.To include a learning-based image registration baseline, we provide the results on the NLST dataset for VoxelMorph (Balakrishnan et al., 2019) here. The method was trained using the the default settings of the author's PyTorch implementation on GitHub (commit: ca28315), employing seven diffeomorphic integration steps of the vector field. The only change was to reduce the network channel dimensions from 16 to 6, which was necessary to fit a set of full-size NLST images (2 × 336 × 288 × 366) into GPU memory (3090, 24GB) during training and test-time optimization. This network was trained for 200 epochs in 5fold cross-validation. At test-time, the to-be-registered image pair was repeatedly processed by the VoxelMorph pipeline, using the same training settings as used in the original training code; each optimization step took 0.95 seconds. © 2024 L.D. van Harten, R.L.M. van Herten & I. Išgum.Generated keypoints are available at https://github.com/louisvh/cbct_generated_keypoints 


Decision: The authors modulate the weights of a Neural Deformable Image Registration Network via meta-learning. Overall, the reviewers found the paper interesting, but of minimal difference (experimentally) to existing methods. In most situations, the improvement is minimal or non-existent.  Unfortunately, in 2 of the 4 reviews, the reviewer score went down after rebuttal, with the reviewers being unsatisfied. I believe all reviewers here acted in good faith and were thorough. 

Overall, the paper paper remains borderline. It may be insightful to communicate at the conference, but I believe it will also be good for the authors to take all this feedback and strengthen the paper.

Reviews: review: paper_summary: paper_summary: The paper presents REINDIR, a method for enhancing deformable image registration using implicit neural representations (INRs). By infusing image embeddings into INRs, REINDIR optimizes the registration process, significantly reducing optimization time during inference. This approach is demonstrated to be efficient and robust across different datasets, including challenging scenarios with severe domain shifts. The method's innovation lies in its use of meta-learning and embedding infusion, setting a new standard for quick and reliable deformable image registration in medical imaging.
main_review: main_review: After reviewing this paper and other reviewer's suggestion, I think this paper is novel and has solid contributions. If possible, the author can try to polish the work again and provide more comprehensive explanation and experiment in the camera ready version.
strength_weakness: strength_weakness: The paper are well constructed and with no obvious weakness. This paper has sufficient amout of experiments, clear explanation, and the idea is very novel. If possible, the authors can bring part of the experiment details to the main body.
questions: questions: N/A
paper_summary: paper_summary: This paper aims to address the cost of running test-time optimization for implicit neural representation (INR) based image registration methods. This paper proposes to use an image encoder to map a pair of test images to an embedding that is used to modulate the INR's weight. The whole framework is trained end-to-end with Reptile [1]. The authors demonstrated that per-example modulation requires a smaller number of test-time optimization steps to achieve the same level of registration performance.


[1] On First-Order Meta-Learning Algorithms
main_review: main_review: I don't feel the authors sufficiently addressed my concern. 

- On the dataset that the proposed method works the best (NLST), it seems that if your test-time budget is 0.1 seconds, then REINDIR yields 30% lower error than the Reptile baseline. However, with a test-time budget >= 0.25 seconds, REINDIR and Reptile essentially yields the same error. Moreover, by just setting a relatively larger test-time budge of 0.5 second, the error of both methods improved significantly (>50%) compared to a test-time budget of 0.25 second. If the motivation of the method is for problems that warrant faster registration, the authors should demonstrate significant performance gain at a certain test-time inference budget that makes sense for a specific/concrete application/task. Currently, the take-away from reading these figure 3 and figure 8 is that the proposed REINDIR is not that different from Reptile.
- The authors are proposing a new hypernetwork that "broadcast the encodings to fill the modulation weight matrices." I don't think it's an unreasonable request to ask for how existing hypernetworks do in the same experimental setup. I agree it's perhaps not enough time to run these additional experiments, and perhaps it's better if the authors could run these experiments and make the current paper more complete.
strength_weakness: strength_weakness: - My main concern is that the proposed approach is better than the Reptile baseline when the number of test-time optimization steps is small. The gain is either nonexistent or marginal for larger number of test-time optimization steps, e.g., 10. The registration performance is greatly improved if do a few more optimization steps, implying that realistically people would do more than 10 optimization steps anyways where the proposed approach do not have significant advantage over simple baselines.
- This paper lacks sufficient references on how past works have modulate network weights and how their approach is different. Related, this paper lacks sufficient comparisons to alternative ways to modulate network weights. More details in further comments below.
questions: questions: Adress problems raised in Weaknesses and additional comments. Specifically, ablation on modulation proposed and additional reference on how modulation is done in the past.
paper_summary: paper_summary: The paper proposes a module that modulates the weight matrices of an registration INR with learned embeddings. This results in a individual initialization of the network, resulting in better zero-shot results, which benefits the inference time.
main_review: main_review: The authors addressed my questions well. The additional evaluation of runtime is helpful to asses the actual benefit of the methods. While the benefit is indeed minor, the author make clear that their method does outperform Reptile when using very few iterations, useful in real-time applications.
My initial concerns still stand. However, the novelty and clear presentation of the method alongside the new runtime analysis justify an **boderline** acceptance.
strength_weakness: strength_weakness: While the benefit of the method is shown for the first few optimization steps, the benefit when training until convergence is not evaluated.
The inference time of the embedding network is not taking into account.
It is unclear how dependent the method is on the exact embedding network. An ablation study would be beneficial.
questions: questions: Repeating the embedding to gain a properly sized modulation matrix seems arbitrary. This is especially the case since the lengths of the embeddings are fixed. Would it be negative to predict the full modulation matrix?
It seems like the better initialization saves only around ten optimization steps compared to random initialisation, saving about 1 second in inference time, even less when compared with the meta-learning approach. How many steps and/or time is saved through the better initialization when properly training the network until convergence?
How much time does the inference of the embedding network take?
paper_summary: paper_summary: This paper proposes REINDIR: Repeated Embedding Infusion for Neural Deformable Image Registration. This meta-learning framework combines an image encoder and template representations that are infused with image embeddings to specialize them for image pairs. 
The presented approach improves the efficiency of deformable registration with INRs and remains robust to domain shifts from the distribution the method is trained on.
main_review: main_review: Some concerns have been addressed.
The additional evaluation comparison with VoxelMorph registration methods and updated figure captions help address my concerns about the experiment and the paper's clarity. However, the VoxelMorph is the SOTA method in 2019.
 While fine-tuning can be leveraged to optimize registration CNNs/Transformers to specific image pairs, the authors make clear that the encoder employed in their work only requires a single forward pass and is never tuned at test time. However, my initial concerns still stand, cause the encoder often contains meta-information rather than task-specific information which needs tunning.
strength_weakness: strength_weakness: - The submission lacks a comprehensive qualitative and quantitative comparison with current state-of-the-art (SOTA) registration methods, as well as a benchmark of runtime performance.
- The motivation behind using a meta-learning framework for the registration task is not well-defined, particularly in distinguishing it from fine-tuning approaches. The data presented in Figure 3 indicates comparable performance between Reptile and the proposed REINDIR, which calls for a more detailed justification.
- The methodology is not adequately elucidated, with Figures 1 and 2 failing to convey the nuances of the proposed approach.
questions: questions: Concerns about unconvincing experiment verification and unclear motivation.

Comments: title: Response to Reviewer KSKH (1/1)
Response to Reviewer pFD7 (1/1)
Response to Reviewer YWF6 (2/2)
Response to Reviewer YWF6 (1/2)
Response to Reviewer s7rH (1/1)
comment: The authors addressed my questions well. The additional evaluation of runtime is helpful to asses the actual benefit of the methods. While the benefit is indeed minor, the author make clear that their method does outperform Reptile when using very few iterations, useful in real-time applications.
My initial concerns still stand. However, the novelty and clear presentation of the method alongside the new runtime analysis justify an acceptance.
We thank the reviewer for their comments, which we have addressed below.

__W1:__ The submission lacks a comprehensive qualitative and quantitative comparison with current state-of-the-art (SOTA) registration methods, as well as a benchmark of runtime performance.

__A:__ We agree that a comparison to a registration baseline could benefit the paper. Since there is unfortunately not enough space to include this in the main text, we have added an additional appendix in which we evaluate the performance of VoxelMorph (Balakrishnan et al., 2019) on our largest dataset, i.e. NLST. We present both a comparison with this method as well as runtime benchmarking performance in __Appendix C__. The figure shows VoxelMorph performance using a single forward pass (red dotted vertical line), as well as performance given fine-tuning steps in a test-time training setting (red line). The figure further shows the inference time for all three previously discussed INR-based approaches, and we present results for both Jacobian determinant regularization (left) and Bending energy penalty regularization (right). Finally, the orange dotted line represents the overhead in REINDIR resulting from the forward pass of Encoder _E_, which is equivalent to 1.5 steps for Jacobian determinant regularization and 0.23 steps for Bending penalty regularization.

__W2:__ The motivation behind using a meta-learning framework for the registration task is not well-defined, particularly in distinguishing it from fine-tuning approaches. The data presented in Figure 3 indicates comparable performance between Reptile and the proposed REINDIR, which calls for a more detailed justification.

__A:__ Fine-tuning can indeed be leveraged to optimize registration CNNs/Transformers to specific image pairs at test-time, as well. A downside to this approach is that it requires optimization of a large network, therefore eliminating the efficiency of such methods compared to optimization-based methods (Zhu et al., 2021). In comparison, the encoder employed in our work only requires a single forward pass and is never tuned at test time, thus requiring minimal overhead while preserving the benefits of an optimization-based registration method. This is now illustrated in __Appendix C__, where we show fine-tuning performance for VoxelMorph (Balakrishnan et al., 2019) over time.    
We agree that performance gains vary across different sets, which we discuss in the __Discussion (Section 5, paragraphs 2 and 3)__. We discuss the role of the training set size and the effect of noise. In future work, we will train the proposed method on a larger dataset, with the aim to demonstrate the feasibility of zero-shot registration, with test-time optimization reserved for out-of-distribution images. We now state this more explicitly in the __Discussion (Section 5, paragraph 3)__:

_“Future work may therefore include a large set of more diverse training data, or research more expressive encoding architectures. This could further boost zero-shot performance and limit the necessity of test-time optimization to out-of-distribution data.”_

__W3:__ The methodology is not adequately elucidated, with Figures 1 and 2 failing to convey the nuances of the proposed approach.

__A:__ We apologize for any potential unclarity in Figures 1 and 2. We have updated the figure captions to explicitly cover all modules presented in the figures, and hope they explain the specific parts more clearly now. They now read:

__Figure 1:__ Overview of the proposed method. A source image $S$ and target image $T$ are embedded into a combined representation $\overline{e}$ using a trainable convolutional encoder $E$. We subsequently infuse the parameters of two template implicit neural representations (INR) $\phi_{S\rightarrow T \/ T\rightarrow S}$ by modifying their weights using the embedding $\overline{e}$ in the infusion module. This results in two task-informed INRs~($\phi^*$) capable of rapidly fitting the deformable registration of coordinates in $S$ to coordinates in $T$ in a cycle-consistent fashion.

__Figure 2:__ The REINDIR infusion module. For each implicit neural representation (INR) layer $l$, two modulation vectors $\gamma_l$ and $\delta_l$ are inferred from $\overline{e}$. In order to perform pointwise modulation of weight matrix $W_l^\phi$, the $\gamma_l$ vector is broadcasted to $W_l^\gamma$ through repetition ($R$). Meanwhile, bias $\beta_l^\phi$ is directly modulated by the same-size vector $\delta_l$. This allows the model parameters $W_l$ and $\beta_l$ to instantly incorporate motion pattern information between source and target images. This is repeated for all $L$ layers of each INR. Note that no modulation matrices or bias vectors are predicted for the first ($x,y,z\rightarrow a_0$) and final ($a_L\rightarrow dx,dy,dz$) INR layers.
We thank the reviewer for their helpful remarks, for which we list our answers on a per-point basis below.

__W1:__ While the benefit of the method is shown for the first few optimization steps, the benefit when training until convergence is not evaluated.    
__Q2:__ It seems like the better initialization saves only around ten optimization steps compared to random initialisation, saving about 1 second in inference time, even less when compared with the meta-learning approach. How many steps and/or time is saved through the better initialization when properly training the network until convergence?

__A:__ To give a sense of the results for longer test-time optimization, we have included an alternate version of Figure 3 in __Appendix C__ that shows results for the NLST set up to 50 optimization steps, rather than 25. This also shows that with Jacobian determinant regularization, the result after optimizing for 2 seconds from Reptile-based initialization is equivalent to the result with the proposed method after 1.34 seconds (i.e. a 33% speed increase).

__W2:__ The inference time of the embedding network is not taking into account.    
__Q3:__ How much time does the inference of the embedding network take?

__A:__ We thank the reviewer for pointing this out; the inference time of the embedding network was indeed not taken into account. This network runs in approximately 62 milliseconds on our hardware. We have added a figure showing the runtimes of the different approaches in __Appendix C__. The results show inference times for the embedding network and test-time optimization steps (with both Jacobian determinant regularization and Bending penalty regularization). We also indicate the number of optimization steps equaling an image embedding forward pass, which is approximately 1.5 steps for optimization with Jacobian determinant regularization and 0.23 steps with Bending penalty regularization.

__W3:__ It is unclear how dependent the method is on the exact embedding network. An ablation study would be beneficial.

__A:__ We understand the reviewer’s comment regarding alternatives for image embedding architectures. We opted to use the encoder part of the Large Kernel U-Net (LKU-Net, Jia et al., 2022), which is listed as one of the best-performing networks on the Learn2Reg NLST leaderboard. The paper shows little differences between convolutional and transformer-based backbones, with their CNN-based LKU-Net performing best. In agreement with this, we expect any sufficiently parameterized model to perform well as an image encoder. Future work may explore the impact of image encoder choice more explicitly. We now include this reasoning in __Appendix A.1__ to clarify our choice of image encoder:

_“The image encoder used in this work is based on the encoder of the LKU-Net architecture. The authors of the paper extensively explore differences between convolutional and transformer-based backbones, and are listed as one of the best-performing algorithms on the Learn2Reg NLST leaderboard. Nevertheless, we expect any sufficiently parameterized model to perform well as an image encoder.”_

__DC1:__ Tancik et al. (2021) should be cited.    

__A:__ We now cite Tancik et al. (2021) in the __Introduction (Section 1, paragraph 2)__:

_“At test time, the INR is then able to quickly adapt to new similar tasks (Nichol et al., 2018; Tancik et al., 2021).”_

__Q1:__ Repeating the embedding to gain a properly sized modulation matrix seems arbitrary. This is especially the case since the lengths of the embeddings are fixed. Would it be negative to predict the full modulation matrix? 

__A:__ Generating the full INR weight matrices has indeed been proposed in earlier works (Babu et al., 2023), but it would be challenging for the proposed method. Previous INR-based registration methods typically employ weight matrices of 256x256. Predicting such weight matrices from our image encoder bottleneck (with embedding size 1025 in our current work) with an additional linear layer would result in a large number of additional trainable parameters (i.e. over 130 million in that one layer). Given that even our current encoder requires fairly large datasets to generalize properly to new images, we do not expect to have enough data to properly train such a large translation layer, likely resulting in overfitting.
__DC4:__ It would be nice to provide some information of the overhead of using an additional encoder that generates embeddings of image pairs during training and compare that with that of the Reptile baseline.

__A:__ Thank you for pointing out this oversight. We have added a figure presenting the runtimes of method variations in __Appendix C__. The results also show inference time for the embedding network and test-time optimization steps (with both Jacobian determinant regularization and Bending penalty regularization). We also indicate the number of INR optimization steps equaling an image embedding forward pass, which is approximately 1.5 steps for optimization with Jacobian determinant regularization and 0.23 steps with Bending penalty regularization.

__DC5:__ Figure 3: It would be helpful to include "Runtime" in additional to "optimization steps' in the x-axis or in caption/text to understand how much time could be saved by using REINDIR vs. Reptile.

__A:__ See __DC4:__ we now also refer to the runtime benchmark results in the __caption for Figure 3__.

Figure 3 caption: _“Registration error over time, comparing random initialization (ccIDIR), learned initialization (Reptile), and the proposed learned initialization with embedding infusion (REINDIR). Median results plotted for 10 different random seeds; solid lines indicate the median, shaded areas indicate the range among seeds. Extended results regarding performance benchmarks and runtimes are listed in Appendix C.”_

__DC6:__ It would be helpful if the authors write a few sentence on the motivation for outputing two modulation vectors, and broadcast one to the size of the 2D matrix through repetition. It would be helpful if the authors provide ablations on why doing so is helpful. For example, compare with (1) just modulate the bias term in each layer, (2) just modulate the 2D matrix in each layer, (3) modulate the matrix term by predicting a 2D matrix instead of a vector then repeat, (4) modulate the weight matrix by addition instead of point-wise product, (5) other modulation methods like [1,2]. The authors should either run the ablations or explain why it's not necessary.    
__DC7:__ Related to previous point, it would helpful if the authors explain how prior works modulate neural network weights using hypernetworks differently, e.g., modulate weights using product vs. sum. A few examples of this body of work: [2,3] and I'm sure there are many diferent ways to modulate INR in addition to the 1 reference mentioned in discussion.

__A:__ We originally considered generating the full weight matrices (option 3, as in Babu et al. 2023), but to do so for the same size networks used in previous literature on INR-based registration, this would require a linear layer from the bottleneck size (1025 in our current work) to 2x256x256, which results in over 130 million parameters in that single layer. This is several times larger than our proposed image encoder. Given that even our current encoder requires large datasets to generalize well to new images, we do not have enough data to properly train such a large translation layer, likely resulting in severe overfitting.   
Though other modulation options (bias only, weight matrix only, activation only, addition, multiplication) are indeed interesting ablations, re-running the experiments is unfortunately infeasible given the limited rebuttal time. We hypothesize that the hypernetworks could indeed already boost performance significantly using shift modulation only, i.e. only modulating the bias vector (either through multiplication or addition), as indicated by Dupont et al. [2]. Though [1] modulates the activations rather than the weight and bias matrices (as also used by Papa et al.), we theorize that having a network modulate both does not negatively impact results - at worst, it could cause minimal additional overhead. We have added a note on this in the __Discussion (Section 5, paragraph 5)__:

_“Recent works (Dupont et al., 2022; Papa et al., 2023) proposed to modulate INR activations rather than the weight matrices themselves, which allows the network to condense meaningful sample-specific information into a low number of parameters. Future work may investigate such modulation variations.”_

We now also cite FiLM and provide a short description in the __Introduction (Section 1, paragraph 3)__:

_“... to directly predict the weights or modulate the activations of an INR. … (Ha et al., 2017; Perez et al., 2018; Babu et al., 2023).”_
We thank the reviewer for their detailed assessment of the paper; we have tried to address all raised concerns, as listed below (note that this response was split in 2 parts due to the character limit).

__W1:__ My main concern is that the proposed approach is better than the Reptile baseline when the number of test-time optimization steps is small. The gain is either nonexistent or marginal for larger number of test-time optimization steps, e.g., 10. The registration performance is greatly improved if do a few more optimization steps, implying that realistically people would do more than 10 optimization steps anyways where the proposed approach do not have significant advantage over simple baselines.

__A:__ Our primary aim is to design a registration method for clinical applications where long optimization times are unfeasible, such as mesh propagation in real-time settings or in registration of long 4D sequences, where large numbers of 3D volumes need to be registered towards each other. In such applications, registration performance within a short temporal window is critical for clinical viability, as is stressed in the __Introduction (Section 1, end of paragraph 2)__.  We agree the performance boost varies per evaluated dataset and is notably correlated with the number of available training examples. In our future work, we will train the proposed method on a larger dataset, with the aim to demonstrate the feasibility of zero-shot registration, with test-time optimization reserved for out-of-distribution images. We now state this more explicitly in the __Discussion (Section 5, paragraph 3)__:

_“Future work may therefore include a large set of more diverse training data, or research more expressive encoding architectures. This could further boost zero-shot performance and limit the necessity of test-time optimization to out-of-distribution data.”_

__W2:__ This paper lacks sufficient references on how past works have modulate network weights and how their approach is different. Related, this paper lacks sufficient comparisons to alternative ways to modulate network weights. More details in further comments below.

__A:__ We agree with the reviewer that there are many ways to modulate weights, and many versions have been proposed over the past years for various applications. However, none have been applied to image registration, which is a peculiar application of INRs: the information density of these representations is much higher than for a typical neural radiance field or implicit shape network (which are generally extremely sparse with respect to their coordinate space). The amount of information required to fully predict a deformation vector field for any given pair of images is highly dependent on the specific subjects being registered: a (near) rigid transformation could be fully represented with an encoding of only six elements, whereas a highly elastic deformation between two small bowel images may contain multiple megabytes of information. This is our reasoning for keeping the amount of information infused in the INRs a free parameter in the method. We plan to investigate the impact of different encoding lengths for different datasets in a follow-up paper, but as this will require extensive additional experiments, we consider this out-of-scope for the current work. We have updated the __Discussion (Section 5, paragraph 5)__ accordingly:

_“Future work may further explore the impact of different encoding lengths and types of embedding infusion.”_

__DC1:__ Abstract: introduce INR before using INR

__A:__ Thank you for pointing this out; the “INR” acronym is now introduced in the first sentence of the Abstract: 

_“The use of implicit neural representations (INRs) …”_

__DC2:__ Introduction: talk a bit about what are the advantages of INR over optimization / DL based methods.

A: We have added a short description of the benefits of INRs in the __Introduction (Section 1, paragraph 2)__:

_“This optimization framework has several advantages over both classic optimization-based methods and deep learning-based methods, such as precise regularization of spatial derivatives, robustness to domain shift, and the ability to capture small details in the DVF, …”_

__DC3:__ Move Section 2 Data to Section 4 Experiments to avoid repeating description of the datasets.

__A:__ We thank the Reviewer for this suggestion, this indeed reads somewhat redundantly. We have therefore swapped positions of the Method and Data sections, with the Method Section now being Section 2, and the Data Section being Section 3. The following text has been removed from the start of Section 4, as it is now redundant:

_“We evaluate our method on three chest CT datasets: the DIR-LAB set (10 image pairs) (Castillo et al., 2019), a subset of the NLST set (100 image pairs) (Aberle et al., 2011), and the CBCT set (Balik et al., 2013) (26 image pairs), the latter two as included in the Learn2Reg challenge (Hering et al., 2022).”_
We thank the reviewer for their encouraging assessment of the paper.

__W1:__ The paper are well constructed and with no obvious weakness. This paper has sufficient amout of experiments, clear explanation, and the idea is very novel. If possible, the authors can bring part of the experiment details to the main body.

__A:__ We thank the reviewer for recognizing our work and for their positive feedback. Regarding the experimental details, we agree that it would be useful to address some key aspects of our method more directly in the main body. Therefore, we now include general details about the image encoder (E) and INR modulation in __Section 2.2 paragraph 2 and 3__:

_“The INRs used in this work are SIRENs (Sitzmann et al., 2020) with three hidden layers containing 256 features each. Modulation is performed for the two weight matrices between these hidden layers.”_

_“... first embedded by a convolutional encoder based on the LKU-Net architecture (Jia et al., 2022), which is used to generate modulation vectors.”_

Due to space constraints, remaining extensive experimental details are described in __Appendix A__. Additionally, a more extensive runtime analysis was highly suggested by other reviewers, for which we now show the results in __Appendix C__.