Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 3: Borderline
review_content: summary: The authors apply and evaluate implicit neural representations (INRs) in the context of deformable image registration: instead of training a neural network to predict deformation fields between pairs of images, the network is used to represent a transform. The work analyzes the effect of choosing periodic (SIREN) activation functions over ReLU and explores the benefit of different regularization terms. Accuracy in terms of target registration error (TRE) is computed for the DIR-LAB dataset consisting of lung-CT scans from 10 subjects, and compared to results reported by other studies, where it outperforms deep-learning (DL) methods.
strengths: 1. The work evaluates INRs in the context of image registration, using only standard DL tools such as the PyTorch library and stochastic gradient descent, and demonstrates the use of INRs as a means of parameterizing deformable transforms.

2. The authors show that INRs can accurately represent transformations between lung-CT images from the same subject. Their approach achieves registration accuracy in terms of TRE surpassing the performance of DL methods reported in the literature.

3. The paper reproduces and confirms the superiority of SIREN over ReLU activation for representing dense transforms, previously demonstrated by Sitzmann et al. for images, wavefields, video, sound, and their derivatives.
weaknesses: 1. The comparison with DL methods, with the stated disadvantage of requiring “large training data sets”, appears inadequate. The appeal of DL methods is the ability to train on one set of images to later register unseen images from a different set, whereas INRs need to be learned separately for every new image pair. The DL-typical GPU-enabled libraries and stochastic gradient descent could be used in the same way to optimize a voxel-wise displacement field instead of an INR. As an alternative parameterization, INRs are not in competition with DL registration. Rather than spinning the work as a stand-alone registration method, it would be preferable to show if, and how, INRs can be as accurate as competing parameterizations - for an identical optimization strategy.

2. Parameterizing transforms with INRs is technically interesting, but some of the advantages presented in the introduction are not convincing. First, the benefit of image-size independence is unclear as the network needs to be retrained for every new image pair. Second, it is unclear how the ability to use a range of regularizers and to represent various deformation scales differentiates INRs from alternative parameterizations.

3. The evaluation is limited to images from 10 subjects and only lists TRE achieved by competing methods in the respective publications.
detailed_comments: The authors report that increasing network size improves performance, which is well known (see e.g. [1]). It would be interesting to analyze what capacity is needed to match the accuracy of competing parameterizations (likely not in the rebuttal).

[1] Hoffmann et al. SynthMorph: learning contrast-invariant registration without acquired images. IEEE Transactions on Medical Imaging (2021).
recommendation: Poster
review_title: null

final_score: rating: 5: Strong accept
review_content: summary: This is an exiting work that explores the possibility of rephrasing iterative medical image registration as an instance optimisation over neural network parameters that can represent a spatial transformation. The method can employ any mathematical regulariser without crude fixed-point approximations and the authors further extend the idea to higher order derivatives by using a periodic activation (SIREN). That way smoother deformations can be obtained with Jacobian-determinant or hyper-elastic regularisation. The approach is validated on the DIRlab 4DCT dataset is very accurate and has a reasonably runtime at inference (but it is still much slower than current DL-based or conventional algorithms).
strengths: - The concept is novel, convincingly motivated and a very nice contribution to MIDL
- The accuracy (and likely the deformation smoothness, zero negative Jacobian determinants are reported) is excellent outperforming most (if not all) DL-based registration approaches
- The paper is very well written, easy to follow and nicely illustrated.
- The method could be further extended using other cost functions (e.g. semantic features) and will be mode open source.
weaknesses: - I am missing some better intuition why a simple random sampling was performed. Lung images exhibit prominent geometric features (e.g. bifurcations) that could substantially reduce the required samples (cf Ruehaak).
- It would be good to discuss potential extensions w.r.t. to the network architecture. While it is impressive that a basic fully-connected network works well and can be realised with a reasonable number of trainable parameters, it could also be conceivable that a GraphCNN (e.g. Deep learning based geometric registration for medical images IPMI 2021) could work well in this scenario (where a sparse number of coordinate points is sampled). Currently the runtimes are pretty low (although that is usually not a problem).
- Some details about the implementation, in particular the metric, normalised cross correlation, are missing. What window size was employed?     
- A discussion of the relations to one-shot learning is missing: see e.g. "One Shot Learning for Deformable Medical Image Registration and Periodic Motion Tracking" T Fechter, TMI 2020
recommendation: Oral
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: This paper presents IDIR, a deformable registration method for medical images that iteratively optimizes a neural network as implicit representation of the deformation field per image pair. The authors build their work on recent advances in neural rendering and propse an approach to learn a continuous function that outputs the displacement for a given voxel coordinate of the input image.

By adopting the recent SIREN activation function instead of ReLUs, the network can be differentiated multiple times, which allows for straightforward implementation of regularizers that make use of higher-order derivatives. The authors further use three different regularizers to impose smoothness constraints on the displacement field.

The method is experimentally evaluated on 4D CT images from the DIR-LAB data set and is compared against different state-of-the-art methods. While the method does not outperform all comparisons, it does outperform recent deep learning based methods without the need for any prior training. Using SIREN activations and proper regularization, the determinants of the Jacobians suggest a diffeomorphic registration.
strengths: First of all, I really enjoyed reading this paper. It is well-written and easy to follow, although presenting a sophisticated method. The presented approach is interesting and a logical application of recent advances in neural rendering. The method is explained clearly and its beauty lies in its simplicity. Moreover, the method performs very well and does not need prior training on large data sets. The use of higher-order derivatives is especially interesting in the context of regularizers in deformable registration, which is exploited by the proposed method. The experimental evaluation highlights the potential of the method. I applaud the authors for boldly making a comparison with a method (isoPTV) that outperforms the approach presented. This helps a lot to judge the capabilities of the method. I appreciate the (anticipated) publication of the code.
weaknesses: While the application of implicit neural representations to registration is of high interest, the approach is not as radically novel as the authors claim. Laves et al. (2019) have already applied the very similar concept of Deep Image Prior to deformable registration and there exist some works that have used similar approaches for, e.g., CT reconstruction, inpainting, or denoising of medical images (Baguer et al., 2020; Tölle et al., 2021). I admit that there are some differences between Deep Image Prior and the coordinate-based interpolation used in the presented paper. However, the relevant prior work should properly be cited. Additionally, the presented method comprises very little novelty compared to the seminal works from neural rendering. I still think that the work contains enough novelty to be published, but the authors should clearly state their contribution.

The method is only evaluated on a single task and data set. The experimental evaluation could be more elaborate. Especially since the method does not require prior training. Other than that, the paper is valid and sound.

Laves, M. H., Ihler, S., & Ortmaier, T., 2019. Deformable medical image registration using a randomly-initialized CNN as regularization prior. MIDL 2019. arXiv:1908.00788.

Baguer, D.O., Leuschner, J., Schmidt, M., 2020. Computed tomography recon- struction using deep image prior and learned reconstruction methods. Inverse Problems 36, 094004.

Tölle, M., Laves, M. H., & Schlaefer, A., 2021. A Mean-Field Variational Inference Approach to Deep Image Prior for Inverse Problems in Medical Imaging. MIDL 2021.
detailed_comments: Some minor comments:

* Abstract: "a single three-layer multi-layer perceptron" should be rephrased.
* I don't understand the use of $ \omega $ and the weight initialization in § 2.1. This could be explained a bit more detailed or removed and refered to Sitzmann et al. (2020).
* I would appreciate a sentence about the intuition behind bending energy as it is done for the other two regularizers.
recommendation: Oral
review_title: null

Meta-review: metareview: All reviewers and myself appreciate the interesting idea here in combining the INR representation with registration, and appreciate the cleanliness of the paper and execution of the idea. The thorough discussion has also indeed improved the paper -- I appreciate the authors's thorough response and willingness to build on the feedback -- which is what we want to see from these discussions.

I would encourage the authors to explore the MLP-vs-DIP aspect a bit more in the future. I'm not really convinced that either is the optimal representation for a deformation field, and might even vary by task. It would be an interesting analysis and paper on its own.

Congratulations to the authors.


confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)