Multimodal Image Registration Guided by Few Segmentations from One Modality

Authors: başar  demir
marc  niethammer
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 18:45:48

Abstract: Registration of multimodal images is challenging, especially when dealing with different anatomical structures and samples without segmentations. The main difficulty arises from the use of registration loss functions that are inadequate in the absence of corresponding regions. In this work, we present the first registration and segmentation approach tailored to this challenge. In particular, we assume the practically highly relevant scenario that only a limited number of segmentations are available for one modality and none for the other. First, we augment our few segmented samples using unsupervised deep registration within one modality, thereby providing many anatomically plausible samples to train a segmentation network. The resulting segmentation network then allows us to train a segmentation network on the target modality without available segmentations by using an unsupervised domain adaptation architecture. Finally, we train a deep registration network to register multimodal image pairs purely based on predictions of their segmentation networks. Our work demonstrates that using a small number of segmentations from one modality enables training a segmentation network on a target modality without the need for additional manual segmentations on that modality. Additionally, we show that registration based on these segmentations provides smooth and accurate deformation fields on anatomically different image pairs, unlike previous methods. We evaluate our approach on 2D medical image segmentation and registration between knee DXA and X-ray images. Our experiments show that our approach outperforms existing methods. Code is available at https://github.com/uncbiag/SegGuidedMMReg.

Main paper: 
 introduction: 
 Image registration is a crucial medical imaging task in establishing correspondences between image pairs. It is commonly used for disease diagnosis (Khalil et al., 2018) , disease progression monitoring (Viergever et al., 2016) , and organ motion tracking (Fu et al., 2020) . At its core is the solution of an inverse problem to determine the unknown spatial transformation between a moving and a fixed image, such that the deformed moving image matches the fixed image well. Solving this inverse problem involves solving a possibly high-dimensional optimization problem. One can directly optimize over the parameters of a chosen transformation model (e.g., a displacement vector field) or over the parameters of a deep neural network which then indirectly produces the desired transformation by regression. Deeplearning approaches (Balakrishnan et al., 2019; Yang et al., 2017) are typically significantly faster than direct optimization approaches (Avants et al., 2009; Heinrich et al., 2014) and can achieve state-of-the-art (SOTA) accuracies (Tian et al., 2023; Mok and Chung, 2020) especially when combined with instance optimization.All of these methods aim to optimize a loss function that is a compromise between image pair dissimilarity and transformation irregularity. Multimodal image registration is still significantly more challenging than monomodal image registration because it requires image dissimilarity costs that can assess whether two images from different modalities differ or should be considered spatially well-matched. Further, Fig. 1 illustrates that the visibility of different anatomical structures across different modalities may cause a registration model to predict incorrect correspondences between different anatomical structures. We hypothesize that a registration model trained using supervision based on segmentations available for both modalities can assist the registration model in focusing on common regions of image pairs. However, this presupposes the availability of these segmentations. Our focus in this work is, therefore, to develop an approach that solves both image segmentation and registration for multimodal image pairs, assuming that 1) anatomies are not consistently visualized between the modalities (e.g., one might focus on bone visualization while another might visualize both bones and soft tissue) and 2) only a few manual segmentations are available in one modality and none in the other. In summary, the contributions of our work are:• We propose the first deep image registration approach for multimodal images where image pairs visualize different anatomical parts. Our approach is based on registering segmentations while only requiring a few segmentations from one modality and none for the other.• We test our approach for 2D knee datasets between DXA images which focus on imaging bone, and radiographs (termed X-rays in what follows) which visualize bones and soft tissue.• Compared to existing loss functions and approaches, our approach can reliably register DXA and X-ray image pairs and predicts smooth deformation fields on anatomically different image pairs. 
related work: 
 Image Registration. Image registration is a key tool of medical image analysis to estimate spatial correspondences between images. Recently, deep neural networks have enabled population-based registration approaches that require one-time training but then provide very fast inference (Balakrishnan et al., 2019; Miao et al., 2016; Greer et al., 2021; Tian et al., 2023; Greer et al., 2023) . However, registration performance still heavily depends on selecting a correct task-specific image similarity measure.For monomodal applications, mean squared error (MSE) (Fan et al., 2019; Hoopes et al., 2021; Kim et al., 2021) and normalized cross-correlation (NCC) (Kim et al., 2022; Tang et al., 2020) are popular image similarity measures. In contrast, multimodal registration cannot rely on matching image intensities directly. Hence, measures of statistical dependence such as NCC and Mutual Information (Wells III et al., 1996) are popular choices. Another option, Modality Independent Neighbourhood Descriptors (MIND) (Heinrich et al. , 2012), operates by comparing local image neighborhoods in both the moving and fixed images. Many other image similarity measures exist (e.g., structural similarity index (SSIM) (Wang et al., 2004) and normalized gradient fields (NGF) (Haber and Modersitzki, 2006) ). However, none of these similarity measures are usable for image pairs which show different anatomical parts (see Fig. 1 ). Specifically, registrations based on these similarity measures fail to find appropriate correspondences even between common anatomical structures when one of the images contains anatomical structures that are not visualized in the other image.Recent works (Xu and Niethammer, 2019; Canalini et al., 2019; Cai et al., 2022) show that segmentation can supervise registration and improve registration performance with better border alignment. Further, such supervision encourages segmentations to match after registration, ensuring the preservation of anatomical consistency. Inspired by these works, we propose a registration strategy that can find correspondences fully based on given segmentations tailored for anatomically different and multimodal image pairs. Domain Adaptation for Segmentation. Machine learning models often struggle with data that differs significantly from their training data. This is a major challenge in medical imaging, where datasets can vary greatly in modality (e.g., MRI vs. CT) even when imaging the same body region. Moreover, manual segmentations are typically available for only a limited number of images and modalities. Unsupervised domain adaptation (UDA) offers a solution to this challenge. UDA techniques adapt models trained on one domain (source) to perform well on a related but different domain (target) without requiring segmented target data. This is crucial for generalizing models in medical imaging, where acquiring large segmented datasets is expensive and time-consuming. Building on the success of adversarial image translation methods (Liu and Tuzel, 2016; Isola et al., 2017; Zhu et al., 2017) , several techniques have been proposed for pixel-level UDA (Murez et al., 2018; Pizzati et al., 2020; Li et al., 2020) . These methods initially establish a mapping function between the source and target domains, enabling inter-domain image translation.Recent works (Hoffman et al., 2018; Xie et al., 2018) enhance UDA by combining feature alignment and pixel-level adversarial training. In our approach, we use an CycleGAN (Zhu et al., 2017) unsupervised domain adaptation strategy with semantic losses at each step of translation (Gogoll et al., 2020) . These semantic losses ensure that modality-translated images not only appear to belong to the target image domain but also maintain consistent anatomical boundaries during image translation. This is an essential property for our registration approach. Further, the approach allows us to train a segmentation network on the target modality using only source domain segmentations. 
overview: 
 Our source datasetD S = {(I i )} i∈[n] includes at least one segmentation {(S i )} i∈[l] (1 ≤ l ≤ n).No segmentations are available for our target dataset D T = {(I i )} i∈ [m] . First, we augment the source dataset D S with a registration network (Sec. 3.2). Subsequently, we train the segmentation network S S for the source domain using the augmented dataset and freeze weights of the network. Next, we obtain a segmentation network S T for the target dataset with the guidance of S S via an unsupervised domain adaptation network (Sec. 3.3). We predict segmentations for the non-segmented part of the source dataset with S S and apply S T to the target dataset. Finally, we train a registration network solely based on the segmentations we obtained (Sec. 3.4). This registration network then enables us to register two images according to their predicted or provided segmentations.A non-parametric registration network predicts a displacement field u = R(I m , I f ; θ) between the moving image I m and the fixed image I f for given network parameters θ. We can warp the moving image to the fixed image as I w m = I m • Φ -1 , where Φ -1 = u + id is a transformation map, and id is the identity transform. Since the registration network can find correspondences between images without supervision, we use predicted transformation maps to create artificial samples by warping the images with segmentations to images without available segmentations. First, we sample moving image segmentation pair (I m , S m ) ∈ D S and fixed image I f ∈ D S and we obtain the transformation map Φ -1 between I m and I f . Next, we create a new sample by warping both the moving image I m and its segmentation S m asEQUATIONWarping both the image and its segmentation helps preserve semantic consistency between them. This artificially created sample, (I w m , S w m ), is anatomically similar to I f . When a limited number of segmented samples is available, we can use this approach to create additional samples. Sec. B.2 provides details of the implementation.Inspired by (Gogoll et al., 2020) , we use a CycleGAN (Zhu et al., 2017) -based domain adaptation network for medical image segmentation. We assume that we already have a pretrained segmentation network S S for the source domain. Our goal is to train a segmentation network S T for the target domain by transferring information from S S . We use an adversarial training procedure. Two generators, G ST and G T S , are designed to learn mapping functions from the source domain to the target domain and vice versa. Concurrently, discriminators D S and D T are responsible for distinguishing between real and translated samples. We integrate these generators and discriminators into the image translation process with adversarial loss (ℓ GAN ) and cycle consistency loss (ℓ cycle ) terms, following the principles of CycleGAN (Zhu et al., 2017) . However, it is important to note that this process does not ensure the semantic consistency between real and translated images, i.e., while translated images look realistic for a CycleGAN approach the synthesized image will not necessarily be spatially consistent, and anatomical boundaries might shift. This would, of course, be problematic for any subsequent registration. Hence, we incorporate semantic segmentation networks S S and S T to encourage the semantic segmentation of the images at each step of the cycle to remain consistent (Gogoll et al., 2020) . The work by (Gogoll et al., 2020) assumes that all segmentations are available for the source domain, and hence manual segmentations can be used for network training. However, in our case, only a few segmentations are available for the source domain. Therefore, we augment our source dataset with registration (see Sec. 3.2), and use all images in D S for training.To encourage consistent segmentation predictions at each cycle phase we use the lossesℓ sem-ST :=E (I S ,M S )∼D aug S L sem (S T (G ST (I S )) , M S ) + L sem (S S (G T S (G ST (I S ))) , M S ) , ℓ sem-TS :=E I T ∼D T L sem (S T (I T ), S S (G T S (I T ))) + L sem (S T (I T ), S T (G ST (G T S (I T )))) (2)where D aug S is augmented source dataset containing image segmentation pairs (I S , M S ). These losses enable us to train S T based on S S which was trained with only a few segmented samples. The total loss isEQUATIONIn summary, we use an unsupervised domain adaptation method for medical image segmentation (Gogoll et al., 2020) . In doing so, we jointly learn image segmentation and translation using a pre-trained segmentation network trained with only a few segmentations. Sec. B.3 provides implementation details. Instead of proposing a new registration approach, we use an inverse-consistent registration network (Tian et al., 2023) . Since our goal is to register anatomically different pairs, we register images based on their segmentations. Note that while our domain adaptation approach of Sec. 3.3 generates translated images and segmentations, we only utilize its capability to train a segmentation network for the target domain.Our image-segmentation pairs for moving and fixed images are (I m , S m ) and (I f , S f ). In our registration setting, the registration network R finds correspondences between images based on their segmentations. The registration network predicts the transformation map Φ = R (S m , S f ; θ) + id, where θ denotes the network parameters. We denote by Φ θ [S m , S f ] the transformation mapping the moving segmentation, S m to the space of the fixed segmentation, S f . During training, we register segmentations of image pairs, using gradient inverse consistency of the transformation maps as the regularizer and 1-Dice score on the segmentations as the similarity loss L sim . The training loss function isEQUATIONDuring inference, we first register the segmentation pair (S m , S f ) and obtain the corresponding transformation map Φ θ [S m , S f ]. Then, we use this transformation map to warp the real multimodal image pair (I m , I f ) asI w m = I m • Φ θ [S m , S f ].In summary, we use our pre-trained segmentation networks (see Sec. 3.2, Sec. 3.3) for inference. First, we predict the segmentations for a given image pair. Then, we predict the transformation map based on the predicted segmentation pair. Lastly, we warp the real images using this predicted transformation map. Thus, we preserve anatomical consistency while finding correspondences between images. Sec. B.4 provides implementation details. 
experiments: 
 We conduct experiments to evaluate the effectiveness of each step of our approach. We use 123 DXA samples provided by the UK Biobank (Sudlow et al., 2015) and 123 X-rays from The Osteoarthritis Initiative (OAI) (Nevitt et al., 2006) 1 . We resized each sample to 256 × 256 and manually segmented the femur, tibia, and fibula for 100 samples from each dataset. We performed affine pre-alignment on all samples to one random sample using the affine layers of (Greer et al., 2023) . See Table 2 for the dataset splits. Augmentation with Registration: We compare the effectiveness of our augmentation strategy with raw data (without any augmentation) and elastic augmentation (Ronneberger et al., 2015) . Our method outperforms elastic augmentation in most cases. Additionally, we investigate the impact of the number of segmented training samples on test performance. Table 5 (highlights in orange cells) shows that we achieve comparable performance to the segmentation network trained on 75 segmented samples using only 10 segmented samples for DXA (86.6% fewer) or 15 samples for X-rays (80% fewer). Additionally, our method does not require any parameters unlike elastic augmentation (e.g., number of control points and maximum displacement). Domain Adaptation for Segmentation: We evaluate our domain adaptation network in both the DXA to X-ray and the X-ray to DXA adaptation directions. We assume that we already have a pre-trained segmentation model on the source domain trained with varying numbers of segmented samples. We investigate the impact of different ways of creating the source dataset. We conduct experiments using the following approaches: 1) "Baseline," which only uses images with available segmentations; 2) "Baseline + Predictions," which additionally uses pre-trained network predictions for all available images without segmentation; and 3) "Baseline + Augmentation," which augments the source dataset with our augmentation strategy. We observe that augmenting the source dataset is the most reliable approach when the number of segmentations is less than 5. However, for 15 segmentations or more, all approaches converge to similar Dice scores.The domain adaptation network significantly improves performance on the target modality. The "Baseline + Augmentation" approach improves over directly using a segmentation network trained only on the source domain and then applied to the target domain ("w/o Adaptation"). For example, when only 5 segmented samples are available for the source domain, we observe a 7.1% increase in mean Dice score when comparing the "Baseline + Augmentation" and the "w/o Adaptation" approaches when adapting from the X-ray to the DXA domain, and an increase of 50.5% in Dice score when adapting from the DXA to the X-ray domain. Table 6 shows quantitative results, and we highlight the results discussed herein orange cells. Figures 7 and 8 provide visual comparisons. Registration: We compare our approach to deep registration approaches that differ with respect to their input (either images or segmentations) and the losses that were used during training (either using 1-Dice for segmentation-based losses or various image-based similarity measures). Specifically, we compare our registration approach with 1) An oracle approach that has access to manual segmentations for all images and uses manual segmentations as input and for the loss (to provide an upper bound on performance); 2) Multimodal registration where inputs are real images and our loss uses images; 3) Multimodal registration where inputs are real images and our loss uses images but only within manually segmented regions (Sec. C.1); 4) Multimodal registration where inputs are real images and our loss uses manual segmentations (Sec. C.2); 5) Monomodal registration where inputs are realfake image pairs generated by our image translation network, where loss uses images (Sec. C.3). Note that while our approach only assumes that few segmentations are available on the source domain, approaches 1, 3, 4, and 5 above may use manual segmentations for all images in the source and target domain to provide strong competing approaches.Table 1 shows that our approach achieves a mean Dice score of 95.7% with only 5 segmentations from the DXA domain or 96% with 10 segmentations from the X-ray domain, which is better than the non-oracle best-performing approach (5) that uses 150 manual segmentations in the training process and results in a Dice score of 94.6%. For more detailed results, please refer to Table 7 where we compare different approaches with varying training losses. Additionally, Figures 9 and 10 demonstrate that our method aligns bone borders with smooth deformation fields. 
conclusion: 
 We demonstrated that our proposed approach enables the registration of 2D multimodal image pairs visualizing different anatomical structures using a few segmentations from one modality. This work is essential when we have limited numbers of segmented samples for one modality and potentially none for other modalities. To the best of our knowledge, this is the first work addressing this particular problem. Our results showed that our approach using registration-based data augmentation, unsupervised domain adaptation, and segmentationbased registration outperforms existing multimodal registration approaches. Future work will explore 3D extensions to our approach (see Sec. A for a related discussion on the limitations of our approach). the MONAI Framework (Cardoso et al., 2022) setting channels to (4, 8, 16, 32, 64, 128) , strides to (1, 1, 1, 1, 1) and using 8 residual units. 
 
 Appendix: We implement generators using ResNet (He et al., 2016) , discriminators following the Patch-GAN (Isola et al., 2017) approach, and segmentation models using a Residual U-Net foot et al., 2019) that shares the same architecture introduced in Section B.2. We embed our source segmentation network, which is pre-trained with our augmentation strategy, into the domain adaptation architecture and freeze it. Additionally, we initialize the segmentation network for the target modality with the same weights. We set λ GAN to 1, λ cycle to 10, and λ sem to 5 and train the network for 400 epochs with a batch size of 1. We use 1-Dice as a semantic loss L sem .For our registration task, we prefer using a single-stage registration network (Tian et al., 2023) over a two-stage network. Since registering segmentations is an easier task than registering images, even a single-stage model can converge easily and achieve high performance.We set the learning rate to 10 -4 , batch size to 32, and use 1-Dice as a similarity loss for 5000 steps. During inference, we do not employ any instance optimization.We train our registration network for augmentation for ∼4.5 hours, the segmentation network on the source dataset for ∼1 hour, the UDA network for ∼3.5 hours, and the registration network for registering segmentations for ∼0.5 hours. We use one NVIDIA GeForce RTX 3090 for all of our experiments. EQUATIONWe use gradient inverse consistency of the transformation maps as the regularizer and 1-Dice score on the segmentations as the similarity loss L sim . During inference, this approach does not require any segmentations since it operates on image space.Table 7 shows that this approach (registration approach 4 of Sec. 4) performs worse when registering the fibula, even if we incorporate all manual segmentations for both modalities. We hypothesize that in contrast to a registration network that uses segmentations as inputs, a registration network that uses the acquired images directly will likely have to internally form some notion of a region of interest, essentially performing a form of latent segmentation. Hence, using the segmentations already as input is likely an easier task.In our approach, we use a CycleGAN-based image translation network to train a segmentation network on the target domain, and we use this segmentation network for registration. However, we can also use our image translation network to convert our multimodal registration problem into a monomodal one. To obtain a strong competitor to our approach, we assume that we have manual segmentations for all images of both domains in contrast to our approach.We train both a standard CycleGAN and a CycleGAN using manual segmentations for both modalities as semantic losses. Then, we translate all images in our training dataset to the same domain and create DXA & fake DXA and X-ray & fake X-ray image pairs. Based on these datasets, we train monomodal registration networks. During inference, we initially translate images to the same domain and then use these real-fake image pairs as the input to a monomodal registration network. Finally, we predict deformation fields.Table 7 shows that registration via a CycleGAN using manual segmentations for both modalities as semantic losses is better than using a standard CycleGAN. However, because registering based on images is still difficult, the registration accuracy of our proposed approach remains better. Therefore, in our approach, it is beneficial to use the segmentationguided CycleGAN to train the segmentation networks and work with the predicted segmentations.Figure 5 illustrates the moving, fixed, fake fixed, and registered images produced using the CycleGAN monomodal registration approach with semantic loss for several cases. Table 7 presents the Dice scores of the CycleGAN monomodal registration approach, comparing results with and without the use of semantic loss. In our registration setting, we register pairs based on their provided or predicted segmentations. Although registering based on segmentations aligns the borders of the bones well, it does not establish correspondences between the inner regions of the bones. To decouple registration problems due to visualization of different anatomical regions from registration problems due to image appearance differences for the different modalities within the same anatomical regions, we conducted an experiment by using images that only visualize common anatomical structures. We achieve this by masking the images of both modalities based on our manual segmentations. We thereby remove uncommon structures, in this case, soft tissue. This masking results in an artificial experimental dataset for multimodal registration that only shows anatomically similar regions. Then, we trained a registration network with several image similarity loss functions. Figure 6 illustrates predictions of grid-based displacement fields based on these different image similarity measures. We observe each loss causes the network to behave significantly differently for inner bone regions due to appearance differences between the DXA and X-ray images. Combined with the multi-layer structure of the knee (including the presence of the patella), this leads to some similarity measures to highly spatially irregular deformation fields.Figure 6 : Qualitative results for multimodal registration using different similarity measures.The DXA and X-ray images have been masked to obtain pairs showing the same anatomical structure. Due to appearance differences within the bones, registration results strongly differ between the different similarity measures. DXA images reproduced by kind permission of the UK Biobank ® .Our approach assumes that a small number of segmentations are provided for only one of the modalities. However, we also explore the applicability of our approach to a case where we have segmentations for both modalities. In this setting, the unsupervised domain adaptation phase is no longer necessary. Instead, we directly train segmentation networks for each modality with augmentation based on registration (see 3.2) for which we illustrated its effectiveness in Table 5 . Then, we register image pairs again with predicted segmentations as we do in our proposed approach.Table 3 compares two approaches: one that assumes segmentations are available from only one modality (ours) and another that utilizes segmentations from both modalities (ours w/o UDA). When the same number of segmentations are provided, our approach can outperform the approach without unsupervised domain adaptation (UDA) in some cases. For instance, ours reaches a 95.7% Dice score with 10 DXA segmentations and 96% with 10 X-ray segmentations, while ours w/o UDA with 5 segmentations for each modality only achieves a 94.2% Dice score. We observe similar results for the case of 20 segmentations: our approach reaches a 96.4% Dice score with 20 DXA segmentations and 96.5% with 20 X-ray segmentations while ours w/o UDA with 20 segmentations achieves a 96% Dice score. However, ours w/o UDA slightly outperforms with a 97% Dice score when 40 segmentations are provided, while ours reaches 96% with DXA and 96.8% with X-ray segmentations. These results demonstrate that our method, even when utilizing segmentations from only one modality, performs competitively compared to the approach that benefits from segmentations from both modalities. This highlights the effectiveness of unsupervised domain adaptation in our approach. Training a robust CycleGAN typically requires a large and diverse dataset. However, collecting such a dataset may be a challenging task. Under our experimental constraints, where we have a small number of images, this task becomes even more difficult. To investigate the robustness of our unsupervised domain adaptation approach, we run our experiments with 5 random seeds and report the mean and standard deviation of Dice scores for each class. Table 4 shows that our approach is reliable for domain adaptation, with the highest standard deviation being 0.028 for the worst-case scenario. Table 6 : Class-based Dice scores for unsupervised domain adaptation for segmentation. We test our networks on the target modality with a limited number of source segmentations. Additionally, as an oracle method, we trained and tested a segmentation network on the target modality with full supervision. We observe that Dice scores converge when around 15 segmentations are provided. The results of domain adaptation are competitive with the oracle method. The orange cells indicate the results discussed in the main text. This registration task is motivated by our desire to extract spatially localized biomarkers across multiple large-scale, but modality-diverse datasets, e.g., to enable imaging genetics studies for large sample sizes. 


Decision: Reviewers' initial critiques were mostly about experimental detail rather than the significance of the proposed method. The authors' feedback addressed those critiques, and all reviewers appreciated the authors' responses and confirmed that they clarified all their questions.

Reviews: review: paper_summary: paper_summary: The authors present a novel registration and segmentation approach tailored to the challenging scenario of dealing with different anatomical structures and the limited availability of segmentations. The authors assume that only a small number of labeled examples is available for the source domain, and no examples for the target domain. The proposed algorithm proceeds in three stages: First, the small number of labeled examples for the source domain is augmented using unsupervised deep registration. Second, a domain adaptation approach by Gogoll et all. is applied to train a segmentation algorithm on the target domain. Third, an inverse-consistent registration network by Tian et al. is used to register the segmentations for the source and target domain.
main_review: main_review: The authors propose a novel registration and segmentation approach tailored to the challenging scenario of the limited availability of segmentations for different anatomical structures. The approach utilizes unsupervised deep registration, domain adaptation, and inverse-consistent registration to register image pairs from different modalities. Despite leveraging existing techniques, the authors provide extensive ablation studies in the appendix, demonstrating the effectiveness of their proposed approach.
strength_weakness: strength_weakness: The paper leverages and combines multiple existing techniques as described and cited in the manuscript (Grogoll et al. and Tian et al.'s work in particular). This may reduce the novelty of the overall approach.
questions: questions: Could you provide more details on the robustness of training the CycleGAN? 
Will the code be made publicly available upon publication?
Can you give more details on the computational cost of training this method?
paper_summary: paper_summary: This paper tackles the complex issue of unsupervised multimodal image registration with an innovative approach that leverages a small set of labels available for one modality. The proposed method involves a three-step pipeline. Initially, a segmentation network is trained on the modality with labels, utilizing an augmented training and label the entire dataset. Subsequently, an image-to-image translation network is trained to convert images between the two modalities, incorporating an image segmentation objective to maintain anatomical accuracy. This network facilitates the generation of labels for the second modality. The final step involves training an unsupervised image registration network using these generated segmentation masks, making the registration process modality-independent. The methodology is validated on knee images from two distinct modalities.
main_review: main_review: Having read the updated manuscript, I have no further questions at this time. I appreciate the availability of the source code. However, the paper lacks sufficient novelty to justify a 'strong accept,' so I will remain with my previous decision of 'weak accept.' Great work!
strength_weakness: strength_weakness: The paper could be strengthened by including experiments on a broader variety of images, such as 3D brain scans, to demonstrate the method's applicability across different anatomical regions. Additionally, the rationale behind using only labels for registration, rather than directly inputting the original images for a more authentic multimodal registration, remains unclear to me.
questions: questions: Could you clarify if there was any experimentation using the original images as inputs for the registration network, and the labels only for optimization? This clarification might address potential confusion regarding the chosen methodology.
paper_summary: paper_summary: This paper introduces a novel pipeline for multimodal image registration, particularly effective when limited segmentation data is available for one modality. The authors hypothesize that a registration model, when trained with supervision from segmentations across both modalities, can enhance focus on common image regions, thereby improving registration accuracy.
Proposed Pipeline Overview:
1.	Augmenting modality/dataset A with segmentations using a registration network
2.	Train segmentation network on dataset A
3.	Train a segmentation network on modality B using guidance an unsupervised domain adaptation network
4.	Train registration network which registers segmentations of modality A and B

The authors apply their pipeline on the task of registering 2D knee DXA and X-ray image pairs showing promsing results on this dataset. However, the experimental setting is not completely clear.
main_review: main_review: The authors provided a detailed and clear explanations during the rebuttal. Integrating this information into the original manuscript will undoubtedly enhance its quality and improve its comprehensibility.  On the condition that the explanations and improvements are incorporated into the original manuscript, I recommend that the paper be accepted.
strength_weakness: strength_weakness: -	The clarity of the experimental design is lacking, with a disconnect between the experiments listed in Table 1 and their detailed descriptions, complicating the understanding of the methodologies employed (see detailed comments)
-	For the presented registration application of 2D knee DXA and X-ray images, no information is provided why those images need to be registered.
-	The necessity for the proposed pipeline, specifically for the chosen modalities, is not adequately justified. The use case appears somewhat contrived, with an unclear rationale behind the specific choice of segmentation and training approach, raising questions about the necessity of the unsupervised domain adaptation. 
-	The discussion on the pipeline's limitations is notably brief and could benefit from a more detailed exploration.
questions: questions: -	Please rework the experiment section to make it easier to follow.  
-	Please add the experiment restricting the similarity measure to the available structures on the fixed image. This experiment should be early implemented and don’t take too much time. 
-	Please address as many points as possible from the detailed comments.
-	Add a more intensive discussion on limitations of the work.

Comments: title: Thanks to Reviewer J4kr
Thanks to Reviewer vdWQ
Kind Reminder to Reviewer J4kr
Kind Reminder to Reviewer vdWQ
Thanks to Reviewer qPPp
Answer to Reviewer qPPp - continued
Answer to Reviewer qPPp - continued
Answer to Reviewer qPPp
Answer to Reviewer J4kr
Answer to Reviewer vdWQ
comment: Thank you for your reviews and kind feedback! Your suggestions have helped us improve the clarity of our work. We appreciate your time in reviewing our paper.
Thank you for your reviews and kind feedback! Your suggestions have helped us improve the quality of our work. We appreciate your time in reviewing our paper.
Thank you for addressing my questions and making the source code publicly available! I appreciate your answers.
Thank you very much for revising the manuscript and addressing my questions. Having read the updated manuscript, I have no further questions at this time. I also appreciate the availability of the source code.
Thank you again for sharing your valuable feedback and comments with us. As the discussion period ends soon, we kindly request that you let us know if our rebuttal and the updated manuscript have addressed your concerns. We would love to clarify and explain any remaining concerns you may have.
Thank you again for sharing your valuable feedback and comments with us. As the discussion period ends soon, we kindly request that you let us know if our rebuttal and the updated manuscript have addressed your concerns. We would love to clarify and explain any remaining concerns you may have.
Thank you for your detailed reviews, feedback, and raising the score. Your insightful suggestions have helped us improve the clarity and quality of our work. We truly appreciate your time and patience in reviewing our paper.
Thank you very much for providing such detailed and clear explanations. Integrating this information into the original manuscript will undoubtedly enhance its quality and improve its comprehensibility. Additionally, the rebuttal has effectively addressed all of my outstanding questions. Consequently, I am adjusting my evaluation to a weak accept.
**- Please add the experiment restricting the similarity measure to the available structures on the fixed image.** 

Thank you for suggesting an additional experimental setting, we added this scenario to our experiments, please see Appendix C.1 for more details. Table 7 shows that our approach, with 3 segmentations from either the DXA or X-ray modality, outperforms this approach (registration approach 3 of Sec. 4) trained with several similarity losses while using 75 segmentations from each of the modalities. Therefore, optimizing over images, even if constrained by segmentations, is not the best way for this registration task.

**- A discussion on the practicality of manual segmentation in real-world scenarios would enhance the paper's relevance. Please give examples where this is the case and why it is not an option to annotate on each modality a few samples and only use your registration augmentation.**

Our approach is motivated by registering many different datasets. Where images can come from different modalities, may have been acquired on different scanners, or with different imaging protocols. With the ultimate goal of supporting the pooled analysis of several large-scale studies (such as data from the UK Biobank, the Osteoarthritis Initiative, the Multicenter Osteoarthritis Study (MOST), and the Johnston County Study for example). All these studies are longitudinal and during such studies (e.g., for MOST) imaging protocols may also change over time as new, improved imaging capabilities become available. While it is true that one could segment all relevant structures on all different kinds of images, our goal was to develop an approach that could ultimately be used much more flexibly in such a scenario without manual (2D and 3D) segmentations for each image type.

**- To me, UDA is not necessary in this example. In which cases it is necessary? Revisiting the hypothesis in the discussion to explicitly address its validation or rejection based on the experimental results would strengthen the paper's narrative.**

Taking into account your valuable feedback, we now also conducted an ablation study where we assumed that a small number of segmentations are provided for both modalities. As you suggested, in this setup, we no longer need unsupervised domain adaptation (UDA), and our augmentation strategy is sufficient for training segmentation networks for both domains. In Appendix C.5, we reported the registration performance of our strategy without the UDA phase.

We observe that our approach can outperform our approach without unsupervised domain adaptation (UDA) when the same number of segmentations are provided in some cases. For instance, ours reaches a 95.7\% Dice score with 10 DXA segmentations and 96\% with 10 X-Ray segmentations, while ours w/o UDA with 5 segmentations for each modality only achieves a 94.2\% Dice score. We observe similar results for the case of 20 segmentations as well. However, ours w/o UDA slightly outperforms with a 97\% Dice score when 40 segmentations are provided, while ours reaches 96\% with DXA and 96.8\% with X-ray segmentations. These results demonstrate that our method, even with segmentation from only one modality, shows competitive performance compared to the approach that benefits from segmentations from both modalities and highlights the effectiveness of unsupervised domain adaptation.
**- The discussion on the pipeline's limitations is notably brief and could benefit from a more detailed exploration.**

Limitations are explained in the Appendix A. We are now also discussing computational cost in the paper. The main limitation of our approach is that it only focuses on bone shape (via the segmentations) and does not attempt to register the bone interiors based on image information. The primary reason for this limitation is the multi-layered structure of the knees. Since we are working with 2D images that are projected versions of 3D volumes, they are insufficient to accurately capture anatomical structures. Therefore, we have chosen to register only the bone shapes using predicted segmentation masks. Additionally, our approach consists of several phases, and the success of the registration depends on the performance of each trained intermediate model. Although we have demonstrated the robustness of unsupervised domain adaptation (UDA) in our case (see Appendix C.6), UDA may perform poorly in other more complex domains, which may in turn lead to inaccurate registrations or even registration failures.

**- Confusion about the choice of the regularizer and the architecture across the experiments and what ours refers to?**

We are sorry about the confusion. Yes, all GradICON results refer to the approach by (Tian et al., 2023) and we use the same architecture and regularization for all of our experiments. We only change input types and how we measure the similarity. Ours indicates using the GradICON registration model with segmentations as input which are obtained via our unsupervised domain adaptation approach. To prevent any future confusion, we have redesigned Table 1 and Table 7.

**- For 3), there are multiple rows given in the table, e.g. Real “DXA - Fake DXA 75 75 GradICON + SSIM “ – however, it is not completely clear what the 75 segmentation masks have been used for.**

For our approach, we use the CycleGAN to obtain segmentations which we then use for registration. Here, manual segmentations are only provided for one domain and then used in the cycleGAN to provide a semantic loss to train segmentation networks for **both** modalities. In the Real DXA - Fake DXA experiments we explore the registration quality when converting the multi-modal registration problem to a monomodal registration problem by image translation via a CycleGAN. To obtain the strongest comparable competitor to our approach we assume here that we have manual segmentations for **both** domains (which is precisely what our approach tries to avoid). We see that registration via a CycleGAN using manual segmentations for both modalities as semantic losses is indeed better than using a standard CycleGAN (without using the manual segmentations during training). However, because registering based on images is still difficult, the registration accuracy of our proposed approach remains better. In effect, we show that for our setting it is beneficial to use the segmentation-guided CycleGAN to train the segmentation networks and then work with these segmentations; the synthesized images are only a byproduct for us that we do not end up using. We also discussed this approach in detail in the **Appendix C.3**.

**- Does 4) correspond to ‘ours’ in the table? If so, it doesn’t fit because in 4) the images are used as input and the network is guided by the segmentation masks in the penalty, whereas ‘ours’ uses the ‘predicted segmentations’ as input.**

Thanks for pointing out this potential source of confusion. Ours refers to our proposed approach and it does not correspond to 4. We reworked the experiment section and tables to make the flow more clear.

**- “Manual segmentations are used as an oracle model for registration.” – Why is this experiment only performed with the baseline registration method and not with the suggested method or is the actual registration method the same (see above)? Using the manual segmentation in the proposed pipeline would give an upper bound on the registration performance.**

All baseline approaches and our approach use the same registration network and regularizer. When segmentations are already available (i.e., the setup of the Oracle experiment), we do not require the augmentation and UDA phases which are meant to train a segmentation network. The Oracle model uses gradICON in the same way as all the other approaches we compare to. The only difference is that we assume manual segmentations are available during the inference. I.e., segmentations for registrations are not provided by a trained segmentation network.
We thank the reviewer for the time, advice, and feedback. We address the main questions and issues below:

**-The clarity of the experimental design is lacking, with a disconnect between the experiments listed in Table 1.**

We agree, and thank you for pointing out the clarity issue. We have edited the methodology section to enhance understanding and readability. Also, we added numbers to the tables to make it easier to match text with results. Here is a summary of our baseline approaches and the questions that we asked ourselves while designing these approaches:

1. **An oracle approach that has access to manual segmentations for all images and uses manual segmentations as input and for the loss:** We assume all manual segmentations are provided and perform registration based on these segmentations. This experiment tests the ideal scenario with correct segmentations on both modalities, aiming to achieve an upper bound of registration accuracy achievable with high-quality segmentations. **Question:** What level of registration accuracy can be achieved using manual segmentations for registration?

2. **Multimodal registration where inputs are real images and our loss uses images:** This baseline approach tests common registration practices where the network attempts registration based on a penalty term applied to intensity values. We tested various registration losses to observe their behavior when image pairs depict different anatomical structures. **Question:** What level of registration accuracy can be achieved using common registration practices that work directly with images?

3. **Multimodal registration where inputs are real images and our loss uses images but only within manually segmented regions:** Thanks to your valuable suggestion, we now also conducted additional experiments by constraining similarity measures to available structures in one modality. Here, we optimize our network by calculating the loss only over segmented regions of the images. This is a form of loss function masking. **Question:** What level of registration accuracy can be achieved if one knew exactly where to look for the structure of interest? 

4. **Multimodal registration where inputs are real images and our loss uses manual segmentations:** We investigated using images as inputs and optimizing the network with their corresponding segmentations, testing the necessity of using segmentations as inputs. **Question:** What level of registration accuracy can be achieved by penalizing the network with segmentations while providing real images as inputs?

5. **Monomodal registration where inputs are real-fake image pairs generated by our image translation network, where loss uses images:** Here, we convert our multimodal problem into a monomodal setting using cycleGAN and an additional semantic loss. We generate image pairs consisting of a real image and a fake image translated to the modality of the real image, exploring the registration of artificially created monomodal image pairs. **Question:** What level of registration accuracy can be achieved by converting our multimodal image registration task to a monomodal setting with image translation?

6. **Ours:** It refers to our full proposed approach. We first augment the source dataset, then train a segmentation network on the source dataset, then perform unsupervised domain adaptation to obtain a segmentation network for the target dataset, and finally register images based on predictions of the segmentation networks.

**-For the presented registration application of 2D knee DXA and X-ray images, no information is provided why those images need to be registered.**

Our goal is to support large-scale imaging genetics studies. To obtain large sample sizes, pooling different studies is useful. However, these different studies may not use the same kind of imaging. For example, the Osteoarthritis Initiative provides knee radiographs whereas the UK Biobank only has DXA images. By being able to register these images (ultimately to a common atlas space) we will be able to extract image biomarkers that are *spatially corresponding* which we will then be able to relate to gene expression. We will add this motivation to the final paper.

**-The necessity of the unsupervised domain adaptation.**

Since we are training based on segmentations and we do not have segmentations for one of the modalities, UDA is necessary. Please see the answer above as to why we are targeting this particular registration problem.
We thank the reviewer for the time, advice, and feedback. We address the main questions and issues below:

**-The paper could be strengthened by including experiments on a broader variety of images, such as 3D brain scans.**

We agree. This is a first step and we are planning to extend our approach to 3D in future work.

**-What is the rationale behind using only labels for registration, rather than directly inputting the original images for a more authentic multimodal registration?**

The rationale for using segmentations only is based on the specific DXA/radiograph registration task. As these are projection images we are dealing with layered anatomical structures. Hence, we opted for registering the bone shapes only. However, our approach is general and could also be used with images instead of segmentations as inputs. This is what we are planning in future work for 3D registration where projection issues are no longer present. 

**-Could you clarify if there was any experimentation using the original images as inputs for the registration network, and the labels only for optimization?**

Note that Table 1 (Real DXA - Real X-ray - 75 75 - Dice) refers to the case where we pass real images to the network and penalize them based on their segmentations. This scenario is also mentioned as “*4) Multimodal registration where inputs are real images and our loss uses manual segmentations*” in the experiments section. Additionally, we have now provided a more detailed explanation in **Appendix C.2**. In this scenario, we observe that the registration network performs significantly worse when registering the fibula, even if we incorporate all manual segmentations for both modalities. Our hypothesis is that in contrast to a registration network that uses segmentations as inputs, a registration network that uses the acquired images directly will likely have to internally form some notion of a region of interest, essentially performing a form of latent segmentation. Hence, using the segmentations already as input is likely an easier task.
We thank the reviewer for the time, advice, and feedback. We address the main questions and issues below:

**-Details on the robustness of training the CycleGAN.**

We have conducted additional experiments to demonstrate the robustness of our domain adaptation network by training it with **5 different seeds**. The mean and standard deviation of Dice scores are now reported in **Appendix C.6**. Based on these results, our approach is reliable for domain adaptation, with the highest standard deviation being **0.028 in the worst-case scenario** (where only 1 manual segmentation is provided). With the incorporation of 15 segmentations, the standard deviation is as low as **0.004**.

**-Will the code be made publicly available upon publication?**

Yes, it will be publicly available at: https://github.com/uncbiag/SegGuidedMMReg. We also added this link to the abstract.

**-Details on computational cost of training this method.**

We train our registration network for augmentation for $\sim$4.5 hours, the segmentation network on source dataset for $\sim$1 hour, the UDA network for $\sim$3.5 hours, and the registration network for registering segmentations for $\sim$0.5 hours. We use one NVIDIA GeForce RTX 3090 for all of our experiments. We also added this information to the paper in Appendix B.5.