Upscaling Prostate Cancer MRI Images to Cell-level Resolution Using Self-supervised Learning

Authors: yaying  shi
yonghong  yan
Venue: MICCAI_org_2024_Workshop_COMPAYL
Published: 2024-06-30 03:05:54

Abstract: Magnetic Resonance Imaging (MRI) plays a pivotal role in medical imaging, particularly in the diagnosis and treatment of cancers via radiography. However, one of the limitations of MRI is its low spatial resolution, which can hinder the accurate detection and characterization of cancerous lesions, especially those that are small or subtle in nature. There is a growing need for advancements in MRI technology to improve the resolution of MRI, particularly in the field of oncology, where precise detection and segmentation of tumors are crucial for effective treatment planning and optimal patient outcomes. In this paper, we proposed a self-supervised deep learning technique to upscale cancer MRI images to cell-level resolution with pathology Whole Slide Imaging (WSI). By integrating information from pathology WSIs with MRI images, this approach aims to create hybrid images that offer a more detailed and comprehensive view of cancer tissue structures. We evaluated our techniques using prostate lesions both on the similarity metrics and downstream segmentation tasks. For the similarity, our reconstructed fusion images can achieve an average 0.933 in structural similarity index. We improved lesion segmentation dice score from 57.3% to 64.0% on the test cases. Such fusion of the two imaging modalities shows promise for improving the accuracy and reliability of cancer diagnosis, guiding treatment decisions, and ultimately improving patient outcomes.

Main paper: 
 introduction and background: 
 Radiological imaging constitutes a cornerstone in the study of cancer, spanning critical stages from foundational research to diagnostic elucidation, therapeutic strategizing, and ongoing surveillance. Modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) furnish intricate depictions of internal anatomical structures, affording clinicians invaluable insights into tumor localization, metastatic dissemination, and anomalous tissue proliferation.The interpretation of radiographic imagery presents challenges, particularly in distinguishing between malignant and benign tissue, which can be subjective even for experienced experts. Manual demarcation of cancerous lesions on radiological scans, though essential, often introduces inaccuracies that may underestimate tumor dimensions or miss less con-spicuous lesions due to low image resolution. In contrast, pathology whole slide images (WSI) offer gigapixel-level resolution, providing pathologists unprecedented insight into cellular morphology, tissue architecture, and aberrant features with exceptional precision. By aligning histopathology images with corresponding radiological slices, clinicians can overlay cancerous regions identified from histopathological analyses onto radiological scans. This approach enables precise tumor segmentation, including lesions not easily visible on MRI scans, thereby enhancing cancer evaluation.However, fusing images from different modalities presents technical hurdles due to inherent disparities in resolution, particularly when integrating images of large resolution gaps such as pathology and radiology images. Currently, image registration techniques are primarily developed for medical images with similar resolutions, such as PET, CT, and MRI scans Hering et al. (2022) ; Wang et al. (2020); Sokooti et al. (2017) . These methods typically fall into two categories: traditional approaches, which often suffer from low computational efficiency, and machine learning-based methods. The machine learning-based methods are limited in their applicability as they tend to work only on specific datasets and struggle to extend to larger resolution gaps, such as between MRI and pathology WSI. Furthermore, these machine learning approaches face challenges due to the lack of paired and well-labeled data available for training purposes.To address the limitations of existing machine learning-based registration methods and enhance the generation of fusion images with improved diagnostic capabilities, we present a novel self-supervised learning framework specifically designed for the registration of radiological and pathological images. This is achieved through the utilization of a self-supervised transformer-based feature extraction network and a feature-matching network. Our approach is able to enhance the resolution of MRI images to the cell level using pathology data, with the registration process serving as a crucial tool in achieving this goal. By leveraging advanced self-supervised learning, our framework overcomes the constraints of current registration methods which need multiple paired and well-labeled data. Our contributions be summarized as follows:1. We developed a novel image fusing network for upscaling prostate MRI images to celllevel resolution, leveraging registration as a key tool. The framework enables the creation of fused images that seamlessly integrate high-resolution pathology data with MRI scans. We have achieved 39 times resolution enhancement between the original MRI and the new fusion image. Using the fused images for downstream tasks such as cancer segmentation, it can offer enhanced details and accuracy crucial for improving diagnostic capabilities.2. We evaluated our framework using prostate cancer datasets, comparing it with the original paper that presented those datasets Shao et al. (2021) . Our framework demonstrated an enhancement in accuracy from 56.3% to 64.6% for prostate cancer. We also evaluated the similarity of our new fused image and up-scaled MRI image by Structural Similarity Index (SSIM). We achieved an average SSIM of 0.939 and a minimum of 0.933, suggesting that our new fused images have high similarity to the MRI images.3. We tackled the issue of insufficient labeled or paired pathology and radiology images using self-supervised learning, which enables model learning using unlabeled datasets. The efficiency of the pre-trained self-supervised learning can be seen in the Appendix C. 
proposed framework: 
 For medical images, which are multimodal in nature, fusion and registration of images of different modality have been well studied topics. Our method advanced the state-of-theart by using self-supervised learning, which alleviates the dependency on labeled data by leveraging the inherent structure and redundancy within the data itself. The self-supervised learning framework is inspired by DINO Caron et al. (2021) and Prosregnet Shao et al. (2021) . In contrast to DINO/Prosregnet that are mainly used for image classification, ours is an image fusion network for images that have high-resolution gaps, such as the tissue-level MRI and the cellular-level pathology images, as illustrated in Fig. 1 . The key component of the network includes 1) Two self-supervised feature extractors, each devoted to MRI and whole-slide pathology images respectively. 2) A correlation mapping block responsible for generating correlation maps of features extracted by the previous feature extractors. 3) A feature-matching sub-network designed to align and map distinctive features extracted from both image types. 4) Post-processing techniques for fusing pathology and MRI patches based on the correlation maps obtained in the previous step. In total, we have 82 paired MRI and Pathology WSI from this dataset. The second dataset Madabhushi (2016), referred to as fused MRI-Prostate, comprises 28 cases, each containing 3 Tesla T1-weighted, T2-weighted, Diffusion weighted, and Dynamic Contrast-Enhanced prostate MRI scans, accompanied by corresponding digitized histopathology (H&E stained) images of radical prostatectomy specimens. For training purposes, we utilized all images from the first dataset and 26 cases from the second dataset. The remaining two cases, along with 6 slides annotated with lesions, were reserved as the test dataset. The first dataset was not used for the test cases due to the absence of lesion annotations necessary for evaluating downstream segmentation tasks. Given the limited size of the datasets, comprising only 82 paired data points, we relied on machine learning-based methods. To mitigate the data limitations, we extended the training data by incorporating additional unlabeled MRI and pathology images from TCIA Archive and TCGA for Cancer Genomics, respectively. Each pre-training dataset consisted of 500 cases, enabling us to pre-train a feature extractor in a self-supervised manner.In the initial stage, we trained the feature extractor using unlabeled MRI and pathology image data. We subdivided the acquired WSI from TCGA and MRI scans from TCIA into smaller patches measuring 256 * 256 pixels. For the subsequent step, we trained our feature matching subnet utilizing two paired MRI/pathology datasets. We standardized the dimensions of these datasets to match those of our pre-trained feature extraction data. Because the data of the two datasets are paired, corresponding MRI slides and pathology images were aligned, as depicted in the P athologyImage and RadiologyImage sections of Fig. 1 . Furthermore, when performing downstream tasks on both pathology slices and MRI scans, such as lesion annotation, we applied the same methodology. This ensured alignment and correspondence between the annotations, as illustrated in the RadiologyImage section of Fig. 1 .The first component of our framework comprises a self-supervised feature extractor inspired by DINO Caron et al. (2021) . DINO is a self-supervised learning method for visual representation learning. It achieves cutting-edge performance by aligning representations of the same image across different layers of the neural network through self-distillation. In this paper, the initial phase involves pre-training a pathology and MRI feature extractors by leveraging the DINO framework. Given the disparity in resolution and format between MRI and pathology images, we undertake separate pre-training processes for each modality. Consequently, we develop distinct feature extractors tailored to MRI and pathology images, as illustrated by the two feature extractors depicted in Fig. 1 .To optimize performance, we leverage pre-trained weights obtained from the TCIA dataset for both the student and teacher networks, as outlined in Caron et al. (2021) . Given the larger size of pathology images compared to natural images, we decompose each image into smaller patches, each measuring 256 × 256 pixels. These patch images are fed into both the student and teacher networks, which share identical network structures. The networks employed standard vision transformer blocks, depicted in the feature extractor block at the bottom of Fig. 1 , featuring components such as Layer Normalization, multiheaded self-attention, residual connections, and a multi-layer perceptron. We adopted the ViT-small as the backbone for our feature extractor, maintaining the parameter settings as per Caron et al. (2021) . During the pre-training phase, the teacher network is frozen with weight updates solely in the student network. A distillation loss across the teacher-student predictions is imposed to train the self-supervised framework (see Fig. 1 ). The parameters in the teacher network is updated using Exponential Moving Average (EMA).Our feature-matching network, inspired by the methodology outlined in the work of Shao et al. Shao et al. (2021) , consists of two main components: correlation mapping and feature matching. Initially, we obtained features for both MRI and pathology images from two separate feature extractors. Each feature map, denoted as f, represents an image with dimensions (w, h, d), where d represents the number of features, w represents the width, and h represents the height. Subsequently, the feature maps f A and f B were downsampled into a smaller dimension representation to reduce computational costs. These downscaled feature maps were then input into a correlation layer, which computes the dot product of input features to quantify the similarity between the two images. This correlation layer combines f A and f B to generate a correlation map C ab of the same size. The computation of the correlation map is expressed as:EQUATIONThe equation k = h(j k -1) + i k is used to calculate the index variable k based on the indices i k and j k , with h representing the width of the feature map. The resulting correlation map C ab indicates the similarity of features from f b at position (i, j) with all features from f a . To address potential ambiguous matches, normalization is applied to obtain the correspondence map f ab . This map then undergoes processing by a feature-matching network, which is a regression network responsible for estimating the parameters of the geometric transformation associated with the input MRI and pathology images. Following the architecture described in paper Shao et al. ( 2021), the regression network comprises two layers. Each layer begins with a convolutional unit, followed by batch normalization and ReLU activation. A final fully-connected (FC) layer performs the regression of parameters for the geometric transform, outputting the affine matrix θ. The matrix θ serves as the affine transformation for the registration process.The loss function was determined as the sum of squared differences (SSD) between the original input MRI and the deformed pathology image. The formula of the loss function was shown as:loss = H i W i ∥I A (i, J) -I B (i, J) • ϕ Θ (i, j)∥ 2 (2)where ϕ Θ (i, j) is the related transformation vector from the output of feature matching sub-network, H is the height of the image and W is the weight of the image.After completing the image registration process, the pathology images, MRI scans, and annotated lesions were aligned with the corresponding MRI slices using the estimated composite affine transformation θ. It is important to note that pathology images typically have larger dimensions than sliced MRI images, which are usually 512 × 512. Consequently, the deformed pathology images maintain their original size as high-resolution images, as the affine transformation is only applied to the original image. Once the affine transformation matrix θ is determined based on the input image, it remains fixed. During inference for high-resolution pathology images in gigapixel scale, we first resize them to the same smaller size as the MRI slides. These resized images are then fed into our network to obtain the transformation matrix θ. Finally, we apply θ to the original resolution pathology image to obtain the registered high-resolution pathology image. 
evaluation: 
 In this section, we demonstrate the effectiveness of our proposed registration framework on two prostate cancer datasets Madabhushi A (2018). Our evaluation consists of three parts. In the first part, we evaluate the fusion image from the resolution comparison, and Structural Similarity (SSIM). In the second part, we will display some qualitative results for our fusion image. In the last part, we will have a quantitative evaluation of downstream lesion segmentation tasks.We evaluated the difference between the fusion image and the original MRI both for resolution enhancement, and SSIM on the prostate cancer dataset Madabhushi A (2018).The resolution enhancement from the original MRI to the fused images for each test case is from 320*320 MRI to a resolution of 2000*2000, which is 39x times higher resolution.We also evaluated the structural similarity of the fusion image and MRI image by Structural Similarity Index (SSIM). SSIM is a metric used to measure the similarity between two images. The calculation of SSIM is located at Appendix B. The SSIM value ranges from -1 to 1, where a value of 1 indicates perfect similarity between the two images. The result of SSIM in all six cases is shown in Fig. 2 . As the value shows, the lowest SSIM score we have for aaa0069 is still 0.933 which is close to the max SSIM value which is 1. The average SSIM is 0.939. That suggested that our fusion image has high similarity with the up-scaled MRI in both luminance, contrast, and structure.3.2 Quantitative and Qualitative Result on Downstream Segmentation TaskWe assessed the quantitative results for lesion segmentation using two metrics: the Euclidean distance and the Dice score, applied to MRI and registered pathology images across six test cases. A higher Dice score indicates increased similarity, while a lower Euclidean distance(ED) signifies reduced separation. In this evaluation, lesions manually annotated on both MRI and pathology images served as landmarks for assessing the segmentation quality. These lesion annotations were solely used for evaluation purposes and were not We present the visualization of our registered results in Fig. 3 . The figure showcases the following components: the upscaled MRI in section (a), the pathology image with the related mask in section (b), the fusion image in section (c), and the lesion labels in section (d).Each row corresponds to a visualized patient, with cases aa0069 Cslides, aa0069 Dslides, and aa0069 Eslides displayed from top to bottom, respectively. The lesion labels were annotated by an expert pathologist reported on the original dataset. We applied the same transformation to the labels and registered them with the fusion image, as illustrated in sub-figures (d) and (e).  
conclusion, limitation and future work: 
 In this paper, we proposed a registration framework for aligning radiological and pathological images. Through our experimental analysis, we have demonstrated the capability of self-supervised learning approach to bridge the resolution gap between these two modalities, facilitating accurate registration without the need for labeled training data. The fusion of high-resolution pathology data with low-resolution radiological scans have shown great promise in enhancing the diagnostic potential of medical imaging, particularly in tasks such as cancer segmentation. However, it is important to acknowledge the limitations of our work. Our experiments were conducted on limited datasets, and the fused high-resolution image was not directly generated but created through the registration process. These limitations highlight areas for future research and improvement. Moving forward, further research and validation studies will be essential to validate the clinical utility and robustness of our framework across diverse datasets and clinical scenarios. 
 
 Appendix:  



Reviews: review: title: title: Review for "Upscaling Prostate Cancer MRI Images to Cell-level Resolution Using Self-supervised Learning"
main_review: main_review: The authors propose a novel self-supervised deep learning framework that integrates pathology WSIs with MRI images, creating hybrid images with detailed cancer tissue structures. The multi-modal pathology/MRI learning is an underresearched area and there are some interesting ideas in this paper. 

However, there are multiple shortcomings to the study overall:

1)There is no comparison to any baseline or state-of-the-art.

2)Would be useful to also report the Euclidean distance between landmarks of anatomical structures i.e urethra and not only lesions.

3)The correlation mapping process used to align MRI and pathology images might result in ambiguous matches, especially when dealing with highly heterogeneous tissue structures. This ambiguity could affect the accuracy of the registration and the quality of the fused images.

4) As the test set size is very small it would be useful to see on which areas does the segmentation improves with the help of the proposed method.

5) The authors do not mention anything about making the code public.

6) The authors report SSIM, Euclidean distance and Dice for each case and even make a bar plot for SSIM of each of the 6 testing cases. It is usually enough to report an average metric with a standard deviation / median per dataset.
title: title: A hard problem, an interesting solution, but what application?
main_review: main_review: The authors propose a deep learning-based approach to affinely register high-resolution histopathological images with MRI images of the prostate. Their method employs two domain-specific feature extractors pretrained in a self-supervised manner, alongside two correlation mapping and feature matching modules to predict the registration parameters. Despite the significant challenge of addressing a large domain shift, their method has shown promising results, highlighting the potential of self-supervised learning for feature representation relevant to registration. However, there are several areas for improvement and some questions that need to be addressed.

M1 - The first question is about the application of this task. The authors suggest "enhancing cancer evaluation," but the specifics are unclear. Are they referring to tasks like tumor segmentation in MRI images? If the prostate tumor has already been resected for histology, what is the benefit of segmenting the tumor in an MRI image? In this scenario, the ground truth is the existing whole-slide image (WSI), so there is no need to revisit the MRI image. If the goal is to create a reliable dataset for tumor segmentation in MRI scans, this should be discussed in the manuscript. Additionally, it should be demonstrated that this method is superior to manual annotation by radiologists on MRI images. Authors should clearly mention or cite relevant publications about the application of this task.

M2 - Similar to the previous comment, the need for "fused" images is unclear. The fused images appear to be histology images transformed to fit MRI images. In clinical use, why not directly examine the histology of WSIs?

M3 - The manuscript lacks clarity in certain methodological explanations. For example:
- In section 2.5, it states that MRI scans are aligned with corresponding MRI slides, which seems to be a typo.
- In section 2.5, the phrase "During inference for ..." is poorly written. It would be better to mention the resizing step in the feature extraction or matching sections and then explain how the obtained transformation function is applied to the original high-resolution histology image. Also, clarify if the WSI was transferred in a patch-wise manner.
- In section 2.1, it mentions that MRI slides and pathology images were aligned. Was this done manually?
- In section 2.4, it is unclear how the ground truth transformation vector is obtained for use in the loss function.

M4 - Given the small test dataset, why not use cross-validation (each time using different parts of the MRI-Prostate dataset for testing) to gain a more robust understanding of the model's performance?

M5 - Is it possible to use existing "Foundation models" for feature extraction instead of pretraining the models? This should be discussed or mentioned in the limitations/future work section.

Furthermore, there are some minor comments:

m6 - It is common to use present tense in scientific writing. For example, on page 3, the phrase "Our method advanced the state-of-the ..." should use "advances" to align with the rest of the manuscript.

m7 - Citations are used inappropriately throughout the manuscript. When a citation is the object or subject of a sentence, it should be used outside of parentheses, e.g., "Shao et al. (2021) found that ...". When a citation is used to refer to a paper, it should be in parentheses, e.g., "In the MRI-Prostate dataset (Madabhushi et al., 2016)...". The authors should revise the manuscript to correct citation formatting.

m8 - In Table 1, instead of naming the baseline "original," cite the publication or name of the method, or simply call it "baseline."
title: title: Review of paper 24: A method for MRI-HE registration
main_review: main_review: The authors proposed a method for the registration of MRI and HE images. However, the paper is not well-written, and many sections are unclear. The proposed framework lacks significant novelty, as both modules of the network are adopted from existing works. Additionally, the results are reported using a limited dataset and inadequate evaluation metrics.

Pros:
- The paper is well-structured

Major Comments:
- There is a wide literature on the registration of MRI and histology images and the use of deep learning for image registration. The authors need to rewrite the introduction and cite relevant literature.

- In the contributions section, point three is a claim that self-supervision helps with an insufficient dataset, which is not entirely accurate, and they lack experiments to support this claim.

- In equation 2, what is J? The authors also need to clarify whether the dot in the formula represents an inner product or something else.

- The size of the test dataset is very small (6 cases) and cannot adequately reflect the efficiency of the proposed framework. The authors need to test the model on a larger dataset, as evaluating segmentation on only 6 cases is not significant.

Minor comments: 
- Table 1 can be restructured as two columns containing N/A values.
- Section 2.2 can be made easier to understand by providing more explanation about the work from which this module is adopted (DINO).