Leveraging Side Information with Deep Learning for Linear Inverse Problems: Applications to MR Image Reconstruction

Authors: Arda Atalik, Sumit Chopra, Daniel Sodickson
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: mr image reconstruction, side information, linear inverse problems, trust-guided variational network

Abstract:
Reducing the time it takes to acquire a Magnetic Resonance Imaging (MRI) scan is an important problem in healthcare, as it can improve patient care and reduce costs. One way to achieve this is by acquiring only a fraction of the frequency space data and reconstructing diagnostic-quality images from it. This problem can be formulated as a linear inverse problem (LIP), where the forward operator, which maps the structure of the imaged object to the acquired frequency space data, can become rank-deficient or exhibit many small singular values. This leads to ambiguities in the reconstruction process, where multiple images (most of them non-diagnostic) can map to the same set of acquired data. To resolve these ambiguities, it is essential to leverage domain knowledge and, whenever possible, exploit additional context (a.k.a., relevant side information) when solving the LIP. We present a novel, end-to-end trainable deep learning-based method, called Trust-Guided Variational Network (TGVN), that reliably incorporates side information into LIPs to eliminate undesirable solutions from the ambiguous space of the forward operator, while remaining faithful to the acquired data. We demonstrate its effectiveness through applications in multi-coil, multi-contrast MR image reconstruction, where incomplete or low-quality measurements from one contrast are used as side information to reconstruct a high-quality image of another contrast from heavily under-sampled data. Its robustness is validated by reconstructing images from different contrasts across different anatomies and field strengths. Compared to a set of baselines that also use side information, our method reconstructs high-quality images in the presence of heretofore challenging levels of under-sampling, thereby speeding up the acquisition drastically while providing protection against hallucinations. Our approach is also versatile enough to incorporate many different types of side information into any LIP.

Decision: reject
Reviewer scores: 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729007102089

thread_type:
official_review

<thread_object>
creator: reviewer_tg2y

<content>
summary:
This paper presents a novel deep learning-based method called Trust-Guided Variational Network. This method addresses the problem of reconstructing high-quality MRI images from undersampled frequency space data. The primary challenge in MRI reconstruction is that undersampling results in ambiguities due to rank-deficient forward operators, leading to multiple non-diagnostic solutions. TGVN integrates relevant side information to constrain the solution space and eliminate undesirable solutions, enhancing image quality.

strengths:
1. This paper incorporates side information into the MRI reconstruction process. It introduces an "ambiguous space consistency constraint" to tackle the ill-posed nature of the problem.
2. The paper is generally clear, with well-organized sections on background, methodology, and empirical validation. The explanations of the core concepts, such as the linear inverse problem and the ambiguous space consistency constraint, are detailed and easy to follow for readers familiar with the domain.

weaknesses:
1. limitation of experiments in terms of the methods that have been compared, please consider comparing some other famous methods such as VN, E2E VarNet.

2. Imposing additional constraints is not a new topic. For example, the paper: Bian, Wanyu, et al. "A learnable variational model for joint multimodal MRI reconstruction and synthesis." is also an optimization-based recon model that utilizes the multicontrast information for enhancing performance. Authors should cite some related paper that leveraging additional contrast.

3. Since it is a model based recon paper, authors need to add optimization based deep learning methods such as ADMM-net, ISTA-net, in the related work. End-to-end and generative methods are not too close to the topic.

4. Authors didn't provide a network structure or whole framework of this work.

questions:
1. From the figure 3 and 12, DMSI provides more details of the tissue and close to the ground truth. The proposed method smooth out some features of the knee joint, fat part. 

2. Please indicate the box of the zoom in region in all of the figures. 

3. Conduct a doctor study for evaluating the performance of these compared results should strengthen  robustness of the study.

4. In equation 3 and 4, author didn't explain what is \phi.

5. H(s, \gamma) is assumed to be learnable, and s is fixed variable, how does author choose s? Why not updating s iteratively with s^t and x^t alternatively?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730519777568

thread_type:
official_review

<thread_object>
creator: reviewer_36te

<content>
summary:
This paper proposes a Trust-Guided Variational Network (TGVN), an end-to-end deep learning (DL) method for accelerating MRI reconstruction. Existing physics-based unrolled networks face the problem of ambiguities in the solution space, degrading reconstruction image quality. By introducing the concept of ambiguous space consistency, this work proposes to incorporate side information (i.e., diverse modalities here) into unrolled DL frameworks for further enhancing MRI reconstruction. This approach appears technically and theoretically sound. And extensive experiments confirm the effectiveness of the proposed TGVN model.

strengths:
**Motivation and Novelty**: This work introduces the concept of ambiguous space consistency to address ambiguities in the solution space of existing unrolled methods and proposes using side information to resolve this issue. Although the work largely follows the general framework of unrolled methods, its novelty meets the standards for ICLR conference publication.

**Clarity and organization**: This paper is well-written and easy to follow.

**Experimental evaluation**: This work conducts extensive experiments on two MRI datasets to validate the proposed method, including: 1) comparisons with other unrolled methods without side information and with existing side-information-based methods, and 2) ablation studies on key components of the proposed method. The results confirm the effectiveness of the TGVN model.

weaknesses:
**Clarity**: In this work, ambiguous space consistency is a key concept. Providing some intuitive explanations would improve readability.

**Type of side information**: In this work, different MR image modalities are used as side information. I wonder if the differences between the main information (i.e., measurements) and side information could significantly affect model performance. More specifically, could complementary main and side information (e.g., T1w and T2w) better improve performance? If so, this use of side information appears similar to multi-modality MRI reconstruction. Please clarify the differences between the proposed approach and existing multi-modality MRI reconstruction methods.

**Compared methods**: The proposed method follows a supervised learning paradigm, requiring extensive high-quality MRI images for pre-training. However, state-of-the-art MRI reconstruction methods are not included, such as supervised Transformer-based methods [1][2][3] and unsupervised diffusion methods [4][5].

**Generalization evaluation on different datasets**: On the two datasets, the training, validation, and test are conducted independently. I wonder if the use of side information can enhance the model's generalization to out-of-domain data.

**Under-sampling mask**: The under-sampling masks used in this work are based on Cartesian sampling. Please evaluate the model performance on other types of sampling, such as radial sampling.

**Experimental results**: 

1.	I observe that the performance of the E2E VarNet in Exp. 3 is significantly decreased compared to its performance in Exp. 1 and 2. However, other baselines and the proposed method do not show this trend. Please explain this result.

2.	In the ablation studies, only the SSIM metric is reported. Please provide other two metrics (PSNR and NRMSE).

> [1] Guo, Pengfei, et al. "ReconFormer: Accelerated MRI reconstruction using recurrent transformer." IEEE transactions on medical imaging (2023).

> [2] Huang, Jiahao, et al. "Swin transformer for fast MRI." Neurocomputing 493 (2022): 281-304.

> [3] Feng, Chun-Mei, et al. "Task transformer network for joint MRI reconstruction and super-resolution." Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24. Springer International Publishing, 2021.

> [4] Chung, Hyungjin, and Jong Chul Ye. "Score-based diffusion models for accelerated MRI." Medical image analysis 80 (2022): 102479.

> [5] Peng C, Guo P, Zhou S K, et al. Towards performant and reliable undersampled MR reconstruction via diffusion model sampling[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 623-633.

questions:
See the Weaknesses above, please.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730569010145

thread_type:
official_review

<thread_object>
creator: reviewer_haqd

<content>
summary:
This paper deals with solving the inverse problem in magnetic resonance imaging by leveraging domain knowledge. Therefore, the authors introduce the so-called Trust-Guided Variational Network (TGVN).  Despite the data consistency where the reconstructed images should align with the measurements the ambiguous space consistency is included which ensures that the projected data aligns with the side information.

strengths:
* The idea to incorporate side information is good. Reducing acquisition time and therefore gathering undersampled data which leads to artifacts in the reconstruction process will require sophisticated but trustable reconstruction algorithms as MRI is a versatile application.
* “Easy” integration of the ambiguous space consistency constraint.
* The approach is end-to-end trainable.

weaknesses:
* There exist already many attempts to incorporate side information.
* The paper includes a lot of general information in the paper but a quite short explanation of the novel work.
* In this paper, four research questions are mentioned but only two are discussed whereby also the other ones, especially Q3, would be also considered as important as it is part of the main contribution of this paper. The general stuff should be shortened and a focus of the new method and the results should be 
* The authors advertise that different types of side information can be included but not exactly how. Furthermore, it is mentioned that this is part of future work 
* The generalizability is not shown across different anatomies, as stated. The brain and knee dataset are handled separately.
* A lot of stuff is described generally but the key points are missing.

questions:
* Why should we use this approach and not another one to incorporate side information? What is the benefit of this method (beyond more feasible solutions which is the goal when incorporating side information)?
* How does the threshold effect the results?
* Which projection was chosen?
* How is the module learned?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730764155960

thread_type:
official_review

<thread_object>
creator: reviewer_gu7v

<content>
summary:
The paper with title: Leveraging Side Information with Deep Learning for Linear Inverse Problems: Applications to MR Image Reconstruction presents a Deep learning-based framework that can leverage the information from other contrast to help the reconstruction, the key framework uses a Null space projection method to disentangle the side information. Results on fastMRI dataset show improved image quality compared to other SOTA methods.

strengths:
1. Using prior knowledge from other contrast is a very interesting topic in terms of MRI reconstruction since for clinical routine, usually multiple contrasts are acquired. 

2. Using Null-space to disentangle the contrast-information is interesting.

weaknesses:
1. The presentation and story telling is not clear and there isn't any figure illustration of the problem/method/novelty, which makes the paper very difficult to follow, especially the null-space and theory part. 

2. Using multi-contrast information for joint reconstruction has been a popular topic, but the authors didn't compare with any-of-them.

3. Following the previous weaknes, instead of using Null-space, one way to extract information from side contrast is by learning an representation, you should incorporate this experiments - like for example in E2E experiment, try inputing two channels.

4. Instead of using aliased side information (3x undersampled image from side contrast), it makes sense to use a reconstructed version of other contrast, like using E2E, have you done this experiments?

5. ~20x undersmapling doesn't make sense at all, in practice, or in fastMRI dataset, only a few lines are sampled beside the ARC region, how do you expect recovering any details? especially for pathologies.

questions:
1. Do you use paired data for training - like the main and side contrasts from the same object/ same anatomy? how do you do the alignment?

2. I need more ablation study to understand how the side information contributes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732488939654

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We hope this message finds you well. 

We wanted to kindly remind you that we have carefully addressed all your comments and questions in the revised manuscript and provided additional clarifications where needed. As the discussion period is nearing its end, we would greatly appreciate it if you could revisit your evaluations in light of the responses and updated information we have shared.We are sincerely grateful for your time and thoughtful feedback throughout this process, and we hope that our efforts to address your concerns are reflected in your final review. Please don’t hesitate to let us know if there is anything further we can clarify or improve.

Thank you again for your invaluable insights and contributions.

Best regards,
The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an end-to-end deep learning-based method, called that reliably incorporates side information into LIPs to eliminate undesirable solutions from the ambiguous space of the forward operator. However, the reviewers believe that imitation of experiments in terms of the methods that have been compared. The description of the relevant work is not very appropriate, while optimization based deep learning methods should be added. More intuitive explanations about the key concept should be provided. 

Based on the average rating of the reviewers, I believe the paper is not yet ready for publication at ICLR.