Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In order to eliminate the need for large training sets, one can consider a transition from (1) fully-supervised to (2) self-supervised methods, and then from (2) self-supervised methods to (3) single-instance reconstruction methods. 

In the context of accelerated MRI reconstruction which is considered in this work, the above categories translate into models that are trained based on (1) having access to a fully-sampled dataset, (2) having access to a dataset of under-sampled measurements, and (3) having access to only one under-sampled measurement. The paper targets (3), that is proposing a zero-shot learning approach for accelerated MRI reconstruction.

The algorithm is based on the idea proposed in paper [1] combined with building a dataset for the given sample in order to eventually perform self-supervised training on the synthesized dataset. In order to prevent overfitting, the authors propose a way to do automatic early stopping.

[1] Yaman, Burhaneddin, et al. "Self-supervised physics-based deep learning MRI reconstruction without fully-sampled data." 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020.
main_review: ####################################################

Strengths

+ Overall, this is a very interesting paper with impressive experimental results. The proposed algorithm yields high-quality reconstructions and the rationale behind designing such an algorithm is very well-motivated by the authors. Indeed, it is important to come up with reconstruction algorithms that do not require large training sets, especially in medical imaging where creating datasets is a tedious task.

+ The idea of self validation is also interesting since it mimics the presence of a validation set for training and prevents the network from overfitting.

+ The robustness investigations are appealing and very related to the problem considered. Moreover, from the robustness perspective, any effort toward single-instance reconstruction is of great value provided that it results in more robustness. Regarding robustness evaluations, please see the reviewer’s comment below.

################################################

Weaknesses/comments

- The paper states several times that ZS-SSL-TL significantly reduces convergence time. Yet what is missing is a computational comparison among single-instance reconstruction methods such as traditional sparsity-based, DIP-based, ZS-SSL, ZS-SSL-TL. This is important since one of the critical challenges of single-instance reconstruction algorithms is their inefficiency at the inference. Adding such an experiment would also make the paper even stronger.

- Table 2 in the appendix contains very interesting results. However, there are several surprising scores achieved by ZS-SSL-TL. For instance, how come for ``trained on brain & tested on knee`` ZS-SSL-TL achieves 40.4 dBs, yet according to Table 1, ``trained on knee & tested on knee`` achieves 40.1 dBs? This is counter-intuitive in that it suggests pre-training a model on ``brains`` is marginally better than pre-training it on ``knees`` if one wants to get a good PSNR on ``knees``!

- Page 7, Section 4.5, Paragraph 2, last sentence: ``ZS-SSL has no prior domain information and is inherently not susceptible to such generalizability issues.`` Is there any evidence to suggest that this is the case? For example, DIP-based methods have been suggested to be susceptible to distribution shifts although there’s no ``prior domain information`` involved. But their hyper-parameters are tuned on a specific domain. Doesn’t ZS-SSL rely on any tunable hyper-parameters that differ from one domain to another?

- The reviewer perceives the point made in Figure 2 regarding automatic early stopping, however, there are the following two comments in terms of the consistency of the results:

    - Second plot from the left: Can the authors provide intuition on why the curves are uncorrelated w.r.t K? Specifically, at the convergence (epoch 300), the curve for k=10 is on top of k=1, the curve for k=25 is on top of k=10, but then somehow the curve for k=50 is below k=25. Moreover, the breakout point of k=50 has a sudden drop from the breakout point of k=25; does this mean k=100 would go down even further?

    - The right-most plot: The reviewer is unsure what the authors mean by ``ZS-SSL with TL converges faster compared to ZS-SSL`` in the caption, in that the validation loss implies no benefit comes with combining ZS-SSL and TL and it’s not a matter of convergence. 

- Table 1 suggests that DIP-TL performs strangely poorly compared to PG-DLR and ZS-SSL-TL. Does this mean that the pretrained network used for DIP-TL performs poorly and DIP has not been able to improve its performance, and thus the low score has nothing to do with the DIP itself?

summary_of_the_review: Decision: Accept (6)

The reviewer finds the paper of great interest to the community and the thorough experimental analysis of the proposed algorithm is the main motivation for accepting the paper. However, 

(1) several concerns/comments mentioned above, and 

(2) the fact that the major difference between the proposed algorithm and the prior work [1] is the self-validation step combined with dataset synthesis 

prevent the reviewer from giving a higher score.

[1] Yaman, Burhaneddin, et al. "Self-supervised physics-based deep learning MRI reconstruction without fully-sampled data." 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This article propose a "zero-shot" method for MRI reconstruction, which is a well-studied inverse problem. The method is based on the ideas of deep image prior, i.e. the ability of correctly sized neural networks to learn about the structure of a single image, sufficiently well for denoising tasks. This self-supervised learned network can then be used in a plug-and-play architecture to solve inverse problem with a variational approach, i.e. as if the learned denoiser were a Total Variation (TV) minimiser. Their denoiser can be improved in a transfer learning approach to benefit from a more complex network trained on similar images than those at hand, and fined-tuned on the image to be reconstructed

The authors go on to apply their plug-and-play architecture to solve the MRI reconstruction problem. They provide experiments and comparisons.
main_review: The paper is well written and clear with great illustrations and only a few typos. It applies the combined principles of zero-shot learning [1] plug-and-play [2] iterative architectures and transfer learning to solve the MRI reconstruction problem on single images.

The idea of zero-shot learning is not new, in the particular context of imaging, it has long been known that the denoising problem could be solved by training on the very same image one would seek to denoise [3,4]. Plug-and-play architectures combine a general-purpose denoiser with a Tikhonov-like least-square solver to solve arbitrary inverse problems. MRI is often regarded as a classical inverse problem in the same range as CT reconstruction or deblurring. The range of solutions provided to solving inverse problems in medical imaging is too large to mention [5].

From the methodological point of view, I see very little novelty in this article.

There are many aspects to using zero-shot learned approaches for denoising, in particular the fact that their regularity is not established and so they can only be applied for a limited number of iterations, and the need to use early-stopping. One contribution of this article is their proposed condition for early stopping, but it is not studied mathematically.





[1] Romera-Paredes B, Torr P. An embarrassingly simple approach to zero-shot learning. InInternational conference on machine learning 2015 Jun 1 (pp. 2152-2161). PMLR.
[2] S. Venkatakrishnan, C. A. Bouman and B. Wohlberg, "Plug-and-play priors for model based reconstruction", IEEE Global Conference on Signal and Information Processing, pp. 945-948, 2013.
[3] Huang DA, Kang LW, Wang YC, Lin CW. Self-learning based image decomposition with applications to single image denoising. IEEE Transactions on multimedia. 2013 Oct 7;16(1):83-93.
[4] Liu C, Szeliski R, Kang SB, Zitnick CL, Freeman WT. Automatic estimation and removal of noise from a single image. IEEE transactions on pattern analysis and machine intelligence. 2007 Dec 18;30(2):299-314.
[5] Senouf O, Vedula S, Weiss T, Bronstein A, Michailovich O, Zibulevsky M. Self-supervised learning of inverse problem solvers in medical imaging. InDomain adaptation and representation transfer and medical image learning with less labels and imperfect data 2019 Oct 13 (pp. 111-119). Springer, Cham.
summary_of_the_review: The authors make a big deal of using TL, 0-shot and plug-and-play to achieve good results on MRI reconstruction. All of these elements are known and have been applied before to this problem. They do achieve better results than Senouf et al [5], thanks to a better TL regimen.

The paper is correct overall. The early stopping condition is debatable and not studied theoretically.

Overall the paper is pretty good, but I think it does not clear the bar for acceptance at ICLR due to the lack of technical novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission deals with MR images reconstruction in a context where the raw data is under-sampled. This data (which is in the Fourier domain) can be under-sampled to accelerate the imaging exam and thus improve clinical workflow and it is thus a very relevant research topic. 

The submission relies on recent deep learning models that explicitly incorporate some physical aspects of MR image acquisition (multiple coils, coil sensitivity and Fourier transform) and achieve MRI reconstruction based on supervised training examples. The training examples do not need to be (under-sampled / fully sampled) pairs. Indeed, some entries of under-sampled data can be deleted to create an input for which correct reconstruction of the deleted entries can be required through a computable loss term.

main_review: While the paper is pleasant to read and rigorous it does not offer a significant improvement over ref [Yaman et al., 2020] which by the way is regrettably not included in the set of benchmark methods to which the submission is compared in the result tables of the experimental section.

I am not sure to understand the argument w.r.t. transfer learning and single dataset reconstruction. The proposed approach also retrains the model on a new dataset with patient specific properties. I think that what the authors mean is that this dataset does not need to be supervised. But this is a built-in property of the model from [Yaman et al., 2020] from which the authors are building upon.

The proposed approach as well as supervised PG-DLR seem to fail to reconstruct texture feature visible in ground truth images (the reconstructed ones are smoother). This might be a problem for diagnosis or for other trained ML based models that rely on texture features such as radiomics and take MR images as inputs. Is this a known issue in MRI reconstruction in general or in DL based reconstruction ?
summary_of_the_review: Pros : 
- the paper is well written and ideas are clearly explained and stated.
- the paper is technically sound.

Cons : 
- the contribution is too incremental compared to [Yaman et al., 2020] as it consists in re-using the Fourier domain partitioning idea in order to have a validation loss allowing to detect overfitting and stop learning.
- patient specific training demands more computation time and resources than usual clinical workflow.
review_title: null

Meta-review: The paper considers the problem of accelerated magnetic resonance imaging where the goal is to reconstruct an image from undersampled measurements. The paper proposes a zero-shot self-supervised learning approach for accelerated deep learning based magnetic resonance imaging. The approach partitions the measurements from a single scan into two disjoint sets, one set is used for self-supervised learning, and one set is used to perform validation, specifically to select a regularization parameter. The set that is used for self-supervised learning is then again split into two different sets, and a network is trained to predict the frequencies from one set based on the frequencies in the other set. This enables accelerated MRI without any training data. 
The paper evaluates on the FastMRI dataset, a standard dataset for deep learning based MRI research, and the paper compares to a trained baseline and an un-trained baseline (DIP). The paper finds their self-supervised method to perform very well compared to both and shows images that indicate excellent performance. It would have been even better to compare the method on the test set of the FastMRI competition to have a proper benchmark comparison.

Here is how the discussion went:
- Reviewer pt6r is supportive of acceptance, but notes a few potential irregularities, such as the method pre-trained on brain and tested on knees performing better than the method pre-trained on knees and tested on knees, and not providing a comparison of the computational cost. The authors added a table to the appendix revealing that the computational costs are very high, much higher than for DIP even. The reviewer was content with the response and raised the score.  

- Reviewer mBMk argues that the contribution is too incremental compared to prior work, in particular relative to the results of [Yaman et al., 2020], and also argues that the idea of partitioning the measurements is not new. The authors argue in response that their approach of partitioning the measurements is new, and the reviewer was inclined to raise the score slightly, but still thinks that the novelty on the technical ML side remains limited, and doesn't want to back the submission too much, and did not raise the score at the end in the system.

- Reviewer 19v3 has the concern that the all elements used (transfer learning, plug-and-play, etc) are well known techniques and have been applied before to MRI, and therefore thinks that the paper does not clear the bar for acceptance. The paper points out that while those ideas might be applied for the first time to MRI, they have been used before in other image reconstruction problems, in particular denoising. 

I've read the paper in detail too, and am somewhat on the fence: I think it's very valuable to see that a clever application of self-supervised learning works so well for MRI. I agree with the reviewers that the technical novelty is relatively small, but on the other hand this is the first time that I see self-supervised learning being applied that successfully to MRI. I don't share the concern about novelty --- yes, the paper's approach builds on prior work, but it's not clear from the literature how well such a well tuned self-supervised learning approach would work. 
What I would have liked to see in addition to the experimental results is a proper evaluation on the FastMRI dataset: An advantage of the FastMRI dataset is that it provides a benchmark and if researchers evaluate on that benchmark (on the testset/validation set) we can compare different methods well. The paper under review doesn't do that, it only evaluates on 30 test slices, and thus it's hard to benchmark the method. Also, the paper would benefit from more ablation studies.

In conclusion, I would be happy to discuss this paper at the conference, and think that other researchers in the intersection of deep learning and inverse problems would be too, and therefore recommend acceptance.
Decision: Accept (Poster)