Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper propose to use score-based generative models for robust compressed MRI. The paper claims that CSGM framework performs poorly on out-of-distribution samples, and the proposed posterior sampling-based method is robust to changes in the image distribution and measurement process. 
main_review: I list my main concerns below. 

- The main contribution of this paper seems to be in using score-based generative models for MRI. The main approach/algorithm is based on prior work that uses a score-based generative model to perform Langevian dynamics for posterior sampling. 
- The paper claims that they are the first to apply CSGM framework on clinical MRI data. This seems to be a strong claim because several other paper have used fastMRI data with generative model-based reconstruction. 
- The paper also discussed the challenges related to shifts in the distribution of images and measurements. 
- The challenge of shift in image distribution at train and test times seems real and important to me, but I do not understand why measurement distribution is of any concern/relevance. It seems obvious to me that the generative model-based methods can work with any measurement pattern. This is different from end-to-end networks that learn to map measurements to the images, and difference between measurement patterns at training and testing is a concern.
- The problem of mismatch between train and test images of generative models is important. This paper provides empirical results that show the proposed method is robust to such mismatch. At the same time, most of other methods they compare against in Figure 4c also see robust to such image distribution mismatch. I wonder what is the significance of the proposed method in that case? 
- The paper does not compare with the original CSGM framework in which you use a trained generative model and solve an optimization problem similar to the one in Eq. 2. 
- SSIM is a metric often used for natural image quality assessment. It is not necessarily a good metric for MR images. Please report PSNR as well. 
- Theoretical results: I feel there is a mismatch between the theoretical and empirical results in the paper. Gaussian measurements do not exist in MRI, so what is the purpose of discussing those results? Theorem 3.4 talks about general operators, but it is also a generic result that does not take into account any specific component of the proposed generative model-based method. 

limitations_and_societal_impact: Brief discussion on limitation in the conclusion section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper applies deep generative priors successfully to accelerated MRI on clinical data. 

The paper trains a score-based generative prior on brain scans from the fastMRI dataset. To recover an image from sub-sampled multi-coil data, the paper imposes the prior by performing posterior sampling via Langevian dynamics. The paper provides numerical results showing that this approach gives very good image quality. Besides the standard setup of in-distribution evaluation, the paper also provides numerical results for different sampling patterns and for a distribution shift from one anatomy to another, and again shows very good results even when applied to one anatomy and trained on another anatomy. 
The paper also emphasizes that their method can obtain multiple reconstructions with different random initializations which can be used to quantify the uncertainty of the reconstructions. 
Finally, the paper provides theoretical results for posterior sampling. 
main_review: Comments:
1. The ranking of algorithms on in-distribution images (Section 4.1, Figure 4(a)) is inconsistent with the results in the literature. In the FastMRI leaderboard, Varnet outperforms ConvDecoder (Deep Decoder) on the Brain dataset, while in Figure 4a, Varnet is by 0.1 in SSIM  worse than ConvDecoder. This is a significant difference. 
Perhaps the reason is that the paper evaluates on MVUE images, but uses the VarNet pre-trained on RSS images (but fine-tuned on MVUE images). 
In any case, it is important to understand where this difference comes from, because if in fact, the Varnet outperforms ConvDecoder as consitently reported in the literature, then Varnet would outperform Lagevian (i.e., the method in the paper), since ConvDecoder is very close to Lagevian. This in turn would change a major conclusion of the paper saying that 'we achieve competitive performance compared to end-to-end deep learning methods when the test-time data are sampled within the distribution'. 

2. Overall the experimental setup is very good and well described. However, why use MVUE images instead of RSS as the ground truth? The paper's numerical results are based on the FastMRI dataset and the FastMRI competition and all associated results are based on using the RSS images as groundtruth. Choosing MVUE as the groundtruth makes a comparison to the literature difficult and also leads to the complication that the VarNet used in the paper under review is trained on the RSS images, but then applied to a MVUE images (albeit it is fine-tuned on MVUE images). 

3. In the distribution shift section: It would be interesting to specify whether the results are obtained by applying the CSGM method with the same hyperparameters for changing sampling patterns, or if the paper tunes the hyperparameters of the methods on the new sampling patterns. This is important to know, as [20] has shown that hyperparameter tuning on the target distribution significantly impacts the performance on out-of-distribution data. 

4. The paper emphasizes that the Lagevian sampling method can obtain multiple reconstructions with different random initializations which can be used to quantify the uncertainty of the reconstructions. The paper writes in the corresponding Section 4.3 'as shown in Fig. 3, the pixel-wise standard deviation is a good estimate of the ground truth error': I really don't see this conclusion from Figure 3. It would be great to have a quantitative result for this statement.

5. Theorem 3.4 states that if there is an algorithm that reconstructs $x^\ast$ drawn from some distribution $\mu$ from $y = A(x^\ast)$ with $\epsilon$ accuracy, then posterior sampling succeeds with $2\epsilon$-accuracy. That's an interesting statement, but it would be good to also discuss the catch: for a given forward operator, it is typically difficult to check whether an $\epsilon$-accurate algorithm exists, and we don't have access to the distribution of $\mu$, but only to an estimate through the generative model. Also, it would be great if the paper could comment on whether the Langevin Dynamics converges to a sample from the posterior distribution, in order to clarify whether the recovery results Thm 3.3 and 3.4 apply to the algorithm that is used in the numerical results section.

Significance and originality:
The paper provides evidence that imposing a prior (i.e., recovery via the SCGM framework) can give very competitive results for clinical MRI. This is very interesting, as prior works did not achieve competitive results for real-world medical imaging data by imposing a generative prior. Imposing a prior has a number of advantages, in particular, that the method requires no retraining when changing the forward model. If the SCGM framework can really give results that are competitive to the state-of-the-art end-to-end neural networks, this would be a significant development.

Quality and clarity:
The paper is well written overall and describes the setup and experiments well along with the assumption and the setup (apart from a few details pointed out above and in the minor comments below). The paper evaluates the method well, and the experimental setup makes a lot of sense, the only major concern outlined in comment 1 above is the significant discrepancy of the relative ranking of algorithms relative to the literature. 

Summary:
I really enjoyed the paper and would be happy to see a (revised version) at the conference. My reason for not rating it higher, my concern in explained in comment 1. If this can be convincingly resolved, without the need to change the main claim of the paper, the paper would be a clear accept in my opinion, and I'll change my rating accordingly then.



Minor comments:
- Fig 1: Would be good to state the acceleration factor in the caption for reference, and whether this is single or multi-coil reconstruction.
- The paper considers 'test time sampling patterns shifts': I fully agree that a major advantage of the CSGM framework is that it can be easily adapted to changes in the sample process, without the need of re-training. I wouldn't call this a distribution shift as the image distribution doesn't change, but again this is a minor point. 
- A related work is 'Compressible Latent-Space Invertible Networks for Generative Model-Constrained Image Reconstruction' which also showed good results on the FastMRI dataset, albeit not on the original k-space data and not on multi-coil data, and with a different algorithm, but also by imposing a generative prior via the CSGM framework.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed the robust MR image reconstruction framework based on CSGM (Compressed Sensing with Generative Modeling). Compared to the other MR reconstruction algorithms, the proposed method shows consistently superior reconstruction results for both in-distribution (same k-space sampling mask/direction, same anatomy, and same contrast) and out-of-distribution (different k-space sampling direction/mask, different anatomy, and different contrast). Also, it gives the uncertainty estimation of its reconstruction results in pixel-wise manner by running multiple reconstruction samplers.
main_review: Originality  
- The most of the backbone algorithms are comes from the other works: NCSNv2, RefineNet, etc. However, the originality of this paper sufficiently comes from the how to build the robust reconstruction performance on out-of-distribution samples & its theoretical and mathematical background support. 

Quality & Clarity
- The author tackles the out-of-distribution problem for MR reconstruction. And their CSGM based method makes sense for its robustness and it shows the superior MR reconstruction results for various out-of-distribution cases which are fit for their assumption and mathematical background.

Significance
- Many of MR reconstruction algorithms are still focusing on the end-to-end supervised learning (accelerated/sampled data to full-sampled data) and it results to the poor performance on out-of distribution datasets while it results to the excellent performance on in-distribution datasets. Since the proposed method shows impressive robustness of various out-of-distribution datasets (sampling direction/anatomy/contrast), it might give positive impact on real clinical MR acceleration fields.
limitations_and_societal_impact: The followings are the suggestions and my questions  to improve the paper for better understanding of readers:

- Flow of Algorithm might be helpful to the readers: Sample preparation steps such as coil-sensitivity extraction, different sampling masks/different anatomy images for the training/test phase for out-of-distribution, etc.

- Too little explanation of the main score model, NCSNv2 & backbone model, RefineNet.

- Actually, in clinic, they usually fixed the k-space sampling pattern at each acceleration rate (by MRI manufacturer) and they want to maximize the performance of reconstruction for that fixed sampling masks. Thus, the comparison results for the different sampling direction (Phase Encode for anterior-posterior VS. Phase Encode for lateral-later) seems less effective to support the usefulness of the robust algorithm. I think the different contrast images are better examples (T1w, T2w, FLAIR, etc.)

- The author mentions that their results are superior because the the comparative reconstructions show some aliasing artifacts or miss some details. However, some radiologist might score better on  the comparative one to examine the anatomy/lesion even it has some artifacts. The blind examination by the radiologist might be helpful as further work.

-- Figure.4 (a): The 'in-distribution' reconstruction performance of MoDL and E2E-Varnet seems too bad. e.g. less than SSIM 0.8 while the acceleration rate R is only 2? And the PSNR of MoDL and E2E-Varnet results are less than 25dB @only R=2 while the proposed method 40dB (appendix Fig.19(a))? 

-- Figure.4 (b) : Why some results of out-of-distribution ((b)Mask Shift Brain: @R=8, R=12 ) show superior than (a) in-distribution method.

-- Figure.4 (d): Is there any discussion why the proposed method shows worse results @ higher acceleration rates, R=8 & R=12?

Minor concerns
- Please, make sure that figures&tables stand alone, complete and informative in itself.(e.g.Fig.1 what 'mask shift'/'anatomy shift' means)

- Mention about 'inverse crime' (line#214): the authors carefully slice the 3D knee scans in proper dimension to simulate the appropriate real MR acquisition. However, most of the MR image in the dataset(I think all of them), oversampled in k-space Read-Out direction(usually x2) and remove the out of the FOV part in the reconstructed images. Thus, I was wondering if the author did the proper inversion of this process (1.attach the blank bins on the image in Read-Out direction, then  Fourier Transform + retrospective sampling). 

- Use capital letter correctly in references(mri, Modl, mr, etc)

- Please, specify the parameters for SSIM metric in appendix.

- Figure.3 the figures have same intensity scale? GT, mean reconstruction VS. std, error? The intensity of error images seems higher than the GT. Please, specify them.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers the problem of compressed sensing (CS) using generative priors, which can also specifically be characterized on a multi-coil MRI setup. The algorithm itself is a more general version of the CS setup which typically considers Gaussian measurements.

The authors consider a posteriori sampling procedure to invert the measurements $y = Ax^* + w$ by estimating $\hat{x} \sim \mu(\cdot|y)$ (modeled via deep generative prior) assuming $x^* \sim \mu$, given $A$ (MRI setup or general), $y$ and zero-mean $\sigma^2$ variance Gaussian noise $w$ . Authors use an annealed Langevin Dynamics approach to sample from the posterior, which has been previously used in [73] for sampling from generative priors. Here the score function is modeled via a deep generative neural network.

Authors also present two theoretical results: one on the closeness of reconstruction $\hat{x}$ and $x^*$ under distribution shift of the deep generative prior and true distributions (assuming Gaussian A), and another on the quality of posteriori sampling as compared to any other kind of reconstruction procedure under aribitrary (not neccessarily Gaussian) measurements assuming no distributional shift.

Authors present several experiments on the fastMRI and Stanford MRI datasets and compare to baselines such as L1/wavelet, MoDL, E2EVarNet and ConvDecoder and show  improvement reconstructions on both in-distribution and out-of distribution samplings in terms of SSIM.
main_review: This paper considers a measurement agnostic approach to inverting compressively sensed images under deep generative prior. Specifically authors use the model NCSNv2 from [74] for this purpose, which is a score based generative model. This paper uses an annealed Langevin dynamics algorithm to sample from posterior $\nu$ and show experimentally that this approach works better in terms of SSIM to prior art on both in-distribution and out-of-distribution reconstructions at various acceleration rates (under sampling rates).  

Through theorem 3.3, the authors present a theoretical result for $\hat{x}$ estimated according distribution $\nu$, assuming a distributional shift from the true distribution $\mu$ of $x^*$ under gaussian measurements $A$. The main claim of this result is that if $\mu,\nu$ are Wasserstein close by factor $\epsilon$, then the $\ell_2$ estimation error between $x^*$ and $\hat{x}$ is lower bounded with high probability.

Through theorem 3.4, the authors show that this posterior sampling procedure produces a reconstruction within an additional $\epsilon$-error  ($2\epsilon$) of any other reconstruction procedure that is bounded by $\epsilon$ with high probability (for any sampling process A).

The main contributions of this paper are theoretical results and the application of annealed langevin dynamics to CSGM via posteriori sampling (the algorithm itself is incremental in terms of derivation from prior art). 

The paper is very well written highlighting the key contributions which is backed by their theorem statements and experiments. Experiments on Fast MRI are convincing as well. The paper can benefit from some minor changes as follows.

Figure 1 can benefit from presenting reconstruction SSIM or MSE.

Section 2.1. is the regime of experiments $compressive$? Since CSGM typically refers to $M<N$. Please make the relation between samples M and dimensionality N explicit. 

Line 113 is $N_c \cdot L = M$?

Prior to Line 220, the exact formulation of the deep generative model isn't clear or explicit. Please introduce a paragraph discussing choice of deep generative prior. 
limitations_and_societal_impact: Deep neural networks have been successfully used for MRIs and are of importance to society for furthering medical imaging research. I do not think there are negative societal impacts of this work since the authors present a clear ground truth based on MVUE (no regularization, full samples).
review_title: null

Meta-review: The authors provide the first demonstration that Compressed Sensing with Generative Priors can be a competitive approach for MRI reconstruction relative to end-to-end, L1, and untrained neural network methods.  The authors additionally demonstrate that the method is more robust than baselines in the context of certain distribution shifts between training and inversion.  The paper also provides novel theoretical guarantees about distributional robustness of posterior sampling approach.  

There was some concern from one of the reviewers about consistency with the literature about the performance of baseline methods, likely due to training being on RSS images with fine-tuning on MVUE images.  The authors should provide commentary about this issue, as discussed with the reviewers during the rebuttal. 

The authors should add additional comments that the paper should not be used for medical purposes without subsequent study by medical professionals, as per the ethics review.

Decision: Accept (Poster)