Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores the benefits of current deep Reinforced learning in accelerating MRI sampling and concludes that the benefit brought from current SOTA RL methods is not aligned with the computational complexity of these methods.
main_review: strength: 
1. This paper works on a very important topic, accelerating MRI by optimizing signal sampling.
2. This paper compares several SOTA RL methods to present an interesting conclusion.

weakness:
1. I'm not convinced by the novelty of this paper. Honestly, this paper may be more suitable in a medical imaging journal.
2. I'm not sure if this research is really meaningful. Does accelerating MRI really need RL for signal sampling? I am not optimistic that RL based sampling methods can be really applied in MR imaging industry. 

concern:
1. As for the experiments, are these experimental settings well set for a fair comparison?


summary_of_the_review: Though this paper donot introduce quite new algorithm, it is still valuable since it concludes the recent RL based sampling methods in accelerating MRI and analyze the benefit. However, I'm not sure if this paper is suitable for submission in ICLR. Maybe it is more suitable in a medical imaging journal.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors report on the use and current limitations of using deep reinforcement learning for accelerated sampling in MR imaging. They focus on two specific works that use deep RL for sampling: Pineda and Bakker. They mainly compare these methods against a previously described non-RL technique: Stochastic Learning-based Compressive Sampling (LBCS). This method trains a non-adaptive, greedy sampling policy that selects as a measurement candidate in each acquisition step the column that leads to the greatest average improvement over a sample from a training dataset. 

The authors conclude that the ‘return on investment’ of adopting RL over LBCS is generally marginal compared to what changes in the modeling pipeline can make. Improving the reconstruction architecture or using masking regimes adapted to the sampling horizon yield more significant gains than using RL-based methods. The authors state that their results confirm the stated limitation in both Pineda and Bakker, namely that neither adaptivity nor long-term planning methods are currently worth the increased cost. 

main_review: Strengths: 
The authors are commended for recreating results and systematically evaluating previous works on the topic. 
The authors were charitable in their presentation of Bakker and Pineda. 
The Conclusion section is very good. They provide a clear and distinct summary of their contribution in their work AND charitably restate the limitations of the discussed works. I think this was a graceful presentation of the limitations of other investigators’ works without being overly critical. 
The authors did a great job pointing out the importance of openness and reproducibility. This is absolutely critical to good research and reproducing results. The authors are commended for pointing this out in the paper explicitly!

Weakness:
While I think the authors report on an important topic, the work lacks a strong enough negative or positive contribution to meet the threshold for publication in ICLR. The authors correctly demonstrate the limitations of deep RL in accelerated MR sampling; however, the previous authors also stated these limitations. The ROI on any clinically relevant ML system can be called into question due to computational costs. However, these costs are in no way insurmountable and rather simple to overcome in practice. 

The lack of performance boost is relevant and the authors point this out. However, this fact alone is insufficient to warrant publication here and is better suited for another venue. 

summary_of_the_review: Overall, I am in complete agreement with the motivation and sentiment of the paper, but as presented, it does not make a sufficient contribution to warrant publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper first surveys several recent Fast MRI Recon methods, including RL-based and greedy-policy-based ones, that reconstruct high quality images from highly undersampled data from the perspective of optimizing the sampling of k-space data and then concerns the RL-based methods. Further it considers a typical Fast MRI Recon framework that contains three main blocks of preprecessing (cropping and/or resizing, using magnitude only etc.), subsampling (random, LtH, LBCS, adaptive, etc.), and post-precessing and reconstruction, and evaluates its performance changes triggered by seemingly trivial changes at even a single of these stages in a series of experiments. It shows that the RL-based method, which adpatively computes the sampling mask, does not offer benefits over fixed masks.
main_review: Pros:
- The paper focuses on a clinically relevant topic
- The paper presents a solid comparison
- The writing is clear and easy to follow.

Cons:
- There is a lack of methological novelty.
- There is a lack of visual comparison. Although visualizations are somehow subjective, the reconstructed images are important for doctors to compare lesions/patterns in medical diagnosis and hence visual comparions are important. Will these different Fast MRI Recon methods result in similar visual quality (lesion details/organ boundary shapes) or they show different reconstruction performance for different tasks (lesion detection/organ checking)? All these answers should be asked by comparative visual readings.

Typos: 
"following equation equation 1"  in the last line of  Paragraph 5, Section 3;
“preprocessing” ->  “post precessing” in the last line of Paragraph 7, Section 3.

summary_of_the_review: Overall, the paper contains certain empirical value, arguing that RL-based approach is not effective. But it lacks the typical methological novelty necessary for an ICLR paper.

This paper is more suited for MICCAI or medical imaging related conferences.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the effectiveness of deep RL methods for accelerating the acquisition of MRI scans. It compares two recent methods by Bakker et al. as well as Pineda et al. to non-RL baselines. The results indicate that deep RL provides little benefits over the LBCS baseline.
main_review: Pros:
The paper studies the real world problem of how to accelerate the acquisition of MRI scans and whether RL methods can help in this scenario. The paper considers two relevant RL approaches to the MRI acceleration task and compares it to a strong non-DL baseline (LBCS). The paper presents empirical results for a large variety of settings using different preprocessing and postprocessing options.

Cons:
- The paper sets out to unify the results of recent improvements in MRI acceleration but does not offer comparisons of the two considered RL methods. This generally makes it hard to objectively judge the presented results in the paper. The paper would benefit from an "apples to apples" comparison of the different aspects that are being considered including all steps of preprocessing, sampling strategy, and postprocessing.
- The paper mentions that different previous works rely on different possible metrics and that different metrics lead to different conclusions. The display of the results is not very clear to me, e.g.: What's the difference between the AUC for sampling rate or acceleration rate in Table 3? How are they computed? As such it would be useful to present the results of all experiments with all possible metrics and state which metrics would be favourable to use in which scenario. Metrics such as MSE are mentioned but don't seem to be reported.

Minor:
- Reversals are described as cases where the RL approach is suddenly outperformed by LBCS. However, later it is enough that the performance of the RL methods are matched. This leads to misunderstandings and seems to misrepresent the performance gains.
- The display of the results in the table would benefit from reiteration of the design choice made. Rather than just using "cvb" it would be easier to use "cropped, vertical, Backer parameters or explicitly offer an explanation in the caption. Alternatively, one could add a table to explain the possible options for the different design axes.
- There are various typos in the paper:
  - Just after eq. 3, "estimates from training data" -> "estimated from ..."?
  - Page 6, "Except fir the deterministic LtH and NA Oracle,we report" -> missing space after the comma
  - Page 8, "and of the corresponding AUC consists of 20 lines out of the 100 acquired. While is true..." -> "While it is true"?
summary_of_the_review: My main issue with the paper is that it is hard to follow the analysis. It would benefit the paper to combine the analysis of both RL methods and rather ask questions about which design choices have the highest impact. Similarly, the analyses should incorporate the different choices of evaluation metrics. This would enable the authors to propose a recipe of how to approach the problem of MRI acceleration.
review_title: null

Meta-review: While all reviewers acknowledge the relevance of such an evaluation work for the MRI reconstruction field, they all agree that the contribution has a limited fit with a ML conference like ICLR. The work is solid experimentally and will surely interest the audience of conferences like ISMRM or MICCAI. For this reason, the work can unfortunately not be endorsed for publication.
Decision: Reject