Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 3: Borderline
review_content: summary: This work presents a vision transformer (ViT) model for fast MRI reconstruction. The original ViT architecture is modified (from a classifier) to have a reconstruction head comprising layer norm and linear layers. Experiments on the fastMRI dataset with reconstruction of knee and brain MR data show the proposed transformer method fare comparably with U-net baseline.   
strengths: 
* This work explores a pure ViT model for MRI reconstruction which is novel. 
* Experiments on two anatomical structures (knee, brain) are presented from the fastMRI challenge dataset, with comparable results compared to U-net baselines. 
* Experiments checking domain shifts show ViT models are similar in performance with VarNet. 
* Pre-training ViT model shows improved gains compared to pre-training U-net baseline models.
* The method claims improved throughput and lower-memory consumption, which can be beneficial in reconstruction tasks. 
weaknesses: * **Performance in Table 1**: The ViT models reported in Table 1 do not show a large improvment even compared to the simplest of the U-net models (taking the standard deviation into consideration). Perhaps reporting the number of parameters of U-net models could also help contextualize these results.

* **Memory consumption**: In Introduction it is claimed that the ViT models consume less memory compared to baselines. Is this measured as the maximum batch size? If so, that is also higher for U-net-S. Were there any other experiments performed to validate this claim? Also, why do the ViT models consume less memory? Elaborating this could actually strengthen the paper. 

* **Performance compared to VarNet**: Why is VarNet not reported in Table 1? And in Table 2, clearly VarNet performs better than ViT-L. Also, pre-training only one of the models gives an advantage to the specific model. 


* Is the difference in input reason the VarNet outperforms the ViT model? Why can this information not be used for the ViT models? This is not very clear in the discussion presented in Page 6 where authors say:
> Contrary, the End-to-end VarNet, uses information about the MRI forward model (1)
throughout the network. >


detailed_comments: See points above. 
recommendation: Poster
review_title: null

final_score: rating: 3: Borderline
review_content: summary: the authors propose a simple yet effective pre-training, which solely relies on big natural image datasets and only 100 MRI training images.  Even when only fine-tuned on 100 MRI images, a pre-trained Vision Transformer yields sharp and detailed reconstructions, showing that the ViT is very suitable for low-data regimes in accelerated MRI.
strengths: The author propose a simple yet effective pre-training, which solely relies on big natural image datasets, such as ImageNet. We show that pre-training the Vision Transformer drastically improves training data efficiency for accelerated MRI, and increases robustness towards anatomy shifts. In the regime where only 100 MRI training images are available, the pre-trained Vision Transformer achieves significantly better image quality than pretrained convolutional networks and the current state-of-the-art.  Ans the code is available .
weaknesses: 1. The author has mentioned that “Motivated by potential accuracy gains and computational savings, we study Vision Transformers for accelerated magnetic resonance image reconstruction”. However, as far as I know, Transformer does not save computing power but increases it. 
2. Besides, the results exceeding UNet cannot prove the effectiveness of this method, because UNet tends to smooth the image in the process of reconstruction, as can be seen from the results in Fig. 4 and 6. The authors should use classical reconstruction algorithms as a baseline, such as MoDL and the recently proposed Complex reel-based DONet (DONet: Dual-Octave Network for Fast MR Image Reconstruction (IEEE Transactions on Neural Networks and Learning Systems)), which are designed for the reconstruction of MRI data.
3. What are the inductive biases that the authors have mentioned in the introduction?
4. The authors should reclarify their statement that “it is yet unclear whether, or to what extent, such results are applicable to the arguably more different realm of accelerated MRI,” because there are many works focused on the MRI reconstruction. Please see an incomplete list here,
-Task Transformer Network for Joint MRI Reconstruction and Super-Resolution (MICCAI 2021)
- Accelerated Multi-Modal MR Imaging with Transformers, https://arxiv.org/abs/2106.14248
-Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers https://arxiv.org/abs/2105.08059
5. Training data is no longer scarce thanks to the release of fastMRI. Therefore, the author should reorganize their motivation. It is also unclear whether the target and auxiliary images provided to the network inputs and outputs were complex or magnitude? Did the network have separate channels for representing complex numbers? 



detailed_comments: 1. The author has mentioned that “Motivated by potential accuracy gains and computational savings, we study Vision Transformers for accelerated magnetic resonance image reconstruction”. However, as far as I know, Transformer does not save computing power but increases it. 
2. Besides, the results exceeding UNet cannot prove the effectiveness of this method, because UNet tends to smooth the image in the process of reconstruction, as can be seen from the results in Fig. 4 and 6. The authors should use classical reconstruction algorithms as a baseline, such as MoDL and the recently proposed Complex reel-based DONet (DONet: Dual-Octave Network for Fast MR Image Reconstruction (IEEE Transactions on Neural Networks and Learning Systems)), which are designed for the reconstruction of MRI data.
3. What are the inductive biases that the authors have mentioned in the introduction?
4. The authors should reclarify their statement that “it is yet unclear whether, or to what extent, such results are applicable to the arguably more different realm of accelerated MRI,” because there are many works focused on the MRI reconstruction. Please see an incomplete list here,
-Task Transformer Network for Joint MRI Reconstruction and Super-Resolution (MICCAI 2021)
- Accelerated Multi-Modal MR Imaging with Transformers, https://arxiv.org/abs/2106.14248
-Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers https://arxiv.org/abs/2105.08059
5. Training data is no longer scarce thanks to the release of fastMRI. Therefore, the author should reorganize their motivation. It is also unclear whether the target and auxiliary images provided to the network inputs and outputs were complex or magnitude? Did the network have separate channels for representing complex numbers? 



recommendation: []
review_title: null

final_score: rating: 5: Strong accept
review_content: summary: The paper investigates the use of Vision Transformers (ViT) to reconstruct undersampled (i.e., accelerated) magnetic resonance imaging (MRI) acquisition. The authors compare their proposal against the traditional U-net model and the End-to-End Variational network using the fastMRI knee and brain datasets. The authors report superior results to the U-net and End-to-End Variational Network when pre-training the ViT using ImageNet.

strengths: - The paper is well written and the experiments are well designed.
- ViTs are relatively unexplored in the field of MRI reconstruction, making this paper extremely interesting.
- The results are very encouraging.

weaknesses: - The authors report the results in the validation set of the fastMRI datasets. This is okay, but if the authors used the test sets, their results would be directly comparable to all the other entries in the fastMRI public leaderboard (i.e, the current state-of-the-art). 

- Using the root-sum-of-squares of the multi-coil zero-filled reconstructions as the input is probably not optimal. This prevents the addition of data consistency to the proposed model. 

- It seems the authors are reporting results in the validation set and not a proper holdout set.
detailed_comments: - The GitHub link for the authors' code seems to be broken.

- In the results tables, the meaning of the emboldened and underlined metrics is not defined in the captions.

- As future work, it would be interesting to consider including data consistency in the ViT model.
recommendation: Oral
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: This paper applied a vision transformer model to MR image reconstruction task and showed performance on-par with U-Net when enough training data were given and better performance than U-Net when pre-trained on ImageNet dataset and fine-tuned on small MR dataset. The proposed ViT MR reconstruction even beat state-of-the-art VarNet when test on another anatomy.
strengths: There are two strengths in the paper: first, this paper brings freshness into the MR image reconstruction community using deep learning as it is one of the first work applying ViT to MR reconstruction problems; second, the appendix of this paper discussed training details such as training behaviors of U-Net and ViT and performance change with different patch sizes, which are quite informative for readers working on deep learning MR reconstruction and interested in applying ViT to their work. 
weaknesses: I have a few comments:

1. The GitHub link is expired. Please update.
2. How many fine-tuning/training data were used for PT-ViT-L/E2E-VarNet inTable 2 and Fig. 4? Please provide.
3. Following comment 2, it is interesting to see how different training/fine-tuning data affect the anatomy shift performance of PT-ViT-L and VarNet.
4. If possible, it will be very interesting to see how combining VarNet with ViT (e.g. changing the denoising module of VarNet to a ViT-based denoiser) can push the current state-of-the-art MR recon performance.
recommendation: Oral
review_title: null

Meta-review: metareview: After rebuttal, the manuscript receives 2 weak accept, 1 strong accept, and 1 borderline. The authors have clearly clarified the issues raised by the reviewers. 
The only comments are adding more baselines and double-checking the results in figures. 
Overall, most reviewers are satisfied with the response given by the authors and are glad to see that the quality of the paper has been improved substantially. It reaches the minimum requirement for publication.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)