Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed the HUMUS-Net that used to achieve the goal of reconstructing accelerated  multi-coil MRI images. HUMUS-Net consists several cascades of HUMUL-Blocks, each block complete a step in unrolled iterations of an optimization algorithm to solve inverse problem. Each HUMUL-Block contains three blocks: high-resolution feature extraction, low-resolution feature extraction and high-resolution image reconstruction. Further, the core component of HUMUL-Block was specified. It is a multi-scale hybrid feature extraction component named MUlti-scale residual Swin Transformer network (MUST). It adopted the Residual Swin Transformer Blocks (RSTB) in a hierarchical encoder-decoder architecture to facilitate the multi-scale processing. The MUST is comprised several blocks of RSTB-B, RSTB-D and RSTB-U.  The usage of HUMUL-Block dedicate to solve the regularization term in image domain was indicated. At last, the benefit of reconstruct image using adjacent slices to remove artifacts were explained.
Performance of several state-of-the-art reconstruction techniques on the fastMRI knee multi-coil ×8 dataset were compared with respect to metrics of SSIM, PSNR and NMSE. HUMUS-Net and E2E-VarNet were validated as well on three different datasets.  And the ablation studies have done to justify the proposed design.

strengths_and_weaknesses: Strength：
The proposed HUMUS-Net claimed to have several strengths:
1. The utilization of self-attention based Transformer can overcome the weakness of convolutions, like content independence and the inability to model long-range dependencies. 
2. The multi-scale structure in its core block MUST not only could reconstruct fine details but also reduced the computational cost.
3. The model has achieved the second position in fastMRI leaderboard of ×8 multi-coil knee challenge.
4. The proposed method performs good at three various dataset: fastMRI knee dataset, Stanford2D and Stanford3D dataset.

Weakness：
1. The proposed HUMUS-Net achieved better results than the other methods with respect to metrics of SSIM, PSNR and NMSE. But the radiologist's evaluation on specific images is missing.
2. The HUMUL-Block in HUMUS-Net claimed acts like an intermediate denoiser. But the analysis of this effect is missing. 
3. The ablation experiment of cascade with respect to reconstruction performance is missing.

questions: Detailed comments:
1. The model split input images into patches, and embed the patches into lower-dimensional tokens. The way to produce tokens is not clear.
2. Line 180-182: As shown in Figure 6 in supplementary, HUMUL-Block only dedicates to the regularization term in (Eq 2.2). The high-level unrolled architecture is based on the k-space domain. The expression on line 180-182 seems tell us that each cascade works on the image domain, which is confusing.
3. Line 192: The used resolution of MRI image is not indicated, makes it hard to compare with (32x32-256x256).
4. Line 216-218: The explanation of “token representation”, “learned linear mapping” and “positional encoding” are missing.
5. Line 219: what does the phrase of 'Tokens corresponding to different image patches....in a concise latent representation' means? It is too abstract for me.
6. Figure 2 and 3: The font of the dimension indicator is too small.
7. Figure 3: The reshape blocks are not explained.
8. Line 268: Please specify the dimension of concatenated k-space slices and the dimension of input into the HUMUL-Block
9. Line 329: How does 10 times was calculated?

limitations: some ablation study is not provided.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes HUMUS-Net, a NN architecture that combines convolutions and transformer blocks for compressed sensing MRI reconstruction. The approach achieves SOTA on the benchmark fastMRI dataset with 8x acceleration. The code is available with the supplementary materials.
strengths_and_weaknesses: Strengths:
1. The novel approach paves way for applied MRI models with hybrid convolutional-transformer architectures.
2. The paper carefully describes design decisions and includes ablation study of the model.

Weaknesses: -
questions: 1. The experiments are done on 8x acceleration. It seems that this approach might also be fitting well for further acceleration ratios >8x. Did you consider validating the method on higher acceleration ratios and if not, how do you expect the method to perform? How about lower acceleration, 4x?
2. You mention that the current architecture requires fixed-size inputs. Could you explain why? Both extractor and reconstruction blocks are convolutional, so it shouldn't be the problem.
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper describes a method for accelerated parallel MRI reconstruction using deep neural networks. This is inspired by recent advancements of vision transformer (ViT) and deep unfolding (also known as algorithm unrolling) network for medical imaging. In specific, a new network architecture (including convolutional and self-attention layers) was proposed and used as a denoising prior in the gradient descent unrolling iterations. The proposed reconstruction network, named as HUMUS-Net, is then trained in an end-to-end supervised fashion, where ground truth images are used during training. The authors compare their proposal against existing SOTAs such as U-Net and the End-to-End Variational network (E2E-VarNet) using the fastMRI and Stanford 2D/3D datasets. Numerical results show that HUMUS-Net achieves comparable or slightly better results to E2E-VarNet and significantly outperforms traditional U-Net.
strengths_and_weaknesses: This paper is overall well written and reports competitive results with both quantitative and qualitative validation for fMRI reconstruction. However, the methodological contributions and rationale for using transformer in unrolling are unclear or at least not well illustrated in its current state. The supportive numerical ablation studies (e.g., running time, memory usage, robustness to model shifting, comparison to existing transformer-based fMRI methods) are missing. Please find my technical comments bellow.
questions: 1), The HUMUS-Net models reported in Table 1 do not show a large improvement compared to E2E-VarNet in terms of PSNR and NMSE. This especially true for when models trained on the fastMRI using training dataset only. Since the proposal in this work uses exactly the same unrolling algorithm with jointly coil estimation, it is unclear the rational for using sophisticated transformer as denoising priors, compared to the simple U-Net used in E2E-VarNet. 

2), As mentioned by the authors, The MUST module in this work is inspired by the residual Swin transformer blocks. Consider explaining and discussing the relations and differences between the proposal MUST and Swin transformer in more details. A deeper understanding and intuition will strengthen this work and make its contribution more solid.

3), In abstract it is claimed that the HUMUS-Net consume less memory compared to existing transformer-based methods. The memory consumption of proposal should scale with number of iterations. Will unroll more iterations results in larger memory usage? In addition, are the weights of HUMUS-Net shared cross each iteration? If not, this will also affect the disk storage size. 

4), The ablation studies in Sec. 5.2, comparisons to SwinIR transformer denoiser seems to be unfair and not very informative. As mentioned by the authors, SwinIR was not originally designed for fMRI reconstruction. Using existing transformers for fMRI will be more precise. In addition, I do not think the simple SwinIR (reduced embedding dimension due to lack of gpu memory) is fine-tuned as much as HUMUS-Net in this work. Maybe the authors can use memory efficient training such as checkpoint or deep equilibrium model (DEQ), while keeping the model’s structure as in its original work. 

5), As mentioned earlier, some important numerical results, ablation studies and technical details are missing. 

    (a) What is training loss used in this work?
    (b) PSNR values are missing in Table 2. and Table 3.
    (c) Visual comparisons of the proposal to baseline methods are missing in the main manuscript. 
    (d) Different acceleration ratios (e.g., X4, X6, etc.) to E2E-VarNet are missing.
    (e) Number of parameters and running time comparisons to baseline and other transformer-based fMRI methods, such as [R1, R2] are missing.

6), While this may be orthogonal to this proposal, it would be helpful to discuss the robustness of HUMUS-Net to model shifting (e.g., anatomy shift etc.) and how is the HUMUS-Net generalizable to other medical imaging modalities. These will strength the current paper.

Ref:
[R1] Lin. et al., “Vision Transformers Enable Fast and Robust Accelerated MRI”, MIDL, 2022. 
(Github: https://github.com/MLI-lab/transformers_for_imaging)

[R2] Korkmaz. et al., “Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers”, IEEE TMI, 2022. 
(Github: https: //github.com/icon-lab/SLATER.)

limitations: The limitations of this proposal are missing and need to be discussed in the revision.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new state-of-the-art for accelerated MRI reconstruction. Different components of the model are properly explained by the authors and most of the design choices are well-justified. There are also several concerns/comments pointed out below.
strengths_and_weaknesses: ++ The paper is very well-written and easy to follow. Having a new state-of-the-art for accelerated MRI reconstruction is also highly appreciated as it has not been pushed since 2020.

++ Most of the architectural design choices are properly justified by the authors.

+- The proposed method is very intuitive in most of its aspects (e.g., high- and low-res processing paradigms). However, some explanations are vague, please see the comment below in the questions/suggestions part.

-- Ablation studies are not conclusive (please see the comment below in the questions/suggestions part).

-- Evaluation experiments can be improved w.r.t efficiency of the model and fairness of the comparisons.
questions: The reviewer understands the point with long-range-dependancy modeling of Transformers. However, it is unclear why that should be beneficial in reconstruction tasks such as MRI reconstruction. Note that locality matters for image reconstruction which is captured by convolutions.

While the ablation studies are interesting, two important parts are missing: (1) Adjacent Slice Reconstruction (ASR), and (2) Residual learning. It would be interesting to see how the E2E-VarNet performs if these two elements are added. This is important since if with these modifications, the performance of E2E-VarNet matches HUMUS-Net, then **it sends the message that a complicated HUMUS-Block is not necessary and one can simply regularize the unrolled problem with a simple U-Net (as E2E-VarNet currently does)**.

Given the marginal improvement of HUMUS-Net over E2E-VarNet, what are some competitive advantages of HUMUS-Net that could convince someone to use it instead of E2E-VarNet? e.g., is it computationally cheaper to train? Does it require less training data? Is it more robust? etc. Any explanations in this direction are highly appreciated.

It would be nice to add the number of parameters of each model to Table 2 to assure the reader of the fairness of the comparisons.

Typos

- Line 193: a significant challene=
- Line 199: In this work, 
limitations: The reviewer can discuss limitations of the work better if the authors answer the questions/concerns raised above in the Questions/Suggestions section.
review_title: null

Meta-review: metareview: Four reviewers generally favour accepting the paper, and I agree. The authors have done a good job of addressing the most pressing concerns of the reviewers in the rebuttal period.
confidence: Certain
Decision: Accept