Venue: MIDL 2022 Conference

Reviews:

final_score: rating: 3: Borderline
review_content: summary: This work proposes a method to compute super-resolved MRI images using a combination of generative CNN models, vision transformers (ViT) with an adapted loss function encouring feature disentanglement (structure and texture).

The method is tested on the fastMRI and IXI datasets and compared quantitatively with five different superresolution methods and with different combinations of the proposed loss functions.
Also a few qualitative results are shown.
According to these experiments, the proposed method outperforms the other approaches by a considerable margin.
strengths: The paper presents an interesting combination of several advanced approaches (e.g. vision transformers, self-supervised training, perceptual losses) to tackle MR superresolution. The quantitative results are very good.
weaknesses: The methods are not very well explained in this paper, and this makes it very hard to judge the results for me. This might be an effect of the limited size for a conference paper, but in this case the authors might consider writing a longer, more detailed paper for a journal. In its current form, the paper does not allow the reader to follow the methodological details in my opinion.

The paper should also provide more qualitative (visual) results to e.g. judge the effect of the loss ablations.


detailed_comments: I don't have enough expertise in the field of ViT, but I am a bit puzzled by the quality of the results. From figure 2 it seems like the method is able to reconstruct details that are at the resolution of single pixels and even some noisy texture patterns. Could this be an effect of memorization or overfitting? Can you rule out that this is happening?

How many parameters does the ViT model have? What was it trained on?

The MR images in Fig.2 have the wrong aspect ratio. I assume this is only by accident, but it should be corrected. 

Would it be possible to publish the code? I think this would be important.
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: The authors proposed an MRI super-resolution pipeline incorporating the vision transformer and features disentangle techniques. As a part of the discriminator, the transformer is employed to extract the long-range global features while the feature disentangle aims to pose the constraints on sematic and texture aspects. The experiments show that the pipeline achieves better performance than other most recent baselines.
strengths: The experiments show its better quantitative and qualitative performance than other baselines for MRI SR tasks. Ablation studies also demonstrate the effectiveness of each component in network. Paper is well-organized and easy to follow for readers.
weaknesses: Some important works are missed. 
[1].Feedback Graph Attention Convolutional Network for MR Images Enhancement by Exploring Self-Similarity Features
[2]. Hi-net: hybrid-fusion network for multi-modal mr image synthesis
[3]. Image synthesis in multi-contrast mri with conditional generative adversarial networks.

Though the transformer has a powerful ability to extract global and long-range knowledge, it suffers from memory and computational cost. It is better to provide the time and GPU memory of the network both in the training and test stage.

Please provide the details of the generator and discriminator structure which is easy to be reimplemented.

The transformer backbone of Vit is used here. How about the performance of other transformer backbones such as swin transformer[4]?
[4].Hierarchical Vision Transformer using Shifted Windows

recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: The authors propose to inherit the complementary benefits of combining local information of CNNs and global information from ViTs for MR image super resolution.

The LR image is passed through a generator to generate an HR image.
The LR image and the HR output is passed through both the pretrained Vision transformer and the Variational autoencoder (feature disentanglement network - FDN)
The Pretrained vision transformer classifies the primary region in both LR input and HR output and it includes a cosine loss between them
The FDN gives out content and texture features for both LR and HR output. Model includes a cosine loss term for both the features
The FDN is pre-trained by selecting 2 images at random and while using the second image texture feature is replaced and swap loss is included.
Finally a discriminator is included in the model to ensure the generated image is realistic.

strengths: The work considers preservation of information at different levels of abstraction like semantic information, global and local context, detail (structure and texture) recovery and self-similarity. For an image restoration operator like super-resolution, inheriting these benefits is important.

The authors have taken an emerging trend of using vision transformers from computer vision based deep learning.

The authors propose to combine the structure component of one image with the texture component of another image, which is quite interesting.
weaknesses: Section 3.3 -> should still lead to accurate reconstruction. ----> This seems to be a strong claim without citation or adequate reasoning. The reasoning behind the assistance should be explained clearly. The intuition behind including the swap loss is not clear. In the original paper they swap the texture to actually replace the texture of the original image. 

Equation 8, the image reconstruction loss component is missing. Does it mean that it is not used? There is a separate generator in the model too. 

It is not clear if VAE or AE is used for the concept mentioned in the paper. There is no constraint in the latent space. The base paper is mentioning it as autoencoder https://arxiv.org/pdf/2007.00653.pdf
detailed_comments: outptut -> typo error the introduction

Section 3.3 -> anda

The notations for z_{s} and z{t} are missing in Figure 1b. Instead different notations are provided which affects readability of the paper.

In figure 1b, there is a character x between the red and blue portions, its not clear what this means. Similarly there is another character below the green box.

In the table, the entry for the method proposed by authors should be "Ours" or "Proposed method". In both tables left column has no heading.
recommendation: Poster
review_title: null

Meta-review: metareview: The authors have addressed all the points raised by the Reviewers to some extent, including more experiments and discussion points to support the described method. AnonReviewer3 indicated that "this is a strange observation. But I think the discussion points for consistent reduction in the method by Feng et al needs to be highlighted somehow.". I do agree that it would need further clarification and discussion in the final camera-ready version. However, all Reviewers agree that the paper represents a contribution as a methodology paper and, considering the rebuttal effort, I recommend accepting this work for publication. 
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)