Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed a transformer-based model for medical image segmentation, which adopts the pyramid structure and adversarial training. The proposed method is validated on three benchmark datasets, and obtains advantaged results.
strengths_and_weaknesses: Strengths: 
1. the class-aware transformer module is interesting. 
2. the paper is well written, and there are careful experimental analysis conducted.

Weakness:
1. the novelty and contribution are limited. The key contribution is the class-aware transformer module, a revised transformer, to learn the class aware context. Pyramid structure and adversarial training are common approaches used in medical image analysis.
2.The motivation is unclear. Why the adversarial network is needed in this model?
3. the comparison of experimental results is unfair. The proposed model is equipped with newly-added CAT and GAN, which is a bigger model than others. Even the pre-trained model is compared with other models.
questions: As indicated in this paper, it is a pre-trained model on public computer vision datasets. In the experiments, which computer vision dataset is employed? For medical image segmentation task, is the pre-trained model necessary?
limitations: see weakness
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors introduce an effective generative adversarial transformer called CASTformer for 2D medical image segmentation. The main idea behind the design principle is to integrate the multi-scale pyramid structure to capture rich global spatial information as well as local multi-scale context information. The suggested class-aware transformer module enables CASTformer to discover useful aspects of objects incrementally and selectively. The generator-discriminator design is used to improve segmentation performance, enabling the transformer-based discriminator to capture both low-level anatomical information and high-level semantics.
strengths_and_weaknesses: The proposed generative adversarial transformer called CASTformer for 2D medical image segmentation sounds interesting and useful. Both quantitative and qualitative results show the effectiveness of the proposed model. A number of ablations are performed to show that the suggested mechanisms are worth considering.

The authors provide clear justification for their work. This is a well-written manuscript and easy to follow indeed. The supplementary improves the overall attractiveness of the manuscript as well. The suggested mechanism is straightforward but effective. The authors validate their claims through extensive experiments and significant performance gain.

The study leverages pyramid structure to construct multi-scale representations and handle multi-scale variations. Some of the ideas here may be familiar to readers of the following papers [1, 2]. If these studies have any relevance to the topic at hand, it would be great if the authors would highlight it.

[1] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.

[2] T. -Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, "Focal Loss for Dense Object Detection," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 318-327, 1 Feb. 2020, doi: 10.1109/TPAMI.2018.2858826.
questions: - Could you please check the reference papers and make comments on it with respect to your work?
limitations: Due to nature of the work, as per authors, research will not pose significant risks of societal harm to society. Rather, the study has the potential to positively contribute to a number of real-world clinical applications possibly.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed Transformer based segmentation model for medical image called CASTformer. Compared to the other segmentation algorithms, it utilizes 1) the multi-scale approach with CNN+Transformer hybrid architecture, 2) the class-aware transformer module for progressive sampling strategy, and the adversarial training scheme. Also, the pre-trained model in computer vision dataset helps to start their training in good starting point. The results show that CASTformer improves the segmentation performance impressively.
strengths_and_weaknesses: - (significance) CASTformer shows the dramatic improvement of segmentation performance with large gap to the other algorithms. The author proposed better training processes to utilized the adversarial loss and the computer vision dataset as pre-training to improve the performance of segmentation task for the (relatively-) limited medical dataset.

- (originality & quality) Some reader might claim that they combined many concepts already exists in computer vision field such as multi-scale feature extraction, feature sampling, adversarial loss and transfer-learning. However, it is important that they make that work and it reaches the dramatic performance improvement on their task.

- (clarity)  They support various ablation studies and comparison with many deep network algorithms. However, there are some lack of clarity and I asked those questions in the followings.
questions: - Please refer and explain the differences between your model and the deformable sampling CNN/Transformers: DCN(Deformable convolutional networks), DAT(Vision Transformer with Deformable Attention), Deformable DETR(Deformable DETR: Deformable transformers for end-to-end object detection), and etc. They are using same concept for feature sampling with irregular grid. The comparison results with the other deformable sampling module and the reason why it works better might be helpful for the readers.

- Is there any performance drop using the bilinear interpolation in the feature domain? How much the performance drop for discrete sampling such as nearest neighbor sampling? I have some worries about the existence of continuity in 2D grid features. Please, refer some theoretical background proof or practical evidence for that. 

- Fair comparison between pre-trained model and from-scratch model.
I wonder the epoch numbers for the training of from-pretrained model and from-scratch model is same or not. 

- Ablation study.
As you shown in the analysis, the proposed train processes (such as pre-training on natural image dataset and adversarial loss) seems very effective to improve the performance dramatically. Then, to support the proposed architectural excellence, it is necessary to compare between SwinUnet(or other SOTA architecture) with the proposed  training process(adversarial loss+pre-training with computer vision dataset) and the proposed CASTformer. It is necessary to show that the proposed architecture (class-aware Transformer module, Multi-scale feature extraction, etc) is superior to others.

limitations: 
- Why the term ‘generative’ is used for your model? Does your model generate the image which not exists in the input image? The proposed method utilizes discriminator to improve the performance of segmentation with adversarial auxiliary task for the ROI reconstruction not the generation of some parts not in the input. The outputs are the reconstruction of the attended region of input image, not the generation. In my opinion, the term ‘generative’ should be used at a minimum to maximize the persuasive power of your claim because some (quiet large portion of the medical field) peoples in real clinic have a lot of worries to use generative model for the medical imaging. 


- The term ‘class-aware’ is confusing. 
Some readers in computer vision field or machine learning field might expect that the proposed ‘class-aware’ module dynamically/adaptively utilizes the class information like in StyleGAN. However, the authors used the term ‘class-aware’ for the module which samples discriminative locations by iteratively estimated the sampling offsets based on the current input features. 
The author might say the class-aware Transformer module samples different point according to the type of organ. However, does the class-aware module have ability to distinguish kidney/liver/spleen? I think the sampling module does not distinguish the type of organ. It focuses on the current feature pixel is related/connected to the other feature pixel or not according to the global information. 
To support your claim, the author should provide the experimental support which adding a linear layer on the proposed class-aware module with the freezed weight has some ability to classify the organ type class.
review_title: null

Meta-review: metareview: The paper proposes a generative adversarial approach to 2D medical image segmentation. The problem is a standard one for MRI and improvements in this direction can have real-world impact. The reviewers were on the whole positive in their opinions of the paper. They found the design to be well-motivated and the paper to be well-written and easy to follow. The improvement to current methods was significant enough to be a good reason for acceptance. The reviewers generally found the feedback period to be helpful in swaying them in a more positive direction for accepting the paper.
confidence: Less certain
Decision: Accept