Venue: MIDL 2023 Conference

Reviews:

final_score: rating: 4: Weak accept
review_content: summary: This work proposed a lightweight transformer model (MAT) in integrating 1) gated axial-attention blocks and 2) warm-upped self-distillation from last mini-batch (DLB) method. The proposed method was evaluated on both BraTS2018 and BraTS2021 datasets. The experimental results showed that MAT outperformed 2 2D and 2 3D SOTAs with fewer parameters, especially on the BraTS2018 dataset.
strengths: 1. The manuscript is overall well-written.
2. The proposed MAT achieved better or similar performance compared with 4 SOTA methods on BraTS2018 and BraTS2021 datasets.
3. An ablation study has been conducted to demonstrate the importance of warm-upped in DLB, especially for small datasets.
weaknesses: 1. The novelty of MAT is somewhat limited since the proposed structure is fundamentally the same as the local branch of MedT (Valanarasu et al., 2021).
2. The generalizability of MAT may be limited since it is only evaluated on brain tumor segmentation (BraTS datasets) and the improvement becomes less significant with more training data.
detailed_comments: 1. Please provide mean and std in Table 1, Table 2, Table 3, and Figure 3.
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: The author developed a lightweight 3D segmentation transformer based upon the axial attention. A self-distillation method is involved during the optimization of the network. The evaluation is conducted on the BraTS dataset and the results on BraTS2018 is well improved comparing with other methods. The 'lightweight' property is shown by the number of parameters and the effectiveness of the self-distillation is proved by an ablation study.
strengths: - The proposed method is clearly explained so the idea is very easy to follow.
- The experiment result looks promising and the stats of the model parameters and the ablation study illustrate the effectiveness of the axial attention mechanism and the self-distillation.
weaknesses: (1) There is lack of novelty in terms of methodology since the paper is a direct implementation work that combines:
>- The axial attention idea from (Valanarasu et al, 2021)
>- The self-distillation idea from (Shen et al, 2022)

(2) In the evaluation, the UNETR should be included as a baseline method as it is one the most popular application of 3D transformer in medical image segmentation.

(3) Some of the annotation in the paper need to be modified to avoid potential confusion. 
>- In the equation for the axial attention, the query, key and value vectors should be lowercase. The current version is confusing as they are the same with the superscripts. The subscript indicating the position should be consistent.
>- In 2.2.2, avoid using T1, T,2 T3 to annotate the stages as T1 and T2 are modalities as well.

detailed_comments: (1) In the evaluation, the UNETR should be included as a baseline method as it is one the most popular application of 3D transformer in medical image segmentation.

(2) Some of the annotation in the paper need to be modified to avoid potential confusion. 
>- In the equation for the axial attention, the query, key and value vectors should be lowercase. The current version is confusing as they are the same with the superscripts. The subscript indicating the position should be consistent.
>- In 2.2.2, avoid using T1, T,2 T3 to annotate the stages as T1 and T2 are modalities as well.
recommendation: Poster
review_title: null

final_score: rating: 2: Weak reject
review_content: summary: The paper proposes a lightweight transformer model for 3D medical image segmentation. The model consists of transformer encoder and CNN decoder. The method incorporates axial attention, which applies attention to each dimension separately, and self-distillation to achieve remarkable performance on the Brats18 and Brats21 datasets.
strengths: - relevant selection of the baselines
- the method description is clear, easy to follow, and the architecture seems to be easy to implement
- the transformer-based method that can directly utilize 3d images
weaknesses: - poor organization of the results section. Large images from Qualitative results could be moved to the appendix or compressed. It's confusing that validation on the larger dataset is missing in the main text. it would be helpful to have data from appendix A in the results table, as well as FLOPs count and computational time, as transformers typically are more computationally expensive.
Data from Appendix C would be also helpful in Tables 1 and 2.

- design of the experiments 
In section 3.1.1, were hyperparameters tuned using the validation split, and how is the final model selected? CV scheme or testing dataset is missing in the experiment design.


minor:
Evaluation metrics: formulas are redundant and can be moved to appendix if needed. The freed space can be used for brats 2021 results
What does "HD95 is the output of HD multiplied by 95%" mean?
recommendation: Poster
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: a lightweight 3D transformer was proposed for medical image segmentation. The authors mainly use axial transformer to reduce the model size, and combine it with a U-Net like structure. Results on BraTS dataset showed its effectiveness. It was compared with an existing 3D transformer. The authors also employed self-distillation of the model for performance improvement. 
strengths: 1, The motivation of developing lightweight models is clear, and the proposed model can reduce the computational cost for medical image segmentation. 

2, The authors conducted experiments on two public datasets to show its effectiveness. 

3, The paper is generally smooth and easy to follow. 

weaknesses: 1, The novelty of the proposed method is a bit limited. The main idea in this work is axial transformer, but it has been previously proposed by Valanarasu et al., 2021). The self-distillation method has also been previously proposed by Shen et al. 2022. 

2, In the experiment, the authors only compared the proposed model with one 3D transformer network (AFTerUNet). Recently, there have been several works on 3D transformer such as CoTR mentioned by the authors. The proposed method should be compared with these models. 

3, The authors claimed to propose a lightweight model, but the model size was not listed in the text. It would be good to provide such information in Table I. 

detailed_comments: 1, The Dice and HD values were only given with mean values. How about the std? In addition, statistical testing results are encouraged to be given to show if the performance improvement was significant.  

2, The implementation of self-distillation was not clear. Shen et al keep half of the mini-batch the same in two neighboring forward passes, did the authors implement in the same way? Also, there was no ablation study to show the effectiveness of the self-distillation. 

recommendation: []
review_title: null

Meta-review: metareview: The reviewers appreciated the experimental and writing quality of this paper, which introduces a lightweight Transformer model. A majority of the reviewers expressed their recommendation of “weak accept”. The rebuttal for the negative review (“weak reject”) is adequate. Thus, my recommendation leans towards acceptance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)