Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the problem of real-time semantic segmentation with Transformer. The authors proposed an RTFormer block with two attention models to aggregate information on different-resolution features. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: [Strengths]
+ The proposed method achieves great performance in the serval datasets.
+ Compared to the baselines, the proposed methods could bring constant improvements 

[Weaknesses] Some important ablation studies are missing.
- The choice of architectural design. The author put the proposed RTFormer block on the last two stages and does not provide the results to support this design.
- The baseline which does not use any attention is needed to be included in Table 3 (a).
- Comparison with the other lightweight attentions is also needed.
questions: The authors argue that GPU-Friendly attention is quite friendly for GPU-like devices. However, according to Table 3, GPU-Friendly attention has a minor effect on acceleration. (189.6->189.8 or 187.9->189.8). Meanwhile, there is no fair and clear comparison between EA or other lightweight attentions and the proposed attention.

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The manuscript presents an efficient model for semantic segmentation. The main contribution corresponds to GPU friendly attention layer which improves the efficiency by using keys and values as learnable parameters. Dimensionality of keys and values is a hyperparameter that is much less than N (HxW). Furthermore, the MLP from the standard transformer is replaced with plain convolutions. The resulting module is somewhat similar to the classic self-attention layer from pre-transformer era. Finally, some further performance improvement is obtained through cross-resolution attention. Experiments address Cityscapes and ADE20k. 

strengths_and_weaknesses: Strengths
- a resonable hybrid model with convolutions on higher-resolution representations and self-attention on lower-resolution representations
- state-of-the-art ratio between performance and computational complexity

Weaknesses
- incremental contribution; GPU friendly attention appears quite related to previous work [13], as well as to Linformer and Nystromformer (see below).
- hybrid convolutional-transformer models have been proposed before (eg DPT hybrid, see below) 
- the proposed improvements perform only slightly better than baselines in Fig.3
- missing configuration in Fig.3a: EA + CA 
- large training footprint: it appears that only 3 crops 512x1024 can fit into a V100 
- incomplete related work in the field of efficient models for semantic segmentation (eg HardNet, SwiftNet, see below)

Missing related work:
- René Ranftl, Alexey Bochkovskiy, Vladlen Koltun. Vision Transformers for Dense Prediction. ICCV 2021: 12159-12168
- Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, Vikas Singh. Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention. AAAI 2021.
- Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma. Linformer: Self-Attention with Linear Complexity. CoRR abs/2006.04768 (2020).
- Marin Orsic, Sinisa Segvic. Efficient semantic segmentation with pyramidal fusion. Pattern Recognit. 110: 107611 (2021).
- Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, Youn-Long Lin. HarDNet: A Low Memory Traffic Network. ICCV 2019
questions: It appears that it would be more appropriate to call this model as a hybrid convolutional/attention model than a transformer.

It would be beneficial to show experiments on some embedded platform.

Please clarify whether you use V100 16GB or V100 32 GB.
limitations: It appears that large memory footprint precludes training on single GPU systems.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes RTFormer for real-time semantic segmentation.  The RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. The authors demonstrate the efficacy of their method on several benchmarks.
strengths_and_weaknesses: Strengths:
1. The proposed method achieves good performances on the benchmarks.
2. This paper is well organized and clearly described.
3. Efficient segmentation is a valuable problem.

Weaknesses:
1. The method proposed in the paper is a hybrid of various existing methods such as linear-complexity self-attention, HRNet, and CNN and transformer hybrid model. Therefore, the novelty is weakened by previous works.
2. This paper does not say whether to use tensorrt to accelerate the model. So I don't know if the comparison is fair.
3. In terms of performance and model size, there is no significant advantage between this method and the compared methods.

questions: Please refer to “Paper Weaknesses”.
limitations: Please refer to “Paper Weaknesses”.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes RTFormer, an efficient transformer for real-time semantic segmentation, which achieves better trade-off between performance and efficiency than CNN-based models. To achieve high inference efficiency on GPU-like devices, the RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. Besides, the cross-resolution attention is more efficient to gather global context information for high-resolution branch by spreading the high level knowledge learned from low-resolution branch. Extensive experiments on mainstream benchmarks demonstrate the effectiveness of the proposed RTFormer, it achieves state-of-the-art on Cityscapes and CamVid, and shows promising results on ADE20K
strengths_and_weaknesses: Strengths of this paper are as follows :
1. A novel RTFormer block is proposed, which achieves better trade-off between performance and efficiency on GPU-like devices for semantic segmentation task.
2. A new network architecture RTFormer is proposed, which can make full use of global context for improving semantic segmentation by utilizing attention deeply without lost of efficiency.
3. RTFormer achieves state-of-the-art on Cityscapes and CamVid, and show promising performance on ADE20K. In addition, it provides a new perspective for practice on real-time semantic segmentation task.

Weakness of this paper are as follows:
1. The proposed cross-resolution attention is just the variant of self-attention, which is widely used in network design. This paper is incremental compared with previous work DDRNet. The novelty of this paper is limited.
2. The performance  improvement on Cityscape dataset is limited. As shown in Table 1, the mIoU score and FPS improvement are both limited, not obvious enough.
3. The experiments on semantic segmentation is thorough enough and can be used to support the proposed method. But my concern is that the method is simple and not novel enough. In addition, why not apply this method to other vision tasks, like classification, object detection. 

questions: The questions are as follows:
1. is it possible to generalise the proposed network to other vision tasks, like image classification, object detection, etc.
2. It seems the attention used is just MHA or self-attention, this is not a new technique. So the main contribution is limited, right?

limitations: yes
review_title: null

Meta-review: metareview: Reviewers agree that the proposed RTFormer block and overall network architecture achieves good trade-off between performance and efficiency on several datasets. The design of GPU-friendly attention and cross-resolution attention improves the computational efficiency over multi-head attention, and well captures global context information when updating high-resolution embeddings.

The main concern, as mentioned by several reviewers, is the overall novelty as some ideas are related to previous work (GPU-friendly attention and hybrid convolutional-transformer architecture). Other issue includes missing baselines that are based on light-weighted attention designs, or do not use attention at all, but this have been well resolved in the author feedback. In summary, the pros outweigh the cons and therefore AC recommends acceptance.
confidence: Certain
Decision: Accept