RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer
Authors: Jian Wang
Chenhui Gou
Qiman Wu
Haocheng Feng
Junyu Han
Errui Ding
Jingdong Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Recently, transformer-based networks have shown impressive results in semantic segmentation. Yet for real-time semantic segmentation, pure CNN-based approaches still dominate in this field, due to the time-consuming computation mechanism of transformer. We propose RTFormer, an efficient dual-resolution transformer for real-time semantic segmenation, which achieves better trade-off between performance and efficiency than CNN-based models. To achieve high inference efficiency on GPU-like devices, our RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. Besides, we find that cross-resolution attention is more efficient to gather global context information for high-resolution branch by spreading the high level knowledge learned from low-resolution branch. Extensive experiments on mainstream benchmarks demonstrate the effectiveness of our proposed RTFormer, it achieves state-of-the-art on Cityscapes, CamVid and COCOStuff, and shows promising results on ADE20K.

RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer
Jian Wang Chenhui Gou Qiman Wu
Haocheng Feng'
Junyu Han'
Errui Ding' Jingdong Wang"
Baidu VIS Australlian National University(ANU)
(wangjian33, wuqiman, fenghaocheng, hanjunyu, dingerrui, vangjiangdong)@aaddu..oo
u71 u7194588@anu. weteeaaaauueeeuuaaa... edu.au au
Abstract
Recently, transformer-based networks have shown impressive results in semantic segmentation. Yet for real time semantic segmentation, pure CNN-based ap- proaches still dominate in this field, due to the time-consuming computation mechanism of transformer. We propose RTFormer, an efficient dual-resolution transformer for real-time semantic segmenation, which achieves better trade-off between performance and efficiency than CNN-based models. To achieve high inference efficiency GPU-like devices, RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. Besides, we find that cross-resolution attention is more efficient to gather global context in- formation for high-resolution branch by spreading the high level knowledge learned from low-resolution branch. Extensive experiments demonstrate the effectiveness of our proposed RTFormer, it achieves state-of- state-of-the-arr on Cityscapes, Vid and COCOStuff, and shows promising results on ADE20K. Code is available at PaddleSeg 24] https://github.com//
mainstream benchmarks
1 Introduction
Semantic segmentation is fundamental computer vision task which usually serves as critical percep- tion module in autonomous driving, mobile applica- tions, robot sensing and so on. pace with the development of these applications, the demand of executing semantic segmentation in real-time grows stronger increasingly. Existing real-time segmenta- tion methods mainly focus on exploiting CNN ar- chitectures, including designing high-efficiency back- bones and decoders 48,
AlFoauz: Bac
KTTaman-Sir
Accumey (mloU%
SPNH
STACI Seg
BitaNen
ICNA
42
17
21
13
6
28
DFANS
by handcraft and exploring neural architecture search
methods to find better trade-off between accuracy and
efficiency [47 23 22 99. And significant improvement has been achieved by these great works so far.
Inference Speed (FPS)
More recently, vision transformers have been drawn lots of attention for their strong visual recognition capability [2 31 25
Figure 1: Accuracy(mloU % vs. Infer- ence (PPed(FPSS) on Cam Vid test set. Our methods are presented in red dots while other methods are presented blue dots.
35 36) And inherited from them,
Equual Contribution. Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

series of transformer-base architectures like
38 44 30 are proposed and show very promising
performance on general semantic segmentation task. Comparing with CNN-based networks, the main distinction of these transformer-basee architectures is the heavy usage of self-attention, and self-attention is good at capturing long-range context information which is essential in semantic segmentation. So, we consider that attention structure should also be effective in real-time semantic segmentation task.
But up to now, only few works 38] have explored the application of attention in this field, and the state-of of-the-arts are still dominated by CNN- architectures. We suppose the main obstacles of applying attention in real-time setting might come from the following two aspects. One is that the computation properties of most existing types of attention are not quite inference friendly for GPU-liked devices, such as the quadratic complexity and the multi-head mechanism. The quadratic complexity introduces large computation burden when processing high resolution features, especially in dense prediction tasks like semantic segmentation. Although several works such as 35| 38 shrink down the size of keys and values, the property of quadratic complexity remains. While the multi-head mechanism splits the matrix multiplication into multiple groups, which makes the attention operation to be time consuming on GPU-like devices, analogue to the situation that executing group convolution. The other is that conducting attention only on high resolution feature map itself like 44 may not
be the most effective way for capturing long-range context with high level semantic information, as a single feature vector from high resolution feature map has limited receptive field.
We propose a novel transformer block, named RTFormer block, as shown in Figure which aims to achieve better trade-off between performance and efficiency on GPU-like devices with transformer. For the low-resolution branch, a newly proposed GPU--riendly Attention, derived from external attention [5]. is adopted. It inherits the linear complexity property from external attention, and alleviates the weakness of multi-head mechanism for GPU-like devices by means of discarding the channel split within matrix multiplication operations. Instead, it enlarges the number of external parameters and splits the second normalization within double-norm operation proposed by external attention into multiple groups This enables GPU-Friendly Attention to be able to maintain the superiority of multi-head mechanism to some extent. For the righuresolttion branch, we adopt cross-resolution attention instead of only conducting attention within high- resolutioll feature itself. Besides, unlike the parallel formulation of multi-resolution fusion from 32 17 44], we arrange the two resolution branches into stepped layout. Therefore, the high-resolution branch can be enhanced more effectively by the assistant of the high level global context information learned from low-resoloe rew-uesouuion branch. Based on the proposed RTFormer block, we construct new real-time semantic segmentation network, named RTFormer. In order to learn enough local context, we still use convolution blocks at the earlier stages and place RTFormer block at the last two stages. By taking extensive experiments, we find RTFormer can make use of global context more effectively and achieve better trade-off than previous works. Figure D shows the comparison between RTFormer with other methods on Cam Finally, we summarize the contribution of RTFormer as following three aspects:
A novel RTFormer block is proposed, which achieves better trade-off hetween performance and efficiency on GPU-like devices for semantic segmentation task.
A new network architecture RTFormer is proposed, which can make full use of global context for improving semantic segmentation by utilizing attention deeply without lost of efficiency.
RTFormer achieves state-of-the-ar on Cityscapes, Cam and COCOStuff, and show promising performance ADE20K. In addition, it provides a new perspective for practice on real-time semantic segmentation task.
2 Related Work
Generic Semantic segmentation. Traditional segmentation methods utilized the hand-crafted fea- tures to solve the pixel-level label assigning problems.e.gg threshold selection 29 the super-pixel
and the graph algorithm 3 With the success of deep learning, a series of methods 7 2 50) based on
FCN (fully convolutional neural network) [26] achieve superior performance
various benchmarks.
These methods improved FCN from different aspects. The Deeplabv3 7 and the PSPNet (50) enlarge
the receptive field and fused different level features by introducing the atrous spatial pyramid pooling

---

Coom-Attenion
Pooling
NEd
Illustration of RTFormer block. For low resolution, GPU-Friendly Attention is applied. And for high resolution, we use Cross-resolution Attention which draws K and from low resolution branch. Besides, we make up FFN with two 3 x 3 convolution layers.
Figure 2:
module and the pyramid pooling module. The SegNet recovers the high-resolution map through the encoder-decoder structure. HRNet [32] introduces multi-resolution architecture which maintains high-resolution feature maps all through the network. OCRNet [43 enhances the feature outputted from backbone by querying global context.
Real-time Semantic segmentation. To solve the real-time segmentation problem, various 13 9 have been proposed. ICNer(48) solves this problem by using a well-designed multi-resolution image cascade network. FasterSeg[9 [ utilizes neural architecture search (NAS) to reach the goal of balancing high accuarcy and low latency. BiSeNetV [[42] and BiScNetV2 41 adopt two-stream paths network and feature fusion module achieve a well balance between speed and segmentation performance. STDC[13] rethinks and improves BiSeNet by proposing single-steam structure with detail guidance module. DDRNets [7] achieves better performance by designing two deep branch network with multiple bilateral fusions and a Deep Aggregation Pyramid Pooling Module.
methods 48
42
41
17
Attention mechanism. The attention mechanism has been vigorous developed in computer vision field [37 45
14, 39 SE block proposed by [19] applies the attention function to channels and improves the representation capability of the network. [8]] uses attention-module model object relation and help the objection detection. 377 presents a non-local operation which can capture the rangrange dependencies and shows promising results video classification task. 39 uses attention in point cloud recognition task. Self-attention is a special case of the attention mechanism that has been widely used in recent years [45] 33 However, the quadratic complexity limits its
18
19
usage. Some works 15 34 40 reform self-attention to achieve linear complexity. But they are still
friendly for inference on GPU. Inspired by external attention [15], we developed a GPU-Friendly attention module that has low latency and high performance on GPU-like devices.
Transformer in Semantic segmentation. Very recently, transformer shows promising performance in semantic segmentation. DPT (30] applies transformer encoder to improve the performance of dense prediction task. SETR [1 proposes a sequence-to sequence method and achieves impressing
result. SETR uses pretrained VIT [46] as its backbone and has downsampling in spatial resolution. However, it is difficult to use it for real-time segmentation task due to its heavy backbone and very high resolution. SegFormer (38 increases efficiency by introducing hierarchical transformer encoder and a lightweight all MLP decoder. Compared with SETR, SegFormer has both higher efficiency and higher performance. However, the efficiency of SegFormer is still relatively low compared to some state-of the-art-tt CNN based real-time segmentation model. By introducing our RTFormer block, our method can take advantage of the attention mechanism while achieving the real-time speed.
3 Methodology
In this section, we elaborate the details our proposed approach. We first describe the RTFormer block, then present how to construct RTFormer based on RTFormer block.

---

3.1 RTFormer block
RTFormer block is dual-resolution module which inherits the multi-resolution fusion paradigm from 17 44]. In contrast to the previous works, RTFormer block is comprised of two types of attention along with their feed forward network, and arranged as stepped layout, as shown in Figure In the low-resolution branch, we use GPU-Friendly Attention to capture high level global context. While in the high-resolution branch, we introduce cross-resolution attention to broadcast the high level global context learned from low-resolution branch to each high-resolution pixel, and the stepped layout is served to feed more representative feature from the low-resolution branch into the cross-resolution attention.
Comparing the dif- ferent existing types of attention, we find that external attention (13) can be potential choice for being executed on GPU-like devices due to its gratifying property of linear complex- ity, and our GPU-Friendly Attention(GFA) is derived from it. Thus, before detailing GFA, we review EA first. Let X E Nxd denotes an input feature, where N is the number of el- ements(or or pixels in images) and d is the feature dimension, then the formulation of EA can be expressed as:
GPU-Friendly Attention.
Nocza
Impui
Aliziim (8 N
Outpst (N dd
KM dd
Muhi heed EA
Nem
EA(X, DN(X K²) V
(L)
Input
Mg]
Outpor (V *v)
where K, V E Mxd are learnable parameters, M is the parameter dimension, DN is the Dou- ble Normalization operation proposed by[[5]. And the meati-head version of EA can be ex- pressed as:
Kg (Mg
(M 710
GFA
Figure 3: Comparison between Multi-Head Exter- nal Attention and GPU-Friendly Attention. Multi- head external attention splits the matrix multiplica- tion into several groups while our GPU-Friendly Attention makes matrix multiplication to inte- grated which is more friendly for GPU-like de- vices.
MHEA(X) = Concal(hi I2,,.,, hr) h4 EA(X,, K' VV), H
(2)
where K", V E Mxd d' ===1 H/ and is the number of heads, while X; is the ith head of X. As shown in upper part of FigureB3 the multi-head mechanism generates H attention maps for improving upon the capacity of EA, and this makes the matrix multiplication be splitted into several groups, which is similar as group convolution. Although EA uses shared K' and V for different heads, which can speed up the calculation a lot, the splitted matrix multiplication remains.
To avoid the latency reduction GPU-like devices due to the multi- head mechanism, propose simple and effective GPU-Friendly Attention. It evolves from the basic external attention expressed by Equation| I which can be formulated as:
GFA(X, V₂) GDN(X KK)) VV,
(3)
where Kg. Vg E M, M, M H and GDN denotes Grouped Double Normalization, which splits the second normalization of the original double normalization into H groups, as shown in the left lower part of Figure From Equation 3 we can find that GFA has two main improvements. the one hand, it makes the matrix multiplication to be integrated, which is quite friendly for GPU-like devices. Benefit from this, we can enlarge the size of external parameters from (M, ) to (My, d). Therefore, more parameters can be tuned for improving the performance. On the other hand, it maintains the superiority of multi-head mechanism to some extent by taking advantage the grouped double normalization. For intuitive comprehension, it can be regarded that GFA also generates H different attention maps for capturing different relations between tokens, but more feature elements involved for computing similarity and all the attention maps contribute to the final output.

---

stem
stagel
singe2
stage5
stage4
stages
head
1000
10-F D
strided conv
upsample
Conv block
block
module
Figure 4: Illustrating the RTFormer architecture. We place RTFormer block at the last two stages which indicated by pink block and use convolution blocks the earlier stages which indicated by blue block. Besides, we add DAPPM module for segmentation head, drawing the successful experience from[17]
Cross-resolution Attention. Multi-resolution fusion has been proven to be effective for dense prediction task. And for the design of multi-resolution architecture, can intuitively apply the GFA in different resolution branches independently, and exchange features after the convolution module attention module being executed like 32 44
But in high-resolution branch, pixels focus on local information more than high level global context. Thus, we suppose that directly conducting attention high-resolution feature map for learning global context is not effective enough. To obtain the global context more effectively, we propose a cross-resolutii full use of the high level semantic information learned from low-resolution branch. As exhibited in Figure 2 unlike GFA, cross-resolution attention is adopted in high-resolution branch for gathering global context. And the calculation of this rros-reesouutonn attention is expressed as:
attention, which aims to make
Xn - KT
CA(X,, Key Vo) -SSflmar
) .V.
ddn
Ke, V. = (X.) X. = 0(xi)
where Xn, X₁ denote the feature maps on high-resolution branch and low-resolution branch respec- tively, Ø is set of matrix operations including splitting, permutation and reshaping, d, means the feature dimension high-resolution branch. is worth to explain that, the feature map X denoted cross cross-feature in the following text, is computed from X by function A which is composed of pooling and convolution layers. And the spatial size of X, indicates the number of tokens generated from the X,. Experimentally, ization, as a single softmax performs better than double normalization when the key and value are not external parameters. Specially, for fast inference GPU-like devices, multi-head mechanism also discarded here.
only adopt softmax upon the last axis of attention map for normal-
Feed Forward Network. In the previous transformer-basee segmentation methods like 38
the Feed Forward (etwork(FFN) is typically consist of two MLP layers and a depth-wise 3 x 3 convolution layer, where the depth-wise 3 layer works for supplementing position encoding or enhancing locality. Besides, the two MLP layers expand the hidden dimension to be two or four times of the input dimension. This type of FFN can achieve better performance with relative less parameters. But in the scenario where latency on GPU-like devices should be considered, the typical structure of FFN is not very efficient. In order to balance the performance and efficiency, adopt two 3 x convolution layers without dimension expansion in the FFN of RTFormer block. And it shows even better result than the typical FFN configuration.
RTFormer
Figure iillustrates the overall architecture of RTFormer.
Backbone Architecture. For extracting enough local information which is needed by high-resolution feature map, combines convolution layers with proposed RTFormer block to construct
5

---

Table 1: Detailed configurations of architecture variants of RTFormer.
Models
#Channels
#Blocks
Spatial size of cross-feature 8 x 8
RTForner-Slim
[32, 64, 64/128, 64/256, 64,256)/26
2, 1,
RTFormer-Base
(64, 128, /28, 128/512, 128/512]
2/2,1/2,1,
12 x 12
RTFormer. Concretely, we let RTFormer start from a stem block consist of two 3 3 convolution layers and make the first two stages with several successive basic residual Blocks[T6] Then, from stage3, we use dual-resolution modules which enable feature exchange between high-resolution and low-resolution branches, inspired by [7]. And for the high-resolution branches of the last three stages, the feature strides keep as 8 unchanged, while for the res-uesouttion branches, the feature strides are 16, 32, 32 respectively. Specially, we arrange the dual-resolution module into stepped layout for boosting the semantic representation high-resolution feature with the help the output of low-resolution branch. Most importantly, we construct the staged and stage5 with our proposed RTFormer block which illustrated in Figure 2 for efficient global context modeling, while the stage3 is still composed by basic residual blocks.
Segmentation Head. For the segmentation head of RTFormer, we add DAPPM module after low-resolution output feature, drawing the successful experience from[[7] And after fusing the output of DAPPM with high-resolution feature, we obtain the output feature map with stride=S. Finally, this output feature is passed into pixel-level classification head for predicting dense semantic labels. And the classification head is consist of a 3 x 3 convolution layer and a 1 convolution layer, with the hidden feature dimension being same with input feature dimension.
Instantiation. We instantiate the architecture of RTFormer with RTFormer- Slim and RTFormer- Base, and the detailed configurations recorded in Table For the number of channels and number of blocks, each array contains elements, which are corresponding to the stages respectively. Especially, the elements with two numbers are corresponding to the dual-resolution stages. For instance, 64/128 /18 means the number of channels is 64 for high-resolution branch and 128 for low- resolution branch. While 1/2 means the number of basic convolution blocks is for high-resolution branch and 2 for low-resolution branch. It is worth to be noted that, the last two elements in block number array denote the number of RTFormer blocks, and they both for RTFormer-Slim and RTFormer- Base. The spatial sizes of cross-feature are set as 64(8 x 8) and 144(12 x 12) for RTFormer-Shi and RTFormer- Base respectively.
Experiments
In this section, we valid RTFormer on Cityscapes TO), Camvid ADE20K 52 and COCOStuff
We first introduce the datasets with their training details. Then, we compare RTFormer with state-of- the-art real-time methods on Cityscapes and Vid. Besides, more experiments on ADE20K and COCOStufi(S) are summarised to further prove the generality of our method. Finally, ablation
studies of different design modules within RTFormer block on ADE20K 52) are provided.
4.1 Implementation Details
Before finetuning on semantic segmentation, all models are pretrained on ImageNet And the training details for ImageNet will be provided in the supplementary material. We apply mloU and FPS as the metrics for performance and efficiency respectively, and the FPS is measured on RTX 2080Ti without tensorrt acceleration by default.
Cityscapes. Cityscapes 10 is widely-used urban street scene parsing dataset, which contains 19 classes used for semantic segmentation task. And has 2975, 500 and 1525 fine annotated images for training, validation, and testing respectively. We train all models using the AdamW optimizer with the initial learning rate 0.0004 and the weight decay of 0.0125. We adopt the poly learning policy with the power of 0.9 to drop the learning rate and implement the data augmentation method including random cropping into 512 x 1024, random scaling in the range of 0.5 to 2.0, and random trained with 484 epochs (about 120K iterations), batch size
horizontal flipping. All models of 12, and syncBN on four V 100 GPUs For a fair comparison with other algorithms, online hard example Oini) is not used.

---

Table Comparisons with other state-of-the-art real-time methods on Cityscapes and Cam Vid. Performances are measured with a single crop of 1024 x 2048, 720 x 960 for Cityscapes and Cam Vid respectively. #Params refers to the number of parameters. FPS is calculateted under the same input scale as performance measuring. In this table, means we retrain this method follows its original training setting, and + means we measure the FPS on single RTX 2080Ti GPU.
Cityscapes
Cam Vid
Method
Encoder
#Params
GPU
FPS1
val mloU($))
FPS1
test mloU(S))
ICNet
49
TitanX M
30.3
67.7
27.8
67.1
DFANet A
20
Xception A Xception
7.8M 4.8M
TitanX
100.0
120.0
64.7
DFANet
20
TitanX
120.0
160.0
59.3
CAS 47
TitanX
108.0
71.6
169.0
71.2
GAS
23
TitanX
72.4
72.8
DFI-Seg-d8
DF1
1080Ti
136.9
72.4
DFI-Seg 22
DFI
GTX 1080Ti
106.4
74.1
DH2-Seg1
22
DH2
GTX 1080Ti
67.2
75.9
DF2-Seg2
DF2
GTX 1080Ti
56.3
76,9
BiSeNet1
42
Xception39
5.8M
GTX 1080Ti
105.8
69.0
175.0
65.6
BiSeNei1
ResNet18
49.0M
1080Ti
116.3
68.7
BiS
GTX 1080Ti
156.0
73.4
124.5
72.4
BiSeNetV2-L
41
GTX 1080Ti
47.3
75.8
32.7
73.2
SFNet 211
ResNet18
12.9M
RTX 2080Ti
62.9
73.8
FasterSeg 0
4.4M
RTX 2080T:
136.2-
73.1
71.1
STDC1-Seg75 1+
STDCI
14.2M
RTX 2080TI
74.6
74.5
STDC(2-Seg75 13
STIX2
22.2M
RIX 208011
73.5+
77.0
STDCI-Seg
STDC1
14.2M
RTX 2080TI
125.6+
73.0
STDDC2-Seg
STDC2
22.2M
2080T;
100.5+
73.9
DDRNet-23-Sim
5.7M
2080TI
76.1
217.0
74.7
DDRNet-23 17
20.1M
RTX 2080T:
38.34
78.9*
97.1+
76.3
RTFormer-SSim
4.8M
RTX 2080TI
110.0-
76.3
190.74
81.4
RTFormer- (urr))
16.8M
RTX 2080TI
39.1
79.3
94.0
82.5
Cam Cam Vid[4] contains 701 densely annotated frames and the resolution of each frame is 720 x 960. These frames are divided into 367 training images, 101 validation images, and 233 testing images. 44d have 32 categories which has the subset of 11 classes are used for segmentation experiments. We merge the training set and validation set for training and evaluate our models on the testing set. We set the initial learning rate 0.001 and the weight decay to 0.05. The power of poly learning policy is set to 1.O. We train all models for 968 epochs. Data augmentation includes color jitter, random horizontal flipping, random cropping into 720 960 and random scaling of [288, 1152]. Unlike previous methods [33, we do not pretrain our model on Cityscapes 10]. All other training details are the same for Cityscapes[10) [0].
ADE20K. ADE20K
scene parsing dataset covering 150 fine-graine semantic concepts,
which split 20K, 2K, and 3K images for training, validation, and testing, respectively. Our models are trained with batch size of 16 for 160k iterations. And we set the initial learning rate to 0.0001 and the weight decay to 0.05, and the other training settings are identical to those for Cityscapes 10)
COCOStuff. (OCOStufT(5 is a dense annotated dataset derived from COCO. It contains 10K images (9K for training and IK for testing) with respect to 182 categories, including 91 thing and 91 stuff classes. And 11 of the thing classes have annotations. We train RTFormer 110 epochs on COCOStuff with Adam W optimizer, and the initial learning rate and weight decay are set as 0.0001 and 0.05 respectively. In the training phase, we first resize the short side of image to 640 and randomly crop 640 x 640 patch for augmentation. While in the testing phase, we resize all images into 640 x 640. Other training settings are identical to Cityscapes.
4.2 Comparison with State-of-the-arts
In this part, compare RTFormer with state-of-the-art methods Cityscapes [0] and Cam 4). Table 2 shows our results including parameters, FPS and mloU for Cityscapes [10] and Cam 4..
7

---

Image
(b) Ground Truth
(c) DDRNet-23
(d) RTFormer--Bee
Figure 5: Qualitative results on Cam [id testing set. Compared to 23RNet-23 RTFormer
predicts masks with finer details and reduces long-range errors as highlighted in white.
Table 3: Comparisons with other state-of-the-art real-time methods ADE20K. The #Params, FLOPs and FPS are measured at resolution 512 x 2048. #Params refers to the number of parameters. In this table, means that retrain this model by ourself ADE20K + means we measure the FPS using single RTX 2080Ti GPU. Method without + is using its reported #Params, FLOPs, FPS and mloU.
Method
Encoder
#Params
FLOPs|
FPSt
val mloU(()t
FCN 27
MobileNetV2
9.8M
39.0G
64.4
19.7
PSPNet
50
MobileNetV2
13.7M
52.9G
57.7
29.6
Deepl.ab
MobileNetV2
15.4M
69.4G
43.1
34.0
SegFormer [38]
MïT-BO
3.8M
8.4G
50.5
37.4
DDRNet-23-Slim
5.6M
18.2G
189.14
33.3*
DDRNet-23(17)
20.
71.6G
71.2+
38.8*
RTFormer-Slim(Our) WTFormer-BaseeOurr
4.8M
17.5G
187.9
36.7
16.8M
67.4G
71.4
42.1
Results. On Cityscapes [0], our RTFormer owns the best speed-accuracy trade-off among other real-time methods. For example, RTFormer-SHe achieves 76.3% mloU at 110.0 FPS which is faster and provides better mloU compared to TTDC2-Seg755 [3] and DDRNet-22. DORRNet223-Sim 14]. Besides, RTFormer- EFFormee-BBsc achieves 39.1 1 FPS and 79.3% mloU which establishes new state-of-the-an result. Further more, using only ImageNet([I] pre-training, our method achieves 82.5% mIoU at 94.0 FPS Cam CamVid[4], significantly outperforms all other real-time methods including STDC2-Seg [3] which uses additional Cityscapes 10 pre-training. Moreover, Our RTFormer-Shin yields 81.4 mloU 190.7 FPS with only 4.8M, which is faster and better than other models like STDC2-Seg, [3] 125.6FPS and DDRNet-23 [7] at 97.1FPS. Figure shows the qualitative results on Cam Vid testing set, where RTFormer-bar provides better detail than 2DRNet-23 especially for the Column Pole class. which requires more global context. In summary, these results demonstrate the superiority of RTFormer real-time semantic segmentation terms of accuracy, latency, and model size.
4.3 Generalization Capability
To further prove the effectiveness of our RTFormer on more generalized scene, we show additional results on ADE20K 52 and 3OCOStuft
Results. Table 3 presents our result on ADE20K Our RTFormer-Base archieves the superior mloU of 42.1 42.1% % and with 71.4FPS, which outperforms all other methods. For instances, in contrast to DDRNet-23-Slim [17], RTFormer-Slim achieves better mloU 36.7% and maintains nearly the same speed. Figure 6 shows qualitative results ADE20K validation set. Compared with 23

---

Table 4: Comparisons with other state-of-the-art real-time methods on COCOStuff. The #arams, FLOPs and FPS are measured at resolution 640 x 640.
Method
GPU
#Params.
FLOPs4
FPSt
test ml)
PSPNet50 50
6.6
32.6
[C9
TitanX M
35.7
29.1
BiSeNetV2
GTX 1080Ti
87.9
25.2
BiSeNerV2-1.
GTX 1080Ti
42.5
28.7
DDRNet-23
RTX 2080Ti
20.
28.1G
146.1
32.1
RTFormer-Bas
RTX 2080Ti
16.8M
26.6G
143.3
35.3
(a) Image
(b) Ground Truth
(c) DDRNet-23
(d) RTFormer-bar
Figure 6: Qualitative results ADE20K validation set. As shown, RTFormer is good at
focusing on global context.
our RTFormer shows better details and context information. In summary, these results demonstrate that RTFormer also shows very promising performance on real-time semantic segmentation in generalized scene. While on COCOStuff, as shown in Table 4 our RTFormer-Bas achieves 35.3 mloU 143.3 FPS, which outperforms the DDRNet-23 about 3% with comparable inference speed and sets new state-of-the-as
4.4 Ablation study on ADE20K
Training Setup. We provide ablation results with RTFormer-Slin To make quick evaluations, train RTFormer-slim from scratch with the initial learning rate being set to 0.001, and the other training settings same with experiments ADE20K [52 above. More experimental details and analyses are claborated in the supplementary material.
Comparison on different types of attention. To verify the effectiveness of our proposed attentions, we replace the attentions used in RTFormer block with different types and combinations. shown in Table 5a] we give the results of different combinations of multi-head self attention, multi-head external attention, GPU-Friendly attention and cross-resolution attention. For instance, "GFA+CA" means using GFA in low-resolution branch and CA in high-resolution branch. In addition, we adjust multi-head external attention by M x r, where is a reduction ratio.
the hyper parameter We can find that GPU-Friendly attention outperforms all settings of multi-head external attention and is faster than the best one when M = d, and meanwhile, GPU-Friendly attention is much more efficient than multi-head self attention with comparable performance. That indicates GPU-Friendly attention achieves better trade-off between performance and efficiency than both multi-head self attention and multi-head external attention GPU-like devices. When we introduce cross-resolution attention, the performance improves further, while the FPS only drops less than 2.
Comparison on different types of FFN. Table 5b illustrates the results of typical FFN which is consist of two MI.P layers and a x depth-wise convolution layer and our proposed FFN containing two 3 x convolution layers. It is shown that our proposed FFN outperforms typical FFN not only

---

Table 5: Ablation studies on different types attention, FFN and different settings of hyper parameters.
(a) Comparison on different types of attention. SA. EA. GFA, CA denote Self Attention, Exter- nal Attention, GPU-Friendly Attention and Cross- resolution Attention respectively.
(b) Comparison of different types of FFN. The typical FFN is composed of two MLP layers and a 3 3 depth- wise convolution layer, and our FFN design two 3 convolution layers.
Attention
FPSt
ml)
SA+SA
97.4
32.7
Method
FPSt
mloU(*))
EA+EA (r=1)
180.8
32.2
Typical FFN Our FFN
178.5
32.15
EA+EA (r=0.125)
196.9
31.9
187.9
33.0
EA+EA (r25.)
189.6
32.0
GFA+GFA
189.8
32.8
GFA+CA
187.9
33.0
(c) Comparison of different number of groups in Grouped Double Normalization.
(d) Comparison of different spatial size of the cross- feature Cross-resolution Attention.
of Groups
FPSt mloU((()f
Parameters
FPSt
mloU(%))
1, 1
189.8
32.2
6 x 6
191.6
32.85
4, 1
189.8
32.3
189.8
33.00
8, 2
189.8
32.8
12 X 12
175.6
32.94
on mIoU but also FPS. That indicates that our proposed FFN is more suitable in the scenario when latency on GPU-like devices should be considered.
Influence of the number of groups within grouped double normalization. We study the influence of the number of group in grouped double normalization under the setting using GPU-Friendly Attention for both branches. And Table 5c shows the results of different configurations. For example, "8+2" means using groups in low-resolution branch and groups high-resolution. Specially, when the number of groups is set to 1, grouped double normalization degrades the original double normalization. Here, the best mloU is achieved when the numbers of groups are 8 and 2, which illustrates that the grouped double normalization performs better than the original double normalization. And it is worth to be noted that, changing the number of groups in grouped double normalization does not affect the inference efficiency, which makes GPU-Friendly attention being able to keep high FPS when the number of groups is large.
Influence of the spatial size of cross-feature in Cross-resolution Attention. We also investigate the spatial size of cross-feature in cross-teso attention, including applying 6
and 12 x 12. As presented in Table 5d 8 x 8 spatial size of cross-feature for STFormer-Slime is the best according to the trade-off between FPS and mloU. To some extent, indicates that the spatial size of cross-feature which is close to the dimension of high-resolution feature is appropriate, as the high-resolution feature dimension of RTFormer-Slin is 64 which equals x
Conclusion
In this paper, we present RTFormer which can efficiently capture the global context to improve the real-time semantic segmentation performance. Extensive experiments demonstrate that our method not only achieves new state-of state-of-the of-the-art results on common datasets for real-time segmentation but also shows superior performance on challenging dataset for general semantic segmentation. Due to the efficiency of RTFormer, we hope our method can encourage new design of real-time semantic segmentation with transformer. One limitation is that while our RTFormer-Slim only has 4.8M parameters, more parameter efficiency may needed in chip of edge device. We leave for future work.
10

---

References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):22 1)2274-222 2012.
[2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. JEEE Transactions on Pattern Analysis and Machine Intelligence.
(9(12):241-2499) 2017.
[3] Y.Y. Boykov and M.-P. Jolly. Interactive graph cuts for optimal boundary amp; region segmentation of objects in n-d images. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 1. pages 125-112 vol. 2001.
[4] Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition
using structure from motion point clouds. In ECCV (I), pages 44-57, 2008.
[5] Holger Caesar. Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedingy of the IEEE conference
computer vision and pattern recognition, pages 1209-1218 2018. [6] Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, and Youn-Long Lin. Hardnet: A low memory traffic network. In Proceedings of the IEEE/CVF international conference on computer vision,
pages 3552-3561, 2019.
[7] Liang- Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution
for semantic image segmentation. CoRR, abs/1706.055$3 2017.
[8] Liang- Cann-Chichh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European
conference on compuder vision (ECCV), pages 801-818, 2018
[9] Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang. Yuan Li, and Zhangyang Wang. Fasterseg: Searching for faster real-time semantic segmentation. arXiv preprint 1rXiv:/9/2.1099 2019.
[10] Marius Cordts, Mohamed Omran. Sebastian Ramos, Timo Rehfeld. Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscapes dataset for semantic urban scene understanding. CoRR, abs/1604.016$3 (0004,01655..111 2016,
[11] Jia Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Ieee, 2009.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. CoRR,
abs/2010.1 1929, 2020.
[13] Mingyuan Fan, Shenqi Lai, Junshi Huang, Xiaoming Wei. Zhenhua Chai, Junfeng Luo, and Xiaolin Wei. Rethinking bisenct for real-time semantic segmentation. Proceedings of the IEEE/CVF conference on 2021. [14] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3146-330 13566311,2222 2019.
computer vision and pattern recognition, pages
[15] Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. arXiv preprint orXiv:2105. 10.00238, 2021.
[16] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Yuanduo Hong, Huihui Pan, Weichan Sun, and Yisong Jia. Deep dual-resolution networks for real-time
and accurate semantic segmentation of road scenes. arXiv preprint arXiv:2 arXiv:210I.06ii 101.06085, 2021.
[18] Han Hu, Jiayuan Gu, Zheng Zhang. Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3588-3597, 2018. excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 71 7132-7141, 2018
[19] Jie Hu, Li Shen, and Gang Sun. squeeze-and--ee wetee-annndde-iiiioooe
[20] Hanchao Li, Pengfei Xiong, Haoqiang Fan, and Jian Sun. Dianet: Deep feature aggregation for real time semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9522-9531, 2019.
[21] Xiangtai Li. Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In European Conference Computer
Vision, pages 775-793. Springer, 2020.
[22] Xin Li, Yiming Zhou, Zheng Pan, and Jiashi Feng. Partial order pruning: for best speed/accuracy trade-off in neural architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9145-9153, 2019.
[23] Peiwen Lin, Peng Sun, Guangliang Cheng, Sirui Xie, Li, and Jianping Shi. Graph-guided architecture search for real-time semantic segmentation. In Proceedings of the IEEÉ/CVE Conference on Computer Vision and Pattern Recognition, pages 4203-4212, 2020.
[24] Yi Liu, Lutao Chu, Guowei Chen, Zewu Wu, Zeyu Chen, Baohua Lai, and Yuying Hao. Paddleseg: A high-efficient development toolkit for image segmentation. arXiv preprint 2XI.:210100611 2021.
[25] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer:
Hierarchical vision transformer using shifted windows. CoRR, abs/2103. 14030, 2021.
11

---

[26]
Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta-
tion. CoRR, abs/1411 1.44888. 2014.
[27]
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, puges 3431-3440, 2015.
[28] Marin Oršié and Siniša Segvić. Efficient semantic segmentation with pyramidal fusion. Pattern Recognition.
110:10761,, 1, 2021.
[29]
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems,
Man, and Cybernetics, (1):62266, 1979.
[30] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12179- 12188, 2021, [31] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012. 12877, 2020.
[32] Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 13(10))3399-3344 2020.
[33] Qiangchang Wang, Tianyi Wu, He Zheng. and Guodong Guo. Hierarchical pyramid diverse attention networks for face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8326 8335, 2020.
[34] Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[35] Wenhai Wang. Enze Xie, Xiang Li, Deng-Pingg Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 568-578, 2021.
[36] Wenhai Wang. Enze Xie, Xiang Li, Deng- -Pig Fan, Kaitao Song. Ding Liang. Tong Lu,, Ping Luo, and Ling Shao. Pvt Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media.
pages 1-10, 2022.
[37] Xiaolong Wang. Ross Girshick, Abhinav Gupta, and Kaiming Hc. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018. [38] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems. 34, 2021.
[39] Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. Attentional shapecontextnet for point cloud recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4606-4615, 2018.
[40] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: nyström-based algorithm for approximating self-attention. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 35. pages 14138-14145 48, 2021.
[41] Changqian Yu, Changxin Gao, Jingbo Wang. Gang Yu, Chunhua Shen, and Nong Sang. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. International Journal of Computer Vision, 129( (19(1) 0:3051-3066 2021.
[42] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018
[43] Yuhui Yuan, Xilin Chen, and lingdong Wang. Object-contextual representations for semantic segmentation.
In European conference on computer vision, pages 173-190. Springer, 2020.
[44] Yuhui Yuan, Rao Fu. Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution vision transformer for dense predict. Advances in Neural Information Processing Syxtems, 34, 2021.
[45] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Object
context network for scene parsing. arXiv preprint arXiv: 1a0::0009000000 2018.
[46] Xiaohua Zhai Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CoRR,
abs/2106.04566 04560000044660.00000 2021.
[47] Yiheng Zhang, Zhaofan Qiu, Jingen Liu, Ting Yao, Dong Liu, and Tao Mei. Customizable architecture search for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 11641-11650, 2019.
[48] Hengshuang Zhao, Xiaojuan Qi. Xiaoyong Shen, Jianping Shi, and Jaay Jia. Ienet for real-time semantic segmentation on high-resolution images. In Proceedings of the European conference on computer vision (ECCV), pages 405-420, 2018.
[49] Hengshuang Zhao, Xiaojuan Qi. Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Ienet for real-time semantic segmentation on high-resolution images. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
[50] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiyy Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
12

---

[51] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo. Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, Rethinking semantic segmentation from reeueenc-fo--eeueeee perspective with transformers. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages
2021. [52] Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. CoRR,
2016.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "hT375AvjL0", "reviewer": "Reviewer_HQEz", "summary": "The manuscript presents an efficient model for semantic segmentation. The main contribution corresponds to GPU friendly attention layer which improves the efficiency by using keys and values as learnable parameters. Dimensionality of keys and values is a hyperparameter that is much less than N (HxW). Furthermore, the MLP from the standard transformer is replaced with plain convolutions. The resulting module is somewhat similar to the classic self-attention layer from pre-transformer era. Finally, some further performance improvement is obtained through cross-resolution attention. Experiments address Cityscapes and ADE20k. \n", "questions": "It appears that it would be more appropriate to call this model as a hybrid convolutional/attention model than a transformer.\n\nIt would be beneficial to show experiments on some embedded platform.\n\nPlease clarify whether you use V100 16GB or V100 32 GB.", "limitations": "It appears that large memory footprint precludes training on single GPU systems.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- a resonable hybrid model with convolutions on higher-resolution representations and self-attention on lower-resolution representations\n- state-of-the-art ratio between performance and computational complexity\n\nWeaknesses\n- incremental contribution; GPU friendly attention appears quite related to previous work [13], as well as to Linformer and Nystromformer (see below).\n- hybrid convolutional-transformer models have been proposed before (eg DPT hybrid, see below) \n- the proposed improvements perform only slightly better than baselines in Fig.3\n- missing configuration in Fig.3a: EA + CA \n- large training footprint: it appears that only 3 crops 512x1024 can fit into a V100 \n- incomplete related work in the field of efficient models for semantic segmentation (eg HardNet, SwiftNet, see below)\n\nMissing related work:\n- René Ranftl, Alexey Bochkovskiy, Vladlen Koltun. Vision Transformers for Dense Prediction. ICCV 2021: 12159-12168\n- Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, Vikas Singh. Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention. AAAI 2021.\n- Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma. Linformer: Self-Attention with Linear Complexity. CoRR abs/2006.04768 (2020).\n- Marin Orsic, Sinisa Segvic. Efficient semantic segmentation with pyramidal fusion. Pattern Recognit. 110: 107611 (2021).\n- Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, Youn-Long Lin. HarDNet: A Low Memory Traffic Network. ICCV 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CHlR3e7eS9v", "reviewer": "Reviewer_dfdA", "summary": "This paper studies the problem of real-time semantic segmentation with Transformer. The authors proposed an RTFormer block with two attention models to aggregate information on different-resolution features. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.", "questions": "The authors argue that GPU-Friendly attention is quite friendly for GPU-like devices. However, according to Table 3, GPU-Friendly attention has a minor effect on acceleration. (189.6->189.8 or 187.9->189.8). Meanwhile, there is no fair and clear comparison between EA or other lightweight attentions and the proposed attention.\n", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n+ The proposed method achieves great performance in the serval datasets.\n+ Compared to the baselines, the proposed methods could bring constant improvements \n\n[Weaknesses] Some important ablation studies are missing.\n- The choice of architectural design. The author put the proposed RTFormer block on the last two stages and does not provide the results to support this design.\n- The baseline which does not use any attention is needed to be included in Table 3 (a).\n- Comparison with the other lightweight attentions is also needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AU6nXOR3c1", "reviewer": "Reviewer_jocB", "summary": "This paper proposes RTFormer, an efficient transformer for real-time semantic segmentation, which achieves better trade-off between performance and efficiency than CNN-based models. To achieve high inference efficiency on GPU-like devices, the RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. Besides, the cross-resolution attention is more efficient to gather global context information for high-resolution branch by spreading the high level knowledge learned from low-resolution branch. Extensive experiments on mainstream benchmarks demonstrate the effectiveness of the proposed RTFormer, it achieves state-of-the-art on Cityscapes and CamVid, and shows promising results on ADE20K", "questions": "The questions are as follows:\n1. is it possible to generalise the proposed network to other vision tasks, like image classification, object detection, etc.\n2. It seems the attention used is just MHA or self-attention, this is not a new technique. So the main contribution is limited, right?\n", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths of this paper are as follows :\n1. A novel RTFormer block is proposed, which achieves better trade-off between performance and efficiency on GPU-like devices for semantic segmentation task.\n2. A new network architecture RTFormer is proposed, which can make full use of global context for improving semantic segmentation by utilizing attention deeply without lost of efficiency.\n3. RTFormer achieves state-of-the-art on Cityscapes and CamVid, and show promising performance on ADE20K. In addition, it provides a new perspective for practice on real-time semantic segmentation task.\n\nWeakness of this paper are as follows:\n1. The proposed cross-resolution attention is just the variant of self-attention, which is widely used in network design. This paper is incremental compared with previous work DDRNet. The novelty of this paper is limited.\n2. The performance  improvement on Cityscape dataset is limited. As shown in Table 1, the mIoU score and FPS improvement are both limited, not obvious enough.\n3. The experiments on semantic segmentation is thorough enough and can be used to support the proposed method. But my concern is that the method is simple and not novel enough. In addition, why not apply this method to other vision tasks, like classification, object detection. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "8Bx8-EvhYTJ", "reviewer": "Reviewer_qoXt", "summary": "This paper proposes RTFormer for real-time semantic segmentation.  The RTFormer leverages GPU-Friendly Attention with linear complexity and discards the multi-head mechanism. The authors demonstrate the efficacy of their method on several benchmarks.", "questions": "Please refer to “Paper Weaknesses”.", "limitations": "Please refer to “Paper Weaknesses”.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method achieves good performances on the benchmarks.\n2. This paper is well organized and clearly described.\n3. Efficient segmentation is a valuable problem.\n\nWeaknesses:\n1. The method proposed in the paper is a hybrid of various existing methods such as linear-complexity self-attention, HRNet, and CNN and transformer hybrid model. Therefore, the novelty is weakened by previous works.\n2. This paper does not say whether to use tensorrt to accelerate the model. So I don't know if the comparison is fair.\n3. In terms of performance and model size, there is no significant advantage between this method and the compared methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept