SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Authors: Enze Xie
Wenhai Wang
Zhiding Yu
Anima Anandkumar
Jose M. Alvarez
Ping Luo
Venue: NeurIPS 2021 Poster (2021)
Keywords: Semantic Segmentation
Transformers

Abstract: We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perceptron (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation on Transformers.  We scale our approach up to obtain a series of models from SegFormer-B0 to Segformer-B5, which reaches much better performance and efficiency than previous counterparts.
For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C.

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Enze Xie Wenhai Wang² Zhiding Yu Anima Anandkumar Jose M. Alvarez Ping Luo
The University of Hong Kong "Nanjing University *NVIDIA 4Caltech
xieenze@hku. hk, wangwenhai162211 com, (zhidingy, josea, aanandkumar Invidia.......i....i. com, pluoêcs hku.hk
Abstract
We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perceptron (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation Transformers. We scale approach up obtain series of models from SegFormer-By BB to SegFormer-B5, reaching significantly better performance and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegForme-- achieves 84.0% mloU on Cityscapes validation set and shows excellent zero-shot robustness Cityscapes-C. Code available at: gi github.com. wwt.://oo////ab.///o/o/m SSgp///er.r..rr
Introduction
Semantic segmentation a fundamental task in computer vision and enables many downstream applications. It is related to image classification since it produces per-pixel category prediction instead of image-level prediction. This relation- ship is pointed out and systematically studied in seminal work [1], where the authors used fully convolutional networks (FCNs) for semantic seg- mentation tasks. Since then, FCN has inspired many follow-up works and has become a predom- inant design choice for dense prediction.
Segformatt
Twins
SETR
Transformar
#82
A082200 1002000mm mtou
DeepLabv3+5101
81
HENet W4S+OCR
SeriFFN
mkU
Pererol
Ploysl
FPSt
50
FO4R50
FCH4-50
Daspicas #300
44.1
52.70
255.5
183
Since there strong relation between classi- fication and semantic segmentation, many state- of-the-art semantic segmentation frameworks are variants of popular architectures for image classi- fication on ImageNet. Therefore, designing back- bone architectures has remained active area
70.544
304.5
17.0
SETR.
45.5
310,300
201.15
3.4
Segi crmeen4
04.30
13.4
50
150
200
2.0
300
350
Parame (Millions)
Figure Performance model efficiency on achieves state-ofthhe-att 51.0% mloU while being significantly more ellicient tham previous methods.
SegFormer
"Corresponding authors: Zhiding Yu and Ping Luo
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in semantic segmentation. Indeed, starting from
early methods using VGGs [1, 2]. to the latest methods with significantly deeper and more powerful backbones [3], the evolution of backbones has dramatically pushed the performance boundary of semantic segmentation. Besides backbone architectures, another line of work formulates semantic segmentation as a structured prediction problem, and focuses on designing modules and operators, which can effectively capture contextual information. A representative example in this area is dilated convolution [4, 5J, which increases the receptive field by "inflating" the kernel with holes.
Witnessing the great success in natural language processing (NLP), there has been a recent surge of interest to introduce Transformers to vision tasks. Dosovitskiy et al. [6] proposed vision Transformer (ViT) for image classification. Following the Transformer design in NLP, the authors split an image into multiple linearly embedded patches and feed them into a standard Transformer with positional embeddings (PE), leading to an impressive performance on ImageNet. In semantic segmentation, Zheng et al. [7] proposed SETR to demonstrate the feasibility of using Transformers in this task.
SETR adopts ViT as a backbone and incorporates several CNN decoders to enlarge feature resolution. Despite the good performance, ViT has two important limitations: 1) ViT outputs single-scale low- resolution features instead of multi-scale ones, and 2) it has very high computational cost on large images. To address these limitations, Wang et al. [8] proposed pyramid vision Transformer (PVT), natural extension of ViT with pyramid structures for dense prediction. PVT shows considerable improvements over the ResNet counterpart on object detection and semantic segmentation. However, together with other emerging methods such as Swin Transformer [9] and Twins [10]. these methods mainly consider the design of the Transformer encoder, neglecting the contribution of the decoder for further improvements.
This paper introduces SegFormer, a cutting-edge Transformer framework for semantic segmentation that jointly considers efficiency, accuracy, and robustness. In contrast previous methods, our framework redesigns both the encoder and the decoder. The key novelties of our approach are:
A novel caattiionllencodingg free and hierarchical Transformer encoder.
A lightweight AII-MLP decoder design that yields a powerful representation without complex and computationally demanding modules.
As shown in Figure 1, SegFormer sets new state-of-the-arr in terms of efficiency, accuracy and robustness in three publicly available semantic segmentation datasets.
First, the proposed encoder avoids interpolating positional codes when performing inference on images with resolutions different from the training one. As a result, our encoder can easily adapt to arbitrary test resolutions without impacting the performance. In addition, the hierarchical part enables the encoder to generate both high-resolution fine features and low-resolution coarse features, this is in contrast ViT that can only produce single low-resolution feature maps with fixed resolutions. Second, we propose a lightweight MLP decoder where the key idea is to take advantage of the Transformer-inducee features where the attentions of lower layers tend to stay local, whereas the ones of the highest layers are highly non-local. By aggregating the information from different layers, the MLP decoder combines both local and global attention. As result, we obtain a simple and straightforward decoder that renders powerful representations.
We demonstrate the advantages of SegFormer in terms of model size, run-time, and accuracy on three publicly available datasets: ADE20K, Cityscapes, and COCO-Stuff. On Citysapces, our lightweight model, SegFormer-BO, without accelerated implementations such as TensorRT, yields 71.9% mIoU at 48 FPS, which, compared to ICNet [1]]. represents a relative improvement of 60% and 4.2% in latency and performance, respectively. Our largest model, SegFormer-BS, yields 84.0% mIoU, which represents a relative 1.8% mloU improvement while being 5 x faster than SETR [7]. On ADE20K, this model sets a new sate-oofthhe-a of 51. mloU while being 4 x smaller than SETR. Moreover, our approach is significantly more robust to common corruptions and perturbations than existing methods, therefore being suitable for safety-critical applications. Code will be publicly available.

---

Encoder
Decoder
xx Exc,
x4c
Ex K.
Embeddings
Overlap Patch
Transfarmew mer Block
mranstorme Block-2
Transtomme
Layer
MLP
Ex
Overlap Patch
2
Self-Atin
Efficient
Mix-FFN
Merging
UpSample
MLP
XN
Figure 2: The proposed SegFormer framework consists of main modules: A hierarchical Transformer encoder to extract coarse and fine features; and lightweight AII-MLP decoder to directly fuse these multi-level features and predict the semantic segmentation mask. "FFN" indicates feed-forward network.
Related Work
Semantic Segmentation. Semantic segmentation can be seen extension of image classification from image level to pixel level. In the deep learning era [12-14], FCN [1] the fundamental work of semantic segmentation, which is fully convolution network that performs pixel-to-pixel classification in an end-to-end manner. After that, researchers focused on improving FCN from different aspects such as: enlarging the receptive field [15-17, 5, 2, 4, 18]; refining the contextual information [19- 27]; introducing boundary information [28-35]; designing various attention modules [36-44]; or using AutoML technologies [45-49]. These methods significantly improve semantic segmentation performance at the expense of introducing many empirical modules, making the resulting framework computationally demanding and complicated. More recent methods have proved the effectiveness of Transformer-basee architectures for semantic segmentation [7, 44]. However, these methods are still computationally demanding.
Transformer backbones. VIT [6] is the first work to prove that a pure Transformer can achieve state-of- the-art performance in image classification. ViT treats each image sequence of tokens and then feeds them to multiple Transformer layers to make the classification. Subsequently, DeiT [50] further explores a data-efficient training strategy and distillation approach for VIT. More recent methods such as T2T ViT [51], CPVT [52], TNT [53], CrossViT [54] and Local [55] introduce tailored changes VIT to further improve image classification performance.
Beyond classification, PVT [8] the first work to introduce a pyramid structure Transformer, demonstrating the potential of pure Transformer backbone compared to CNN counterparts in dense prediction tasks. After that, methods such as Swin [9], CvT [56], CoaT [57], LeViT [58] and Twins [10] enhance the local continuity of features and remove fixed size position embedding to improve the performance of Transformers in dense prediction tasks.
Transformers for specific tasks. DETR [50] is the first to use Transformers for end-to-end ob- ject detection framework without non-max maximum suppression (NMS). Other works have also used Transformers in tasks such as tracking [59, 60], super-resolution [61], re-id [62], colorization [63], re- trieval [64] and multi-modal learning [65, 66]. For semantic segmentation, SETR [7] adopts VIT [6] as backbone to extract features, achieving impressive performance. However, these Transformer-basee methods have very low efficiency and, thus, difficult to deploy real-time applications.
Method
As depicted in Figure 2, SegFormer consists of two main modules: (1) hierarchical Transformer encoder; and (2) a lightweight AII-MLP decoder predict the final mask. Given an image with size H x W x 3, we first divide it into patches of size 4 4. Unlike ViT which uses 16 x 16, using fine-grained patches favors semantic segmentation. Second, we use these patches as input to the hierarchical Transformer encoder get multi-level features with resolution (1/4, 1/8, 1/16, 1/32)

---

of the original image. We then pass these multi-level features to the All-MLP decoder to predict the segmentation mask with a I J x Nats resolution, where Nets the number of categories. In the remainder of this section, we first detail the proposed encoder and decoder designs and then summarize the main differences of our approach compared to SETR.
If
3.1 Hierarchical Transformer Encoder
We design a series of Mix Transformer encoders (MiT), MîT-BO to MiT-B5, with the same architec- ture but different sizes. On top of the hierarchical architecture and efficient self-attention module in PVT [8], we further propose several novel features including overlapped patch merging and positional-encoing--ree design which will be shown to greatly benefit the segmentation tasks.
Hierarchical Feature Representation. Unlike ViT [6], our encoder generates multi-level multi-scale features given an input image. These features provide both high-resolution coarse features and low- resolution fne-grained features that boost the performance of semantic segmentation. Specifically, given input image with size H x W x 3, we perform patch merging obtain hierarchical feature gETt X C₁ where E [1,2,3 3, 4). and C.+1 is larger than C,.
map F; with a resolution of g
Efficient Self-Attentin.. A major bottleneck of the above hierarchical feature representation is the quadratie self-attention complexity with long sequence inputs from higher resolution features. Recall that in the original multi-head self-attention, each of the heads Q, K, V have the same dimensions N X C, where = H x W is the length of the sequence, the self-attention is estimated as:
QKT
Attention(Q. (Q, K, V) Softmax(
VV.
Vdhead
We instead adopt the sequence reduction process introduced in [8]. This process uses a reduction ratio R to reduce the length of the sequence of as follows:
N
K Reshapee
C. ())
R
(2)
K = Linear (C R, (R),
where is the sequence to be reduced, Reshape( C. (K) refers to reshape K to the one with R
shape of N R (C- R), and Linear(Cin. (.)t)()) refers to linear layer taking a dimedinnni tensor as input and generating a Cout -immensioo tensor as output. Therefore, the new K has dimensions R C. As result, the complexity of the self-attention mechanism is reduced from
O(N2) to 0 N22 ). In our experiments, we set R to [64, 16, 4, 1] from stage-1 to stage-4. R.
Overlapped Patch Merging, Given an image patch, the patch merging process used in ViT, unifies a N x N x 3 patch into 1 x 1 x C vector. This can easily be extended to unify a 2 x 2 x C, feature path into a 1 x 1 x Ce+1 vector to obtain hierarchical feature maps. Using this, we can shrink our hierarchical features from F1 ( x 4 x C1) to F2 ( x W
X C2). and then iterate for any other feature map in the hierarchy. This process was initially designed to combine non-overlapping image or feature patches. Therefore, it fails to preserve the local continuity around those patches. Instead, we use an overlapping patch merging process. To this end, we define K, S, and P. where K is the patch size, S is the stride between two adjacent patches, and P is the padding size. In our experiments, we set K = 7, S = 4, P 3 ,and patch merging to produces features with the same size as the nve-overlgp process. Similar to the original patch embedding in VIT [6], this operation can implemented by "nn.Conv2D" PyTorch.
W
8
3, S = 2, P = 1 to perform overlapping
Encooing--Free Design. The resolution of the PE in ViT is fixed. One thus needs to interpolate the PE when the test resolution differs from training. This leads the drop of accuracy, which is undesirable since the resolution mismatch is common in semantic segmentation. We instead introduce Mix-FFN where we consider the effect of zero padding to the leak location information [67] by directly using 3 x 3 Conv in the feed-forward network (FFN). Mix-FFN is formulated as:
Xout MLP (MLP(X..)) Xins
(3)
where X is the feature from the self-attention module. Mix-FFN mixes a x convolution and an ML. into each FFN. In our experiments, we will show that 3 x 3 convolution is sufficient to provide positional information for Transformers. In particular, we use depth-wise convolutions for reducing the number of parameters and improving efficiency.

---

It should be mentioned that CPVT [52] also alleviates this issue by using 3 x Conv to generate conditional PE at different resolutions and then add the feature map. Our work conceptually goes one step further we argue that adding PE to feature map is not necessary in semantic segmentation. Another recent work CvT [56] introduced 3 x Convs to model the spatial relationship among tokens. Despite the converging design, our work differs in both motivation and application as we aim to totally remove PEs to handle the training/testing resolution mismatch issue in semantic segmentation. Our intuition started from [67] whereas the same intuition was not discussed in CvT.
3.2 Lightweight AlI-MLP Decoder
SegFormer incorporates a lightweight decoder consisting only of MLP layers and this avoiding the hand-crafied and computationally demanding components typically used in other methods. The key enabling such a simple decoder is that our hierarchical Transformer encoder has larger effective receptive field (ERF) than traditional CNN encoders.
The proposed AlI-MLP decoder consists of four main steps. First, multi-level features F; from the MiT encoder go through an MLP layer to unify the channel dimension. Then, in second step, features up-sampled 1/4th and concatenated together. Third, MLP layer is adopted to fuse the concatenated features F. Finally, another MLP layer takes the fused feature to predict the segmentation mask M with This lets us formulate the decoder as:
W. Nole resolution, where Nels is the number of categories.
F Linear C))F),
W W
F Upsample(
(F.),
(4)
F Linear(4C, (Concat| (.)).
M Linear(C,
where M refers to the predicted mask, and Linear( Cin, (-) refers to linear layer with C., and Cout as input and output vector dimensions respectively.
Effective Receptive Field Analysis.
Slage-1
Stage-2
Stage-3
Slage-4
Head
For semantic segmentation, maintain- ing large receptive field to include con- text information has been central is- sue [5, 17, 18]. Here, we use effec- tive receptive field (ERF) [68] as toolkit to visualize and interpret why our MLP decoder design so effec- tive on Transformers. In Figure 3, we visualize ERFs the four encoder stages and the decoder heads for both DeepLabv3+ and SegFormer. We can make the following observations:
SegFemeer
Figure 3: Effective Receptive Field (ERF) on Cityscapes (aver- over 100 images). Top row: Deeplaby3+. Bottom row: Seg- Former. ERFs of the four stages and the decoder heads of both architectures visualized. Best viewed with zoom in.
The ERF of DeepLabv3+ is relatively small even at Stage-4, the deepest stage.
SegFormer's encoder naturally produces local attentions which resemble convolutions at lower stages, while able to output highly non-local attentions that effectively capture contexts at Stage-4.
As shown with the zoom-in patches in Figure 3, the ERF of the MLP head (blue box) differs from Stage-4 (red box) with a significant stronger local attention besides the non-local attention.
The limited receptive field in CNN requires one resort to context modules such as ASPP [16] that enlarge the receptive field but inevitably become heavy. Our decoder design benefits from the non-local attention Transformers and leads to larger receptive field without being complex. The same decoder design, however, does not work well CNN backbones since the overall receptive field is upper bounded by the limited one at Stage-4, and we will verify this later in Table 1d,
More importantly, our decoder design essentially takes advantage of a Transformer induced feature that produces both highly local and non-local attention same time. By unifying them, our ML. decoder renders complementary and powerful representations by adding few parameters. This is
5

---

another key reason that motivated our design. Taking the non-local attention from Stage- alone is not enough to produce good results, as will be verified in Table 1d.
3.3 Relationship to SETR.
SegFormer contains multiple more efficient and powerful designs compared with SETR [7]:
We only use ImageNet-1K K for pre-training. ViT in SETR is pre-trained on larger ImageNet-22K.
SegFormer's encoder has hierarchical architecture, which is smaller than VIT and capture both high-resolution coarse and low-resolution fine features. In contrast, SETR's s VIT encoder can only generate single low-resolution feature map.
We remove Positional Embedding in encoder, while SETR uses fixed shape Positional Embedding which decreases the accuracy when the resolution at inference differs from the training ones.
Our MLP decoder is more compact and less computationally demanding than the one in SETR. This leads to a negligible computational overhead. In contrast, SETR requires heavy decoders with multiple 3x3 convolutions.
4 Experiments
4.1 Experimental Settings
Datasets: We used four public datasets: Cityscapes [69], ADE20K [70], and COCO-Stuff [71]. ADE20K is a dataset covering 150 fine-grained semantic concepts consisting of 20210 images. Cityscapes is driving dataset for semantic segmentation consisting of 5000 mee-annttteed high resolution images with 19 categories. COCO-Stuff covers 172 labels and consists of 164k images: 118k for training. 5k for validation, 20k for test-dev and 20k for the test-challenge.
We used the mmsegmentation" codebase and train server with 8 Tesla V100. We pre-train the encoder on the Imagenet- 1K dataset and randomly initialize the decoder. During training, applied data augmentation through random resize with ratio 0.5-2.0, random horizontal flipping, and random cropping to 512 x 512, 1024x 1024, 512 X 512 for ADE20K, Cityscapes and COCO-Stuff. Following [9] we set crop size to 640 640 on ADE20K for our largest model B5. We trained the models using AdamW optimizer for 160K iterations on ADE20K, Cityscapes, and 80K iterations on COCO-Stuff. Exceptionally, for the ablation studies, we trained the models for 40K iterations. We used a batch size of 16 for ADE20K, COCO- -tuff and batch size of 8 for Cityscapes. The learning rate was "poly" LR schedule with factor 1.0 by default. For simplicity, we did not adopt widely-used tricks such as OHEM, auxiliary losses or class balance loss. During evaluation, we rescale the short side of the image to training cropping size and keep the aspect ratio for ADE20K and COCO-Stu For Cityscapes, we do inference using sliding window test by cropping 1024 x 1024 windows. We report semantic segmentation performance using mean Intersection over Union (mloU).
Implementation details:
an initial value of 0.00006 and then used
4.2 Ablation Studies
Influence the size of model. We first analyze the effect of increasing the size of the encoder on the performance and model efficiency. Figure shows the performance vs. model efficiency for ADE20K as function of the encoder size and, Table la summarizes the results for the three datasets. The first thing to observe here is the size of the decoder compared to the encoder. As shown, for the lightweight model, the decoder has only 0.4M parameters. For MiT-B5 encoder, the decoder only takes up to 4% of the total number of parameters in the model. In terms of performance, we can observe that, overall, increasing the size the encoder yields consistent improvements on all the datasets. Our lightweight model, SegFormer-BB. is compact and efficient while maintaining competitive performance, showing that our method very convenient for real-time applications. On the other hand, our SegFormer- the largest model, achieves state-of-the-a art results on all three datasets, showing the potential of our Transformer encoder.

---

Table Ablation studies related to model size, encoder decoder design.
(a) Accunacy, parameters and llps as function of the model size on the three datasets "5S" and "MS" means singlo/mulii scale test.
Encoder Model Size
Purams
ADE20K
Cityscapes
COCO-Stuff
Fincoder
Decoder
Plops
mloU((SS/MS)
Hops
mloU(S/MMS)
Hlpps
mloU(SS)
MiT-BO
3,4
0.4
8.4
37.4/38.0
125.5
76,2 78,1
8,4
35.6
MiT-B1
13.1
0.6
15.9
42.2/43.1
243.7
78.5 80.0
15.9
40.2
MiT-B2
24.2
3.3
62.4
46.5/47.5
717.1
81.0/ 82.2
62.4
44.6
MrT-B3
44.0
3.3
79.0
49.4/50.0
962.9
81.7 83.3
79.0
45.5
MiT-B4
60.8
3.3
95.7
50.3/511
1240.6
/3.9
95.7
46.5
MiT-B5
81,4
3.3
183.3
51.0/5188
1460.4
82.4/ 84.0
111.6
46.7
(b) Accunccy as a fanction of the MIP dimen- sion C in the decoder on ADE2UK.
Mix-FIN positional encoding (PE) enl lest resolution on Cityscapes.
ADE20K, CNN and Trans- encoder docoder. means stage feature.
C
Flops
Params
mloU
Res
Enc Type
mIoU f
Eacoder
Flags
Purus
256
25.7
24.7
44.9
768 x68x768
77.3
ResNec50 (S1-4)
69.2
29.0
34.7
512
39.8
25.8
45.0
ResNett0:
88.7
47.9
35.7
768
62.4
27.5
45.4
x024x2048
PE
74.0
ResNeXit01 (81-4)
127.5
1024
93.6
29.6
45.2
x68
Mix-FFN
80.5
MIT-U2 (84)
22.3
24.7
43.1
2048
304.4
43.4
45.6
1024x2048
Mix-FFN
79.8
MIT-B3 (S1-4)
MiT-82
62.4
79.0
27.7
47.3
48.6
Table 2: Comparison to state of the art methods on ADE20K and Cityscapes. SegFormer has significant advantages on #Params (M), #Foops, #Speed and #Accuracy. Note that for SegFormer- BO scale the short side of image (1024, 768, 640, 512) to get speed-accuracy tradeoffs.
ADE20K
Cityscupes
Method
Encoder
Params
Flops
FPS 1
mloU
Flops
FPS
mlolI
FCN [J]
MobileNerV2
9.8
39.6
64.4
19.7
317.1
14.2
ICNet [T]]
30.3
67.7
Real Time
PSPNet [15]
MobileNerV2 MobileNerV2
13.7
52.9
57.7
29.6
423.4
11.2
70.2
Deepl [18]
8.4
50.5
37.4
125.5
15.2
76.2
51.7
26.3
75.3
SegFormer (Ours)
MIT-BO
3.8
37.1
73.7
17.7
47.6
71.9
FCN [1]
ResNet-101
68.6
275.7
14.8
41.4
2203.3
1.2
76.6
EncNet [22]
ResNet-101
55.1
218.8
14.9
44.7
1748.0
1.3
76.9
PSPNet [15]
ResNet-101
68.1
256.4
15.3
2048.9
1.2
Non Real- Time
CCNet [39]
ResNet-101
68.9
278.4
14.1
45.2
2224.8
1.0
80.2
D3eplabV3+ [18]
ResNet-101
62.7
255.
14.1
44.1
2032.3
1.2
80.9
OCRNer [21]
HRNet-W48
70.5
17.0
4,2
GSCNN [33]
WideResNet38
80.8
Axial-Deeplaaa [72]
AxialResNet-XX
2446.8
81.1
Dynamic Routing [73]
Dynamic-L13-PSP
270.0
Auto-Desplab [48]
NAS- NAS-I48-ASPP
44.0
695.0
80.3
SETR [7]
Vare
318.3
5.4
50.2
0.5
82.2
SegFormer (Ours)
MrT-B4
64.1
15.4
51.1
1240.6
3.0
SegFormer (Our)
MIT-B5
84.7
183.3
9.8
51.8
1447.6
2.5
84.0
Influence of C, the MLP decoder channel dimension. We now analyze the influence of the channel dimension C' in the MLP decoder, see Section 3.2. In Table 1b we show performance, flops, and parameters function of this dimension. We can observe that setting C 256 provides very competitive performance and computational cost. The performance increases as C increases; however, it leads larger and less efficient models. Interestingly, this performance plateaus for channel dimensions wider than 768. Given these results, we choose C = 256 for our real-time models SegFormer-Bol B1 and 768 for the rest.
Mix-FFN vs. Positional Encoder (PE). In this experiment, we analyze the effect of removing the positional encoding in the Transformer encoder in favor of using the proposed Mix-FFN, To this end, we train Transformer encoders with a positional encoding (PE) and the proposed Mix-FFN and perform inference Cityscapes with two different image resolutions: 768x 768 using sliding window, and 1024 x024x2048 using the whole image.
Table Ic shows the results for this experiment. As shown, for given resolution, our approach using Mix-FFN clearly outperforms using a positional encoding. Moreover, our approach less sensitive to differences in the test resolution: the accuracy drops 3.3% when using positional encoding with

---

lower resolution. In contrast, when we use the proposed Mix-FFN the performance drop is reduced to only 0.7%. From these results, we can conclude using the proposed Mix-FFN produces better and more robust encoders than those using positional encoding.
Effective receptive field evaluation. In Section 3.2, we argued that MLP decoder benefits from Transformers having a larger effective receptive field compared to other CNN models. To quantity this effect, in this experiment, we compare the performance of our MLP-decoder when used with CNN-based encoders such as ResNet or ResNeXt. As shown in Table 1d, coupling our MLP-decoder with a CNN-based encoder yields a significantly lower accuracy compared to coupling it with the proposed Transformer encoder. Intuitively. as CNN has a smaller receptive field than the Transformer (see the analysis in Section 3.2), the MLP-decoder is not enough for global reasoning. In contrast, coupling our Transformer encoder with the MLP decoder leads to the best performance. Moreover, for Transformer encoder, is necessary to combine low-level local features and high-level non-local features instead of only high-level feature.
Influence of difference encoders. We select 2
Table Ablation study of different Transformer encuders different decoders. All the model are trained on ADE20K with 160K iterations.
representative Transformer encoders, ViT [6]
and Swin [9] and compare with our MiT en-
Decader Ghllos
Decoder Paroms (M)
coder. As shown in Table 3, with same decoder, e.g.MLP decoder, MiT-B2 is 3.1% % higher than Swin-T with similar encoder parameters. More- over, MiT-B5 has much fewer encoder param- eters than ViT-large, but is 3+% mloU higher than ViT-large. These experiments shows our MiT encoder is better than Swin and ViT for semantic segmentataion.
Encoder
Decoder
mloU
FPS
MFT-B2
UperNet (Swin)
46.5
14.2
210.7
29.7
MFT-82
MLA (SETR)
46.
87.7
4.2
MTT-B2
MLP (Ours)
46.5
21.4
42.1
3.3
MIT-B5
UperNot (Swin)
50.7
5.3
210.7
29.7
MET-BS
(SETR)
50.9
87.7
4.2
MYT-RS
MI P (Ours)
$1.0
9.8
42.1
3.3
Swin-T
MI P (Ours)
43.4
20.6
42.8
3,6
Swin-T
UparNet (Swin)
44.5
15.4
211.3
31.4
VITL
MLP (Ours)
47.7
4.7
0.6
0.6
VIT-L
MLA (SETR)
47.7
1.8
3.7
Influence of difference decoders. We also test MiT encoder with different decoders. As shown in Table 3, the mloUs are similar with different decoders while the proposed MLP decoder has the least parameters and is only 1/8 of the UperNet decoder in Swin. The MLP decoder thus an important design towards efficient segmentation.
4.3 Comparison to state of the art methods
We now compare our results with existing approaches on the ADE20K [70] and Cityscapes [69]. More experiments about COCO-Stuff [71] are in appendix.
ADE20K and Cityscapes: Table 2 summarizes our results including parameters, FLOPS, latency, and accuracy for ADE20K and Cityscapes. In the top part of the table, we report real-time approaches where we include state-of-the-ar methods and our results using the MiT-BO lightweight encoder. In the bottom part, we focus on performance and report the results of our approach and related works using stronger encoders.
On ADE20K, SegFormer-BO yields 37.4%
Tablo 4: Comparison state of the methods on Cityscapes test set. IM-IK, IM-22K, Course and MV refer to the ImageNett IK, ImageNet 22K, Cityscapes coarse set Mapillary Vistas.
mloU using only 3.8M parameters and 8.4G FLOPs, outperforming all other real-time coun- terparts in terms of parameters, flops, and la- tency. For instance, compared to Deeplab V3+ (MobileNetV2). SegFormer-BO is 7.4 FPS, which is faster and keeps 3.4% better mloU.
Method
Encoler
Extra Data
mloU
PSPNet [15]
ResNet-
IM-IK
78.4
PSANet [41]
ResNet- 101
IM-IK
80.1
CCNet [39]
ResNet- 101
IM-IK
81.9
OCNet [19]
ResNet-
IM-1K
80.
Axial-DespLab [72]
AsiaiResNa--LL
IM-IK
79.9
Moreover, SegFormer-Bi B5 outperforms all other approaches, including the previous best SETR, and establishes new state-of-the of 51.8%, which is 1.6% mloU better than SETR while being significantly more efficient.
SETR [?]
VIT
IM-23K
81.0
SETR [?]
VIT
IM-22K, Course
81.6
SogFormer
MiT-B5 M:T-B5
IM-1K
82.2
SegTormer
IM-IK,
83.1
As also shown in Table 2, our results also hold on Cityscapes. SegFormer-Bo yields 15.2 FPS and 76.2% mIoU (the shorter side of input image being 1024), which represents a 1.3% mloU improvement and 2x speedup compared to Deeplab Moreover, with the shorter side of input image being 512, SegFormer-Bo runs at 47.6 FPS and yields 71.9% mIoU, which is 17.3 FPS faster and 4.2% better than ICNet. SegFormer-B B5 archives the best IoU of 84.0%, outperforming all existing
8

---

methods by at least 1.8% mloU, and it runs x faster and 4 x smaller than SETR [7]. On Cityscapes test set, follow the common setting [18] and merge the validation images to the train set and report results using Imagenet-1K K pre-training and also using Mapillary Vistas [74]. As reported in Table 4, using only Cityscapes fine data and Imagenet-ll 1K pre-training, our method achieves 82.2% mloU outperforming all other methods including SETR, which uses ImageNet-22K pre-training and the additional Cityscapes coarse data. Using Mapillary pre-training, our sets a new state-of-the-ar result of 83.1 mloU.
4.4 Robustness to natural corruptions
Model robustness important for many safety-critical tasks such as autonomous driving [75]. In this experiment, we evaluate the robustness of SegFormer to common corruptions and perturbations. To this end, we follow [75] and generate Cityscapes-C, which expands the Cityscapes validation set with 16 types of algorithmically generated corruptions from noise, blur, weather and digital categories. We compare our method DecplabV3+ and other methods as reported in [75]. We also compare with SETR with DeiT Transformer backbone. The results for this experiment are summarized in Table 5.
Our method significantly outperforms previous CNN-based methods, yielding relative improvement of up to 588% on Gaussian Noise and up to 295% on snow weather. SegFormer also outperforms SETR in general except for one corruption (snow). The results indicate the strong robustness of SegFormer, which we envision to benefit safety-critical applications where robustness is important.
Table 5: Main results on Cityscapes-C. "DLv3+", "MBv2", "R" and "X" refer to DecpLabv3+, MobileNetv2, ResNet and Xception. The mloUs of compared methods reported from [75].
Method
Clean Motion
Blur Detoc
Noise
Digital
Weather Spatt
Glass
Impal
Shot
Speck
Bright
Contr
JPEG
Snow
Fog
Frost
DLv3+ (MBv2)
72.0
53.5
49.0
45.3
49.1
6.4
7.0
6.6
16.6
51.7
46.7
32.4
27.2
13.7
38,9
47.4
17.3
(R:0)
76.6
58.5
47.2
27.4
22.8
DLv3+ (R10))
77.1
59.1
56.3
47.7
$7.3
13.2
13.9
16.3
36.9
59.2
54.5
41.5
37.4
11.9
47.8
55.1
22.7
DLv3+ (X41)
77.8
61.6
51.0
54.7
17.0
43.7
63.6
51.7
38.5
46.6
20.6
DLv3+ (X65)
78.4
63.9
59.1
52.8
59.2
15.0
10.6
19.8
42.4
65.9
59.1
46.1
31.4
19.3
50.7
63.6
23.8
DLv3+ (X71)
78.6
64.1
52.0
60.4
14.9
47.1
40.2
64.1
20.2
ICNst
65.9
45,8
44,6
47.4
44.7
8.4
8.4
10.6
27.9
41.0
33.1
27.5
34.0
6.3
30.5
27.3
11.0
42.7
37.0
53.3
36.0
21.2
37.6
19.7
DilatedNel
68.6
44.4
36.3
32.5
38.4
15.6
14.0
18.4
32.7
$2.7
32.6
38.1
29.1
12.5
32.3
34.7
19.2
ResNet-38
77.5
43.3
13.7
18.3
46.9
14.7
22.2
PSPNet
78.8
59.8
53.2
44.4
53.9
11.0
15.4
15.4
34,2
60.4
$1.8
30.6
21.4
8.4
42.7
34.4
16.2
GSCNN
58.9
41.9
24.7
75.9
70.7
12.0
32.6
SETR DeiT
78.9
64.9
65.1
59.1
65.3
54.7
60.5
$1.9
69.4
74.9
69.6
74.9
58.5
44.3
64.8
68.2
39.1
SegFormer-B5 B5
82.4
69.1
68.6
64.1
69.8
57.8
63.4
52.3
72.8
81.0
77.7
80.1
58.8
40.7
68.4
78.5
49.9
5 Conclusion
In this paper, we present SegFormer, a simple, clean yet powerful semantic segmentation method which contains positional-encodin -ecodnig-nodddreeeeee hierarchical Transformer encoder and lightweight All- MLP decoder. It avoids common complex designs in previous methods, leading to both high efficiency and performance. SegFormer not only achieves new state of the art results common datasets, but also shows strong zero-shot robustness. We hope our method can serve as a solid baseline for semantic segmentation and motivate further research. One potential limitation that even our lightest model may still be too heavy for some edge devices. Thus mixed-precision training, pruning, fariware-friendly attention designs and energy consumption important parts of our future work.
Broader Impact
Efficiency, accuracy, and robustness are important aspects of AI models. Our work pushes the boundary of semantic segmentation models these three aspects. We envision that the work will benefit wide range of safety-critical applications, such as autonomous driving and robot navigation. The proposed method improves the "in-the-wild" robustness of these applications, ultimately leading to better safety. Despite such improvement, we fully understand this work is by no means perfect and there are still many challenges towards reliable real world application. Our models may be subject to biases and other possible undesired mistakes, depending on how they are trained in reality. Our

---

model may also be used for surveillance similar to other Al recognition methods, even though it is mainly designed for surveillance applications.
Acknowledgments and Disclosure of Funding
We thank Ding Liang, Zhe Chen and Yaojun Liu for insightful discussion without which this paper would not be possible. Ping Luo is supported by the General Research Fund of Hong Kong No.27208720.
References
[1] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015. 2,
[2] Liang-Chieh Chen, George Papandreou, Jasonas Kokkinos, Kevin Murphy, and Alan L. Yuille, Semantic image segmentation with deep convolutional nets and fully connected CRFs. In ICLR, 2015. 2,
Hang Zhang, Chongruo Wu, Zhongyue Zhang. Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun. Tong He. Jonas Mueller, Manmatha, et al. ResNest: Split-attention networks. arXiv, 2020.
[4] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI, 2017. 3
[5] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016. 2. 5
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at seale. arXiv, 2020. 2, 3,4,8
[7] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo. Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from requence-to-sequeee perspective with transformers. CVPR, 2021. 2, 3. 8.9
[8] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone dense prediction without convolutions. arXiv, 2021. 3,4
[9] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv, 2021. 2, 6,
[10] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting spatial attention design in vision transformers. arXiv, 2021. 2,3
[11] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In ECCV. 2018. 2.7
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 3
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
[14] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv, 2014. 3
[15] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. 3, 7,8
[16] Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang. Denscaspp for semantic segmentation in street scenes. In CVPR, 2018. 5
[17] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters improve semantic segmentation by global convolutional network. In CVPR, 2017. 3,5
10

---

[18] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decodlrr with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 3.
Liang-Chieh
[19] Yuhui Yuan and Jingdong Wang. Ocnet: Object context network for scene parsing. arXiv, 2018. 3, 8
[20] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang. Context prior for scene segmentation. In CVPR, 2020.
[21] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. arXiv, 2019. 7
[22] Hang Zhang. Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In CVPR, 2018. 7
[23] Yizhou Zhou, Xiaoyan Sun, Zheng-Jun Zha, and Wenjun Zeng. Context-reinforeed semantic segmentation. In CVPR, 2019.
[24] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR, 2017.
[25] Rudra PK Poudel, Ujwal Bonde, Stephan Liwicki, and Christopher Zach. Contextnet: Exploring context and detail for semantic segmentation in real-time. arXiv, 2018.
[26] Tianyi Wu, Sheng Tang, Rui Zhang. and Yongdong Zhang. Cgnet: A light-weight context guided network for semantic segmentation. arXiv, 2018.
[27] Junjun He. Zhongying Deng. Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In CVPR, 2019. 3
[28] Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary-aware feature propagation for scene segmentation. In ICCV, 2019. 3
[29] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural fields. In CVPR, 2016.
[30] Xiangtai Li. Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, and Yunhai Tong. Improving semantic segmentation via decoupled body and edge supervision. arxiv, 2020.
[31] Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model -annotic boundary refinement for segmentation. In ECCV, 2020.
[32] Mingmin Zhen, Jinglu Wang, Lei Zhou, Shiwei Li, Tianwei Shen, Jiaxiang Shang, Tian Fang, and Long Quan. Joint semantic segmentation and boundary detection using iterative pyramid contexts. In CVPR, 2020.
[33] Towaki Takikawa, David Acuna, Varun Jampani. and Sanja Fidler. Gated-scnn: Gated shape cans for semantic segmentation. In ICCV, 2019.
[34] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Learning a discrimina- tive feature network for semantic segmentation. In CVPR, 2018.
[35] Liang-Chich Chen, Jonathan T Barron, George Papandreou, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with task-specific edge detection using cnns and discriminatively trained domain transform. In CVPR, 2016. 3
[36] Jun Fu. Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019.
[37] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[38] Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei-Shi Zheng, Jonathan Li, and Alexander Wong. Squeezz-and-attentioon networks for semantic segmentation. In CVPR, 2020.
[39] Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Conet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 7.8
[40] Hanchao Li, Pengfei Xiong, Jie An, and Lingxue Wang, Pyramid attention network for semantic segmenta- tion. arXiv 2018

---

[41] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy. Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In ECCV, 2018. 8
[42] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation maximiiiaimmiiimiitttii attention networks for semantic segmentation. ICCV, 2019.
[43] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu, Genet: Non-local networks meet squeeze- excitation networks and beyond. In /CCVW, 2019.
[44] Enze Xie, Wenjia Wang, Wenhai Wang. Peize Sun, Hang Xu, Ding Liang, and Ping Luo. Segmenting transparent object in the wild with transformer. IJCAI, 2021. 3
[45] Albert Shaw, Daniel Hunter, Forrest Landola, and Sammy Sidhu. Squeezenas: Fast neural architecture search for faster semantic segmentation. In ICCVW, 2019. 3
[46] Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang. Yuan Li, and Zhangyang Wang. Fasterseg: Searching for faster real-time semantic segmentation. arXiv, 2019.
[47] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. In CVPR, 2020.
[48] Chenxi Liu, Liang-Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CPVR, 2019. 7
[49] Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid, Fast neural architecture search of compact semantic segmentation models via auxiliary cells. In CVPR, 2019. 3
[50] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End object detection with transformers. In ECCV, 2020. 3
[51] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv, 2021.
[52] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv, 2021. 3,5
[53] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv, 2021. 3
[54] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: cross-attention multi-scale vision transformer for image classification. arXiv, 2021. 3
[55] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality vision transformers. arXiv, 2021. 3
[56] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv, 2021. 3,5
[57] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. arXiv. 2021. 3
[58] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. arXiv, 2021. 3
[59] Peize Sun, Yi Jiang, Rufeng Zhang, Euze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple-obeect tracking with transformer. arXiv, 2020.
[60] Tim Meinhardt, Alexander Kirillov, Laura Laixe and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. arXiv, 2021. 3
[61] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng. Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv, 2020. 3
[62] Shuting Hc, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- object re-identification. arXiv, 2021. 3
[63] Manoj Kumar, Dirk Weissenbom, and Nal Kalchbrenner. Colorization transformer. arXiv, 2021.
12

---

[64] Alaaeldin El-Nouby. Natalia Neverova, Ivan Laptev, and Hervé Jégou. Training vision transformers for image retrieval. arXiv,, 2021. 3
[65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Míshkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv, 2021. 3
[66] Ronghang Hu and Amanpreet Singh. Transformer is all you need: Multimodal multitask learning with unified transformer. arXiv, 2021. 3
[67] Md Amirul Islam, Sen Jia, and Neil DB Brucc. How much position information do convolutional neural networks encode? arXiv, 2020. 5,5
[68] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. arXiv, 2017. 5
[69] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[70] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k datasct. In CVPR, 2017. 8
[71] Holger Caesar, Jasper Uijtings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018. 6,8
[72] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan L. Yuille, and Liang-Chich Chen. Axial- deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV, 2020. 7,
[73] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. In CVPR, 2020. 7
[74] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Buld, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017. 9
[75] Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmentation models. In CVPR, 2020. 9
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please Section 5.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see
Broader Impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] We observed quite stable mloU results for semantic segmentation models.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
13

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please see Section 4,
(b) Did you mention the license of the assets? [Yes] The data used in our work is open source and can be used for adademic research.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We provide our code for this work in the supplemental material.
(d) Did you discuss whether and how consent obtained from people whose data you't 're using/curatingg [N/A] The data used in our work is open source and can be used for adademic research.
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information or offensive content? [N/A] The data does not contain personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "v4C3zqpLWr2", "reviewer": "Reviewer_GD12", "summary": "This paper introduces the SegFormer model, a Transformer-based model for semantic segmentation (i.e. dense pixel classification) in images. The model uses prior architectural innovations such as a pyramidal structure (progressive downsampling) and a factorized version of self-attention for computational efficiency. The main novelties over prior work on Transformer-based semantic segmentation models and image classification models are: 1) the use of convolutional filters to propagate position information from the image boundaries (as opposed to using explicit positional encodings) and 2) skip-connections between individual layers directly to the output layer, using spatial upsampling. Experimental evaluation demonstrates improved mIoU scores on default semantic image classification benchmarks compared to a baseline SETR (Segmentation Transformer) model, while achieving computational benefits thanks to the adoption of a more efficient Transformer architecture.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall well-written, well-structured and easy to follow, although clarity could be improved in several aspects (see below). The architectural choices are well-motivated and experimental results demonstrate clear advantages over prior approaches. The code / model implementation looks clean, and providing this implementation is a big plus and will allow for easy reproduction. Transformer architectures for applications in visual processing are very popular right now, and so the paper should be relevant for a NeurIPS audience, although the presented scope of the method (semantic pixel classification) is rather narrow. \n\nOverall, despite the positive points mentioned above, I am arguing for rejecting this paper in its current form. I lay out the reasons for this decision in the following.\n\nThe presented novelties (1. use of convolutional filters to propagate position information from the image boundaries; 2. skip-connections between individual layers directly to the output layer) are intermixed with techniques from prior work, such as using a pyramidal Transformer architecture (as in [1]) and the use of an efficient, factorized form of self-attention (also from [1]). In the experimental section, it is unclear whether the performance benefits stem from utilizing these components from prior work [1] or whether the benefits stem from the novel contributions 1. and 2. -- the computational efficiency gains appear to be largely due to the use of the architecture from [1], which does not merit a separate conference paper (swapping out an image classification with a pixel classification target is of limited novelty).\n\nThe effective receptive field analysis and the analysis of robustness to natural corruptions are interesting, but they do not compare to the more recent SETR baseline, but instead to a weaker ConvNet-based baseline published in 2018. They thus unfortunately add little to convince the reader about the benefits of the proposed novelties over the current state of the art. The showcased qualitative improvements on Cityscapes in Figure 4 compared to SETR appear to be quite marginal (mask boundaries are marginally sharper, but otherwise unchanged). If this is a representative example of the obtained improvement, then it is unclear how this method will improve prediction quality in downstream applications.\n\nThe statement that the proposed \"encoder can easily adapt to arbitrary test resolutions without impacting the performance\" is not adequately verified. In fact, I would expect this to be false: if I increase the resolution 100x of the input image, I would very much expect the model to struggle (not just computationally, but also in terms of propagating position information throughout the model). The experiment in Table 1c suggests that increasing the resolution and aspect ratio by a small factor leads to only little loss in performance, but this does not justify the above statement. \n\nSince the novel form of (implicit) positional encoding is a core contribution, it should be investigated more. For example, one could try to measure the model's ability to predict positional encodings by either: 1) giving the model PE as inputs (i.e. trivial auto-encoding) vs. 2) using the proposed position-free approach. This would allow for a good comparison of how 1) and 2) compare in terms of generalization to new image sizes.\n\nRegarding the second novelty, i.e. skip-connections to the decoder: SETR introduces a related strategy termed \"Multi-Level feature Aggregation (MLA)\". A direct comparison against this technique (+ upsampling where necessary) would strengthen this contribution.\n\nClarity of the paper could be improved. There are many typos and grammatical mistakes, which should be easy to fix by doing another pass over the paper. The \"Overlapped patch merging\" strategy is described in a confusing way and parameters such as the padding size P are not explained. Looking at the code, this operation is simply implemented by a convolutional layer, and hence it would greatly simplify the description if the authors replaced this paragraph by simply saying that they use a convolutional layer after Transformer blocks, if I understand the method correctly.\n\nLastly, the impact of this method could be greatly increased by considering (and comparing on) other pixel-level prediction tasks such as depth prediction, which the architecture should support out-of-the box.\n\n\nOther comments / questions, suggestions for improvement:\n* The references [2,3] as cited in the sentence \"In the deep learning era [...]\" (Section 2, first sentence) look quite out of place compared to the other (seminal) works cited in this context. Were these cited by accident? I recommend explaining their relevance in the text (or dropping them).\n* SETR uses SGD with momentum as optimizer whereas the present method uses AdamW. How much of the performance difference can be explained by the different choice of optimizers? Doing an ablation study over this choice would help clarify where the performance benefits come from (i.e. train with SGD and momentum in a setting similar to SETR).\n* What is the unit for Params in Table 2?\n* Why did the authors not compare on PASCAL Context (as in the SETR paper)?\n* No error bars are reported and the authors state that the variance between seeds was low (\"quite table mIoU results\"). Would the authors be able to quantify this variance on at least one experiments, to get an idea whether results are significant?\n\n\n[1] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021\n[2] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In CVPR, 2019. \n[3] Wenhai Wang, Xiang Li, Tong Lu, and Jian Yang. Mixed link networks. In IJCAI, 2018.\n\n\n----- UPDATE AFTER REBUTTAL ----\n\nI would like to thank the authors for their very extensive and detailed response to my review. My concerns about positioning of the paper still remain, i.e. it is difficult to clearly pinpoint what is the main contribution of this work, as the method clearly benefits from a careful combination of several architectural contributions in conjunction with careful optimization. In this point I largely agree with the concerns raised by reviewer n5uX. \n\nAt the same time, the authors have done an impeccable job at addressing my other concerns and at carefully investigating various aspects and components of the proposed model. Given the detailed experimental evaluation and the clear usefulness of this model as a strong baseline for the semantic segmentation community (as pointed out by reviewer LMud), I believe that this paper lies marginally above the acceptance threshold despite concerns around positioning and significance of the individual technical contributions. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not clearly discussed, apart from the following limitation mentioned by the authors:\n\"One limitation is that although our smallest 3.7M parameters model is smaller than the known CNN’s model, it is unclear whether it can work well in a chip of edge device with only 100k memory. We leave it for future work.\". -- It is not entirely clear how number of parameters relates to memory consumption, as the latter is often dominated by intermediate activations and gradients. It would be good to discuss limitations specific to this particular model class. What are interesting next steps for future work?\n\nIn terms of (negative) societal impact the authors write \"We do not foresee obvious undesirable ethical/social impacts at this moment.\" -- I would encourage the authors to consider highlighting potential negative societal impact stemming from future applications by techniques such as theirs, e.g. in the area of surveillance.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qoGTzsE8dfh", "reviewer": "Reviewer_n5uX", "summary": "This paper introduces SegFormer for semantic segmentation. The SegFormer consists:\n(1) A Transformer encoder that extracts multi-scale features. The authors claim their Transformer encoder to be novel in the sense that it is \"positional-encoding-free\" and \"hierarchical\".\n(2) A \"MLP\" based decoder that aggregates information from different layers. Although the authors claim it to be \"MLP\", the decoder is essentially a stack of several 1x1 convolutions with bilinear interpolation to upsample low-resolution features.\n\nThe authors further explore scaling the Transformer encoder which ends up with a series of models with different number of parameters and FLOPs. The authors show their models are competitive with SOTA on multiple semantic segmentation datasets and claim that their model is more robust on corrupted images than DeepLabV3+ (on Cityscapes-C).", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Limitations of the paper\n\nFirst of all, I don't agree with the position of this paper. The authors claim that both the Transformer encoder and the new \"MLP\" decoder are equally important to make SegFormer work. However, I don't see any reason why other decoders cannot be used with the SegFormer Transformer encoder, and why other Transformer encoder cannot be used with Segformer decoder. For me, this paper is more like a study of Transformer design that is not making a proper comparison with other Transformers: the authors did not show their number on ImageNet classification, which is fine if the model only works well for semantic segmentation. However, the comparison with other Transformers on semantic segmentation is completely unfair which I will explain later.\n\nAlthough this paper shows good performance on multiple semantic segmentation datasets, it has two major problems: (A) the authors overclaimed their contribution (or did not clearly summarize their contribution) and (B) the experiments in this paper cannot support claims made by the authors. Next, I will explain my reasons.\n\n### A. The authors overclaimed their contribution (or did not clearly summarize their contribution)\n\n1. The authors claim they propose a \"novel positional-encoding-free and hierarchical Transformer encoder\" (L61).\n    * In L165, it looks like the authors follow CPVT [54] by using 3x3 convolution to replace positional encoding. Furthermore, in L166-167, the authors argue \"positional encoding is actually not necessary for semantic segmentation\" without any explanation. I'm not sure how the authors come up with this conclusion, since it is something not obvious. In the Table 1 (c), the authors only compare \"PE\" with \"Mix-FFN\", but it is not enough to say positional encoding is not necessary. Given these facts, I don't think \"positional-encoding-free\" or \"Mix-FFN\" is something contributed by the authors, as there are many works already uncover this fact like CPVT [54] and CvT [58].\n\n    * As for the hierarchical Transformer, I find it is very hard to understand what is the key difference with other hierarchical Transformers like PvT [8] from current description (the current text only describes \"Hierarchical Feature Representation\", \"Overlapped Patch Merging\", \"Efficient Self-Attention\" which seems to be already used by multiple hierarchical Transformer papers).\n\n2. The authors claim their \"All-MLP\" decoder is lightweight and less complex than other decoder. First of all, \"complex\" is very subjective. Why one decoder is more complex than another decoder without a proper metric? Second, the decoder used in the paper is **not** \"All-MLP\": the decoder is essentially several 1x1 convolutions. And here is the question again: why 1x1 convolutions are less \"complex\" than 3x3 convolutions?\n\n### B. The experiments in this paper cannot support claims made by the authors\n\n1. The experiments do not show both proposed components (new Transformer encoder and \"MLP\" decoder) are necessary.\n    * In Table 1, the authors only compares their Transformers with CNN backbones which is not a very fair comparison. In order to really show the benefit of this new Transformer, the authors should compare their Transformer with other Transformer designs (e.g., PvT, CvT, Swin Transformer etc.) to really show this is a better Transformer.\n    * As for the decoder design, the authors only compare their decoder with different channel dimensions. However, the claim made by the authors is the \"MLP\" decoder is better than other \"more complex\" decoder. In order to backup this claim, the authors should compare the \"MLP\" decoder with the \"more complex\" decoder they refer to.\n\n2. Design of the experiments is not consistent with the position of the paper. The authors position their paper as a Transformer design for semantic segmentation. Thus, it is okay to not comparing with other Transformer models on ImageNet. However, the authors show at least have a proper comparison with existing Transformers to show why their design is better than other Transformers. For example, the authors can compare their Transformer with Swin Transformer using **the same decoder and the same training recipe**. However, from the description of the paper, I do not see why this special design is more suitable than other Transformer specifically for semantic segmentation.\n\n## Summary\n\nIn summary: (1) I believe the claims made by the authors in this paper is not strong enough, (2) experiments cannot support the claims made by the authors as well; and (3) the position of this paper is weird. I'm especially concerned about (2) and (3). I do not see why the design of this Transformer is specifically suitable for semantic segmentation, if the model does not perform better in ImageNet classification. Since the paper never offers an apple-to-apple comparison with other Transformer models for semantic segmentation task, I'm afraid the authors might have tuned training parameters very hard to make it work on semantic segmentation (while other Transformers mainly focus on reporting numbers for classification). If this is the case, I don't feel this paper is very valuable and I think it is a clear rejection. But I hope to see authors explanation if I make any mistake.\n\n\n\n--------\n\n\\#### post-rebuttal update ####\n\nI have read carefully all reviews and author response. The authors have addressed most concerns well and the major disagreement between reviewers (e.g., me and Reviewer GD12) and the authors lies in the position of this paper. I have to say that I'm still not fully convinced and I believe it is better to position the paper as a new ViT design instead of a specific model design for semantic segmentation. However, the authors argue they mainly show improvements on semantic segmentation which is the reason that they position it as a model for semantic segmentation specifically, which is a fair point and I respect their decision.\n\nBesides the problem of \"novelty\" raised by all other reviewers, another major problem in the original submission is that there is no fair comparison which leads to my original decision of rejection. The authors have addressed this problem by running the suggested experiments and thus I will raise my score to borderline. However, I believe the paper requires a major revision to take into account all these discussions and I'm not sure if the authors will make the revision as they promised since I cannot see the revised version. So I will change my score to 5 and let the AC to decide whether this should be taken into account.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "qbR4Wde4oFM", "reviewer": "Reviewer_LMud", "summary": "This work proposed an efficient Transformer-based semantic segmentation model called SegFormer. In essence, it consists of a hierarchical Transformer encoder and a lightweight MLP decoder. Compared to ViT, the transformer encoder employs overlapped patch merging, self-attention with reduced (key, value) for efficient computation, and a depthwise convolution inserted between two MLP layers in FFN to replace position embedding. The authors also conducted an analysis based on effective receptive field and revealed that transformer's encoder with large receptive field enables relative lightweight decoder design. Extensive experiments demonstrate that SegFormer achieves strong speed-accuracy trade-off and better robustness.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Overall writing is clear and easy to follow. As for the method, the architectural design is simple yet effective as proven by extensive experiments and analysis. As a result, this could serve as a new baseline for semantic segmentation task. \n- In L165, it was stated that \"CPVT [54] uses 3x3 Conv together with PE to implement a data-driven PE\". To my best understanding, [54] does not require PE. Instead, similar to this work, a 2D Conv with zero padding is employed to replace PE.  \n- Swin Transformers [9] is missing from the comparison with state-of-the-arts. \n- The results in ablation experiment (Tab.1(b)(d)) does not match the final performance in Tab.1(a). For example, in Tab.1(d), the model variant MiT-B3 (S1-4) achieves 48.6 mIoU on ADE20K whereas MiT-B3 in Tab.1(a) attains 49.4/50.0 mIoU on the same ADE20K dataset despite having the same FLOPs and number of parameters. Why is this so?\n___________\nUPDATE AFTER REBUTTAL\n\nI appreciate the authors' detailed rebuttal. Here are my responses after reading other reviewers' comments and the authors' response:\n\nI have to admit that when I first read this paper, I had the same concern as other reviewers that this work shares high similarity with some recent vision transformer works, especially PvT [8] in terms of its hierarchical representation structure and efficient self-attention design. However, the efficient self-attention design was not claimed as the contribution of this work. In fact, the authors admitted in L155 that such design was taken from [8]. Nevertheless, I agree with reviewer GD12 that some of the presented components (e.g. overlapped patch merging, hierarchical representation) are intermixed with techniques from prior works. \n\nThe main reason for me to recommend an accept are two-fold: 1) the effectiveness of the proposed Mix-FFN for addressing train-test resolution mismatch; 2) the simplicity and efficiency of the overall design. The former is similar to LocalViT [57] which employs a 3x3 depthwise conv between two consecutive MLPs in FFN. Nevertheless, [57] retains the use of position encoding. On the other hand, this work demonstrates their Mix-FFN not only removes the need of positional encoding, but also fixes train-test resolution disparity while a fixed shape positional encoding fails to. I believe this finding is important for dense prediction task such as semantic segmentation where the testing resolution might differ from the training ones.\n\nSecondly, the simplicity and efficiency of the design makes it serve as a strong baseline for semantic segmentation. SETR has previously shown that transformer backbones work well on semantic segmentation. Differently, this work demonstrates a compact and efficient solution for the task while performing comparably or even better. Although each component may not be entirely novel or inspiring, combining them to offer a simple yet effective solution is beneficial to the community, similar to the role of DeepLab series is playing in the segmentation community.\n\nAfter reading other reviewers' comments and authors' response, I think most of the concerns were well addressed or at least to a certain extent (e.g. different combinations of encoder and decoder, comparison with more real-time segmentation models, variance of different runs, experiments on PASCAL Context dataset etc.). I agree with reviewer n5uX that the technical contribution may be insufficient for both vision and semantic segmentation community. However, as explained earlier, although each component may not be entirely novel, I think offering a feasible compact solution that could serve as a strong baseline for semantic segmentation would be beneficial to the community. After taking all these into accounts, I have decided to lower my rating to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to \"Main Review\" section above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "WmRurCx9x_p", "reviewer": "Reviewer_vMMz", "summary": "This paper presents a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perceptron (MLP) decoders. The proposed SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The extensive experiments show that the proposed method achieves the state-of-the-art performance on semantic segmentation task.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper presents a transformer based network for semantic segmentation task. Different from the last transformer based network SETR, the proposed SegFormer uses  positional-encoding-free,  hierarchical Transformer encoder and lightweight All-MLP decoder, which is more efficient and flexible than SETR.\n2. My concerns and suggestions about this paper are as follows:\n         1). The author claims that the segformer is novel, but the used overlapped patch embedding, efficient attention, the multi-stage structure are basically from PVT, PVTv2.  There is no any novel module or techniques proposed in this paper. So i think the novelty is limited.\n         2). The author mainly uses the SETR as baseline method, but actually recent Swin and PVT (cited in the paper) also show the SOTA performance, which are not compared in the experimental report.\n         3). The lightweight sseg models are compared in the experimental part, but only mobilenet based networks are listed,   ESPNet, ESPNetv2, BiSeNet, ICNet and other lightweight models are not reported and compared. i recommend the author can give more detailed comparison.\n        4). In addition, UNet-like network is also popular network in semantic segmentation. Does it work for the transformer based network? it's better to give experiments about this.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "the suggestions are presented in Main Review part", "review_text": "", "extra_scores": {"ethical_concerns": "No Ethical Concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an efficient semantic segmentation model composed of a hierarchical Transformer encoder and a lightweight MLP decoder. The transformer encoder employs overlapped patch merging, efficient self-attention module, and a depthwise convolution inserted between two MLP layers in FFN to replace positional embeddings. The empirical evaluation demonstrates better accuracy/speed tradeoffs compared to competing models.
The paper was reviewed by 4 experts who appreciated the clarity of exposition, simplicity of the design, the effectiveness of the proposed approach in dealing with test-train resolution mismatch, as well as competitive performance. During the discussion the reviewers pointed out several concerns related to positioning and the significance of technical contributions which were mostly addressed. Finally, the reviewers agreed that the approach is a simple and competitive baseline for the semantic segmentation community. I will recommend acceptance.
Decision: Accept (Poster)