SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation
Authors: Meng-Hao Guo
Cheng-Ze Lu
Qibin Hou
Zheng-Ning Liu
Ming-Ming Cheng
Shi-min Hu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Semantic segmentation
attention
convolutional neural network.

Abstract: We present SegNeXt, a simple convolutional network architecture for semantic segmentation. Recent transformer-based models have dominated the field of se- mantic segmentation due to the efficiency of self-attention in encoding spatial information. In this paper, we show that convolutional attention is a more efficient and effective way to encode contextual information than the self-attention mech- anism in transformers. By re-examining the characteristics owned by successful segmentation models, we discover several key components leading to the perfor- mance improvement of segmentation models. This motivates us to design a novel convolutional attention network that uses cheap convolutional operations. Without bells and whistles, our SegNeXt significantly improves the performance of previous state-of-the-art methods on popular benchmarks, including ADE20K, Cityscapes, COCO-Stuff, Pascal VOC, Pascal Context, and iSAID. Notably, SegNeXt out- performs EfficientNet-L2 w/ NAS-FPN and achieves 90.6% mIoU on the Pascal VOC 2012 test leaderboard using only 1/10 parameters of it. On average, SegNeXt achieves about 2.0% mIoU improvements compared to the state-of-the-art methods on the ADE20K datasets with the same or fewer computations.
























SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation
Meng-Hao Guo
Ze Lu Ming-Ming Cheng"
Qibin Hou Shi-Min Hu Zheng-Ning Liu
BNRist, Department of Computer Science and Technology, Tsinghua University TMCC, CS, Nankai University
FFitten Tech, Beijing, China
Abstract
We present SegNeXt, simple convolutional network architecture for semantic segmentation. Recent transformer- based models have dominated the field of se- mantic segmentation due the efficiency of self-attention in encoding spatial information. In this paper, we show that convolutional attention is more efficient and effective way to encode contextual information than the self-attention mech- anism in transformers. By re-examining the characteristics owned by successful segmentation models, discover several key components leading the perfor- mance improvement of segmentation models. This motivates us to design a novel convolutional attention network that uses cheap convolutional operations. Without bells and whistles, our SegNeXt significantly improves the performance of previous state-of-the oof-the-arttt art methods on popular benchmarks, including ADE20K, Cityscapes, COCO-Stuff, Stuff, Pascal VOC, Pascal Context, and iSAID. Notably, SegNeXt out- performs EfficientNet-L.22 w/ NAS-FPN and achieves 90.6% mloU on the Pascal VOC 2012 test leaderboard using only V10 parameters of it. On average, SegNeXt achieves about 2.0% mloU improvements compared to the state-of-the methods the ADE20K datasets with the same or fewer computations.
Introduction
As one of the most fundamental research topics in computer vision, semantic segmentation, which aims at assigning each pixel semantic category, has attracted great attention over the past decade. From early CNN-based models, typified by FCN [60] and DeepLab series [4, 5, 7]. to recent transformer-based methods, represented by SETR [108] and SegFormer [90], semantic segmentation models have experienced significant revolution in terms of network architectures.
Table Properties we observe from the successful semantic segmentation methods that beneficial to the boost of model performance. Here, n refers to the number of pixels or tokens. Strong encoder denotes strong backbones, like VIT [16] and VAN [25].
Properties
DeepLabV3+
HRNet
SETR
SegFormer
SegNeXt
Strong encoder
Multi-scale interaction
X
x
Spatial attention
/
Computational complexity
(n))
(n)
O(n))
O
O
*S.-M. Hu is the corresponding author. Project page: https: btttpa//gitthbb cttp:/a//ooo//t/t/////a////
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

83
48
81
46-
80
mlou
79-
78
SegFormer [90]
SeyT VerTormer [90]
40-
HRFommer [100]
HRFormer [100]
MaskFormer [10]
SegNeXt
SegNeXt
76
1000
1500
2000
200
250
Model (G)
Model FLOPs (G)
Figure I: Performance- Computttn curves on the Cityscapes (left) and ADE20K (right) validation sets.
FLOPPs are calculated using an input size of 2, 048 x 1, 024 for Cityscapes and 512 x 512 for ADE20K. The size of the circle indicates the number of parameters. Larger circles mean more parameters. We can see that our SegNeXt achieves the best trade-off between segmentation performance and computational complexity.
By revisiting previous successful semantic segmentation works, summarize several key properties different models possess as shown Tab. 1. Based on the above observation, we argue a successful semantic segmentation model should have the following characteristics: (i) A strong backbone network as encoder. Compared to previous CNN- based models, the performance improvement of transformer-based models mostly from a stronger backbone network. (ii) Multi-scale information interaction. Different from the image classification task that mostly identifies a single object, semantic segmentation is dense prediction task and hence needs process objects of varying sizes in single image. (jii) Spatial attention. Spatial attention allows models to perform segmentation through prioritization of areas within the semantic regions. (iv) Low computational complexity. This especially crucial when dealing with high-resolution images from remote sensing and urban scenes.
Taking the aforementioned analysis into account, in this paper, rethink the design of convolutional attention and propose an efficient yet effective architecture for semantic segmentation. Unlike previous transformer-bit based models that use convolutions in decoders feature refiners, method inverts the transformer-convolttio encoder-decoder architecture. Specifically, for each block in our encoder, renovate the design of conventional convolutional blocks and utilize multi-scale convolutional features to evoke spatial attention via a simple element-wise multiplication following [25]. We found such a simple way to build spatial attention is more efficient than both the standard convolutions and self-attention in spatial information encoding. For decoder, we collect multi-level features from different stages and use Hamburger [22] to further extract global context. Under this setting, our method can obtain multi-scale context from local to global, achieve adaptability in spatial and channel dimensions, and aggregate information from low high levels.
Our network, termed St,NeXt, is mostly composed convolutional operations except the decoder part, which contains decomposition based Hamburger module [22] (Ham) for global information extraction. This makes our SetNeXt much more efficient than previous segmentation methods that heavily rely transformers. As shown in Fig. 1, SegNeXt outperforms recent transformer-based methods significantly. In particular, our SegNeXt-S outperforms SegFormer-Bi B2 (81.3% vs. 81.0%) using only about %/6 (124.6G vs. 717. 7I7.1G) computational cost and V/2 parameters (13.9M vs. 27.6M) when dealing with high-resolution urban scenes from the Cityscapes dataset.
Our contributions can be summarized as follows:
We identify the characteristics that a good semantic segmentation model should own and present a novel tailored network architecture, termed SegNeXt, that evokes spatial attention via multi-scale convolutional features.
We show that an encoder with simple and cheap convolutions can still perform better than vision transformers, especially when processing object details, while requires much less computational cost.

---

Our method improves the performance of state-of saate-of-the-arr by a large margin on various segmentation benchmarks, including ADE20K, Cityscapes. COCO-Stuff, Pascal VOC, Pascal Context, and iSAID.
the-hhe-rtt art semantic segmentation methods
2 Related Work
2.1 Semantic Segmentation
Semantic segmentation is fundamental computer vision task. Since FCN [60] was proposed, convolutional neural networks (CNNs) [1, 71, 98, 106, 20, 99, 79, 21, 51] have achieved great success and become a popular architecture for semantic segmentation. Recently, gransformerbbase methods [108, 90, 100, 73, 70, 50, 10, 9] have shown great potentials and outperform CNN-based methods.
In the era of deep learning, the architecture of segmentation models can be roughly divided into two parts: encoder and decoder. For the encoder, researchers usually adopt popular classification networks (e.g., ResNet [28], ResNeXt [91] and DenseNet [33]) instead of tailored architecture. However, semantic segmentation a kind of dense prediction task, which is different from image classification. The improvement in classification may not appear in the challenging segmentation task [29]. Thus, some tailored encoders appear, including Res2Net [21]. HRNet [79], SETR [108], SegFormer [90], HRFormer [100], MPViT [44], DPT [70], etc. For the decoder, it often used in cooperating with encoders to achieve better results. There are different types of decoders for different goals, including achieving multi-scale receptive fields [106, 6, 88). collecting multi-scale semantics [71, 90, 7]. enlarging receptive field [4, 4, 69), strengthening edge features [107, 2, 15, 48, 102]. and capturing global context [20, 35, 101, 46, 24, 27, 103].
In this paper, we summarize the characteristics of those successful models designed for semantic segmentation and present CNN-based model, named SegNeXt. The most related work to our paper, [69]. which decomposes a x & convolution into a pair of & x and 1 x & convolutions. Though this work has shown large convolutional kernels matter in semantic segmentation, it ignores the importance multi-scale receptive field and does not consider how to leverage these multi-scale features extracted by large kernels for segmentation in the form of attention.
2.2 Multi-Scale Networks
Designing mcati-scale network one of the popular directions in computer vision. For segmentation models, multi-scale blocks appear in both the encoder [79, 21, 75] and the decoder [106, 98, 5] parts. GoogleNet [75] is one of the most related multi-scale architectures to our method, which uses multi-branch structure to achieve multi-scale feature extraction. Another work that is related to our method is HRNet [79]. the deeper stages, HRNet also keeps high-resolution features, which are aggregated with low-resolution features, to enable multi-scale feature extraction.
Different from previous methods, SegNeXt, besides capturing multi-scale features in encoder, intro- duces an efficient attention mechanism and employs cheaper and larger kernel convolutions. These enable our model to achieve higher performance than the aforementioned segmentation methods.
2.3 Attention Mechanisms
Attention mechanism is a kind of adaptive selection process, which aims to make the network focus on the important part. Generally speaking, it can be divided into two categories in semantic segmentation [26], including channel attention and spatial attention. Different types of attentions play different roles. For instance, spatial attentions mainly care about the important spatial regions [16, 13, 64, 58, 23]. Differently, the goal of using channel attention is to make the network selectively attend to those important objects, which has been demonstrated important in previous works [31, 8, 80). Speaking of the recent popular vision transformers [16, 58, 94, 81, 82, 57, 90, 34, 56, 100, 93], they usually ignore adaptability in channel dimension.
Visual attention network (VAN) [25] is the most related work to SegNeXt, which also proposes to leverage the large-kernel attention (LKA) mechanism to build both channel and spatial attention. Though VAN has achieved great performance image classification, neglects the role of multi-scale feature aggregation during the network design, which is crucial for segmentation-lit like tasks.
3

---

Corvolltional Attantion
x1
GELU
Chamnel Mixing
FFN
DOORy 3x3
x1
BN
1 x1
d 7x1
11x
21x1
1 x1
Attention
d 1x7
1x11
1x21
MSCA
GELU
BN
Doorv 5x5
Multi-scall
1 1x1
XL.
Feabures
(a) A stage of MSCAN
(b) MSCA
Figure 2: Illustration of the proposed MSCA and MSCAN. Here, d, k1 2y means depth-wise convolution (d) using a kernel size of kn x k2- We extract multi-scale features using convolutions and then utilize them attention weights to reweigh the input of MSCA.
Method
In this section, we describe the architecture of the proposed SegNeXt in detail. Basically, we adopt an encoder-decoder architecture following most previous works, which simple and easy to follow.
3.1 Convolutional Encoder
We adopt the pyramid structure for our encoder following most previous work [90, 4, 20]. For the building block in encoder, adopt similar structure to that of ViT [16, 90] but what is different is that we do not use the self-attention mechanism but design a novel multi-scale convolutional attention (MSCA) module. As depicted in Fig. 2 (a), MSCA contains three parts: depth-wise convolution to aggregate local information, multi-branch depth-wise strip convolutions to capture multi-scale context, and output of the x convolution is used as attention weights directly to reweigh the input of MSCA. Mathematically, our MSCA can be written as:
convolution model relationship between different channels. The
Att Conv x
(DW-CCoDDn
Out Att 1
(2)
where represents the input feature. Att and Out are the attention map and output, respectively. & is the clement-wise matrix multiplication operation. DW-Cony denotes depth-wise convolution and Scale,, E {0, 1, 2, 3), denotes the ith branch Fig. 2(b). Scale, is the identity connection. Following [69], in each branch, we use two depth-wiss wise strip convolutions to approximate standard depth-wise convolutions with large kernels. Here, the kernel size for each branch is set 7, 11, and 21, respectively. The reasons why we choose depth-wise strip convolutions are two-fold. On one hand, strip convolution lightweight. we only need pair of x and 1 x 7 convolutions. On the other hand, there are some strip-like objects, such as human and telephone pole in the segmentation scenes. Thus, strip convolution can be complement of grid convolutions and helps extract like fcatures [69,30].
mimic a standard 2D convolution with kernel size
Stacking sequence of building blocks yields the proposed convolutional encoder, named MSCAN. For MSCAN, we adopt common hierarchical structure, which contains four stages with decreasing spatial resolutions
& 16 16 32 32 Here, H and W are height and width of the input image, respectively. Each stage contains down down-sampling block and stack of building blocks as described above. The down-sampling block a convolution with stride and kernel size 3 x 3, followed by batch normalization layer [36]. Note that, in each building block of MSCAN, we use batch normalization instead of layer normalization as we found batch normalization gains more for the segmentation performance.
W 1
H W and H

---

Table 2: Detailed settings of different sizes of the proposed Se.NeXt. In this table, 'c.r.' represents the expansion ratio in the feed-forward network. 'C" and 'L' are the numbers of channels and building blocks, respectively. 'ecooder dimension' denotes the MLP dimension in the decoder. 'Parameters' are calculated on the ADE20K dataset [111]. Due to the different numbers of the categories in different datasets, the number of parameters may change slightly.
stage
output size
SegNeXi-T
SegNeXt-S
SegNeXt-B
SegNeXI-L
C
32,
C= 64, =
C= 64, =
64,
4
2
H
8
x
x
64, 3
128, I.
C= 128, 1,
128, 5
3
16
H
x
W
T6
x C
C 160, = 5
320,L 4
C 320,L = 12
C 320,L 27
4
32
8
32
x
C
4
C'
256.L = 2
C 512,L
C C= 512,4 33
C' 512,L 3
Decoder dimension
256
256
512
1,024
Parameters (M)
4.3
13.9
27.6
48.9
desgin four encoder models with different sizes, named MSCAN-T, MSCAN-S, MSCAN-B, and MSCAN-L, respectively. The corresponding overall segmentation models are termed SegNeXt-T, SegNeXI-S, SegNeXI-B, SegNeXt-L, respectively, Detailed network settings are displayed in Tab.
3.2 Decoder
In segmentation models [90, 108, 4], the encoders are mostly pretrained on the ImageNet dataset. To capture high-level semantics, decoder is usually necessary, which is applied upon the encoder. In this work, we investigate three simple decoder structures, which have been shown Fig. The first one, adopted in SegFormer [90], purely MI.P-hased structure. The second one is mostly adopted CNN-based models. In this kind of structure, the output of the encoder is directly used as the input heavy decoder head, like ASPP [4], PSP [106], and DANet [20]. The last one is the structure adopted in our SegNeXt. We aggregate features from the last three stages and use a lightweight Hamburger [22] to further model the global context. Combined with our powerful convolutional encoder, we found that using a lightweight decoder improves performance-computaiie efficiency.
[ sbees
2 abe1s
8 Labees
t Saaee
MD
concat
MLP
sbees
e6egs
Slbees
peay
diw
2 eabeis
{ eBeess
7 abees
2000
head
MLP
(a)
(b)
(c)
Figure 3: Three different decoder designs.
It is worth nothing that unlike SegFormer whose decoder aggregates the features from Stage 1 to Stage our decoder only receives features from the last three stages. This because our St is based convolutions. The features from Stage contain too much low-level information and hurts the performance. Besides, operations on Stage bring heavy computational overhead. In our experiment section, we will show that our convolutional SegNeXt performs much better than the recent state-of-the-arr transformer-based SegFormer [90] and HRFormer [100].
4 Experiments
Dataset. We evaluate our methods on seven popular datasets, including ImageNet-IK IK [14], ADE20K [111], Cityscapes [12], Pascal VOC [17], Pascal Context [65], COCO-Stuff [3], and iSAID [84]. ImageNet [14] is the best-known datasct for image classification, which contains
5

---

Table 3:
Comparison with state-of-the-art methods on ImageNet validation set. 'Acc!'
Table 4: Comparison with state-of-ther -f-t---rtt methods on the remote sensing dataset iSAID. Single-scale (SS) test is applied by default. Our SegNeXt-T has achieved state-of- of-te-of-the-ar the--rteattt performance.
denotes Top-1 accuracy.
Method
Params. (M)
Acc. (%)
Method
Backbone
mloU (%)
MiT-BO [90]
3.7
70.5
VAN-Tiny [25]
4.1
75.4
DenseASPP [95]
ResNet50
57.3
MSCAN-T
4.2
75.9
PSPNet [106]
ResNet50
60.3
SemanticFPN [40]
ResNet50
62.1
MiT-BI [90]
14.0
78.7
RefineNet [54]
ResNet50
60.2
VAN-Small [25]
13.9
81.1
HRNet [79]
HRNetW-18
61.5
MSCAN-S
14.0
81.2
GSCNN [76]
ResNec50
63.4
MrT-B2 [90]
25.4
81.6
SFNet [49]
ResNet50
64.3
Swin-T
28.3
81.3
RANet [66]
ResNet50
62.1
Con-NeXtt-T [59]
28.6
82.1
PointRend [41]
ResNet50
62.8
VAN-Base [25]
26.6
82.8
FarSeg [109]
ResNet50
63.7
MSCAN-B
26.8
83.0
UperNet [89]
Swin-T
64.6
PointFlow [47]
ResNet50
66.9
MiT-B3 [28]
45.2
83.1
Swin-S [58]
49.6
83.0
SegNeXt-T
MSCAN-T
68.3
Con- [58]
83.1
SegNeXt-S
MSCAN-S
68.8
VAN-Large [25]]
44.8
83.9
SegNeXt-B
MSCAN-B 69.9
MSCAN-L
45.2
83.9
SegNeXt-L
MSCAN-L 70.3
1,000 categories. Similar to most segmentation methods, it to pretrain MSCAN coder. ADE20K [111] is challenging dataset which contains 150 semantic classes. It consists of 20.210/2.000/33335 images in the training. validation and test sets. Cityscapes [12] mainly fo- cuses on urban scenes and contains 5.000 high-resolution images with 19 categories. There are 2,975/500/1,52 images for training, validation and testing, respectively. Pascal VOC [17] involves 20 foreground classes and a background class. After augmentation, has 10, 582/1, 449/1, 456 images for training, validation and testing, respectively. Pascal Context [65] contains 59 foreground classes and background class. The training set and validation set contain 4,996 and 5,104 images, respectively. COCO-Stuff [3] is also challenging benchmark, which contains 172 semantic cate- gories and 164k images in total. iSAID [84] is a large-scale aerial image segmentation benchmark, which includes 15 foreground classes and background class. Its training, validation and sets separately involve 1,41 1/458/937 images.
Implementation details. We conduct experiments by using Jittor [32] and Pytorch [68]. Our implementation is based on timm (Apache-2.0) [85] and mmsegmentation (Apache-2.0) [11] libraries for classification and segmentation, respectively. All encoders of our segmentation models pretrained on the ImageNet-I 1 dataset [14]. We adopt Top-1 accuracy and mean Intersection over Union (mloU) as our evaluation metrics for classification and segmentation, respectively. All models are trained on node with RTX 3090 GPUs.
For ImageNet pretraining, our data augmentation method and training settings the same as DeiT [78]. For segmentation experiments, we adopt some common data augmentation including random horizontal flipping, random scaling (from 0.5 to 2) and random cropping. The batch size set to for the Cityscapes dataset and 16 for all the other datasets. Adam W [61] is applied to train models. We set the initial learning rate 0.00006 and employ the poly-learning rate decay policy. We train our model 160K iterations for ADE20K, Cityscapes and iSAID datasets and 80K iterations for COCO-Stuff, Paseal VOC and Pascal Context datasets. During testing, single-scale (SS) and multi-scale (MS) flip test strategies for fair comparison. More details can be found in our supplementary materials.
both the
4.1 Encoder Performance on ImageNet
ImageNet pretraining is common strategy for training segmentation models [106, 5, 90, 100, 4]. Here, we compare the performance of our MSCAN with several recent popular CNN-based and transformer-based classification models. As shown in Tab. 3, our MSCAN achieves better results than the recent state-of-the-ar CNN-based method, ConvNeXt [59] and outperforms popular transformer- based methods, like Swin Transformer [58] and MiT, the encoder of SegFormer [90].

---

Table 5: Performance of different attention mechanisms in decoder. SegNeXt-B w/ Ham means the MSCAN-B encoder plus the Ham decoder. FLOPs are calculated using the input size of 512x512.
Architecture
Params. (M)
GFLOPs
mloU (SS)
mloU (MS)
SegNeXt-B CC [35]
27.8
35.7
47.3
48.6
SegNeXt-B B w/ EMA [46]
27.4
32.3
48.0
49.1
SegNeXt-B NL [83]
27.6
40.9
48.6
50.0
SegNeXt-B w/ Ham [22]
27.6
34.9
48.5
49.9
4.2 Ablation study
Ablation MSCA design. We conduct ablation study MSCA design on both ImageNet and ADE20K dataset. x K branch contains a depth-wist wise x convolution and a x depth-wise convolution. 1 conv means the channel mixing operation. Attention means the element-wiss. product, which makes the network obtain adaptive ability. Results are shown in Tab. 6. We can find that each part contributes to the final performance.
Table 6: Ablation study the design of MSCA. Top-1 means Top-1 I accuracy ImageNet dataset and mloU denotes mloU on ADE20K benchmark. The results are based on MSCAN-T.
7 x 7 branch
11 11 branch
21 x 21 branch
x 1 Conv
Attention
Top-1
mIoU
X
X
74.7
39.6
x X
75.2
39.7
X
75.3
40.0
X
74.8
39.1
X
75.5
40.5
75.9
41.1
Global Context for Decoder. Decoder plays an important role in integrating global context from multi-scale features for segmentation models. Here, we investigate the influence of different global context modules on decoder. As shown in most previous works [83, 20]. attention bbaed decoders achieves better performance for CNNs than pyramid structures [106, 4], we thus only show the results using attention-
decoders. Specifically, we show results with 4 different types of attention- based decoders, including non-local (NL) attention [83] with Onn?) complexity and CCNet [35], EMANet [46], and HamNet [22] with (n) complexity. As shown in Tab. 5, Ham achieves the best trade-off between complexity and performance. Therefore,
Hamburger [22] in decoder.
Table 7: Performance of different decoder structures. SegNeXt-T (a) means Fig. 3 (a) is used in decoder. FLOPs are calculated using the input size of 512x512. SegNeXt-T (c) w/ stage means the output of stage is also sent into the decoder.
Architecture
Params. (M)
GFLOPs
mloU (SS)
mloU (MS)
- (a)
4.4
10.0
40.3
41.1
SegNeXt-T (b)
4.2
4.9
30.9
40.6
SegNeXt-T (c)
4.3
6.6
41.1
42.2
SegNeXt-T (c) w/ stage
4.3
12.1
40.7
42.2
Decoder Structure. Unlike image classification, segmentation models need high-resolution outputs. We ablate three different decoder designs for segmentation, all of which have been shown in Fig. 3. The corresponding results are listed in Tab. We can see that SegNeXt (c) achieves the best performance and the computational cost is also low.
Importance Our MSCA. Here, we conduct experiments to demonstrate the importance of MSCA for segmentation. As comparison, we follow VAN [25] and replace the multiple branches in our MSCA with single convolution with large kernel. shown Tab. 8 and Tab. 3, we can observe that though the performance of the two encoders close in ImageNet classification, SegNeXt

---

Table 8: Importance of our multi-scale convolutional attention (MSCA), SegNeXi-T w/ MSCA means we use only branch with a large kernel convolution as done in [25] replace the multiple
branches in our MSCA. FLOPs are calculated using the input size of 51 x12.
Architecture
Params, (M)
GFLOPs
mloU (SS)
mloU (MS)
SegNeXt-T w/o MSCA SegNeXI-T w/ MSCA
4.2
6.5
39.5
40.9
4.3
6.6
41.0
42.5
SegNeXt-S wo MSCA
13.8
15.8
43.5
45.2
SegNeXt-S w/ MSCA
13.9
15.9
44.3
45.8
SegFormer-Bi
SegNeXt-B
SegFormer-Bi
SegNeXt-B
Figure 4: Qualitative Comparison of SegNeXt-B and SegFormer-B2 the Cityscapes dataset. More visual results can be found in our supplementary materials.
w/ MSCA yields much better results than the setting w/o MSCA. This indicates that aggregating multi-scale features crucial in encoder for semantic segmentation.
4.3 Comparison with state-of-the-art methods
In this subsection, compare method with state-of-the-art CNN-based methods, such as HRNet [79], ResNeSt [104], and EfficientNet [77], and transformer-bart based methods, like Swin Transformer [58], SegFormer [90], HRFormer [100], MaskFormer [10], and Mask2Former [9].
Performance-compptatioon trade-off. ADE20K and Cityscapes are two widely used benchmarks in semantic segmentation. As shown in Fig. 1, we plot the poformanccecomputatte different methods on the Cityscape and ADE20K validation set. Clearly, our method achieves the best trade-off between performance and computations compared to other Late-offthheart methods, like SegFormer [90], HRFormer [100], and MaskFormer [10].
curves of
Comparison with state-of-the-art transformers. We compare SegNeXt with state-of- slate-of-the-art trans- former models on the ADE20K, Cityscapes, COCO-Stuff and Pascal Context benchmarks. As shown in Tab. 9, SegNeXt-L surpasses Mask2Former with Swin-T backbone by 3.3 mloU (51.0 v.s. 47.7) with similar parameters and computational cost on he ADE20K dataset. Moreover, SegNeXt-B yields 2.0 mloU improvement (48.5 v.s. 46.5) compared to SegFormer-Bi B2 using only 56% computations on the ADE20K dataset. In particular, since the self-attention in SegFormer [90] is of quadratic complexity w.r.f., the input size while our method uses convolutions, this makes our method perform greatly well when dealing with high-resolution images from the Cityscapes dataset. For instance, Seg gains 1.6 mloU (81.0 v.s. 82.6) over SegFormer-B2 but uses 40% less computations. In Fig. 4, we also show a qualitative comparison with SegFormer. We can see that thanks to the proposed MSCA, our method recognizes well when processing object details.
Comparison with state-of-the-art CNNs. As shown in Tab. 4, Tab. 10, and Tab. 12, we compare our SegNeXt with tfte-off-the-ae ahet CNNs such as ResNeSt-266 [104]. Efficientett-22 [1] and HRNet- W48 [79] on the Pascal VOC 2012, Pascal Context, and iSAID datasets. SegNeXt-L outperforms the popular HRNet (OCR) [79, 99] model (60.3 v.s. 56.3) using even less parameters and computations, which is elaborately designed for segmentation task. Moreover, SegNeXt-L performs even better than EfficientNer-L.2 (NAS-FPN), which is pretrained on additional 300 million unavailable images, worth noting that EfficientNet-L.2 (NAS-FPN) has 485M parameters, while SegNeXt-L has only 48.7 38.7M parameters.
on the Pascal VOC 2012 2 test leaderboard.

---

Table 9: Comparison with state-of-the-ar methods on the ADE20K, Cityscapes and COCO-Stuff benchmarks. The number of FLOPs (G) is calculated on the input size of 512x512 for ADE20K and COCO-Stuff, and 2,048 1,024 for Cityscapes. means models pretrained on ImageNet-22K.
Params (M)
ADE20K
Cityscapes mloU (SSIMS)
COCO-Stuff
Model
GFLOPs
mloU (SS/MS)
GFLOPs
GFLOPs
mloU (SSIMS)
Segformer-BD [90]
3.8
8.4
37.4
38.0
125.5
76.2
78.1
8.4
35.6
SegNeXt-T
4.3
6.6
41.1
42.2
50.5
79.8
6.6
38.7
39.1
Segformer-Bl [90]
13.7
15.9
42.2
43.1
243.7
78.5
80.0
15.9
40.2
HRFormer-S [100]
109.5
44.0
45.1
835.7
80.0
81.0
109.5
37.9
38.9
SegNeXI-S
13.9
15.9
44.3
45.8
124.6
81.3
82.7
15.9
42.2
42.8
Segformer-B2 [90]
27.5
62.4
46.5
47.5
717.1
81.0
82.2
62.4
44.6
MaskFormer [10]
42
55
46.7
48.8
SegNeXI-B
27.6
34.9
48.5
49.9
275.7
82.6
83.8
34.9
45.8
46.3
SETR-MLA [108]
310.6
48.6
50.1
79.3
82.2
DPT-Hybrid [70]
124.0
307.9
49.0
Segformer-B3 B3 [90]
47.3
79.0
49.4
50.0
962.9
81.7
83.3
79.0
45.5
Mask2Former [9]
47
74
47.7
49.6
HRFormer-B [100]
56.2
280.0
48.7
50.0
2223.8
81.9
82.6
280.0
42.4
43.3
MaskFormer [10]
79
51.0
SegNeXt-I.
48.9
70.0
51.0
52.1
577.5
83.2
83.9
70.0
46.5
47.2
Table 10: Comparison with state-of-the-arr meth- ods on Pascal VOC dataset. means COCO [55] pretraining. denotes JFT-300M [74] pretrain- ing. utilizes additional 300M unlabeled images for pretraining.
Comparison with state-of-the-an real- time methods on Cityscapes test dataset. We test our method with single RTX-3090 GPU and AMD EPYC 7543 32-core processor CPU Without using any optimizations, SegNeXt-T can achieve 25 frames per second (FPS), which meets the requirements of real-time applications.
Table 11:
Method
Backbone
mloU
DANet [20]
ResNet101
82.6
OCRNet [99]
HRNetV2-W- W48
84.5
Method
Input size
mloU
HamNet [22]
ResNet101
85.9
ESPNet [62]
512x1,022
60.3
EneNet" [103]
ResNet101
85.9
ESPNetv2 [63]
512x1,024
66.2
EMANet" [46]
ResNet101
87.7
ICNet [105]
1,024 x 2,048
69.5
DeepLabV3++ [7]
Xception-71
87.8
DFANet [45]
1,024 1,024
71.3
DecpLabbV3+1 [7]
Xception-JFT
89.0
BiSeNet [97]
768 x 1,536
74.6
NAS-FPN [112]
EfficientNet-L2
90.5
BiSeNetv2 [96]
512 x 1,024
75.3
DF2-Seg [52]
1,024 x 2,048
74.8
SegNeXt-T
MSCAN-T
82.7
SwiftNet [67]
1,024 2,048
75.5
SegNeXt-S
MSCAN-S
85.3
SFNet [49]
1,024 2,048
77.8
SegNeXt-B
MSCAN-B
87.5
SegNeXI-L"
MSCAN-L
90.6
SegNeXI-T
768 x 1,536
78.0
Comparison with real-time methods. In addition to the sate-of-theearr performance, our method is also suitable for real-time deployments. Even without any specific software or hardware acceleration, SegNeXt-T realizes 25 frames per second (FPS) using single 3090 RTX GPU when dealing with an image size 768% 1, 1,536. As shown in Tab. 11, our method sets new state-of-the-ar results for real-time segmentation the Cityscapes test set.
4.4 Weakly-Supervised Semantic Segmentation
In this subsection, apply the proposed network the weakly-supervise semantic segmentation task. In this task, pseudo segmentation map is often generated by classification model using CAM [10]. Previous works mostly utilize VGGNet [72] or ResNets [28, 87] as the CAM generator. Here, we test the performance of the CAMs produced by our MSCAN. We use the EPS [43] architecture and follow the training strategies and recipes. The numerical results are shown in Tab. 13. We can see that simply replacing the ResNer38 backbone with our MSCAN can clearly improve the performance compared to the EPS baseline. When using our SetNeXt as the segmentation network, the performance gain increases further.

---

Table 12: Comparison on Pascal Context benchmark. The number of FLOPs is calculated with the
input size of 512x512. means ImageNet-22K pretraining. denotes ADE20K pretraining.
Method
Backbone
Params.(M)
GFLOPs
mloU (SS/MS)
DANett [20]
ResNet101
69.1
277.7
52.6
EMANet [46]
ResNet101
61.1
246.1
53.1
HamNet [22]
ResNet101
69.1
277.9
55.2
HRNet HRNet (OCR) [79]
HRNetW48
74.5
56.2
DecpLabbV3 [7]
ResNeSt-269
58.9
SETR-MLA" [108]
ViT-Large
309.5
54,9
55.8
HRFormer-B [100]
HRFormer-B
56.2
280.0
57.6
58.5
DPT-Hybrid [70]
ViT-Hybrid
124.0
60.5
SegNeXt-T
MSCAN-T
4.2
6.6
51.2
53.3
SegNeXt-S
MSCAN-S
13.9
15.9
54.2
56.1
Neg-eXt-B
MSCAN-B
27.6
34,9
57.0
59.0
NeX-L
MSCAN-L
48.8
70.0
58.7
60.3
NeXi-L
MSCAN-L
48.8
70.0
59.2
60.9
Table 13: Comparisons previous state-of-the-art weakly-supervised semantic segmentation ap- proaches on the PASCAL VOC 2012 validation set. All the segmentation results are based on the ResNet backbone [28, 87] except ours which utilize MSCAN-B.
Methods
Network
Supervision
mIoU (%) Val.
FickleNetg019 [42]
DeeplabV2
Image Saliency
64.9
OAA2019 [37]
DeeplabV1
Image Saliency
65.2
ICD2020 [18]
De
Image Saliency
67.8
[19]
DeeplabV1
Image Saliency
67.2
20S [39]
DeeplabV2
Image Saliency
71.2
Group SSup-W [53]
D2
Image Saliency
68.2
AuxSegNet2021 [92]
DeeplabV1
Image Saliency
69.0
EDAM2021 [86]
DeeplabV1
Image Saliency
70.9
EPS2021 [43]
DeeplabV2
Image Saliency
70.9
L2G [38]
DeeplabV1
Image Saliency
72.0
MSCAN + EPS [43] (Ours)
DeeplabV2
Image Saliency
71.7
MSCAN EPS [43] (Ours)
SegNeXt
Image Saliency
72.2
Conclusions and Discussion
In this paper, we analyze previous successful segmentation models and find the good characteristics owned by them. Based on the findings, we present tailored convolutional attention module MSCA and CNN-style network SegNeXt. Experimental results demonstrate that SegNeXt surpasses current state-of-the-arr caanssfomer--bacee methods by a considerable margin.
Recently, transformer- based models have dominated various segmentation leaderboards. Instead, this paper shows that CNN-based methods can still perform better than seaeformer-based methods when using a proper design. We hope this paper could encourage researchers to further investigate the potential of CNNs.
Our model also has its limitations, for example, extending this method to large-scale models with 100M+ parameters and the performance other vision or NLP tasks. These will be addressed in future works.
Acknowledgment
This work was supported by the National Key R&D Program of China (NO. 3018AAA0100400) and the Natural Science Foundation of China (No. 62220106003, No. 62176130, and No. 62276145). We would like to thank Yi Zhang and Zhengyang Geng for their kind help in experiments.
10

---

References
[1] Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence (92)), 2481-2492 (2017)
[2] Bertasius, G., Shi, J., Torresani, L.: Semantic segmentation with boundary neural fields. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3602-3610 (2016)
[3] Caesar, H., Uijlings, J., Ferrari, V: Coco-stuff: Thing and stuff classes in context. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1209-1218 (2018)
[4] Chen, L.C., Papandreou, G., Kokkinos, L., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(4), 834-848 (2017)
[5] Chen, L.C.,, Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 10t.::170600587777 (2017)
[6] Chen, L.C., Yang, Y., Wang. J., Xu, W., Yuille, A.L.: Attention to scale: Scale-aware semantic image segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3640-3649 (2016)
[7] Chen, I.C.., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: European Conference Computer Vision. pp. 801-818 (2018)
[8] Chen, I.., Zhang, H., Xiao, J., Nie, L.,, Shao, J., Liu, W., Chua, T.S.: Sca-cnn: Spatial and channel- -wiee attention convolutional networks for image captioning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5659-5667 (2017)
[9] Cheng, B., Misra, I. Schwing, A.G., Kirillov, A.. Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1290-1299 (2022)
[10] Cheng, B., Schwing, A., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems 34, 17864-17875 (2021)
[11] Contributors, M.: MMSegmentation: Openmmlab semantic segmentation toolbox and bench- mark. htttp:://githiie cwm/ comm/ope---a mmlab
(2020)
[12] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3213-3223 (2016)
[13] Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: IEEE/CVF International Conference on Computer Vision. pp. 764-773 (2017)
[14] Deng, J., Dong, W., Socher, R., Li, Li, K., Fei-Fei, L.: Imagenet: large-scale hierarchical image database. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 248-255, Ieee (2009)
[15] Ding. H., Jiang. X., Liu, A.Q., Thalmann, N.M., Wang, G.: Boundary- feature propaga- tion for scene segmentation. In: IEEE/CVF International Conference on Computer Vision. pp. 68 6819-6829 (2019)
[16] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., De- hghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Repre- sentations. (2020)
[17] Everingham, M., Van Gool, I.., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88(2), 303-338 (2010)

---

[18] Fan, J., Zhang. Z., Song, C., Tan, T.: Learning integral objects with intra-class discriminator for weakly semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4283 4292 (2020)
[19] Fan, J., Zhang, Z., Tan, T.: Employing multi-estimations for weakly-supervised semantic segmentation. In: European Conference on Computer Vision. pp. 332-348. Springer (2020)
[20] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z.,, Lu, H.: Dual attention network for scene segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3146-3154 (2019)
[21] Gao, S.H., Cheng, M.M., Zhao, K., Zhang, X.Y., Yang, M.H., Torr, P.: Res2net: A new multi- scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(2), 652-662 (2021)
[22] Geng, Z. Guo, M.H., Chen, H., Li, X., Wei, K., Lin, Z.: Is attention better than matrix decomposition? In: International Conference on Learning Representations. (2021)
[23] Guo, M.H., Cai, J.X., Liu, Z.N., Mu, TJJ,, Martin, R.R., Hu, S.M.: Pet: Point cloud transformer. Computational Visual Media 7(2), 187-199 (2021)
[24] Guo, M.H., Liu, Z.N., Mu, T.J., Hu, S.M.: Beyond self-attention: External attention using two linear layers for visual tasks. arXiv preprint arXiv:2105.02355 (2021)
[25] Guo, M.H., Lu, C.Z., Liu, Z.N., Cheng, M.M., Hu, S.M.: Visual attention network. arXiv preprint arXiv:2202.0974: (2022)
[26] Guo, M.H., Xu, T.X., Liu, JJJ., Liu, Z.N., Jiang, P.T., Mu, TJ.., Zhang, S.H., Martin, R.R., Cheng, M.M., Hu, S.M.: Attention mechanisms in computer vision: A survey. Computational Visual Media 8(3), 329-367 (2022)
[27] He, J., Deng, Z., Zhou, L., Wang. Y., Qiao, Y:: Adaptive pyramid context network for semantic segmentation. In: IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7519-7528 (2019)
[28] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 770-778 (2016)
[29] He. T., Zhang, Z., Zhang, H., Zhang, Z., Xie. J., Li, M.: Bag of tricks for image classification with convolutional neural networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 558-567 (2019)
[30] Hou, Q., Zhang, L., Cheng, M.M., Feng, J.: Strip pooling: Rethinking spatial pooling for scene parsing. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4003-4015 (2020)
[31] Hu, J., Shen, L., Sun, G.: Squeeze-and-cxcitatie networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7132-7141 (2018)
[32] Hu, S.M., Liang, D., Yang. G.Y., Yang. G.W., Zhou, W.Y.: Jittor: novel deep learning framework with meta-operators and unified graph execution. Science China Information Sciences 63(222103), 1-21 (2020)
[33] Huang, G., Liu, Z., Van Der Maaten, I.., Weinberger, K.Q.: Densely connected convolutional networks, In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4700- 4708 (2017)
[34] Huang, Z., Shi, X., Zhang, C., Wang, Q., Cheung, K.C., Qin, H., Dai, J., Li. H.: Flowformer: A transformer architecture for optical flow. arXiv preprint arXiv:2003.161 (2022)
[35] Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W.: Cenet: Criss-cross attention for semantic segmentation. IEEE/CVF International Conference on Computer Vision. pp- 603-612 (2019)
12

---

[36] loffe. S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning. pp. 448 456. PMLR (2015)
[37] Jiang, P.T., Hou, Q. Cao, Y., Cheng, M.M., Wei, Y., Xiong, H.K.: Integral object mining via online attention accumulation. In: IEEE/CVF International Conference on Computer Vision. pp. 2070-2079 (2019)
[38] Jiang, P.T., Yang, Y., Hou, Q., Wei, Y.: L2g: A simple local-to- to--lobblll knowledge transfer framework for weakly supervised semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
(2022)
[39] Kim, B., Han, S., Kim, J.: Discriminative region suppression for beakky-supervisee semantic segmentation. In: Association for the Advancement of Artificial Intelligence. pp. 1754-1761 (2021)
[40] Kirillov, A., Girshick, R., He, K., Dollár, P.: Panoptic feature pyramid networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399-6408 (2019)
[41] Kirillov, A., Wu, Y., He, K., Girshick. R.: Pointrend: Image segmentation as rendering. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9799-9808 (2020)
[42] Lee, J., Kim, E., Lee, S., Lee, J., Yoon, S.: Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5267-5276 (2019)
[43] Lee, S., Lee, M., Lee, J., Shim, H.: Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation. In: IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 5495-59555 -495-5505 (2021)
[44] Lee, Y., Kim, J., Willette, J., Hwang, S.J.: Mpvit: Multi-path vision transformer for dense prediction. In: IEEE/CVF Conference Computer Vision and Pattern Recognition. (2022)
[45] Li, H., Xiong, P., Fan, H., Sun, J.: Dfanet: Deep feature aggregation for real-time semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9522-9531 (2019)
[46] Li. X., Zhong, Z., Wu, J., Yang. Y., Lin, Z., Liu, H.: Expectation-maximizzni maximization attention networks for semantic segmentation. In: IEEE/CVF International Conference on Computer Vision. pp. 9167-9176 (2019)
[47] Li, X., He, H., Li, X., Li, D., Cheng, G., Shi, J., Weng, L., Tong, Y., Lin, Z.: Pointflow: Flowing semantics through points for aerial image segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4217-4226 (2021)
[48] Li, X., Li, X., Zhang, L., Cheng, G., Shi, J., Lin, Z., Tan, S., Tong, Y.: Improving semantic segmentation via decoupled body and edge supervision. In: European Conference Computer Vision. pp. 435 452. Springer (2020)
[49] Li, X., You, A., Zhu, Z., Zhao, H., Yang, M., Yang, K., Tan, S., Tong, Y.: Semantic flow for fast and accurate scene parsing. In: European Conference Computer Vision. pp. 775-793. Springer (2020)
[50] Li, X., Zhang, W., Pang. J., Chen, K., Cheng, G., Tong, Y., Loy, C.C.: Video k-net: A simple, strong, and unified baseline for video segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18847-18857 (2022)
[51] Li. X., Zhao, H., Han, L., Tong, Y.. Tan, S., Yang. K.: Gated fully fusion for semantic segmentation. In: Association for the Advancement of Artificial Intelligence, pp. 11418-11425 (2020)
[52] Li, X., Zhou, Y., Pan, Z., Feng, J.: Partial order pruning: for best speed/accuracy trade-off in neural architecture search. IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9145-9153 (2019)
13

---

[53] Li, X., Zhou, T., Li, J., Zhou, Y., Zhang, Z.: Group-wis semantic mining for weakly supervised semantic segmentation. In: Association for the Advancement of Artificial Intelligence. pp. 1984-1992 (2021)
[54] Lin, G., Milan, A., Shen, C., Reid, I: Refinenet: Multi-path refinement networks for high- resolution semantic segmentation. In: IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1925-1934 (2017)
[55] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference Computer Vision. pp. 740-755. Springer (2014)
[56] Liu, R., Deng, H., Huang, Y., Shi. X., Lu, L.. Sun, W., Wang, X., Dai, J., Li, H.: Decoupled spatial-temporal transformer for video inpainting. arXiv preprint arXiv:2104.06637 (2021)
[57] Liu, R., Deng, H., Huang, Y., Shi, X., Lu, L., Sun, W., Wang, X., Dai, J., Li, H.: Fuse- former: Fusing fine fine-grained information in transformers for video inpainting. In: IEEE/CVF International Conference on Computer Vision. pp. 14040-14044 (2021)
[58] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: IEEE/CVF International Conference Computer Vision. pp. 10012-10022 (2021)
[59] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C.,, Darrell, T., Xie, S.: A convnet for the 2020s. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-1 11986 (2022)
[60] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1441-3440 (2015)
[61] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019)
[62] Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., Hajishirzi, H.: Espnet: Efficient spatial
pyramid of dilated convolutions for semantic segmentation. In: European Conference on Computer Vision. pp. 552-568 (2018)
[63] Mchta, S., Rastegari, M., Shapiro, L., Hajishirzi, H.: Espnetv2: A light- power efficient, and general purpose convolutional neural network. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9190-9200 (2019)
[64] Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in Neural Information Processing Systems. pp. 2204-2213 (2014)
[65] Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., Yuille, A.: The role of context for object detection and semantic segmentation in the wild. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 891-898 (2014)
[66] Mou, L.,, Hua, Y., Zhu, X.X.: A relation-augmented fully convolutional network for semantic segmentation in acrial scenes. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14116-14425 (2019)
[67] Orsic, M., Kreso, I., Bevandic, P., Segvic, S.: In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12607-12616 (2019)
[68] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, I.., al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)
[69] Peng, C., Zhang, X., Yu, G., Luo, G., Sun, J.: Large kernel matters-improve semantic segmentation by global convolutional network. IEFE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4353-4361 (2017)
14

---

[70] Ranftl, R., Bochkovskiy, A., Koltun, V. Vision transformers for dense prediction. In: IEEE/CVF International Conference on Computer Vision. pp. 12179-12188 (2021)
[71] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image
segmentation. In: International Conference Medical Image Computing and Computer- Assisted Intervention. pp. 231-241. Springer (2015)
[72] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recog- nition. arXiv preprint arXiv: 100:: 1556 (2014)
[73] Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic seg- mentation. In: IEEE/CVF International Conference on Computer Vision. pp. 7262-7272 (2021)
[74] Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: IEEE/CVF International Conference on Computer Vision, pp. 843-852 (2017)
[75] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-9 (2015)
[76] Takikawa, T., Acuna, D., Jampani, V., Fidler, S.: Gated-senn: Gated shape cnns for semantic segmentation. In: IEEE/CVT International Conference Computer Vision, pp. 5229-5238 (2019)
[77] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105-61 61 14. PMLR (2019)
[78] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data- efficient image transformers distillation through attention. In: International Conference on Machine Learning. pp. 10347-10357. PMLR (2021)
[79] Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al.: Deep high-resolution representation learning for visual recognition. IEEE Transactions Pattern Analysis and Machine Intelligence (2020)
[80] Wang. Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: Eca-net: Efficient channel attention for deep convolutional neural networks, In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11534-11542 (2020)
[81] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyra- mid vision transformer: A versatile backbone for dense prediction without convolutions. In: IEEE/CVF International Conference on Computer Vision. pp. 568-578 (2021)
[82] Wang, W., Xie, R., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T. Luo, P., Shao, L.: Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media 8(3), 415-424 (2022)
[83] Wang, X., Girshick, R., Gupta, A., Hc, K.: Non-local neural networks. In: IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7794-7803 (2018)
[84] Waqas Zamir, S., Arora, A., Gupta, A., Khan, S., Sun, G., Shahbaz Khan, F,, Zhu, F., Shao, L.. Xia, G.S., Bai, X.: isaid: A large-scale dataset for instance segmentation in acrial images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop. pp. 28-37 (2019)
[85] Wightman, R.: Pytorch image models.
httpss//ggithub.ccm///gggg
pytorch-image maraage-mddelsAApaaee
(2019)
[86] WI, T. Huang, J., Gao, G., Wei, X., Wei, X., Luo, X., Liu, C.H.: Embedded discriminative attention mechanism for weakly supervised semantic segmentation. IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16765-16774 (2021)
15

---

[87] Wu, Z., Shen, C., Van Den Hengel, A.: Wider or deeper: Revisiting the resnet model for visual recognition. Pattern Recognition. 90, 119-133 (2019)
[88] Xia, F., Wang, Chen, L.C., Yuille, A.L.: Zoom better to see clearer: Human and object parsing with hierarchical auto-zoom net. In: European Conference on Computer Vision. pp. 648 663. Springer (2016)
[89] Xiao, T., Liu, Y., Zhou, B., Jiang, Y., Sun, Unified perceptual parsing for scene understand- ing. In: European Conference on Computer Vision. pp. 4 418-434 (2018)
[90] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34 (2021)
[91] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1492-1500 (2017)
[92] Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., Sohel, F., Xu, D.: Leveraging auxiliary tasks with affinity learning for weakly supervised semantic segmentation. In: IEEE/CVF (2021)
International Conference on Computer Vision. pp.
[93] Xu, Y., Wei, H., Lin, M., Deng, Y., Sheng, K., Zhang, M., Tang. F., Dong, W., Huang. F., Xu, C.: Transformers in computational visual media: A survey. Computational Visual Media 8(1), 33-62 (2022)
[94] Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L.,, Gao, J.: Focal attention for long-range interactions in vision transformers. Advances in Neural Information Processing Systems 34, 30008-30022 (2021)
[95] Yang, M., Yu, K., Zhang, C., Li, Z., Yang, K.: Denseaspp for semantic segmentation in street scenes. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3684-3692 (2018)
[96] Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., Sang, N.: Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. International Journal of Computer Vision 11). 3051-3066 (2021)
[97] Yu, C.. Wang. J., Peng, C.. Gao, C., Yu. G., Sang, N.: Bisenet: Bilateral segmentation network for real-time semantic segmentation. In: European Conference on Computer Vision. pp. 325-341 (2018)
[98] Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: International Conference on Learning Representation (2016)
[99] Yuan, Y., Chen, X., Wang, J.: Object-contextual representations for semantic segmentation. In: European Conference on Computer Vision. pp. 173-190. Springer (2020)
[100] Yuan, Y., Fu, R., Huang, L.,, Lin, W., Zhang, C., Chen, X., Wang. J.: Hrformer: High-resolution vision transformer for dense predict. Advances in Neural Information Processing Systems 34, 7281-7293 (2021)
[101] Yuan, Y., Huang, L., Guo, J.. Zhang, C.,, Chen, X., Wang. J.: Ocnet: Object context network for scene parsing. arXiv preprint arXiv: 809.00916 (2018)
[102] Yuan, Y., Xie, J., Chen, X., Wang, J.: Segfix: Model-agnostic boundary refinement for segmentation. In: European Conference Computer Vision. pp. 489-506. Springer (2020)
[103] Zhang, H.. Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., Agrawal, A.: Context encoding for semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7151-7160 (2018)
[104] Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J., Manmatha, R., et al.: Resnest: Split-attention networks. arXiv preprint arXiv:2004.0890 0.855 (2020)
16

---

[105] Zhao, H., Qi, X., Shen, X., Shi, J., Jia, J.: Icnet for real-time semantic segmentation on high-resolution images. In: European Conference on Computer Vision. pp. 405-420 (2018)
[106] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2881-2890 (2017)
[107] Zhen, M., Wang, J., Zhou, L.,, Li, S., Shen, T., Shang, J., Fang, T., Quan, L.: Joint semantic seg- mentation and boundary detection using iterative pyramid contexts. In: IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13666-13671 13675 (2020)
[108] Zheng. S.. Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang. Y., Fu, Y., Feng, Xiang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a transformers. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6881-6890 (2021)
perspective with
[109] Zheng, Z., Zhong, Y., Wang, J., Ma, A.: Foreground-aware relation network for geospatial ob- ject segmentation in high spatial resolution remote sensing imagery. IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 4096-4105 (2020)
[110] Zhou, B., Khosla, A.. Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discrim- inative localization. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2921-2929 (2016)
[111] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: IEER/CVF Conference on Computer Vision and Pattern Recognition. pp. 633 641 (2017)
[112] Zoph, B., Ghiasi, G., Lin, T.Y., Cui, Y., Liu, H., Cubuk, E.D., Le, Q.: Rethinking pre-training and self-training. Advances in Neural Information Processing Systems 33, 3833-3845 (2020)
17


Reviews:
[{"review_id": "nHFfaH-IhX6", "reviewer": "Reviewer_rY1T", "summary": "This paper proposes a novel convolution-based architecture that could utilize convolutional attention in an effective way to encode contextual information for the segmentation task. The main contributions of this paper are (1) a new tailored network architecture (SegNeXt) that envokes spatial attention via multi-scale convolution features, (2) the illustration that the encoder with simple and cheap convolutions can exhibit improved performance than the vision transformer, and (3) the proposed network exhibit the state-of-the-art semantic segmentation performance.", "questions": "* Questions & Discussions\n1. The novelty and the contribution of this paper should be more discussed. In recent years, the new methods related to vision transformer and ConvNext are popularly studied, and they showed extremely improved accuracy in terms of classification and segmentation. At the first gland, this paper just addresses the application of the ConvNext to the segmentation task. At this point, the reviewer is curious that the combination of the ConvNext to the segmentation task could be a novel contribution to this Neurips society. \n\n2. Effectiveness of the proposed operation. The authors addressed in the abstract that “convolutional attention is a more efficient and effective way to encode contextual information than the self-attention mechanism in transformers”. However, the manuscript could not effectively exhibit this property which is the main contribution of this paper. The mathematical proof/illustration or experimental analysis for contextual information should be discussed to improve the quality of the manuscript.\n", "limitations": "1. Major issues\n- The limited novelty should be discussed. As illustrated above (Question), the proposed operation exhibits limited novelty. The reviewer expects that the limitations could be improved in the discussion periods.\n\n- The experiments should be improved. As the authors commented in the contribution 2, the proposed module can perform better than vision transformers, “especially when processing object details”. As the reviewer already understand, the evaluation metric of “mean Intersection over Union (mIoU)” can qualitatively measure the object details. However, recent studies [1-3] proposed new evaluation metrics to measure the object detail (especially boundaries of the target objects) quantitatively. Otherwise, the visualization of the feature map could illustrate the novel feature extraction when processing object details. Please refer the activation maps [4]. To clear the authors’ addresses, more experimental or mathematical evidence should be justified.\n\n\n[1] Fernandez-Moral, Eduardo, et al. \"A new metric for evaluating semantic segmentation: leveraging global and contour accuracy.\" 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.\n\n[2] Lee, Kyungsu, et al. \"Boundary-oriented binary building segmentation model with two scheme learning for aerial images.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.\n\n[3] Cheng, Bowen, et al. \"Boundary IoU: Improving object-centric image segmentation evaluation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[4] Selvaraju, Ramprasaath R., et al. \"Grad-cam: Visual explanations from deep networks via gradient-based localization.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n\n2. Minor issues\n- Definition of the contextual information\n\n- The detailed description should be improved. For instance, what does the arrow indicate in figure 3? The reviewer recommends authors to review the figures to improve the explanation details for all readers.\n\n- The reviewer recommends reviewing the grammar and typo errors to improve the quality of the manuscript.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n1. Strengths\n- The reviewer significantly understands the significance and importance of the task proposed in this paper. Segmenting various objects in real-world images is significantly important for many applications. Recent studies have researched the Vision-Transformer to improve the segmentation performance in vision fields, but it exhibits many limitations such as requirements a large number of images and datasets, and even heavy cost. However, this paper address that still the convolution and attention-based operations could exhibit significantly improved performance at a low cost. \n\n- Reproducibility of the manuscript. The manuscript is well organized in terms of exhibiting the hyper-parameters and model architecture. \n\n- The manuscript is well organized and well written.\n\n2. Weakness\n- More detailed descriptions are illustrated in the “Question” and “Limitation” sections. Please see below.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "__Q5h7420zi", "reviewer": "Reviewer_S4w5", "summary": "\nThis paper proposes a simple convolution network SegNeXt for semantic segmentation which include three important properties : strong encoder, multiscale interaction and spatial attention. It shows convolutional attention is a more efficient and effective way to encode contextual information than the self-attention mechanism in transformers. SegNeXt improves the performance of previous state-of-the-art methods on popular benchmarks, including ADE20K, Cityscapes, COCO-Stuff, Pascal VOC, Pascal Context, and iSAID.\n", "questions": "See the weakness part. ", "limitations": "None ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrength:\n1, The paper is well written and easy to follow for both method and experiment part. The summary of Table.1 is interesting. \n\n2. The proposed approach achieve new state-of-the-art results on iSAID, ADE-20k, COCO-stuff and Pascal Context dataset compared with recent transformer based approach. The result in Figure.1 looks good. \n\n3, The proposed approach: convolutional attention is simple yet effective. MSCA is an improved version of large kernel design for segmentation. \n\n4, The results on Pascal VOC dataset rank the first. \n\nWeakness:\n\n1, The technical novelty is very limited. For example, despite the MSCA is simple, it follows the large kernel design which has been proposed in previous works. [1] [2]\n\nMoreover. The decoder adopts multi-scale feature fusion via Hamburger which is a previous work. \n\nThus the technical novelty makes the submission looks like a report. \n\n2, Could not find new insights. The ablation studies are not good which lacks design analysis of component. A better comparison with HRFormer or HRnet is needed to show the effectiveness of spatial attention.\n\n\n3, In Tab10. Why not list the FPS of other method for better comparison. \n\n\n4, Missing ablation on MSCA design on both ImageNet or other segmentation datasets.\n\n\n\n[1]  Large kernel matters–improve semantic segmentation by global convolutional network. CVPR-2017. \n\n[2] Convnext -CVPR-2022\n\n[3] Is attention better than matrix decomposition? ICLR-2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "TxMqvoogAjt", "reviewer": "Reviewer_r2Bq", "summary": "This paper presents SegNeXt, a simple convolutional network architecture for semantic segmentation. The main contribution of this paper is replacing the standard convolutions and self-attention with the spatial attention proposed in this paper. Experimental results demonstrate that SegNeXt surpasses current state-of-the-art transformer-based methods by a considerable margin.", "questions": "Please see the weaknesses.", "limitations": "This work has described some limitations and potential negative social impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n\n1) This paper is well written, and I can easily understand the overall flowchart of this paper.\n\n2) The experimental results of this paper are good.\n\n**Weaknesses**: \n\nAlthough the idea of this work seems interesting, the good performance is not sufficiently documented.  \n\nFirstly, the motivation of this paper is not clearly explained. In Line.112 to Line.137, the authors directly tell us that the pipeline proposed MSCA, while they do not tell us why to design such architecture.  This makes me confused that would multi-scale convolutional attention be really better than the self-attention mechanism?\n\nSecondly, the authors do not provide any visualization results to show the shows the superiority of MSCA compared with self-attention.\n\nIn conclusion, I think the main drawback of this paper is the authors do not tell us that what defects of self-attention are solved by MSCA?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns", "I don’t know"]}, "extra_text": {}}, {"review_id": "8Aqlk1BEqCk", "reviewer": "Reviewer_mij9", "summary": "This paper presents SegNeXt, a new convolution-based model for semantic segmentation. The authors first propose 4 properties that constitute a successful semantic segmentation model: 1) A strong encoder backbone, 2) Multi-scale information interaction, 3) Spatial attention, and 4) low computational complexity. Based on these requirements, the authors propose a convolutional encoder based on a new multi-scale convolutional attention module (MSCA). This module uses multi-branch convolutions of different sizes to generate an attention mask, which is then used to weight the input (via element-wise multiplication) to generate the output of the module.\n\nAs a decoder, the authors propose to concatenate the convolutional features from the last input stages and feed them as input to a Hamburger module.\n\nThe authors evaluate their proposed architecture on several datasets like ADE20K, Cityscapes, Pascal VOC, Pascal context, COCO-Stuff and iSAID. The results of the experiments show that SegNeXt matches or outperforms other models with similar number of parameters while requiring less computation.", "questions": "1) Why does the model need to use spatial attention? What is the property of spatial attention that makes it a key component for semantic segmentation models? (Be it self-attention or computed attention masks like in this paper)\n2) In Section 1, Table 1, the authors mention that “Strong encoder denotes strong backbones, and adopts the advanced training strategy”. What is the “advanced training strategy”? Does this paper use it?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n+ The paper is well written and the new concepts and ideas are explained well and with enough detail. The paper is well structured and the content is presented in a clear and logical way.\n\n+ The paper proposes an extensive evaluation to show the performance of the proposed model and how it compares against previous state-of-the art approaches. Additionally, the authors perform some ablation experiments to backup their choices of multi-scale in the MSCA module, Hamburger as a decoder and the decoder structure.\n\n+ The proposed model outperforms previous state of the art approaches while also requiring less computation.\n\n## Weaknesses\n- A weakness of this paper is that it doesn’t justify enough the four properties argued to be necessary for a successful semantic segmentation model. The authors mention that by looking at previous work (DeepLabV3+, HRNet, SETR and SegFormer) they are able to conclude that a good model needs these four properties (strong backbone, multi-scale, attention and low complexity), but that might just indicate recent trends, not necessarily important properties. For example, a stronger justification to claim that the model needs to use spatial attention would be to design an experiment (or a citation to another paper) that shows why this property is needed, instead of assuming that because recent papers used attention, this property becomes necessary.\n\n- The authors explicitly mention that \n> “taking the aforementioned **analysis** into account, … we propose an efficient yet effective encoder-decoder architecture of semantic segmentation”. \n  \n  However, as I mentioned before, a more detailed analysis of the different properties of recent models would make the claims more significant.\n\n- Just so the authors are aware, there is already another model called SegNext [1].\n\nTo summarize, the paper proposes a novel architecture for semantic segmentation which outperforms previous approaches. The authors perform a thorough evalution to empirically show the superior performance. However, some of the justifications are a bit weak or not supported enough with data.\n\n[1] T. Forbes and C. Poullis, \"Deep Autoencoders with Aggregated Residual Transformations for Urban Reconstruction from Remote Sensing Data,\" 2018 15th Conference on Computer and Robot Vision (CRV), 2018, pp. 23-30, doi: 10.1109/CRV.2018.00014.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept