SegViT: Semantic Segmentation with Plain Vision Transformers
Authors: Bowen Zhang
Zhi Tian
Quan Tang
Xiangxiang Chu
Xiaolin Wei
Chunhua Shen
Yifan liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Semantic segmentation
Transformer
Efficient

Abstract: We explore the capability of plain Vision Transformers (ViTs) for semantic segmentation and propose the SegViT. Previous ViT-based segmentation networks usually learn a pixel-level representation from the output of the ViT. Differently, we make use of the fundamental component—attention mechanism, to generate masks for semantic segmentation. Specifically, we propose the Attention-to-Mask (ATM) module, in which the similarity maps between a set of learnable class tokens and the spatial feature maps are transferred to the segmentation masks. Experiments show that our proposed SegViT using the ATM module outperforms its counterparts using the plain ViT backbone on the ADE20K dataset and achieves new state-of-the-art performance on COCO-Stuff-10K and PASCAL-Context datasets. Furthermore, to reduce the computational cost of the ViT backbone, we propose query-based down-sampling (QD) and query-based up-sampling (QU) to build a Shrunk structure. With our Shrunk structure, the model can save up to 40% computations while maintaining competitive performance.

Seg ViT: Semantic Segmentation with Plain Vision Transformers
Bowen Zhang", Zhi Tian"; Quan Tang", Xiangxiang Chu?, Xiaolin Wei?, Chunhua Shen
Yifan Liu'
1 The University of Adelaide, Australia Meituan Inc.
Zhejiang University, China
South China University of Technology, China
Abstract
We explore the capability of plain Vision Transformers (VīTs) for semantic seg- mentation and propose the SegViT. Previous ViT- segmentation networks usually learn pixel-level representation from the output of the VIT. Differently, we ceapoonent-attntiinn mechanism, to generate masks
make use of the fundamental for semantic segmentation. Specifically, we propose the Attention-to-Mask (ATM) module, in which the similarity maps between set of learnable class tokens and the spatial feature maps are transferred to the segmentation masks. Experiments show that our proposed Seg ViT using the ATM module outperforms its counter- parts using the plain ViT backbone on the ADE20K dataset and achieves new state-of-the-art performance COCO-Stuff-10K and PASCAL-Contex datasets. Furthermore, to reduce the computational cost of the ViT backbone, we propose query-based down-sampling (QD) and query-based up-sampling (QU) to build a Shrunk structure. With the proposed Shrunk structure, the model can save up to 40% computations while maintaining competitive performance.
Introduction
Semantic segmentation is dense prediction task in computer vision that requires pixel-level clas- sification of an input image. Fully Convolutional Networks (FCN) [1] are widely used in recent state-of-the-art methods. This paradigm includes deep convolutional neural network the en- coder/backbone and a segmentation-orientn
decoder to provide dense predictions. A I x convolu- tional layer is usually applied representative feature map to obtain the pixel level predictions. To achieve higher performance, previous works [2-4] focus on enriching the context information or fusing multi-scale information. However, the correlations among spatial locations are hard model explicitly in FCNs due to the limited receptive field.
Recently, Vision Transformers (ViT) [5], which make of the spatial attention mechanism introduced to the field of computer vision. Unlike typical convolution- based backbones, the ViT has plain and non-hierarchical architecture that keeps the resolution of the feature maps all the way through. The lack of the down-sampling process (excluding tokenizing the image) brings differences the architecture to do the semantic segmentation task using ViT backbone. Various semantic segmentation methods [6-8] based on ViT backbones have achieved promising performance due to the powerful representation learned from the pre-trained backbones. However, the potential of the attention mechanism is not fully explored.
Different from previous per-pixel classification paradigm [6-8], consider learning meaningful class token and then finding local patches with higher similarity to it. To achieve this goal, we propose
"The first two authors contributed equally. CS is the corresponding author, e-mail: chunhua@me.com com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the Attention-to to-ioosktt-MMkk (ATM) module. More specifically, we employ a transformer block that takes the learnable class tokens as queries and transfers the spatial feature maps as keys and values. A dot-product operator calculates the similarity maps between queries and keys. We encourage regions belonging to the same category to generate larger similarity values for the corresponding category (i.e. a specific class token). Fig. visualizes the similarity maps between the features and the "Table' and "Chair' tokens. By simply applying a Sigmoid operation, we can transfer the similarity maps to the masks. Meanwhile, following the design of typical transformer block, a Sof operation is also applied to the similarity maps to get the cross attention maps. The 'Table' and 'Chair' tokens are then updated as in any regular transformer decoders, by weighted sum of the values with the cross attention maps as the weights. Since the mask is a byproduct of the regular attentive calculations, negligible computation is involved during the operation.
Building upon this efficient ATM module, we propose a new semantic segmentation paradigm with the plain ViT structure, dubbed SegViTl VÎT. In the paradigm, several AT' modules are employed on different layers, and we get the final segmentation mask by adding the outputs from different layers together. Seg ViT outperforms its ViT-based counterparts with less computational cost. However, compared with previous encoder-decoder structures that use hierarchical networks as encoders, ViT backbones as encoders are generally heavier. To further reduce the computational cost, we employ Shrunk structure consisting of query-based down-samplin The QD can be inserted into the ViT backbone to reduce the resolution by half and QU is used parallel to the backbone to recover the resolution. The Shrunk structure together with the ATM module as the decoder can reduce up to 40% computations while maintaining competitive performance.
(QD) and query-based up- sampliinn (QU).
We summarize our main contributions as follows:
We propose an Attention-to-Matt (ATM) decoder module that is effective and efficient for semantic segmentation. For the first time, we utilize the spatial information in attention maps to generate mask predictions for each category, which can work as a new paradigm for semantic segmentation.
We managed to apply our ATM decoder module to the plain, non- min-hierrrchicll ViT backbones in a cascade manner and designed a structure namely Seg ViT that achieves mloU 55.2% on the competitive ADE20K dataset which is the best and lightest among methods that use ViT backbones. We also benchmark our method on the PASCAL-CCntext dataset (65.3% mloU) and Stuo--Stuf--OK dataset (50.3% mIoU) and achieve new state-of-the-a performance.
We further explore the architecture of VIT backbones and work out a Shrunk structure to apply to the backbone reduce the overall computational cost while still maintaining competitive performance. This alleviates the disadvantage of ViT backbones that are usually more computationally intensive compared their hierarchical counterparts. Our Shrunk version of SiT ViT on the ADE20K dataset reaches mloU 55.1% with the computational cost of 373.5 GFLOPs which is about 40% off compared the original SegViT (637.9 GFLOPs).
2 Related Work
Semantic segmentation. Semantic segmentation which requires pixel-level classification on an input image is fundamental task in computer vision. Fully Convolutional Networks (FCN) used to the dominant approach to this task. Initial per-pixel approaches such as [9, 10) attribute the class label to each pixel based on the per-pixel probability. To enlarge the receptive field, several approaches [11, 12] have proposed dilated convolutions or apply spatial pyramid pooling to capture contextual information of multiple scales. With the introduction of attention mechanisms, [13, 14, 6] replace the feature merge conducted by convolutions and pooling with attention to better capture long-range dependencies.
Recent works [15, 8, 16] decouple the per-pixel classification process. They reconstruct the structure by using a fixed number of learnable tokens and use them as weights for the transformation to apply on feature maps. Binary matching rather than cross-entropy used to allow overlaps between feature maps and learnable tokens are used to dynamically generate classification probabilities. This paradigm enables the classification process to be conducted globally and alleviates the burden for the decoder to do per-pixel classification, which as result, is more precise and the performance is

---

Table
Chair
Similanty
Atlention
Mask
GT/Emage
Figure I: The overall concept of our Attention-to-Me decoder. In a typical attentive process, the dot-product first calculated between queries and keys to measure the similarity (as illustrated on the left). If the similarity map is applied with tmat operation the spatial dimension, the output the typical attention map (multiple heads are summed together). However, if the same similarity map is applied with per-pixel operation Sigmoid, it produces mask that indicates the area with certain similarity. Based on the assumption that the tokens within the same category have higher train token vector have high similarity within tokens of the specific category and low similarity elsewhere. In the meantime, this process does not violate the attention mechanism. Thus, it can process alongside the original transformer layers.
similarity,
generally better. However, for those methods, the feature map is still calculated in a static manner, usually requiring feature merge modules such FPN [4].
Transformers for vision. Attention- based transformer backbones have become powerful alterna- tives standard convolution based networks for image classification tasks. The original VIT [5] is a plain, non-hierarchical architecture. Various hierarchical transformers such as [17-21] I] have been presented afterwards. These methods inherit some designs from convolution based networks such as hierarchical structures, pooling and down-sampling with convolutions. As a result, they can be used straightforward replacement for convolutional based networks and applied with previous decoder heads for tasks such as semantic segmentation.
Plain-backbone decoders. High-resolutee feature maps generated by backbones are important for dense prediction tasks such as semantic segmentation. Typical hierarchical transformers use feature merge techniques such as FPN [4] or dilated backbones to generate high-resolution feature maps. However, for plain, non-hierarchical transformer backbones, the resolution remains the same for all layers. SETR [6] proposed simple strategy to treat transformer outputs sequence-to sgquence-to-sequence perspective to solve segmentation tasks. Segmenter [8] joints random initialized class embeddings and the transformer patch embeddings together and applics several relf-attention layers to the joint token sequence obtain updated class embeddings and patch embeddings semantic prediction. In our study, we consider learning class token and then finding local patches with higher similarities with the help of the attention map, making the inference process more direct and efficient.
3 Method
3.1 Encoder
Given an input image / E Ix plain vision transformer backbone reshapes it into a sequence of tokens F, L2 where L HWW/WW P is the patch size and C' is the number channels. Learnable position embeddings of the same size of Fo are added to capture the positional information. Then, the token sequence Fo is applied with m transformer layers get the output. We define the output tokens for each layer as [F.. Fmn E RT.x Typically, transformer layer consists of a multi-head self-attention block followed by point-wise multilayer perceptron block with layer norm in between and then residual connection is added afterward. The transformer layers are stacked repetitively several times. For a plain vision transformer like ViT, there are no other modules involved and for each layer, the number of the tokens is not changed.

---

bib
siik
TOllS
Self
(LxCC
Add Norm
MatMal
ATM
Mask
Stemodd
VIT
Attention
SoltMax
(Lx((
ATM
MutMal
Ardd & Norm
PC
Food Forward
Class predictions
ATM
Add Nori
Figure 2: The overall ViT structure with the ATM module. The Attention-to-Man (ATM) module inherits the typical transformer decoder structure. It takes in randomly initialized class embeddings as queries and the feature maps from the ViT backbone to generate keys and values. The outputs of the ATM module are used as the input queries for the next layer. The ATM module carried out sequentially with inputs from different layers of the backbone as keys and values in cascade manner. A linear transform is then applied to the output of the ATM module to produce the class predictions for each token. The mask for the corresponding class is transferred from the similarities between queries and keys in the ATM module.
Decoder
Mask-to-Attentinn (ATM). Cross attention can be described as the mapping between two se- quences of tokens. We define two token sequences as E RR with the length equals to applied each of them to form
number of classes and F, E R L..C. First, linear transformations
query (Q), key (K) and values (V), as presented by Eq. (1).
Q = (9) E NxC = on (F.) E R V V
E LXC
(1)
The similarity map is calculated between the query and the key. Following the scaled dot-product attention mechanism, the similarity map and attention map are calculated by:
S(Q,K) QKT
RNL
vd
Attention(G.FF) = (ocftmax(((Q,K))) e NXC.
where v dds a scaling factor with de equals to the dimension of the keys. The shape of the similarity map S(Q, K) is determined by the length of the two token sequences N and L. The attention mechanism is then to update G by a weighted sum of V, where the weight assigned to the summation the similarity map applied with Softmax. along the dimension
Dot-product attention uses the Sof tmax function to exclusively concentrate the attention on the token that has the most similarity. However, we suppose that the tokens other than ones that yield maximum similarities are also meaningful. Based on this intuition, we design a lightweight module directly. To be more specific, we assign G the class
that generates semantic predictions embeddings for the segmentation task and F; as the output of layer of the ViT backbone. We pair a semantic mask each token in G to represent the semantic prediction for each class. The calculation for the mask is:
Mask(g, Fi) Sigmoid K)) NXI.
The shape of the masks is N x L, which can be further reshaped to N H/PxW/P. The structure of the ATM mechanism illustrated in the right part in Fig. Masks are the middle output of the cross attention. The final output tokens from the ATM module are used for classification. We apply a linear transformation followed by tmat activation to the output class tokens get class probability predictions. Note that follow [15] to add 'no object' category (Ø) in case the image

---

errbslting
erbhediing
Cliss eerleddings
P
P
QU
QU
(2 Single
(b) Natvz sbrunk
(4) Skraed
Figure 3: The structure comparison between ViT with a single layer and the Shrunk version. (a) illustrates the Seg ViT structure with ATM module used once with the last layer of the ViT backhone as the input to generate predictions. (b) uses the query-based down-sampling (QD) module to implement naive way to shrink the resolution of the features of the backbone from 1/10 to 1/32 and thus reduces the overall computational cost. (c) is the proposed (shrunk) version which applies the additional query- query-based up-sampling module. The Shrunk version can save up to 40% of computational cost when using the ViT-Large backbone without much sacrifice to the performance.
doesn't 't contain certain classes. During inference, the output is produced by the dot-produe between the class probability and the mask groups.
Plain backbones such as ViT does not have multiple stages with features of different scale. Thus, structures such as FPN to merge features with multiple scales are not applicable. However, fea- tures other than the last layer contain rich low-level semantic information and are beneficial to the performance. We designed structure that can make use of the feature maps from different layers of ViT to compact with our ATM decoder namely Seg ViT. In this study, we also found a way to compact the computational cost for the ViT backbone without sacrificing performance. This proposed Shrunk version of SegViT ViT uses query-based wwmn-sampling (QD) module together with a query-basce up-sampling (QU) module to compress the VIT backbone and bring overall reduction to the computational cost.
The SigViT structure. As illustrated in Fig. 2, an ATM decoder takes in N tokens as the class embeddings and another sequence of tokens as the base to calculate keys and values for the ATM module to generate masks. The output of the ATM is N updated tokens and N masks corresponding to each class token. We use random initialized learnable tokens as the class embeddings and the output of the last layer of the ViT backbone as the base first. To make use of multi-layer information, the output of the first ATM decoder is then used as the class embeddings for the next ATM decoder with the output of another layer of the ViT backbone as the base. This process is repeated another time so that we can get three groups of tokens and masks. Formally, the loss function of each layer can be formulated as, Loverall = Lott + Lmmae = Lole + A LIUU + Addiice Lice
(4)
In each group, the output tokens are supervised by the classification loss (Lrt) which is mentioned above and the masks are summed orderly and supervised by the mask loss (Lmask) which is linear A and Xdice
combination of a focal loss [22] and a dice loss [23] multiplied by respectively in DETR [24]. The loss of all three groups are then summed together. We have further experiments to show that this design is beneficial and efficient.
The Shrunk structure. Plain transformer backbones such as VIT is known to have larger com- putational cost than their counterparts with similar performance. We propose a Shrunk structure using query-based down-sampling (QD) and up-sampling (QU). Since the shape of the output of the attention module is determined by the shape of the query, we can apply down-sampling before the query transformation to realize the QD or insert new query tokens during the cross attention to realize the QU. By changing the resolution with the number of query tokens, the spatial size changed according to the cross attention, providing more flexibility to preserve (recover) important regions. To be more specific, in the QD layer, we use the nearest sampling to reduce the number of the query tokens while keep the size of the key and value tokens. When passing through a transformer layer, the values are weighted and summed by the attention map between query tokens and the key tokens. This is non-linear downsampling that will pay more attention to the important regions. In the QU
5

---

layer, we employ a transformer decoder structure [25] and initialize new learnable tokens as queries based on the desired output resolution.
As shown in Fig. 3, we design the Seg ViT structure with one single layer as the baseline (a). We first try naive approach (b), which is apply the QD once the 1/a depth of the backbone (e.g., the 8th layer of a backbone with 24 layers) to down-sample the resolution of the layer output from 1/16 to 1/32 as reduce the overall computational cost. The performance drops expected since the QD process involves information lose.
To compensate for the information loss in the naive 'shrunk' version, we further apply two QU layers in parallel with the backbone. This is our proposed Shrunk version (c). The first QU layer takes in features with 1/16 resolution from the low level of the backbone. Its output is then used as the query to make cross attention with the down-sampled features with 1/a2 resolution from the last layer of the backbone. The shape of the output of this QU structure is of 1/16 resolution.
Directly reducing the number of the query tokens inevitably harms the final performance. However, with our designed QU layer and the ATM module, the Shrunk structure is able to reduce 40% of overall computational cost while still being competitive in performance.
4 Experiments
4.1 Datasets
ADE20K [26] is a challenging scene parsing dataset which contains 20, 210 images as the training set and 2, 000 images as the validation set with 150 semantic classes.
COCO-Stuff-10K [27] is scene parsing benchmark with 9, 000 training images and 1, 1000 test images. Even though the dataset contains 182 categories, not all categories exist in the test split. We follow the implementation of mmsegmentation [28] with 171 categories to conduct the experiments.
PASCAL-Context [29] is a dataset with 4, 996 images in training set and 5, 104 images in the validation set. There are 60 semantic classes in total, including a class representing 'background'.
4.2 Implementation details
Transformer backbone. We use the naive ViT [5] as the backhone. In particular, we use its 'Base' variation for most ablation studies and provide results the 'Large' variation. Since there can be huge difference with different pre-trained weights, as suggested by Segmenter [8], we use the weights provided by Augreg [30] following the counterparts [8, 31] for fair comparison. The weights are obtained by training on ImageNet-21k with strong data augmentation and regularization. For a simple reference, we report that for pre-trained weights provided by ViT [5] and Augreg [30], the mIoU scores using the same training recipe on ADE20K dataset are 51.7% and 54.6%, respectively. Training settings. We use MMSegmentation [28] and follow the commonly used training settings. During training, we applied data augmentation sequentially via random horizontal flipping, random resize with the ration between 0.5 and 2.0 and random cropping (512 x 512 for all except that we use 480 x 480 for PASCAL-CCntee and 640 x 640 for VaT-large on ADE20K). The batch size 16 for all datasets with a total iteration of 160k, 80k and 80k: for ADE20K, COCO-Stuff-100 10k and PASCAL-Conted (mloU) as the metric to evaluate the performance. 'ss' means single-scale testing and 'ms' test time augmentation with multi-scaled (0.5, 0.75, 1.0, 1.25, 1.5, 1.75) inputs. All reported mloU scores are in percentage format. All reported computational costs in GFLOPs are measured using the fvcore library.
respectively. Evaluation metric. We use the mean Intersection over Union
4.3 Comparisons with the State-of-the-art Methods
Results on ADE2OK. Table 1 reports the comparison with the state-of-the-art methods on ADE20K validation set using ViT backbone. The ViT uses the ATM module with multi-layer inputs from the original ViT backbone, while the Shrunk is the one that conducts QD to the ViT backbone and saves 40% of the computational cost without sacrificing too much performance. Our method achieves

---

Table 1: Experiment results on the ADE20K val. split. 'ms' means that mloU calculated using sulti-scale inference. 't' means the models use the backbone weights pre-trained by AugReg [30].
** represents the model reproduced under the same setrings as the official repo The GFLOPs
measured at single-scale inference with the given crop size.
Method
Backbone
Crop Size
GFLOPs
mIoU (ss)
mloU (ms)
UperNet* [32]
ViT-Base
512 x 512
>250
46.6
47.5
DPT [7]
VIT-Base
512 X 512
219.8
47.2
47.9
SETR-MLA* [6]
ViT- Base
512 x 512
113.5
48.2
49.3
Segmenter* [8]
ViT-Base
512 X 512
129.6
49.0
50.0
StructToken [31]
ViT-Base
512 x 512
>15
50.9
51.8
Seg ViT (Ours)
ViT-Base
512 x 512
120.9
51.3
53.0
DPT* [7]
VIT- ViT-Large
640 x 640
479.0
49.2
49.5
UperNet* [32]
ViT-Large
640 x 640
>700
48.6
50.0
SETR-MLA [6]
ViT-Large
512 x 512
368.6
48.6
50.3
MCIBI [33]
ViT-Large
512 x 512
>400
50,8
Segmenter [8]
ViT-
640 x 640
671.8
51.8
53.6
StructToken [31]
ViT-
640 x 640
>700
52.8
54.2
Seg ViT (Shrunk, ours)
Vir-Large"
640 x 640
373.5
53.9
55.1
SegViT (ours)
VIT- Large 640 x 640
637.9
54.6
55.2
55.2% in terms of mloU with the ViT-Large backbone. It 1.0% better than the recent StrucrTooken [31] using the same backbone. Besides, Shrunk version also achieve similar performance 55.1% with computational cost 373.5 GFLOPs which much less than the ViT-Large backbone alone (612.3 GFLOPs).
Table 2 shows the result the COCO-Stuff-100 dataset. Our method achieves 50.3% which is higher than the previous state-to-the-an Struc'Token by 1.2% with less computational cost. Our Shrunk version achieves 49.4% with 224.8 GFLOPs, which similar to the computational cost of dilated ResNet-101 backbone but with much higher performance.
Results
Table 2: Experiment results on the COCO-Stuff-Iee 10K test. split. Following published methods, we report the results with multi-scale inference (denoted by 'ms'). The GFLOPs is measured single scale inference with crop size of 512 x 512.
Method
Backbone
GFLOPs
mloU (ms)
DANet [34]
Dilated-ResNet--e
289.3
39.7
Faskerrmer [15]
ResNet-101-PB
81.7
39.8
EMANet [35]
Dilated-ResNet-100
247.4
39.9
SpyGR [36]
ResNet-101-fpn
>80
39,9
OCRNet [3]
HRNeIV2-W48
167.9
40.5
GINet [37]
JPU-ResNet-101
>200
40.6
RecoNet [38]
Dilated-ResNet-1O0
2000
41.5
ISNet [39]
Slatted-ResNet-tae 101
228.3
42.1
MCIBI [33]
ViT-Large
>380
44.9
StructToken [31]
ViT-Large
>00
49.1
SegViT (Shrunk, ours)
ViT-Large
224.8
49.4
ViT (ours)
ViT-Large
383.9
50.3
Results on PASCAL-Context. Table 3 shows the results on the PASCAL-Context dataset. We
follow HRNet [40] to evaluate our method and report the results under 59 classes (without background) and 60 classes (with background). Seg ViT reaches mloU 65.3% and 59.3% respectively for those two metrics that outperform the state-of-the-ar methods using the VIT backbones with less computational cost.

---

Table 3: Expperiment results on the PASCAL-Context val. split. Following published methods, we report the results with sulti-scale inference (denoted by 'ms') mloUsg: mloU averaged over 59 classes (without background), mloUso: mloU averaged over classes (59 classes plus background). Both metrics were used in the literature; and we report for the 60 classes. The GFLOPs is measured single scale inference with crop size of 480 480.
Method
Backbone
GFLOPs
mloUse (ms)
mloUG0 (ms)
RefineNet [41]
ResNet-152
47.3
UNct++ [42]
ResNet-101
47.7
PSPNet [11]
Dilated-ResNet-101
157.0
47.8
Ding et al. [43]
ResNet-101
51.6
EncNet [44] Dilated-ResNet-100
192.1
52.6
HRNet [40] HRNerV2-W44
82.7
54.0
48.3
NRD [45] ResNet-101
42.9
54.1
49.0
GFFNet [46]
Dilated-ResNet-101
54.3
EfficientFCN [47]
ResNet-101
52.8
55.3
OCRNet [3]
HRNerV2-W48
143.9
56.2
SETR-MLA [6]
ViT-Large
318.5
55.8
Segmenter [8]
ViT-Large
346.2
59.0
ViT (Shrunk, ours)
ViT-Large
186.9
63.7
57.4
Sig (ours)
ViT-Large
321.6
65.3
59.3
4.4 Ablation Study
In this section, we conduct the ablation study to show the effectiveness of our proposed methods.
Effect of the ATM module. Table shows the effect of the ATM module. We set the SETR-naive as the baseline, which uses two x 1 convolutions to get per-pixel classifications directly from the last layer of the ViT-Base transformer output. We supervise with a regular cross-entropy loss, ATM is capable of providing 0.5% performance boost. However, it is more beneficial decouple the classification and mask prediction process and the mask and classification supervision separately (3.15 % increase).
that by applying the ATM module and
Ablation of using different layers input for Sig Table 5 shows the performance boost that multiple layers input can provide. We can see that the performance boost feature maps from additional lower layers is obvious (+1.3%). We then involved more layers of features and see further performance gains. We empirically choose to use three layers for its best performance.
Table 4: Comparison between proposed ATM module with other methods. 'CE loss' indicates the cross-entropy loss that commonly used in mantic segmentation. The experiments are carried the ViT-Base backbone using ADE20K dataset.
Table 5: Ablation results of using differ-
ent layer inputs to the Sig structure on ADE20K dataset using ViT-Base the back- bone. Involving multi-layer features can bring obvious performance gain.
Used layers
mioU (ss)
Decoder
Loss
mloU (ss)
Single
[12]
49.6
SETR
CE loss
46.5
Cascade
|6. 12]
50.9 (+1.3)
ATM
CE loss
47.0 (+0.5)
Cascade
[6, 8, 12]
51.3 (+1.7)
ATM
Cmakk loss
49.6 (+3.1)
Cascade
[3, 9, 12]
51.2 (+1.6)
Ablation for the ATM Decoder. ATM decoder
We conduct experiments to show the effectiveness of the proposed
SegVIT on hierarchical backbones. Shown in Table 6, the Seg ViT structure is also able to apply to hierarchical backbones. We choose the most competitive methods Maskformer [15] and Mask2former [48] for comparison. Results indicate that even though method is not designed for

---

Table 6: The experiments use the Swin-Tiny [18] backbone and are carried out on the ADE20K dataset. The GFLOPss are measured at single scale inference with crop size of 512 x 512. QD: query-based down-saumping. QU: query-based
Table 7: Ablation of the QD module in terms of the targets and methods down-sample. The experiments are carried out on the ViT-Large backbone of ADE20K dataset.
upsampling.
Applied to
Methods mloU (ss)
Method
mloU (ss)
GFLOPs
Conv
44.5
Maskformer [15]
46.7
57.3
Q. K. Q
Nearest Nearest
52.6
Mask2former [48]
47.7
73.7
53.9
SegViT (Ours)
47.1
48.0
hierarchical backbones, we can still achieve competitive performance while being efficient in terms of computational cost.
Ablation for the QD module. The motivation to use QD is to make use of the pre-train weights of the backbone. As in Table 7, if use a stride convolution with learnable parameters down- sample the query, it will destroy the pre-train weights and dramatically decrease the performance. If the nown-sampling
applied to both Q and (K, V). there will be inevitable loss in information during the down-sampling process which is reflected in the weaker performance. We found that applying x 2 nearest wwm-saamling on query only for the QD module is the better option.
Ablation of the components in Shrunk structure. Shown in Table 8, we studied the effect of each component (QD and QU) in the Shrunk structure. The results presented matches the structures illustrated in Fig. When QD is applied, the performance decreases by 2.7% from the 'Singlee' ATM head. However, by applying QU, the performance is recovered. QD learns a non-linear down- sampling by the attention mechanism between key and query. One query will attend several keys. QU is used to preserve the resolution and at the same time provide low-level feature information. We can see that by using QD and QU jointly, the performance can be retained and the computational cost is reduced. ATM module can also be used as the decoder to form our Shrunk structure to further boost performance.
Table 8: Ablation results of Shrunk version on the ADE20K dataset. The GFLOPs are measured single scale inference with a crop size of 512 x 512 on ViT-Base backbone.
Structure
QD
QU
Head
mIoU (ss)
GFLOPs
Single
SETR
46.5
107.3
Single
ATM
49.6 (+3.1)
115.8
Naive Shrunk
ATM
46.9 (+0.4)
74.1
Shrunk
ATM
50.0 (+3.5)
97.1
5 Conclusion
We proposed an effective structure using plain ViT transformer backbones termed Seg for the semantic segmentation task. For the first time, we utilize spatial information in attention maps for semantic segmentation. implement this idea, we proposed an Attention-to-mask (ATM) module that can derive mask predictions during the attention calculation process. We show number of semantic segmentation benchmarks that our method efficient and achieves state-of-the-an performance. We also proposed a Shrwnk structure which is applied to the backbone and capable of reducing 40% of the computational cost while still maintaining competitive performance. We believe both structures can be strong paradigms, especially for semantic segmentation using ViT backbones. Last but not the least, our method still has some limitations. One of the limitations that the large amount of GPU memory consumed by the global attention mechanism might not be supported by some devices, which might restrict the applicability of our structures.
Acknowledgments C. Shen's participation was in part supported by major grant from Zhejiang Provincial Government. This work was also supported by the start-up funding of the University of Adelaide. [grant number 151300 130411]]. This research was supported by Meituan.

---

References
[1] J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 3431-3440, 2015.
[2] L.-C. Chen, G. Papandreou, Kokkinos, K. Murphy, and A. Yuille, "Deeplab: Semantic image segmenta- tion with deep convolutional nets, atrous convolution, and fully connected CRFs," IEEE Trans. Pattern
Anal. Mach. Intell., vol. 40, no, 4, pp. 834-848, 2017.
[3] Y. Yuan, X. Chen, and J. Wang, "Ohject-contextual representations for semantic segmentation," " in Proc.
Fur. Conf. Comp. Vis., pp. 173-190, Springer, 2020.
[4] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 2117-2125, 2017.
[5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, el al., "An image is worth 16x16 words: Transformers for image recognition at scale," Proc. Int. Conf. Learn. Representations, 2021.
[6] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang. Y. Fu, J. Feng. T. Xiang, P. H. Torr, et al., "Rethinking semantic segmentation from a wwwreeencett--eqeeeee perspective with transformers," in Proc. IEEE Conf. Comp. Vis. Patr. Recogn. pp. -8900, 2021.
[7] R. Ranftl, A. Bochkovskiy, and V. Koltun, "Vision transformers for dense prediction." in Proc. IEEE Int.
Conf. Comp. Vis., pp. 11179--2188, 2021.
[8] R. Strudel, R. Garcia, Laptev, and Schmid, "Segmenter: Transformer for semantic segmentation." in Proc. IEEE Int. Conf. Comp. Vis., pp. 7262-7277 0002-727,22 2021.
[9] C. Farabet, C. Coupric, L.. Najman, and Y. LoCun, "Learning hicrarchical features for seene labeling," " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1915-1929, 2012.
[10] J. I Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proc. IEER Conf. Comp. Vis. Patt. Recogn., pp. 3431-3440, 2015.
[11] H. Zhao, J. Shi, X. Qi. X. Wang, and J. Jia, "Pyramid scene parsing network," in Proc. IEEE Conf. Comp.
Vis. Patt. Recogn., 2017.
[12] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, "Encoder-decoder with atrous separable convolution for semantic image segmentation." in Proc. Eur. Conf. Comp. Vis., pp. 801-818, 2018.
[13] J. Fu, J. Liu, J. Jiang, Y. Li, Y. Bao, and H. Lai, "Scene segmentation with dual relation-aware attention network," JEEE Trans. Neural Netw. & Learn. Syst., vol. 32, no. 6, pp. 2547-2560, 2020.
[14] C. Yu, J. Wang, C. Gao, G. Yu, C. Shen, and N. Sang. "Context prior for scene segmentation"" in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 12416-12425, 2020.
[15] B. Cheng, A. Schwing, and A. Kirillov, "Per-pixel classification is not all you need for semantic segmenta- tion," Proc. Advances in Neural Inf. Process. Syst., vol. 34, 2021.
[16] W. Zhang, J. Pang. K. Chen, and C. C. Loy, "K-net: Towards unified image segmentation." Proc. Advances
in Neural Inf. Process. Syst., vol. 34, 2021.
[17] Wang, E. Xie, X. Li, D.-P. Fan, K. Song. Liang, T. Lu, P. Luo, and L. Shao, "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions," in Proc. IEEE Int. Conf. Comp. Vis.. pp. 568-578, 2021.
[18] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hicrarchical vision transformer using shifted windows," in Proc. IEEE Int. Conf. Comp. Vis., pp. (0012-10022, 2021.
[19] X. Chu, 7. Tian, Y. Wang. B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, "Twins: Revisiting the design of spatial attention in vision transformers," Proc. Advances in Neural Inf. Process. Syst., vol. 34, 2021.
[20] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, "Segformer: Simple and efficient design for semantic segmentation with transformers," " Proc. Advances in Neural Inf. Process. Syst., vol. 34, 2021.
[21] Y.H.. Wu, Y. Liu, X. Zhan, and M.-M. Cheng, "P2t: Pyramid pooling transformer for scene understanding." IEEE Trans. Pattern Anal. Mach. Intell., 2022.
[22] T.-Y. Lin, P. Goyal, R. Girshick, K. He. and P. Dollár. "Focal loss for dense object detection," in Proc. IEEE Int. Conf. Comp. Vis., pp. 2980-2988, 2017.
[23] F. Milletari, N. Navab, and S.-A. Ahmadi, "V-net: Fully convolutional neural networks for volumetric
medical image segmentation. " 3DV, pp. 565-571, JEEE, 2016.
[24] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," " in Proc. Eur. Conf. Comp. Vis., pp. 213-229, Springer, 2020.
[25] A. Vaswani. N. Shazeer, N. Parmar. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and L. Polosukhin, "Attention is all you need," Proc. Advances in Neural Inf. Process. Syst., vol. 30, 2017.
[26] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Torralba, "Scene parsing through ade20k dataset." in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 633-641, 2017.
10

---

[27] H. Caesar. J. Uijlings, and V. Ferrari, "Coco-stuff: Thing and stuff classes in context," in Proc. IEEE Conf. Comp. Vis. Putt. Recogn.. pp. 1209-1218, 2018.
[28]
MMSegmentation, "MMSegmentattin:: OpenMMLab semantic segmentation toolbox and benchmark." https://github.c com open- www.aa/mmmmgmeeatii...mm 2020.
[29] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.W. Lee, S. Fidler, R. Urtasun, and A. Yuille, "The role of context for object detection and semantic segmentation the wild," Proc. IEEE Conf. Comp. Vis. Patt.
Recogn.. pp. 391-898,70 2014.
[30] A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer, "How to train your vit? data, augmentation, and regularization in vision transformers," arXiv: Comp. Res. Repository, 2021. [31] F. Lin, Z. Liang. J. He, M. Zheng. S. Tian, and K. Chen, "Structtoken: Rethinking semantic segmentation with structural prior." arXiv: Comp. Res. Repository, 2022.
[32] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun, "Unified perceptual parsing for scene understanding,"" in
Proc. Fur. Conf. Comp. Vis., pp. 418-434, 2018.
[33] Z. Jin, T. Gong, D. Yu, Q. Chu, J. Wang, C. Wang. and J. Shao, "Mining contextual information beyond
image for semantic segmentation." in Proc. IEEE Int. Conf. Comp. Vis., pp. 7231-7241, 2021.
[34] J. Fu, J. Liu. H. Tian, Y. Li, Y. Bao, Z. Fang. and H. Lu, "Dual attention network for scene segmentation."
in Proc, IEEE Conf. Comp. Vis. Patt. Recogn., pp. 3146-3154, 2019.
[35] X. Li, Z. Zhong, J. Wu, Y. Yang, Z. Lin, and H. Liu, Exxpectation-maximizztlli
attention networks for
semantic segmentation," in Proc. IEEE Int. Conf. Comp. Vis., pp. 9167-9176, 2019.
[36] X. Li. Yang. Q. Zhao, Shen, Z. Lin, and Liu, "Spatial pyramid based graph reasoning for semantic segmentation."" in Proc. JEEE Conf. Comp. Vis. Patt. Recogn.. pp. 8950-8959, 2020.
[37] T. Wu, Y. Lu, Y. Zhu, C. Zhang, M. Wu, Z. Ma, and G. Guo, "Ginet: Graph interaction network for scene
parsing," in Proc. Eur. Conf. Comp. Vis., Pp. 34-51, Springer, 2020.
[38] W. Chen, X. Zhu, R. Sun, J. He, R. Li, X. Shen, and B. Yu, "Tensor low-rank reconstruction for semantic
segmentation." in Proc. Eur. Conf. Comp. Vis., pp. 52-69, Springer, 2020.
[39] Z. Jin. B. Liu. Q. Chu, and N. Yu, "Isnet: Integrate image-level and semantic-level context for semantic segmentation," in Proc. IEEE Int. Conf. Comp. Vis., pp. 7189-7198, 2021.
[40] K. Sun, Y. Zhao, B. Jiang, T. Cheng, B. Xiao, D. Liu, Y. Mu, X. Wang, W. Liu, and J. Wang. "High-
resolution representations for labeling pixels and regions," " arXiv: Comp. Res. Repository, 2019. [41] G. Lin, A. Milan, C. Shen, and Reid, "RefineNct: Multi- path refinement networks for high-rcsolution semantic segmentation, in Proc. JEEE Conf. Comp. Vis. Patt. Recogn., pp. 1934,193,222 2017.
[42] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, "Unet++: A nested U-net architecture for medical image segmentation," " Proc. Deep Learning in Medical Image Analysis Workshop, pp. 3-11, 2018.
[43] H. Ding, X. Jiang, B. Shuai, A. Q. Liu, and G. Wang, "Context contrasted feature and gated multi-scale aggregation for scene segmentation," in Proc. IEEE Conf. Comp. Vis. Part. Recogn., pp. 2393-2402, 2018.
[44] H. Zhang, K. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal, "Context encoding for semantic segmentation," in Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 7151-7160, 2018 2018.
[45] B. Zhang, Z. Tian, C. Shen, et al., "Dynamic neural representational decoders for high-resolution semantic segmentation," Proc. Advances in Neural Inf. Process. Syst. vol. 34, 2021.
[46] X. Li, H. Zhao, L. Han, Y. Tong, S. Tan, and K. Yang. "Gated fully fusion for semantic segmentation," in
Proc. AAAI Conf. Artificial Intell., vol. 34, pp. 11418-11425, 2020.
[47] J. Liu, He, J. Zhang. J. Ren, and H. Li. "EfficientFCN: Holistically-guue guided decoding for semantic segmentation." in Proc. Eur. Conf. Comp. Vis., 2020.
[48] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, "Masked- attention mask transformer for universal image segmentation," 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] We conduct experiments on a fundamental task of semantic segmentation. This technique may be used for editing fake images to mislead the public if being used by someone who has ulterior motives.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either the supplemental material or URL)? [Yes] We included the code for the main experiments in the supplemental materials. All the code will be released upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? Yes] See implementation details.
(c) Did you report
bars (e.g.. with respect the random seed after running experiments random seed other random operators
multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the implementation details and GFLOPs tables.
4. If you using existing assets (e.g., code, data, models) curating/relcasing new assets...
(a) your work uses existing assels, did you cite the creators? [Yes]
(b) Did you mention the license of assets? [No] The license can be found their own homepages.
(c) Did you include any assets either in the supplemental material URL22 [Yes] We
include our trained model in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [N/A] All data are public benchmarks.
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [No] All the data are public benchmarks.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "wb7UM6E56K0", "reviewer": "Reviewer_oFbW", "summary": "The authors deal with ViT-based semantic segmentation. In particular, they use a set of learnable tokens (each corresponding to a semantic class) which decode the outputs of the ViT-based backbone into per-class semantic masks. This is accomplished by multiple layers of cross attention between class token and ViT tokens. Rather than use a dot product like mechanism to produce similarity between a class token and spatial features, they directly supervise the cross attention maps using a sigmoidal output. Furthermore, they introduce a down/upsampling technique to mimic the general idea of an efficient multi-scale prediction head. Their results are quite good even when compared to some of the best recent models and their QD module provides some computation/performance tradeoffs.", "questions": "1. I'm slightly confused about why one needs class predictions on the output tokens? Is this because the attention mechanism is bad at producing low confidence values when the class does not exist? Do you just multiply this probability by the sigmoid output?\n2. Table 4 could be more convincing. The loss formulation is not novel, so why not also apply it to SETR?\n3. It seems reasonable to provide a \"dot product\"-based baseline (similar to the decoder in Segmenter) to tease where attention (which itself is a dot product to some degree in this scenario) actually makes a difference.\n4. Does this work when just supervising the aggregated output (a single per class segmentation loss)?", "limitations": "I believe so.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This is a well written paper and the approach is quite clean\n2. The results presented are quite good as well, achieving at/near SOTA against competitive models\n\nWeaknesses\n\n1. The idea is still related to the idea of dot product based segmentation (from some class embedding). I think a good deal of experiments might need to be performed to actually understand the technical contribution.\n2. While the results are good, related work like Segmenter is not far off from the performance presented here and shares some significant similarities with this method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UtxgVeF5OWb", "reviewer": "Reviewer_YDPR", "summary": "The paper introduces SegViT, a semantic segmentation framework with plain ViTs as backbones. One of the core technical contributions is the proposed Attention-to-Mask (ATM) block, which generating masks from the intermediate attention maps between class embeddings and key maps. In addition, a shrunk structure is then proposed to save computational cost while maintaining the performance. Based on plain ViT networks only, SegViT obtains state-of-the-art results on three semantic segmentation datasets (ADE20K, PASCAL-Context and COCO-Stuff-10K). ", "questions": "SegViT still utilizes FPN-like architecture to fuse low-level and high-level features from the backbone. However, some previous works [*1, *2, *3] challenge the essential of such feature fusion. It will be interesting if the authors show experiments where ATM module only relates to the last layer of ViT encoder. For example, like the experiments in Table 5, the authors may try a cascade design while utilizing the last layer multiple times, or just follow ViTDet fashion [*3].\n\n[*1] Zhang et al. ExFuse: Enhancing Feature Fusion for Semantic Segmentation. ECCV 2018. \n\n[*2] Chen et al. You Only Look One-level Feature. CVPR 2021.\n\n[*3] Li et al. Exploring Plain Vision Transformer Backbones for Object Detection. Tech report. \n", "limitations": "Limitations are mentioned in the conclusion. Although I think more discussion and comparisons with [15] are required in the paper. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n\n1. The paper is well motivated. Recently many works (e.g. [*1]) have realized even plain ViTs could have rich representation capacity, which however requires special optimization (e.g. masked image modeling) or other architectural modifications for downstream tasks. I am pleased that the paper demonstrates that plain ViTs can obtain as good results as the hierarchical counterparts (e.g. [15, 47]) on segmentation tasks, which may encourage simpler and unified network design principles. \n\n2. Strong results are reported in the paper. For example, on ADE20K val a model with ViT-L backbone achieves 55.2 mIoU, which is very competitive even among more sophisticated networks, such as Swin-L and MViT. \n\n3. The motivation of ATM module sounds reasonable to some extent: intuitively a good attention mask should cover the foreground of the given object (or class). Therefore, it is possible to generate mask directly from the attention matrix. \n\nCons\n\n1. My major concern is that the technical novelty is relatively limited. The overall framework is very similar to MaskFormer [15] and Mask2Former [47]. Compared with [15], the major difference on the technical details is, [15] generate masks from the product of the mask embedding and the per-pixel embedding, while in the paper the mask is directly derived from the attention weights. However, I do not think it differs much. Although [15, 47] mainly evaluate on hierarchical backbones, theoretically they can also be equipped with plain networks. In addition, the proposed QU/QD layers are not novel (also sounds irrelevant to the main topic of the paper), since many previous works, e.g. PVT [17], also adopt similar blocks to reduce computational cost. In conclusion, I think the contributions claimed in the introduction seems not significant.\n\n2. According to Table 4 and Line 234-239, in the proposed ATM block, the separated supervision of classification and mask prediction is the most important design principle. However, it is not originally proposed in the paper as [15, 16] already introduces the paradigm. It further weakens the significance of the proposed method. \n\n[*1] Li et al. Exploring Plain Vision Transformer Backbones for Object Detection. Tech report. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PXdCd0hmnqE", "reviewer": "Reviewer_Nzhh", "summary": "This paper present a semantic segmentation method based on the plain vision transformer (ViT). Specifically, it proposes the attention-to-mask (ATM) module to generate the pixel-level mask. In addition, to reduce the computational cost, it designs a query-based downsampling (QD) and upsampling module (QU) in the shrunk version. Experiments are conducted on three datasets and better results are obtained compared with previous methods.", "questions": "Please address the above concerns in the weakness.", "limitations": "The authors have addressed the limitations of the proposed method in large memory consumption.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n1. Exploring plain architecture for semantic segmentation is an interesting and promising direction. This paper make a forward step towards this direction.\n2. The performance of SegVIT seems to be better than previous state-of-the-art methods.\n\n**Weakness**\n\n1. About the Attention-to-Mask module (ATM), it is implemented in cross-attention manner. But in fact, there is little difference with the standard classifier (a fc layer to map features to probability) for per-pixel classification in the normal semantic segmentation framework. Each learned token could be viewed as a classifier layer to map the pixel-level features into a probability with a Sigmoid function. In this sense, the ATM is similar to a standard classification layer.\n\n2. About the Shrunk structure, I am confused about the query-based downsampling operation (QD). In line 172, it says to use the nearest sampling to reduce the token numbers. In this sense, it has nothing with the query based downsampling and is simply a standard downsampling operation. I am also confused about the implementation details on query-based upsampling operation (QU). It says to use a standard transformer decoder structure to upsample features. Is there any special design on the transformer decoder by incorporating the spatial information? More details are required on the decoder design.\n\n3. About the two QU operations in the version (c) of Figure 3,  the downsampling ratio of lower QU is 1/16, and it is natural to think its output have smaller downsampling ratio like 1/8. However, from line 181-183, its output size seems to be 1/16, which is confused for me.\n\n4. I think this paper should compare with the previous works PerceiverIO, which employs a similar downsampling-upsampling architecture for dense prediction with transformers. More discussion on the difference is required to better motivate the proposed method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GNPF-8kiwGv", "reviewer": "Reviewer_Eyo8", "summary": "The paper proposed a plain-ViT based semantic segmentation framework, which uses an Attention-to-Mask decoder to aggregate image features and a Shrunk structure to save computational cost.", "questions": "1. Considering the similarity of MaskFormer and SegViT, I am curious about the result of MaskFormer + ViT + multi-layer-feature (use the same three ViT layer inputs to MaskFormer transformer decoder). I think the result may be similar with SegViT. \n2. In the paper, it claims that the SegViT framework has a quite good performance with Swin Transformer backbone and lists the results with Swin-Tiny. I am curious about SegViT + Swin-Large result and its comparison with MaskFormer and Mask2Former.\n", "limitations": "The authors have addressed the limitations and potential negative societal impacts.", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper proposed SegViT framework and achieved a SOTA performance based on a plain ViT backbone. \n2. The paper is well written and clear to understand.\n\nWeaknesses:\n1. I doubt the novelty of the design of ATM module. Since the MaskFormer framework has been proposed for over half a year, the ATM module is similar to the MaskFormer transformer decoder module. The only difference is that the mask output of MaskFormer is generated by the multiplication of the final output query tokens and image features, while the mask output of ATM is from the multiplication of an intermediate variable K inside the transformer layer and the image features. The difference is not obvious. The SegViT framework is just like MaskFormer + ViT + multi-layer-feature.\n2. Table 4 shows that ATM has a relatively low performance gain of about 0.5% to SETR. It shows that the performance of ATM is even worse than Segmenter (Since the result of Segmenter is 0.8% better than SETR in Talbe 1)?\n3. Also, in table 4, it shows that by using L_mask loss the mIoU result increases about 2.6% than using CE loss only. However, Table 1 shows that the result of SegViT is about 2.3% better than Segmenter baseline (which only uses CE loss). If it shows that the performance gain is all from the new loss design but not from the framework architecture. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept