Per-Pixel Classification is Not All You Need for Semantic Segmentation
Authors: Bowen Cheng
Alex Schwing
Alexander Kirillov
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: semantic segmentation
panoptic segmentation
mask classification

Abstract: Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

Per-Pixel Classification is Not All You Need for Semantic Segmentation
Bowen Cheng
Alexander G. Schwing²
Alexander Kirillov
Facebook Al Research (FAIR)
University of Illinois at Crbana-Champaign (UIUC)
Abstract
Modern approaches typically formulate semantic segmentation as a per-pixel classi- fication task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, simple mask classification model which predicts set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-base method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-basse method outperforms both current sate-offtheeant semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models."
1 Introduction
The goal of semantic segmentation to partition an image into regions with different semantic categories. Starting from Fully Convolutional Networks (FCNs) work of Long el al. [28], most deep learning-bused semantic segmentation approaches formulate semantic segmentation as per-pixel classification (Figure lefft,, applying classification loss to each output pixel [8, 46]. Per-pixel predictions in this formulation naturally partition an image into regions of different classes.
Mask classification is an alternative paradigm that disentangles the image partitioning and classifica- tion aspects of segmentation. Instead of classifying each pixel, mask classification- based methods predict a set of binary masks, each associated with a single class prediction (Figure right). The more flexible mask classification dominates the field of instance-level segmentation. Both Mask R-CNN [19] and DETR [3] yield a single class prediction per segment for instance and panoptic segmentation. contrast, per-pixel classification assumes a static number of outputs and cannot return variable number of predicted regions/segments, which is required for instance-level tasks.
Our key observation: mask classification is sufficiently general solve both semantic- and instance- level segmentation tasks. In fact, before FCN [28]. the best performing semantic segmentation methods like O2P [4] and SDS [18] used a mask classification formulation. Given this perspective, natural question emerges: can single mask classification model simplify the landscape of effective approaches to semantic- and instance-leve segmentation tasks? And can such mask classification model outperform existing per-pixel classification methods for semantic segmentation?
To address both questions we propose simple MaskFormer approach that seamlessly converts any existing per-pixel classification model into a mask classification. Using the set prediction mechanism proposed in DETR [3], MaskFormer employs Transformer decoder [37] compute set of pairs,
Work partly done during an internship at Facebook Al Research. I Project page: https://bowenc/2222 wwt.:22.......... github. io/maskformer
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

per-pisel classification loes
building
sky
K
prediction
prediction
prediction
pexepixel class predictions
per pixel binary mask loss
per-mask classification loss
Figure 1: Per-pixel classification vs. mask classification. (left) Semantic segmentation with per- pixel classification applies the same classification loss to each location. (right) Mask classification predicts a of binary masks and assigns single class each mask. Each prediction is supervised with per-pixel binary mask loss and a classification loss. Matching between the set of predictions and ground truth segments can be done either via bipartite matching similarly to DETR [3] or by fixed matching via direct indexing if the number of predictions and classes match, ie., if N K.
each consisting of class prediction and mask embedding vector. The mask embedding vector is used to get the binary mask prediction via a dot product with the per-pixel embedding obtained from an underlying fully-convolutional network. The new model solves both semantic- and instance-level segmentation tasks in a unified manner: no changes to the model, losses, and training procedure are required. Specifically, for semantic and panoptic segmentation tasks alike, MaskFormer is supervised with the same per per-piee binary mask loss and a single classification loss per mask. Finally, we design simple inference strategy to blend MaskFormer outputs into task-dependent prediction format.
We evaluate MaskFormer on five semantic segmentation datasets with various numbers of categories: Cityscapes [13] (19 classes), Mapillary Vistas [31] (65 classes), ADE20K [49] (150 classes), COCO- Stuff-10K [2] (171 classes), and ADE20K-Ful [49] (847 classes). While MaskFormer performs on par with per-pixel classification models for Cityscapes, which has few diverse classes, the new model demonstrates superior performance for datasets with larger vocabulary. We hypothesize that a single class prediction per mask models fine- tne-grained recognition better than per-pixel class predictions. MaskFormer achieves the new state-of-the- on ADE20K (55.6 mIoU) with Swin- Transformer [27] backbone, outperforming per-pixel classification model [27] with the same backbone by 2.1 mloU, while being more efficient (10% reduction in parameters and 40% reduction in FL.OPs).
Finally, we study MaskFormer's ability to solve instance-level tasks using two panoptic segmentation datasets: COCÓ [26, 22] and ADE20K [49]. MaskFormer outperforms a more complex DETR model [3] with the same backbone and the same post-processing. Moreover, MaskFormer achieves the new state-of-the-art on COCO (52.7 PQ), outperforming prior state-of-the [38] by 1.6 PQ. Our experiments highlight MaskFormer's ability unify instance- and semantic-level segmentation.
Related Works
Both per-pixel classification and mask classification have been extensively studied for semantic segmentation. In early work, Konishi and Yuille [23] apply per-pixel Bayesian classifiers based on local image statistics. Then, inspired by early works on non-semantic groupings [11, 33], mask classification-basedd methods became popular demonstrating the best performance in PASCAL VOC challenges [16]. Methods like O2P [4] and CFM [14] have achieved state-of-the-art results by classifying mask proposals [5, 36, 1]. In 2015, FCN [28] extended the idea of per-pixel classification to deep nets, significantly outperforming all prior methods on mIoU (a per- per-pixel evaluation metric which particularly suits the per-pixel classification formulation of segmentation).
Per-pixel classification became the dominant way for deep-net-basee semantic segmentation since the seminal work of Fully Convolutional Networks (FCNs) [28]. Modern semantic segmentation models focus on aggregating longraanee context in the final feature map: ASPP [6, 7] uses atrous convolutions with different atrous rates; PPM [46] uses pooling operators with different kernel sizes; DANet [17], OCNet [45], and CCNet [21] use different variants of non-local blocks [39]. Recently, SETR [47] and Segmenter [34] replace traditional convolutional backbones with Vision Transformers (ViT) [15] that capture long-range context starting from the very first layer. However, these concur- rent Transformer-base [37] semantic segmentation approaches still use a per-pixel classification

---

formulation. Note, that our MaskFormer module can convert any per-pixel classification model to the mask classification setting, allowing seamless adoption of advances per-pixel classification.
Mask classification is commonly used for instance-level segmentation tasks [18, 22]. These tasks require a dynamic number of predictions, making application of per-pixel classification challenging as it assumes a static number of outputs, Omnipresent Mask R-CNN [19] uses a global classifier to classify mask proposals for instance segmentation. DETR [3] further incorporates a Transformer [37] design to handle thing and stuff' segmentation simultaneously for panoptic segmentation [22]. How- ever, these mask classification methods require predictions of bounding boxes, which may limit their usage in semantic segmentation. The recently proposed Max- DeepLab [38] removes the dependence on box predictions for panoptic segmentation with conditional convolutions [35, 40]. However, in addition to the main mask classification losses it requires multiple auxiliary losses (i.e., instance discrimination loss, mask-ID cross entropy loss, and the standard per-pixel classification loss).
3 From Per-Pixel to Mask Classification
In this section, we first describe how semantic segmentation can be formulated as either per- per-pixel classification or a mask classification problem. Then, we introduce our instantiation of the mask classification model with the help of a Transformer decoder [37]. Finally, we describe simple inference strategies to transform mask classification outputs into task-dependent prediction formats.
3.1 Per-pixel classification formulation
For per-pixel classification, segmentation model aims to predict the probability distribution over all possible K categories for every pixel of an H x XW image: y = {p-lps E A ji=1 Here A is the K- dimensional probability simplex. Training a per-pixel classification model is straight- ground truth category labels y = {v," w/ E {1,..., K)
given
H-W for every pixel, a per-pixel cross- H-W
entropy (negative log-likelihood) loss is usually applied, ie., Lpiset y²t) )
log (v
3.2 Mask classification formulation
Mask classification splits the segmentation task into 1) partitioning/rrouingg the image into N regions
(N does not need to equal K), represented with binary masks (m,.m [0, 1 ):M; and 2) associating each region as a whole with some distribution over categories. To jointly group and classify a segment, i..,, to perform mask classification, we define the desired output z as a set of N probability-mast pairs, i.e, 2 = {P,,, m In contrast to per-pixel class probability prediction, for mask classification the probability distribution p; E K+I contains an auxiliary "no object" label (Ø) in addition to the K category labels. The Ø label is predicted for masks that do not correspond to any of the K categories. Note, mask classification allows multiple mask predictions with the same associated class, making it applicable to both semantic- and instance-level segmentation tasks.
To train mask classification model, a matching a between the set of predictions and the set of Nat ground truth segments z {c,," m² Ie"" E {I,.... , K} m² E {0, 1] # 1 is required." Here of is the ground truth class of the 5hh ground truth segment. Since the size of prediction set |æ| N and ground truth set | = N generally differ, we assume N 2 Ne and pad the set of ground truth labels with "no object" tokens Ø to allow one-to-one matching.
For semantic segmentation, a trivial fixed matching is possible if the number of predictions N matches the number of category labels K. In this case, the 7th prediction is matched to a ground truth region with class label and to Ø if a region with class label i is not present in the ground truth. In our experiments, we found that bipartite matching-based assignment demonstrates better results than the fixed matching. Unlike DETR [3] that uses bounding boxes to compute the assignment costs between prediction z; and ground truth =á for the matching problem, we directly use class and mask predictions, i.e., (p)) + maskk (m; mn)) where Lmask is binary mask loss.
To train model parameters, given a matching, the main mask classification loss Lmask-els is composed of a cross-entropy classification loss and a binary mask loss Luuusk for each predicted segment:
Lmuss-ek (z, 20)
.V
[-
(oggoou)(e (c5)) + 1l M (M) m
2 Different mask classification methods utilize various matching rules. For instance, Mask R R-CNN [19] uses heuristic procedure based on anchor boxes and DETR [3] optimizes bipartite matching between and z
3

---

transformer module
classification loss
segmentation module
transformer decoder
predictions
semantic segmentation
MLP
0
inference only
b queries
Nx(K 1)
mask embeddings
drop
semantic segmentation
pixel-level module
binary mask loss
Emask
XN
backbone
pixel decoder
Epixet
N mask predictions
KxlxWW
CxXHXW
NxHxW
Image features F
per-pixel embeddings
Figure 2: MaskFormer overview. We use a backbone to extract image features F. A pixel decoder gradually upsamples image features extract per-pixel embeddings Epixel. A transformer decoder attends to image features and produces N per-segment embeddings Q. The embeddings independently generate N class predictions with N corresponding mask embeddings &mask. Then, the model predicts N possibly overlapping binary mask predictions via dot product between pixel embeddings Epixel and mask embeddings Emak followed by a sigmoid activation. For semantic segmentation task we can get the final prediction by combining N binary masks with their class predictions using simple matrix multiplication (see Section 3.4). Note, the dimensions for multiplication X are shown in gray.
Note, that most existing mask classification models use auxiliary losses (e.g.. a bounding box loss [19, 3] an instance discrimination loss (38] in addition to Lmmusk cls- In the next section present a simple mask classification model that allows end-to-end training with Lmask alone.
3.3 MaskFormer
We now introduce MaskFormer, the new mask classification model, which computes N probability- mask pairs = {pi, m The model contains three modules (see Fig. 2): 1) a pixel-level module that extracts per-pixel embeddings used to generate binary mask predictions; 2) a transformer module, where stack of Transformer decoder layers [37] computes N per-segment embeddings; and 3) segmentation module, which generates predictions {(p;,m:) m,)] [ from these embeddings. During inference, discussed in Sec. 3.4, Pt and m, are assembled into the final prediction.
Pixel-level module takes an image of size H x W as input. A backbone generates a (typically)
low-resolution image feature map F E RC x 4 where Cf is the number of channels and S the stride of the feature map (C5 depends on the specific backbone and we use = 32 in this work). Then, pixel decoder gradually upsamples the features to generate per-pixel embeddings Epixel E Ce
where Cg is the embedding dimension. Note, that any per-pixel classification- based segmentation model fits the pixel-level module design including recent Transformer-based models [34, 47, 27J. MaskFormer seamlessly converts such model to mask classification.
Transformer module uses the standard Transformer decoder [37] to compute from image features F and N learnable positional embeddings (i.e.. queries) its output, i.e., N per-segment embeddings Q E Co XN of dimension Co that encode global information about each segment MaskFormer predicts. Similarly [3], the decoder yields all predictions parallel.
Segmentation module applies a linear classifier, followed by a softmax activation, on top of the per-segment embeddings Q yield class probability predictions (p; A KK+JN } for each segment. Note, that the classifier predicts an additional "no object" category (Ø) in case the embedding does correspond to any region. For mask prediction, a Multi-Layer Perceptron (MLP) with 2 hidden layers converts the per-segment embeddings Q to N mask embeddings Emask E I Ce of dimension Cg. Finally, we obtain each binary mask prediction m, E [0, 1] x via dot product between the 7th mask embedding and per-pon per-pixel embeddings Epixel computed by the pixel-level module. The dot product is followed by a sigmoid activation, i.e., m,(h, w| = ( i]_ Epixal h, w)).
Note, we empirically find it is beneficial to not enforce mask predictions to be mutually exclusive to each other by using a softmax activation. During training, the Laaskels loss combines a cross entropy classification loss and binary mask loss Lmask for each predicted segment. For simplicity we use the same Cmast as DETR [3], i..., linear combination of a focal loss [25] and a dice loss [30] multiplied by hyper-parameters Apecal and Adiee respectively.

---

3,4 Mask-classificatinn inference
First, we present a simple general inference procedure that converts mask classification outputs Upi mi) to either panoptic semantic segmentation output formats. Then, we describe semantic inference procedure specifically designed for semantic segmentation. We note, that the specific choice of inference strategy largely depends on the evaluation metric rather than the task.
General inference partitions an image into segments by assigning each pixel [h, w] to one of the N predicted probability- pairs via arg ma Pa(c:) my [h, w|. Here is the most likely class label G = arg (1..... (c()) for each probability-man -mak pair i. Intuitively, this procedure assigns a pixel at location [h. w] to probability-mask pair i only if both the most likely class probability Pe(c;) and the mask prediction probability mi(h, w) are high. Pixels assigned to the same probability- mask pair i form a segment where each pixel is labelled with c,. For semantic segmentation, segments sharing the same category label are merged; whereas for instance-level segmentation tasks, the index i of the probability-mase pair helps to distinguish different instances of the same class. Finally, to reduce false positive rates in panoptic segmentation we follow previous inference strategies [3, 22]. Specifically, filter out low-confidence predictions prior to inference and remove predicted segments that have large parts of their binary masks (m; > 0.5) occluded by other predictions.
Semantic inference is designed specifically for semantic segmentation and is done via a simple matrix multiplication. We empirically find that marginalization over probability-
pairs, i.e., arg (1... () K) Ex= Pa(c)- - m. [h, w). yields better results than the hard assignment of each pixel to probability-man pair used the general inference strategy. The argmax does not include the "no object" category (Ø) as standard semantic segmentation requires each output pixel to take label. Note, this strategy returns a per-pixel class probability Σ/= Pe(c) m,(h, w] However, we observe that directly maximizing per-pixel class likelihood leads to poor performance. We hypothesize, that gradients are evenly distributed to every query, which complicates training.
4 Experiments
We demonstrate that MaskFormer seamlessly unifies semantic- and instance-level segmentation tasks by showing state-of-the results on both semantic segmentation and panoptic segmentation datasets. Then, we ablate the MaskFormer design confirming that observed improvements in semantic segmentation indeed stem from the shift from per-pixel classification to mask classification.
Datasets. We study MaskFormer using four widely used semantic segmentation datasets: ADE20K [49] (150 classes) from the SceneParse150 challenge [48], COCO-Stuff- 10K [2] (171 classes), Cityscapes [13] (19 classes), and Mapillary Vistas [31] (65 classes). In addition, we use the ADE20K-Full [49] dataset annotated in an open vocabulary setting (we keep 874 classes that are present in both train and validation sets). For panotic segmenation evaluation we use COCO [26, 2, 22] (80 "things" and 53 "stuff" categories) and ADE20K -Pnnoptic [49, 22] (100 "things" and 50 "stuff" categories). Please the appendix for detailed descriptions of all used datasets.
Evaluation metrics. For semantic segmentation the standard metric is mloU (mean Intersection-over- Union) [16]. a per-pixel metric that directly corresponds to the per-pixel classification formulation. To better illustrate the difference between segmentation approaches, our ablations we supplement mloU with PQ (PQ stuff)
[22], per-region metric that treats all classes as "stuff" and evaluates each segment equally, irrespective of its size. We report the median of 3 runs for all datasets, except for Cityscapes where we report the median of 5 runs. For punoptic segmentation, we use the standard PQ (panoptic quality) metric [22] and report single run results due to prohibitive training costs.
Baseline models. On the right we sketch the used per-pixel classification baselines. The PerPixelBaseline uses the pixel-level module of MaskFormer and directly outputs per-pixel class scores. For fair comparison, we de- sign PerPixelBaseline+ which adds the transformer module and mask em-
per pixel
transformer decader
per-plxel
loss
backbone
decoder
pixel
backbone
decoder
pixel
CxHHW
Kxilsw
(a) PerPixellaaselie
(b) PerPixelBaselinet
bedding MLP to the PerPixelBaseline. Thus, PerPixelBaseline+ and MaskFormer differ only in the formulation: per-pixel vs. mask classification. Note that these baselines are for ablation and we compare MaskFormer with state-of- bate-of-the-an the-art per-pixel classification models well.
5

---

4.1 Implementation details
Backbone. MaskFormer is compatible with any backbone architecture. In our work we use the stan- dard convolution-base ResNet [20] backbones (R50 and R101 with 50 and 101 layers respectively) and recently proposed Transformer-base
[27] backbones. In addition, we use the R10lc model [6] which replaces the first 7 7 convolution layer of R101 with 3 consecutive 3 x convolutions and which is popular in the semantic segmentation community [46, 7, 8, 21, 44, 10].
Pixel decoder. The pixel decoder in Figure 2 can be implemented using any semantic segmentation decoder (e.g., [8-10]. Many per-pixel classification methods use modules like ASPP [6] or PSP [46] to collect and distribute context across locations. The Transformer module attends to all image features, collecting global information to generate class predictions. This setup reduces the need of the per-pixel module for heavy context aggregation. Therefore, for Fasker we design light-weight pixel decoder based on the popular FPN [24] architecture.
Following FPN, we 2x upsample the low-resolution feature map in the decoder and sum it with the projected feature map of corresponding resolution from the backbone; Projection is done to match channel dimensions of the feature maps with 1 x 1 convolution layer followed by GroupNorm (GN) [41]. Next, we fuse the summed features with an additional x 3 convolution layer followed by GN and ReLU activation. We repeat this process starting with the stride 32 feature map until we obtain a final feature map of stride 4. Finally, we apply a single 1 x 1 convolution layer to get the per-pixel embeddings. All feature maps in the pixel decoder have a dimension of 256 channels.
Transformer decoder. We use the same Transformer decoder design as DETR [3]. The NN query embeddings are initialized as zero vectors, and we associate each query with learnable positional encoding. We use 6 Transformer decoder layers with 100 queries by default, and, following DETR, apply the same loss after each decoder. In our experiments we observe that MaskFormer is competitive for semantic segmentation with a single decoder layer too, whereas for instance-level segmentation multiple layers necessary to remove duplicates from the final predictions.
Segmentation module. The multi-layer perceptron (MLP) Figure 2 has 2 hidden layers of 256 channels to predict the mask embeddings mmask, analogously to the box head in DETR. Both per-pixel Episel and mask Emak embeddings have 256 channels.
Loss weights. We use focal loss [25] and dice loss [30] for our mask loss: Cmask Emask(m,, (m, m²) = Afocal focal (m, m²)) + Cdie (m, m²), and set the hyper-parameters to Afocal = 20.0 and Adice 1.0. Following DETR [3], the weight for the "no object" (Ø) in the classification loss is set to 0.1.
4.2 Training settings
Semantic segmentation. We use Detectron2 [42] and follow the commonly used training settings for each dataset. More specifically, we use AdamW [29] and the poly [6] learning rate schedule with an initial learning rate of 10 and a weight decay of 10 for ResNet [20] backbones, and an initial learning rate of 6 10 and weight decay of 10 2 for Swiin-Tranffrmer [27] backbones. Backbones are pre-trained on ImageNet- IK [32] if not stated otherwise. A learning rate multiplier of 0.1 applied to CNN backbones and 1.0 applied to Transformer backbones. The standard random scale jittering between 0.5 and 2.0, random horizontal flipping. random cropping as well as random color jittering are used as data augmentation [12]. For the ADE20K dataset, if not stated otherwise, we use a crop size of 512 x 512, a batch size of 16 and train all models for 160k iterations. For the ADE20K-Full dataset, we use the same setting as ADE20K except that we train all models for 200k iterations. For the COCO-Stuff- 10k dataset, we use a crop size of 640 x 640, batch size of 32 and train all models for 60k iterations. All models are trained with 8 V100 GPUs. We report both performance of single scale (s.s.) inference and multi-scale (m.s.) inference with horizontal flip and scales of 0.5, 0.75, 1.0, 1. 1.25, 1.5, 1.75. See appendix for Cityscapes and Mapillary Vistas settings.
Panoptic segmentation. We follow exactly the same architecture, loss, and training procedure as we use for semantic segmentation. The only difference is supervision: i.e., category region masks in semantic segmentation vs. object instance masks in panoptic segmentation. We strictly follow the DETR [3] setting to train our model on the COCO panoptic segmentation dataset [22] for fair comparison. On the ADE20K panoptic segmentation dataset, we follow the semantic segmentation setting but train for longer (720k iterations) and use a larger crop size (640 x 640). COCO models are trained using 64 V100 GPUs and ADE20K experiments are trained with 8 V V100 GPUs. We use

---

Table Semantic segmentation on ADE20K val with 150 categories. Mask classification- based MaskFormer outperforms the best per-pixel classification approaches while using fewer parameters and less computation. We report both single-scale (s.s.) and multi-scale (m.s.) inference results with ssddd FLOPs are computed for the given crop size. Frames-per-second (fps) is measured on V100 GPU with a batch size of 1. Backbones pre-trained on ImageNet-22K are marked with 2
method
backhone
crup size
mloU (s.s.)
mloU (m...)
Aparams.
FLOPs
Eps
OCRNer
R101e
520 520
45.3
CNN backbones
44.9
44M
DecpLabV3+ [8]
R101e
46.4
63M
14.2
R50
512 x 512
44.5 #0.5
46.7 2006
41M
53G
24.5
MaskFormer (ours)
R101
512 x 512
45.5 +0.5
47.2 4:02
60M
730
19.5
R10le
512 x 512
46.0 10.1
48.1 ::022
60M
80G
19.0
SETR [47]
WTL
512 x 512
50.3
308M
Trunsorme backbones
Swin-T
512 x 512
46.1
60M
2366
18.5
Swin-S
512 x 512
49.3
81M
2596
15.2
Swin UperNet [27,
Swin-B*
640 x 640
51.6
121M
471G
8.7
Swin-L
640 640
53,5
234M
647G
6.2
Swin
512 512
46.7 1.0.7
48.8 1:0.6
42M
55G
22.1
Swin-S
49.8 #0.4
51.0
63M
79G
19.6
MaskFormer (ours)
Swin-B
#0.2
$2.3
Swin-Rt
640
10.4
53.9 1:0.2
195G
Swin-L.
640 x 640
54.1
40.2
55.6 1:0.1
375G
7.9
Table 2: MaskFormer vs. per-pixel classification baselines 4 semantic segmentation datasets.
MaskFormer improvement is larger when the number of classes larger. We use a ResNet-50 backbone and report single scale mloU and PQ for ADE20K, COCO-Stufl and ADE20K-Full
whereas for higher-resolution Cityscapes we use a deeper ResNet- backbone following [7, 8].
Cityscupes (19 classes)
ADE20K (150 classes) PQ
COCO-Stuff (171 classes)
ADE20K-Fall (847 classes)
mloU
PQ"
mioU
mloU
PQ"
mloU
PQ
PerPiselBaasllne
77.4
58.9
39.2
21.6
32.4
15.5
12.4
5.8
PerPiselBaelline
78.5
60.2
41.9
28.3
34.2
24.6
13.9
9.0
MaskFormer (ours)
78.5 (+0.0)
63.1 (+2.9)
44.5 (+2.6)
33.4 (+5.1)
37.1
(+2.9)
28.9
(+4.3)
17.4
(43.5
11.9 (+2.9)
the general inference (Section 3.4) with the following parameters: we filter out masks with class confidence below 0.8 and set masks whose contribution to the final panoptic segmentation is less than 80% of its mask area to VOID. We report performance of single scale inference.
4.3 Main results
Semantic segmentation. In Table 1, we compare MaskFormer with state-of-the-art per-pixel classi- fication models for semantic segmentation on the ADE20K val set. With the same standard CNN backbones (e.g., ResNet [20]), MaskFormer outperforms DeepLabV3+ [8] by 1.7 mloU. MaskFormer also compatible with recent Vision Transformer [15] backbones (e.g., the Swin Transformer [27]), achieving new
of 55.6 mloU, which is 2.1 mloU better than the prior state-of-the art [27]. Observe that Faskerrmer outperforms the best per-pixel classification-base
models while having fewer parameters and faster inference time. This result suggests that the mask classification formulation has significant potential for semantic segmentation. See appendix for results on test set.
Beyond ADE20K, we further compare MaskFormer with our baselines on COCO-Stuff-100 10K, ADE20K-Full well Cityscapes in Table 2 and refer the appendix for comparison with state-of-the-art methods on these datasets. The improvement of MaskFormer over PerPixelBase- line+ is larger when the number of classes larger: For Cityscapes, which has only 19 categories, MaskFormer performs similarly well as PerPixelaaselinet; While for ADE20K-FIi Full, which has 847 classes, MaskFormer outperforms PerPixelBaseline+ by 3.5 mloU.
Although Forkerrmer shows no improvement in mloU for Cityscapes, the PQ metric increases by 2.9 SQ We find MaskFormer performs better in terms of recognition quality (RQSt) while lagging per-pixel segmentation quality (SQSt) (we refer to the appendix for detailed numbers). This observation suggests that on datasets where class recognition is relatively easy solve, the main challenge for mask classification-bast approaches is pixel-level accuracy (i.e., mask quality).
It isn't recommended to compare fps from different papers: speed is measured in different environments. DeepLabV3+ fps are from MMSegmentation [12], and Swin-UperNed fps from the original paper [27].
7

---

Table 3: Panoptic segmentation on COCO panoptic val with 133 categories. MaskFormer seam- lessly unifies semantic- and instance-level segmentation without modifying the model architecture or loss. Our model, which achieves better results, can be regarded as a box-free simplification of DETR [3]. The major improvement comes from "stuff" classes (PQ which are ambiguous to represent with bounding boxes. For MaskFormer (DETR) we use the exact same post-processing as DETR. Note, that in this setting MaskFormer performance is still better than DETR (+2.2 PQ). Our model also outperforms recently proposed Max-DeepLate [38] without the need of sophisticated auxiliary losses, while being more efficient. FLOPs are computed as the average FLOPs over 100 validation images (COCO images have varying sizes). Frames per-eeode (fps) is measured on a V100 GPU with batch size of by taking the average runtime on the entire val set including post-processe processing time. Backbones pre- trained on ImageNet-22K are marked with
method
backbone
PQ
PQ
PQ
SQ
RQ
#params.
FLOPs
Ips
CNN backbone
DETR [3]
Enc
MaskFormer (DETR)
R50 Enc
45.6
(+1.8)
39.0 (+2.7)
80.2
55.8
MaskFormer (ours)
R50 Enc
46.5
$1.0 (+2.8)
39.8 (+3.5)
80.4
56.8
45M
1816
17.6
DETR [3]
RIOI Enc
45.1
50.5
37.0
79.9
55.5
MaskFormer (ours)
R101 Enc
47.6
$2.5 (+2.0)
40.3 (43.3)
80.7
58.0
64M
248G
14.0
Transfommer buckhones
Max-S
48.4
53.0
41.5
62M
324G
7.6
Max- Max-Deepl ah [38]
Max-L
$1.1
57.0
42.2
451M
36920
Swin-T
47.7
51.7
41.7
80.4
58.3
42M
179G
17.0
Swin-S
49.7
54.4
42.6
80.9
60.4
63M
259G
12.4
MaskFormer (ours)
Swin
51.1
56.3
43.2
81.4
61.8
102M
411G
8.4
Swin B²
51.8
56.9
44.1
81.4
62.6
102M
411G
Swin-L'
212M
792G
Panoptic segmentation. In Table 3, we compare the same exact MaskFormer model with DETR [3] on the COCO panoptic val set. To match the standard DETR design, we add 6 additional Transformer encoder layers after the CNN backbone. Unlike DETR, our model does not predict bounding boxes but instead predicts masks directly. MaskFormer achieves better results while being simpler than DETR. To disentangle the improvements from the model itself and our post- prospprcessing inference strategy we our model following DETR post-proce prateesoossing (MaskFormer (DETR)) and observe that this setup outperforms DETR by 2.2 PQ. Overall, we observe a larger improvement in PQ compared to PQTh, This suggests that detecting "stuff" with bounding boxes is suboptimal, and therefore, box- based segmentation models (e.g.. Mask R-CNN [19]) do not suit semantic segmentation. MaskFormer also outperforms recently proposed Max-DeepLat [38] without the need of special network design as well as sophisticated auxiliary losses (i.e., instance discrimination loss, mask-ID cross entropy loss, and per-pixel classification loss in [38]). MaskFormer, for the first time, unifies semantic- and instance-level segmentation with the exacl same model, loss, and training pipeline.
We further evaluate our model the panoptic segmentation version of the ADE20K dataset. Our model also achieves state-of offte----hh-arr performance. We refer to the appendix for detailed results.
4.4 Ablation studies
We perform a series of ablation studies of MaskFormer using single ResNet-50 backbone [20].
Per-pixel mask classification. In Table 4, verify that the gains demonstrated by MaskFromer come from shifting the paradigm to mask classification. We start by comparing PerPixelBaseline+ and MaskFormer. The models are very similar and there only 3 differences: 1) per-pixel vs. mask classification used by the models, 2) MaskFormer uses bipartite matching, and 3) the new model uses combination of focal and dice losses as mask loss, whereas PerPixelBaselinet utilizes per-pixel cross entropy loss. First, we rule out the influence of loss differences by training PerPixelBaselinet with exactly the same losses and observing no improvement. Next, in Table 4a, we compare PerPixelBaselinet with MaskFormer trained using fixed matching (MaskFormer-fixeee) N K and assignment done based on category label indices identically to the per-pixel classification setup. We observe that MaskFormer-fe Forder-fiied is 1.8 mloU better than the baseline, suggesting that shifting from per-pixel classification to mask classification is indeed the main reason for the gains of MaskFormer. In Table 4b, we further compare MaskFormer- bipartite matching MaskkFormer-bipatticc and find bipartite matching not only more flexible (allowing to predict less masks than the total number of categories) but also produces better results.
i.e.
with MaskFormer trained with
8

---

Table Per-pixel vs. mask classification for semantic segmentation. All models use 150 queries for fair comparison. We evaluate the models on ADE20K val with 150 categories. 4a: PerPixel- Baseline+ and MaskFormer-fixg
use similar fixed matching (i.e., matching by category index), this result confirms that the shift from per-pixel to mask classification is the key. 4b: bipartite matching is not only more flexible (can make less prediction than total class count) but also gives better results.
(a) Per-pixel vs. mask classification.
(b) Fixed vs. bipartite matching assignment.
mloU
PQ
mloU
PQ
PerPixelBasselnee
28.1
MaskFormer- fixed
MaskFormer-fixed
43.7
30.3 (+2.0)
Moskerrrmerriiparttte (ours)
44.2 (+0.5)
33.4 (+3.1)
Number of queries. The table to the right shows results of MaskFormer trained with varying number of queries on datasets with dif- ferent number of categories. The model with 100 queries consistently performs the best across the studied datasets. This suggest we may not need to adjust the number of queries w.r.t. the number of categories or datasets much. Interest- ingly, even with 20 queries MaskFormer outper- forms our per-pixel classification baseline.
ADEZOK
COCO-Stul'
ADE2UK-Full
# of queries
mloU
PQ
mloU.
PQ
mIoU
PQ
PerPiselBascline+
41.9
28.3
34.2
24.6
13.9
9.0
20
42,9
32.6
35.0
27.6
14.1
10.8
43,9
32.7
35.5
27.9
15.4
11.1
44.5
33.4
37.1
28,9
16.0
33.4
37.0
28,9
15.5
36.1
1000
35.4
26.7
34.4
27.6
8.0
5.8
We further calculate the number of classes which are average present in a training sel image. We find these statistics to be similar across datasets despite the fact that the datasets have different number of total categories: 8.2 classes per image for ADE20K (150 classes), 6.6 classes per image for
10K (171 classes) and 9.1 classes per image for ADE2OK-Full (847 classes). We hypothesize that each query is able to capture masks from multiple categories.
The figure the right shows the num- ber of unique categories predicted by each query (sorted in descending or-
der) of our MaskFormer model on the
Number uniqas dasddese dasess prodicted by
validation sets of the corresponding datasets. Interestingly, the number of unique categories per query does not follow uniform distribution: some queries capture more classes than oth-
each each sunna on on validation set
(a) ADE20K (150 classes)
We try analyze how Mask- Former queries group categories, but we do not observe any obvious pattern: there are queries capturing categories with similar semanties or shapes (e.g., "house" and "building"), ), but there are also queries capturing completely dif- ferent categories (e.g., "water" and "sofa").
(b) COCO-Stufi 10K (171 classes)
130
60
100
(c) ADE20K Full (847 classes)
Number of Transformer decoder layers. Interestingly, MaskFormer with even single Transformer decoder layer already performs well for semantic segmentation and achieves better performance than our 6-layer-decode PerPixelBaselinet. For panoptic segmentation, however, multiple decoder layers are required to achieve competitive performance. Please see the appendix for a detailed discussion.
5 Discussion
Our main goal is to show that mask classification is general segmentation paradigm that could be a competitive alternative to per-pixel classification for semantic segmentation. To better understand its potential for segmentation tasks, we focus on exploring mask classification independently of other factors like architecture, loss design, or augmentation strategy. We pick the DETR [3] architecture as our baseline for its simplicity and deliberately make as few architectural changes as possible. Therefore, MaskFormer can be viewed "box-free" version of DETR.

---

Table 5: Matching with masks vs. boxes. We compare DETR [3] which uses box-based matching with two MaskFormer models trained with box- and mask- based matching respectively. To use box-based matching in MaskFormer we add to the model an additional box prediction head as in DETR. Note, that with box-based matching MaskFormer performs on par with DETR, whereas with mask-based matching it shows better results. The evaluation is done on COCO panoptic val set.
method
hackhone
matching
PQ
PQ
PQ
DETR
R50 Enc
by
R50 + Enc
by box
43.7
49.2
35.3
MaskFormer
R50 6 Ein
by mask
46.5
51.0
39.8
In this section, we discuss in detail the differences between MaskFormer and DETR and show how these changes are required to ensure that mask classification performs well. First, to achieve a pure mask classification setting we remove the box prediction head and perform matching between prediction and ground truth segments with masks instead of boxes. Secondly, we replace the compute- heavy per-query mask head used in DETR with a more efficient per-image FPN-based head make end-to-end training without box supervision feasible.
Matching with masks is superior to matching with boxes. We compare MaskFormer models trained using matching with boxes or masks in Table 5. To do box-based matching, we add to MaskFormer an additional box prediction head as DETR [3]. Observe that MaskFormer, which directly matches with mask predictions, has a clear advantage. We hypothesize that matching with boxes more ambiguous than matching with masks, especially for stuff categories where completely different masks can have similar boxes as stuff regions often spread over a large area in an image.
MaskFormer mask head reduces computation. Results in Table 5 also show that Mask performs on par with DETR when the same matching strategy is used. This suggests that the difference mask head designs between the models does not significantly influence the prediction quality. The new head, however, has significantly lower computational and memory costs in comparison with the original mask head used in DETR. In MaskFormer, we first upsample image features to get high- resolution per-pixel embeddings and directly generate binary mask predictions at a high-resolution. Note, that the per-pixel embeddings from the upsampling module (i.e., pixel decoder) are shared among all queries. In contrast, DETR first generates low rew-ussouution attention maps and applies an independent upsampling module to each query. Thus, the mask head in DETR N times more computationally expensive than the mask head in MaskFormer (where N is the number of queries).
6 Conclusion
The paradigm discrepancy between semantic- and instance-level segmentation results in entirely different models for each task, hindering development of image segmentation as a whole. We show that simple mask classification model can outperform state-of-the-arr per-pixel classification models, especially in the presence of large number of categories. Our model also remains competitive for panoptic segmentation, without a need to change model architecture, losses, or training procedure. We hope this unification spurs joint effort across semantic- and instance-level segmentation tasks.
Acknowledgments and Disclosure of F Funding
We thank Ross Girshick for insightful comments and suggestions. Work of UIUC authors Bowen Cheng and Alexander G. Schwing was supported in part by NSF under Grant #1 #1718221, 2008387, 2045586, 2106825, MRI #1725729, NIFA award 2020-67021-32799 and Cisco Systems Inc. (Gift Award CG 1377144 thanks for access to Arcetri).
References
[1] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In CVPR, 2014. 2
[2] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In
CVPR, 2018. 2,5
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- End-to-end object detection with transformers. In ECCV, 2020. 1. 4, 6, 8, 10
10

---

[4] Joao Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu. Semantic segmentation with
second-order pooling. In ECCV, 2012. 1,2
[5] Joao Carreira and Cristian Sminchisescu. CPMC: Automatic object segmentation using constrained
parametric min-cuts. PAMI, 2011, 2
[6] Liang-Chieh Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab:
Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs.
PAMI, 2018. 2. 6
[7] siang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv: 05587. 2017. 2. 6,
[8] Liang Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder decoder
with atrous separable convolution for semantic image segmentation. In ECCV. 2018. 1.
[9] Bowen Cheng. Lang-Chich Chen, Yunchao Wei, Yukun Zhu, Zilong Huang, Jinjun Xiong, Thomas S Huang. Wen-Mei Hwu, and Honghui Shi. SPGNet: Semantic prediction guidance for scene parsing. In
ICCV, 2019.
[10] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang- Chich Chen. "anoptic-Deepl.ae
simple, strong, and fast baseline for bottom-up up panoptic segmentation.
In CVPR, 2020. 6
[11] Dorin Comaniciu and Peter Meer. Robust Analysis of Feature Spaces: Color Image Segmentation. In
CVPR. 1997. 2
[12] MMSegmentation Contributors, MMSsgmeenatiin:: OpenMMLab semantic segmentation toolbox and benchmark. https: //ghthub. com/ com/open-nna oppn-mmlaaeeee mm1 aatpammmmmmmmmmeeeeeeeeeeeeeaaae wab/mmmegmentaateeeeeeeeeeeeeeee on, 2020. 6, 7
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benen- son, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2.5
[14] Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object and stuff segmenta-
tion. In CVPR. 2015. 2
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, ct al. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2021. 2.7
[16] Mark Everingham, SM Ali Eslami, Luc Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes challenge: A retrospective. IJCV, 2015. 2,5
[17] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 2
[18] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014. 1,
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 1, 3,3,4, 4,8 8 [20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016. 6. 7,8
[21] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu, CCNet:
Criss-cross attention for semantic segmentation. In ICCV, 2019. 2. 6
[22] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation.
In CVPR, 2019. 2, 3, 6
[23] Scott Konishi and Alan Yuille. Statistical Cues for Domain Specific Image Segmentation with Performance
Analysis. In CVPR, 2000. 2
[24] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature
[25] pyramid networks for object detection. In CVPR, 2017. 6
Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object
detection. ICCV, 2017. 4,
[26] Tsung-Y Yi Lin. Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2, [27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 1rXiv:2103.14000 2021. 2, [28] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta-
tion. In CVPR, 2015. 1,2
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR. 2019.
[30] Fausto Milletari, Nassir Navab, and Scyed-Ahmaa Ahmad Ahmadi. V-Net: Fully convolutional neural networks
for volumetric medical image segmentation. In 3DV, 2016. 4, 6
[31] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder. The mapillary vistas
dataset for semantic understanding of street scenes. In CVPR, 2017. 2.5
[32] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large
Scale Visual Recognition Challenge. IJC,, 2015. 6
[33]
Jianbo Shi and Jitendra Malik. Normalized Cuts and Image Segmentation. PAMI, 2000.
[34]
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv:2 105.00 05663,, 2021. 2,4
[35] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV,
2020. 3
11

---

[36]
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJCV, 2013.
[37]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017. 3, 3,4
[38]
Chich Chen. MaX -eeX-eepLaab: End-to-end Huiyu Wang. Yukun Zhu, Hartwig Adam. Alan Yuille, and Liang-Chich
panoptic segmentation with mask transformers. In CVPR, 2021. 2, 3. 4.
[39] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks, In CVPR,
2018. 2
[40] Xinlong Wang, Rufeng Zhang. Tao Kong, Lei Li, Chunhua Shen. SOLOv2: Dynamic and fast instance
segmentation. NeurlPS, 2020, 3
[41]
Yuxin Wu and Kaiming He. Group normalization. ECCV, 2018.
[42]
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick, Detectron2. https: //github..
aet.rrrreaarcc/eeectzo222, detectron2, 2019.
[43] Tete Xiao, Yingcheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene
understanding. In ECCV, 2018.
[44] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation.
In ECCV, 2020. 6. 7
[45] Yuhui Yuan, Lang Huang, Jiamyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. OCNet: Object
context for semantic segmentation. IJCV, 2021. 2
[46] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid seene parsing
network. In CVPR, 2017. 1, 2.
[47]
Sixiao Zheng. Jiachen Ln,, Hengshuang Zhao, Xiatian Zhu, 7ekun Luo, Yahiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, perspective with transformers. In CVPR, 2021. 4, 7
Rethinking semantic segmentation from sequence-to-sequenee
[48] Bolci Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scone parsing challenge 2016.
csail mit edu/ ht////ndxx_______ challenge html, 2016. 5
[49] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing
through ADE20K dataset. In CVPR, 2017. 2,5
12


Reviews:
[{"review_id": "zYD-5skU6YI", "reviewer": "Reviewer_Bzs5", "summary": "In this paper the authors present a technique to directly predict the mask and class label for semantic and panoptic segmentation. This is different from current state of the aret where per pixel labels are produced which are then process to obtain region masks and labels. The method consists of a backbone encoder which produces an image embedding, whic is fed into the backbon decoder to get pixel level embedding and to the transform decoder, the out of the transformer decode processed by an MLP provides the mask embedding. The mask embeddings and the pixel embeddings are used to get mask prdication. Whereas the class predictions are obtained throught the MLP output.\n\nThe method is tested on semantic segmentation datsets and produce comparable or better results on four datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles the problem of semantic and panoptic segmentation by proposing MaskFormer which simulatenously predict the binary mask and it's corresponding class label. The method can work on any semantic semgentation and panoptic segmentaiton backbone. Following are the main strenghts of the paper.\n\n1) The paper is well-written, the main ideas are effectively presented and it is easy to read.\n\n2) The method is of use to the community, the approach is sound and make intuitive sense.\n\n3) Despite the fact that all the modules used in the setup are form the literature, the setup for the problem is novel.\n\n4) The experiment section shows the effectiveness of the proposed method. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper presents a novel technique for simualtaneous mask segmentaiton and class labelling in segmentation problems. I found the few design choices to be a bit arbitrary and not well motivated.\n\n\n1) Line 149-150: \"Note, we empirically find it is beneficial to not enforce mask predictions to be mutually exclusive to each other by using a softmax activation.\". Line 167: \"We empirically find that marginalization over probability-mask pairs...\". Line 172-173: \"we empirically observe directly maximizing per-pixel class  likelihood for MaskFormer leads to poor performance.\"\nClaims like the obove should be motivate a bit more, probably even just showing the experiments which led to these design choices would suffice.\n\n\n2) I could not find in the paper if the masks output by the algorithm are always contiguous. If yes, why, if not how do we tackle the panoptic segmentation when lets say 2 instance of the same class are labelled in one mask form the algorithm. Are we controlling for this with large N? as this will decouple different instance of same class.\n\n\n3) \"We observe the model that predicts N = 100 masks consistently performs the best across datasets with different numbers of classes, suggesting that N is a stable hyper-parameter\". N, the number of mask seems to be an important hyperparameter for the algortihm and should not be fixed across datasets. It might work for these datasets but in general I dont see why it should work in all cases. ", "review_text": "", "extra_scores": {"ethical_concerns": "Not Applicable", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hourse"}, "extra_text": {}}, {"review_id": "vESq3tRjxx4", "reviewer": "Reviewer_Xtbm", "summary": "This paper focuses on the problem of semantic segmentation. Traditional semantic segmentation methods mainly adopt the FCN kind of structure and treat the task as per-pixel classification. While in this paper, the author reformulates the task into binary mask prediction and extra-label classification. The proposed method named MaskFormer could be utilized for both semantic segmentation and panoptic segmentation tasks, and the experimental evaluations on different datasets look promising.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The way of conducting semantic segmentation as the task of mask prediction and classification is interesting. It is different from previous lots of DL-based semantic segmentation solutions and provides some new insights in this field.\n\n2. The experimental evaluations on different datasets are promising and convincing. Even though some of the results are not SOTA, while some others are nice, e.g., the performance on ADE20K validation set and full set.\n\n3. The writing is clear and easy to follow.\n\nWeaknesses:\n1. The way of utilizing mask classification for panoptic segmentation is previously adopted in MaX-DeepLab, using conditional convolutions. In this paper, the author also uses mask classification while with transformer and bipartite matching. It is not clear which solution is better regarding the accuracy, efficiency, etc.\n\n2. It seems that most of the reported results are in validation sets. It would be nice to include some results on the test sets, like ADE20K, Cityscapes, and COCO.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "j3rZG1VttN6", "reviewer": "Reviewer_VbAq", "summary": "This paper proposes a transformer-based model, called MaskFormer for semantic segmentation task. MaskFormer treats semantic segmentation as a mask classification problem, and predicts a set of binary masks with the corresponding class labels.  The proposed approach can be easily extended for panoptic segmentaion task. Experimental evaluations for semantic segmentation and panoptic segmentation tasks demonstrate the effectiveness of proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The proposed idea of  treating semantic segmentation as a mask classification problem is interesting. \n\n2. Semantic segmentation and panoptic segmentation results are impressive. \n\nWeaknesses:\n1.\tThe proposed framework can be seen as a box-free simplification of DETR.  Hence, the advantages and novel contributions of the proposed approach compared to DETR needs to be further clarified. \n2.\tFor the panoptic segmentation task, DETR predicts boxes around both stuff and thing classes, while the proposed approach directly predicts masks for both stuff and things classes. \nLines 290-293: Please clarify why predicting masks based on boxes is inferior  to predicting masks directly. In many scenarios, the box prediction may provide some prior information indicating possible image regions containing the desired category, which may be helpful even for the stuff classes.   Hence, it will be interesting to investigate if the superior PQ^stuff performance over DETR in Table 3 is really due to this direct prediction of mask, or due to the encoder-decoder structure? \n3.\tWhy the MaskFormer  operation (matrix multiplication operation) is superior to the panoptic head in DETR for the stuff segmentation task. It will be interesting to perform an ablation study that replaces the matrix operation in MaskFormer with the  DETR panoptic head module. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no major concerns regarding the negative societal impact of this work. ", "review_text": "", "extra_scores": {"ethical_concerns": "There is no major concerns regarding ethical issues. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9 hours"}, "extra_text": {}}, {"review_id": "ce0yvsiDhzA", "reviewer": "Reviewer_kk4e", "summary": "This paper discusses per-pixel classification and mask classification for semantic segmentation. It shows that mask classification is sufficiently general to solve semantic and instance segmentation in a unified manner. Thorough experiments are conducted on several benchmarks for both semantic and panoptic segmentation. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengthens:\n1. This paper is well-written and easy to follow.\n2. The most valuable point in this paper is its highlight for pointing out the differences between per-pixel classification and mask classification. This point is not explicitly discussed before, while the authors show that mask classification is better than per-pixel classification with thorough experiments. \n3. The experiments are thorough on many benchmarks. \n\nWeaknesses:\n1. The key insight of this paper is interesting. However, this operation is not first proposed in this paper. The paper Max-DeepLab [39] similarly tackled panoptic segmentation. The authors discussed the differences in Line 81-84. However, the differences in the losses are not significant enough. \n\nOverall, this paper is well-contained with great presentation and thorough ideas, but the methodology part is incremental vs. existing work. Nevertheless, I indeed agree that the key insight here is meaningful for discussion. Hence, my initial rating is borderline acceptance. \n\nI also have several questions. \n1. The method segments images with full resolution. How about the performance with the other resolutions? A trade-off table between performance and efficiency is appreciated. \n2. The method merges semantic masks from different queries. However, during the training process, the masks are trained with one-to-one matching. A visualization or statistical analysis for how the queries are merged during the inference is beneficial for understanding the optimized models. \n3. In Line 261-262, there are several tricks for the inference process. The confidence threshold is 0.7, not 0.5. Ablation studies about the inference process are appreciated. \n\nTypos:\nLine 280: “testbed”. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are unanimously positive about this paper and recommend to accept it. I see no reason to overturn their decision.
The paper is well written, with a clear message and good results. The authors are encouraged to include some of the discussions related to Max-DeepLab and DETR in the main paper (if they can find space for it).
Decision: Accept (Spotlight)