Per-Pixel Classification is Not All You Need for
Semantic Segmentation

Contribution•mask classification模型可以同时解决语义分割和实例分割问题，并且我们发现这个模型
甚至不用做任何改动：包括模型结构(model architecture)，训练的loss，以及训练方法。

•mask classification模型在语义分割上不仅比像素分类模型的结果更好，而且需要更少的参数和计算量。

MaskFormer••

MaskFormer employs a Transformer decoder [41] to compute a set of pairs, each consistingof a class prediction and a mask embedding vector.
The mask embedding vector is used to get the binary mask prediction via a dot product withthe per-pixel embedding obtained from an underlying fully-convolutional network.MaskFormer••

MaskFormer employs a Transformer decoder [41] to compute a set of pairs, each consistingof a class prediction and a mask embedding vector.
The mask embedding vector is used to get the binary mask prediction via a dot product withthe per-pixel embedding obtained from an underlying fully-convolutional network.

MaskFormer••••

The model contains three modules :1) a pixel-level module that extracts per-pixel embeddings used to generate binary maskpredictions;2) a transformer module, where a stack of Transformer decoder layers [41] computes N persegment embeddings;
3) a segmentation module, which generates predictions
from these embeddings.

Pixel-level module•1) a pixel-level module that extracts per-pixel embeddings used to generate binary mask
predictions;backbone down-sample to 1/32.
decoder upsample 32.
此部分与大部分per-pixel classificationbased segmentation 是相同的。

Transformer module•

2) a transformer module, where a stack of Transformer decoder layers [DERT] computes N
per-segment embeddings;
input： features F (value) and N learnable positional embeddings (i.e., queries)
output：N per-segment embeddings

DERT

DERT

K,q,v->F

DERT

FK,q,v->N

DERT

FK, v->FQ->NTransformer module•2) a transformer module, where a stack of Transformer decoder layers [DERT] computes N
per-segment embeddings;
input： features F and N learnable positional embeddings (i.e., queries)
output：N per-segment embeddings

Transformer module•

2) a transformer module, where a stack of Transformer decoder layers [DERT] computes N
per-segment embeddings;
input： features F and N learnable positional embeddings (i.e., queries)
output：N per-segment embeddings
we assume N ≥ Ngt and pad the set of ground truth labels with “no object” tokens to allow
one-to-one matchingSegmentation module•

3) a segmentation module, which generates predictions

from these embeddings.

a linear classifier + a softmax activation class probability predictions
MLP(2 hidden layers )Segmentation module•

3) a segmentation module, which generates predictions

from these embeddings.

a linear classifier + a softmax activation class probability predictions
Dot product +a sigmoid activation

Segmentation module•

3) a segmentation module, which generates predictions

from these embeddings.

a linear classifier + a softmax activation class probability predictions
Dot product +a sigmoid activation

Loss••

for semantic and panoptic segmentation tasks ：a single classification loss per mask (cross entropy) and a per-pixel binary mask loss

The same as DETR: a focal loss and a dice loss

Mask-classification inference•

converts mask classification outputs
output formats.

••

For General inference :For Semantic inference :to either panoptic or semantic segmentation

MaskFormer--mask-classification inference•

converts mask classification outputsoutput formats.

•

For General inference :to either panoptic or semantic segmentation

对pixel(h,w)遍历所有N masks，计算pixel(h,w)在每个图上的
大的那个masks，即为pixel(h,w)的实际label。
注：此时每个mask代表的类别为ci，找到此值最is the most likely class label for each probability-mask pair i (N)MaskFormer--mask-classification inference•

converts mask classification outputs
output formats.

•

For General inference :•reduce false positive rates :1. filter out low-confidence predictions prior to inference
2. remove predicted segments that have large parts of their binary masks (mi > 0:5) occluded
by other predictions.

to either panoptic or semantic segmentation

MaskFormer--mask-classification inference•

converts mask classification outputs
output formats.

•For Semantic inference :to either panoptic or semantic segmentation

marginalization over probability-mask pairs yields better••对pixel(h,w) 求和其在N个mask上的，即，找到此值最大的那个class。
注：此时每个mask代表的类别已经被淡化。这时候ci是由p*m一起决定的，而之前
是只由p决定的。
N==K

Experiments--Implementation details
•••••
••

BackboneResNet backbones and Transformer-based Swin-Transformer
Pixel decoder
for MaskFormer, we design a light-weight pixel decoder based on the popular FPN architecture.
Transformer decoder
the same Transformer decoder design as DETR，
The N query embeddings are initialized as zero vectors
Loss：
focal loss ：dice loss =20：1
MLP：
2 layerExperiments--Training settings
Semantic segmentation
8 V100 GPUs
ADE20K ：512 × 512, a batch size of 16 and train all models for 160k iterations
COCO-Stuff-10k ：640 × 640, a batch size of 32 and train all models for 60k iterations
Panoptic segmentation.
COCO models are trained using 64 V100 GPUs
640 × 640, a batch size of 32 and train all models for 60k iterations
ADE20K experiments are trained with 8 V100 GPUs and 720k iterations and 640 × 640
We follow exactly the same architecture, loss, and training procedure as we use for semantic segmentation.
The only difference is supervision: i.e., category region masks in semantic segmentation vs. object instancemasks in panoptic segmentation.

•

Semantic segmentation on ADE20K val with 150 categories.

•

MaskFormer vs. per-pixel classification baselines on 4 semantic segmentation datasets.

•PerPixelBaseline+ and MaskFormer differ only in the formulation: per-pixel vs. mask
classification.

当类别越多的时候
mask classification
模型的提升越大•Panoptic segmentation on COCO panoptic val with 133 categories.

•Ablation studies

Max-Deeplab••

Max-Deeplab中，一张图会有N（最后为100）个query，每个query对应一个Mask和一个
C分类结果，然后通过C分类的得分，将不符合要求的mask弃置，达到定长预测变成变
长结果的效果，从而完成全景分割。
Max-Deeplab两个分支都用了transformer，模型大很多的重要原因。

相比Max-deeplab，maskformer更为简洁 体量小一点。
Max-Deeplab两个分支都用了transformer，而Maskformer其中一个
分支用了CNN。
auxiliary loss：
PQ-style loss
Instance discrimination
Mask-ID cross-entropy
Semantic segmentation