Pyramid Fusion Transformer for SemanticSegmentation

MaskFormer•

MaskFormer employs a Transformer decoder to compute a set of pairs, each consisting of aclass prediction and a mask embedding vector.

MaskFormer••••

The model contains three modules :1) a pixel-level module extracts per-pixel embeddings used to generate binary mask2) a transformer module, computes N per-segment embeddings;
3) a segmentation module

Single scaleMask2FormerQK,VHigh-resolution features improve model performance,
especially for small objects

Transformer-based Pyramid Fusion Transformer (PFT)•

1. multi-scale transformer decoder•

2. to avoid heavy computation , Neither intra-scale nor cross-scale pixel-to-pixel attention isused in our PFT•

3. The final prediction is the average of the per-scale predictions in the logit space.

••

we recurrently stack three types of attention layers:(1) an intra-scale query self-attention layer that conducts conventional self-attention
between queries within the same scale,(2) a novel cross-scale inter-query attention layer to efficiently communicate scale-aware
information using the limited number of 4K queries of the 4 different scales,(3) an intra-scale query-pixel cross-attention layer that aggregates semantic information fromflattened sequences of pixel tokens.

••

we recurrently stack three types of attention layers:(1) an intra-scale query self-attention layer that conducts conventional self-attention
between queries within the same scale,(2) a novel cross-scale inter-query attention layer to efficiently communicate scale-aware
information using the limited number of 4K queries of the 4 different scales,(3) an intra-scale query-pixel cross-attention layer that aggregates semantic informationfrom flattened sequences of pixel tokens.

Intra-scale query self-attention•(1) an intra-scale query self-attention layer that conducts conventional self-attention
between queries within the same scale,•••conducts self-attention only between category queries within the scale .
The category queries Qs are zero-initialized at the first layer and updated by the stacked
attention layers, the learnable positional encodings Ps are learned and shared at different
depths.

••

we recurrently stack three types of attention layers:(1) an intra-scale query self-attention layer that conducts conventional self-attention
between queries within the same scale,(2) a novel cross-scale inter-query attention layer to efficiently communicate scale-aware
information using the limited number of 4K queries of the 4 different scales,(3) an intra-scale query-pixel cross-attention layer that aggregates semantic information fromflattened sequences of pixel tokens.

Cross-scale inter-query attention•

the number of such scale-aware queries in each scale is much smaller than the number of all
visual tokens in each scale••

we recurrently stack three types of attention layers:(1) an intra-scale query self-attention layer that conducts conventional self-attention
between queries within the same scale,(2) a novel cross-scale inter-query attention layer to efficiently communicate scale-aware
information using the limited number of 4K queries of the 4 different scales,(3) an intra-scale query-pixel cross-attention layer that aggregates semantic information fromflattened sequences of pixel tokens.

Intra-scale query-pixel cross-attention.
•

Within each scale , we fixed sinusoidal positional encodings Pssine to the pixel tokens Ps,
because of the too large number of pixel tokens.

Training losse•

Generating Segmentation Maps.
•probability logits

Each scale•binary category masks•

Sementic masks

averageExperiments on the ADE20K dataset

Experiments on the PASCAL-Context dataset.

Experiments on the COCO-Stuff-10K dataset

Ablations on weight sharing for different modules