Paper
1 Panoptic Segmentation
2 UPSNet
3 Panoptic Feature Pyramid Networks
4 Panoptic-DeepLab

01 Panoptic Segmentation• CVPR2019• Facebook AI Research (FAIR) 、 HCI (海德堡图像处理中心)• Contribution• New task: Panoptic Segmentation• New metrics: panoptic quality(PQ)• Basic algorithm：combine PSPNet and Mask R-CNN

01 Panoptic Segmentation• Panoptic, Instance, Semantic Segmentation
Things: people, animals, tools – received the dominant share of attention
Stuff : grass, sky, road – amorphous regions of similar texture

Task format：

01 Panoptic Segmentation• Panoptic Quality• Segment Matching
IoU is greater than 0.5

True Positive

False Negative

False Positive

01 Panoptic Segmentation• Panoptic Quality• PQ Computation

匹配得分：

24𝑅𝑄 =

44 + 1 + 0.5

1

01 Panoptic Segmentation• Human Consistency Study（1）IOU threshold二分图最大权匹配得到的结果:

0.16（2）SQ vs. RQ balance01 Panoptic Segmentation• Human Consistency Study(3) Stuff vs. things(4) Small vs. large objects

01 Panoptic Segmentation• Basic algorithm：Combine PSPNet and Mask R-CNN（1）we combine those segments with semantic segmentation
results by resolving any overlap between thing and stuff classes infavor of the thing class

Mask R-CNN（2） Starting from the most confident instance.

PSPNet（3）For each instance, remove pixels which have been assigned toprevious segments, if a sufficient fraction of the segment remains,
accept the non-overlapping portion

02 UPSNet—A Unified Panoptic Segmentation Network• CVPR2019• Uber ATG
University of Toronto
The Chinese University of Hong Kong

02 UPSNet—A Unified Panoptic Segmentation Network• Contribution

单个backbone+两个轻量网络头部可以获得很好的语义和实例分割• A end-to-end network: Unified Panoptic Segmentation Network

02 UPSNet—A Unified Panoptic Segmentation Network• Instance Head (Mask R-CNN)

02 UPSNet—A Unified Panoptic Segmentation Network• Semantic Head

Deformable convolution+Multi scale feature cascadePSPNet

02 UPSNet—A Unified Panoptic Segmentation Network• Semantic Head
• Deformable convolution

02 UPSNet—A Unified Panoptic Segmentation Network• Panoptic Headmask大小为28*28，将其reshape成对应b-box
Instance head:

Xmaski：由Semantic结果里面对应通
道上对应b-box的点组成

Semantichead：train: 由ground-truth里面实例个数给定

Ninstpredict: 由mask的个数确定

02 UPSNet—A Unified Panoptic Segmentation Network• 8 Loss function• RPN
box classification : cross entropy loss
box regression : smoth L1• Instance Headbox classification : cross entropy loss
box regression : smoth L1mask segmentation : cross entropy loss

• Semantic Headpixel-wise classification loss: cross entropy loss
ROL Loss: cross entropy loss

• Panoptic Headpixel-wise classification loss: cross entropy loss

ROL Loss:02 UPSNet—A Unified Panoptic Segmentation Network• Results on COCO and Cityscapes

03 Panoptic Feature Pyramid Networks• CVPR2019• Facebook AI Research (FAIR)• Contribution• a single network for instance andsemantic segmentation tasks

03 Panoptic Feature Pyramid Networks• backbone

symmetric decoder

asymmetric, lightweight decoder
top-down pathway has only one block03 Panoptic Feature Pyramid Networks• Instance segmentation branch•

The design of FPN same channel applies a shared network branch to predict a
refined box and class label for each region.

‘×4’ denotes a stack of four consecutive convs.

03 Panoptic Feature Pyramid Networks• Semantic segmentation branchelement-wise summed.

03 Panoptic Feature Pyramid Networks• LossInstance：Semantic：

04 Panoptic-DeepLab• CVPR2020• UIUC（伊利诺伊大学厄巴纳-香槟分校）• Google Research• Contribution•

a bottom-up 、one-stage approach could deliver state-of-the-art
results on panoptic segmentation

04 Panoptic-DeepLab• Difference between the top-down and the bottom-up• the top-down methods(UPSNet, Panoptic FPN)••

Two –stage
attaching another semantic segmentation branch to Mask R-CNN• the bottom-up methods(Panoptic-DeepLab)••

One –stage
semantic segmentation + class-agnostic offset regression

04 Panoptic-DeepLab• Networkan encoder backbone
decoupled ASPP modules

decoupled decoder modules
task-specific prediction heads
语义分割每一个像素点是质心的概率

每一个像素点回归的方向04 Panoptic-DeepLab• Simple instance representation••

max-pooling with kernel size 7 保留Pool前后未改变的坐标，作为实例的中心
top-k highest confidence scores are kept (k = 200)

• Simple instance grouping04 Panoptic-DeepLab• Post processing• generate the final panoptic segmentation result by
the ”majority vote” proposed（多数投票算法）

04 Panoptic-DeepLab• Loss function

cross entropy loss

cross entropy loss

Mean Squared Error (MSE) :to minimize thedistance between the predicted heatmap and mask