Few Shot Segmentation
方致远 张泽康

2021/5/14

Motivation

Defects of Deep-learning-based Segmentation⋯◼ Eager for large amount of samples of target◼ Cannot segment unseen objects

The aims of Few-shot Segmentation⋯⋯◼ Segment unseen object with only a few
references after training⋯2

Implementation◼ Divide classes 𝐶 of dataset into seen 𝐶𝑠𝑒𝑒𝑛 and unseen 𝐶𝑢𝑛𝑠𝑒𝑒𝑛 ，𝐶𝑠𝑒𝑒𝑛 ∩ 𝐶𝑢𝑛𝑠𝑒𝑒𝑛 = ∅◼ 𝐷𝑡𝑟𝑎𝑖𝑛 =◼ 𝑆𝑖 =𝑆𝑖 , 𝑄𝑖

𝑁𝑡𝑟𝑎𝑖𝑛𝑖=1 ， 𝐷𝑡𝑒𝑠𝑡 =𝑆𝑖 , 𝑄𝑖

𝑁𝑡𝑒𝑠𝑡
𝑖=1𝐼𝑐,𝑘 , 𝑀𝑐,𝑘 | 𝑐 ∈ 𝑐𝑠𝑒𝑒𝑛 for training 𝑒𝑙𝑠𝑒 𝑐 ∈ 𝑐𝑢𝑛𝑠𝑒𝑒𝑛 ，𝑄𝑖 =𝐼𝑐,𝑛 , 𝑀𝑐,𝑛◼ 𝑆𝑖 , 𝑄𝑖 called episode is the input to the model during training/testing stage
Support Set

Query Set

3-way 1-shot3

Implementation

Decoder

Encoder

Support Set

feature embedding

Prediction

Query Set

feature embedding

4

Dataset & MetricDataset：◼ Pascal VOC 2012
◼ MS COCO
Metric：◼ Mean Intersection over Union (mIoU)◼ 𝑚𝐼𝑜𝑈 =𝑇𝑃𝐹𝑃+𝐹𝑁+𝑇𝑃5

Paper list

◼ SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation (2018)◼ PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment (2019)◼ Prior Guided Feature Enrichment Network for Few-Shot Segmentation (2020)◼ Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (2021)◼ Edge-Labeling Graph Neural Network for Few-shot Learning (2019)6

SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation

SG-One

Key points:◼ Masked Average Pooling strategy◼ Build relationship between support andquery with cosine similarity◼ Unified network structure8

SG-One𝑣𝑖 =σ𝑤,ℎ𝑥=1,𝑦=1 𝑌𝑥,𝑦 ∗ 𝐹𝑖,𝑥,𝑦σ𝑤,ℎ𝑥=1,𝑦=1 𝑌𝑥,𝑦Unified network
9

SG-OneK-shot Testing:◼ Ensemble the K segmentation mask with equation:
1 ,𝑌
2 ,⋯,𝑌𝐾 )෠𝑥,𝑦෠𝑥,𝑦𝑌෠𝑥,𝑦 = 𝑚𝑎𝑥(𝑌෠𝑥,𝑦◼ Average the K representative vectors

10

PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment

PANet

Key points:◼ Propose a non-parametric metric learning to extract
knowledge and perform segmentation◼ Design a Prototype Alignment structure to fully
exploit knowledge from the support

12

PANet෡𝑞(𝑥,𝑦) = argmin 𝑀
෩ (𝑥,𝑦)𝑀𝑞;𝑗𝑗

13

PANet14

Prior Guided Feature Enrichment Network for Few-Shot Segmentation

PFENetPFENet mainly wants to solve following two problems:◼ High-level features using in a few-shot model may cause performance drop
◼ Using fixed high-level features to yield the prior mask
◼ Spatial Inconsistency◼ Proposing a FPN-like Feature Enrichment Module with residual blocks

16

PFENetcos 𝑥𝑞 , 𝑥𝑠 =𝑐𝑞 =𝑥𝑞𝑇 𝑥𝑠𝑞, 𝑠 ∈ {1,2, ⋯ , ℎ𝑤}
𝑥𝑞 𝑥𝑠max (cos(𝑥𝑞 , 𝑥𝑠 ))𝑠∈ 1,2,⋯,ℎ𝑤

𝐶𝑄 = 𝑐1 , 𝑐2 , ⋯ , 𝑐ℎ𝑤 ∈ ℝℎ𝑤×1

17

PFENet18

PFENet19

Adaptive Prototype Learning and Allocation for Few-Shot Segmentation

Adaptive Prototype Learning and Allocation
for Few-Shot Segmentation(CVPR2021)

M o t i va t i o nUsing one prototype to represent all theinformation may lead to ambiguities

21Adaptive Prototype Learning and Allocation
for Few-Shot Segmentation

22

Adaptive Prototype Learning and Allocation
for Few-Shot Segmentation

23

Adaptive Prototype Learning and Allocation
for Few-Shot Segmentation
S u p e r- p i xe l
Serve super-pixel centroids as prototypes◼ concatenate the coordinates of each pixel with the support feature map◼ Get foreground feature map with mask

◼ Iteration

24Adaptive Prototype Learning and Allocation
for Few-Shot SegmentationP r o to t y p e A l l o c a t i o n
Allocation byCosine Similarity

25

Adaptive Prototype Learning and Allocation
for Few-Shot Segmentation
Adaptability
Compute the number of prototypes with the number of pixels in theforegroundK-shot setting
Directly get all prototypes from K-shot𝑖
𝑁𝑠𝑝 = 𝑚𝑎𝑥 σ𝑘𝑖=1 𝑁𝑠𝑝

26

Adaptive Prototype Learning and Allocation
for Few-Shot Segmentation27

Edge-Labeling Graph Neural Network for Few-shot Learning

Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)G r a p h N e u r a l N e t wo r k （ G N N ）GCN[1]Kipf and M. Welling. Semi-supervised classification with graph convolutional
networks. ICLR, 2017.

29

Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)

Not suitable for few-shot learning

30

Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)31

Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)

Initial edge label

Node update

Edge update

32Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)33

Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)34

Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)
Transductive inference
Classify the entire test set at once to alleviate the low-data problem[2]Liu Y, Lee J, Park M, et al. Learning to propagate labels: Transductive
propagation network for few-shot learning[C]//7th International Conference on
Learning Representations, ICLR 2019. 2019.

35

Edge-Labeling Graph Neural Network for Fewshot Learning (CVPR2019)36