Hyper-correlation Squeeze for Few-Shot Segmentation
POSTECH CSE & GSAI arxiv.2021
Fang Zhiyuan
2021.6.18

Prior knowledge
Few shot segmentation◼ Defects of Deep-learning-based Segmentation◼ Eager for large amount of samples of target◼ Cannot segment unseen objects◼ The aims of Few-shot Segmentation◼ Segment unseen object with only a few references after training

⋯⋯

⋯

⋯

Prior knowledge
Few shot segmentation◼ Divide classes 𝐶 of dataset into seen 𝐶𝑠𝑒𝑒𝑛 and unseen 𝐶𝑢𝑛𝑠𝑒𝑒𝑛 ，𝐶_𝑠𝑒𝑒𝑛 ∩ 𝐶_𝑢𝑛𝑠𝑒𝑒𝑛 = ∅◼ 𝐷𝑡𝑟𝑎𝑖𝑛 =◼ 𝑆𝑖 =𝑆𝑖 , 𝑄𝑖𝑁𝑡𝑟𝑎𝑖𝑛
𝑖=1 ， 𝐷𝑡𝑒𝑠𝑡 =𝑆𝑖 , 𝑄𝑖

𝑁𝑡𝑒𝑠𝑡
𝑖=1𝐼𝑐,𝑘 , 𝑀𝑐,𝑘 | 𝑐 ∈ 𝑐𝑠𝑒𝑒𝑛 for training 𝑒𝑙𝑠𝑒 𝑐 ∈ 𝑐𝑢𝑛𝑠𝑒𝑒𝑛 ，𝑄𝑖 =𝐼𝑐,𝑛 , 𝑀𝑐,𝑛◼ (𝑆𝑖 , 𝑄𝑖 ) called episode is the input to the model during training/testing stage
Support Set

3-way 1-shot

Query Set

Prior knowledge
Hyper-correlation◼ A collection of 4D correlation tensors
Learning visual correspondence◼ Find reliable correspondences under challenging degree of variations◼ Build correspondences upon convolutional features pretrained on classification task
◼ Exploiting different levels of convolutional features 1◼ Employ 4D convolutions on dense feature 2

1. Robust Image Matching By Dynamic Feature Selection – BMVC 2020
2. Correspondence Networks with Adaptive Neighborhood Consensus – CVPR 2020

Motivation
Few shot semantic segmentation equals to:◼ understand diverse levels of visual cues
◼ analyze fine-grained correspondence relations between the query and the support images
Proposed methods◼ multi-level feature correlation◼ efficient 4D convolutions

MethodMethod◼ Capture multi-level semantic and geometric patterns◼ Query and support images 𝐼 𝑞 , 𝐼 𝑠 ∈ ℝ3×𝐻×𝑊◼ 𝐿 pairs of intermediate feature maps𝑞

𝐹𝑙 , 𝐹𝑙𝑠◼ Mask each 𝐹𝑙𝑠 with support mask 𝑀 𝑠 ∈ 0,1𝐿𝑙=1
𝐻×𝑊:◼ 𝐹෠𝑙𝑠 = 𝐹𝑙𝑠 ⨀𝜁 𝑀 𝑠◼ ⨀ - Hadamard product◼ 𝜁 - bilinearly interpolates◼ Form a 4D correlation tensor 𝐶መ𝑙 ∈ ℝ𝐻𝑙 ×𝑊𝑙×𝐻𝑙 ×𝑊𝑙 :◼ 𝐶መ𝑙 𝑥 𝑞 , 𝑥 𝑠 = 𝑅𝑒𝐿𝑈𝑞

𝐹𝑙 𝑥 𝑞 ⋅𝐹෠𝑙𝑠 (𝑥 𝑠 )𝑞
𝐹 𝑥𝑞𝐹෠𝑙𝑠 (𝑥 𝑠 )𝑙

◼ Collect 4D tensors having the same spatial sizes 𝐶መ𝑙 𝑙∈ℒ at some pyramidal layer 𝑝𝑝◼ Form a hyper-correlation 𝐶𝑝 ∈ ℝ ℒ𝑝 ×𝐻𝑝 ×𝑊𝑝 ×𝐻𝑝 ×𝑊𝑝 by concatenating 𝐶መ𝑙 𝑙∈ℒ along channel.
𝑝Method◼ Squeeze 𝐶 = 𝐶𝑃 𝑃𝑝=1 into 𝑍 ∈ ℝ128×𝐻1×𝑊1𝑠𝑞𝑧◼ Two blocks: 𝑓𝑝and 𝑓𝑝𝑚𝑖𝑥◼ Each block consists of◼ multi-channel 4D convolution◼ Group normalization◼ ReLU𝑠𝑞𝑧◼ 𝑓𝑝squeeze last two (support) spatialdimensions◼ 𝑓𝑝𝑚𝑖𝑥 has a FPN-like fuse structure◼ Up-sample first two spatial (query)dimensions◼ element-wise addition◼ Average pooling the out of 𝑓1𝑚𝑖𝑥 on last two (support) spatial dimensions

Method◼ 2D-convolutional context decoder◼ 2D convolutions◼ ReLU◼ Upsampling layers◼ Softmax

Method
Center-pivot 4D convolution◼ 4D convolution and its limitation:◼ quadratic complexity is too high◼ over-parameterization of the high-dimensional kernel

Method
Center-pivot 4D convolution◼ Goal: design a lightweight and efficient 4D kernel◼ Main idea: disregard a large number of activations◼ Given 4D position (𝑥, 𝑥′), define two respective sets:◼ 𝑃𝑐 𝑥, 𝑥 ′ = { 𝑝, 𝑝′ ∈ 𝑃 𝑥, 𝑥 ′ ∶ 𝑝 = 𝑥}
◼ 𝑃𝑐 ′ (𝑥, 𝑥′) = { 𝑝, 𝑝′ ∈ 𝑃 𝑥, 𝑥 ′ ∶ 𝑝′ = 𝑥′}

Method
Center-pivot 4D convolution◼ Origin: 𝑐 ∗ 𝑘 𝑥, 𝑥 ′ = σ 𝑝,𝑝′ ∈𝑃(𝑥,𝑥′) 𝑐 𝑝, 𝑝′ 𝑘(𝑝 − 𝑥, 𝑝′ − 𝑥′)◼ Improve: 𝑐 ∗ 𝑘𝐶𝑃 𝑥, 𝑥 ′ = σ𝑝′ ∈𝑃(𝑥′) 𝑐 𝑥, 𝑝′ 𝑘𝑐2𝐷 (𝑝′ − 𝑥′) + σ𝑝∈𝑃(𝑥) 𝑐 𝑝. 𝑥 ′ 𝑘𝑐2𝐷′ (𝑝 − 𝑥)Experiments

Experiments

Experiments