Video Object Segmentation & Video Instance Segmentation

1

Video Object Segmentation using Space-Time Memory Networks

2

Fast End-to-End Embedding Learning for Video Object Segmentation

3
4

Collaborative Video Object Segmentation by Multi-Scale
Foreground-Background Integration

Video Instance Segmentation

0Video Object Segmentation (视频对象分割)• Target• 将前景对象与背景区域进行分离的二值标记• Unsupervised• 不需要任何手动注释• 通常基于物体运动与周围环境不同进行分割• Semi-supervised• 利用第一帧的Mask 跟踪和分割给定的对象

二者都将目标对象视为一般对象，而不关心语义类别

0Dataset• DAVIS(CVPR2016)• DAVIS-2016数据集为单对象分割数据集，包含30个训练集，20个验证集• DAVIS-2017数据集为多对象分割数据集，一共有90个视频序列，包含60个训练视频，30个验证视频，验证集含59个对象组成0Dataset• Youtube-VOS （for Video Object Segmentation ）• 由4453个高分辨率的YouTube视频和94个常用对象类别组成。
• 验证集由474个视频组成，包括91个对象类别，其中65个为训练集中类别，其余26个为不可见类别对象。
• 每个视频的长度约为3到6秒。
• 30fps的帧速率每5帧手动跟踪对象边界• Youtube-VIS(ICCV2019) （for Video Instance Segmentation ）• 由2883个高分辨率的YouTube视频组成，含40个类别的标签集

0Metrics –( A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation)• Region Similarity J(区域相似度)• Contour Accuracy F(轮廓精确度)

将Mask 看成一系列闭合轮廓的集合，并计算基于轮廓的 F 度量

MaskFNFPGround Truth

𝑇𝑃𝑃𝑐 =𝑇𝑃 + 𝐹𝑁
𝑇𝑃𝑅𝑐 =𝑇𝑃 + 𝐹𝑃

01 Video Object Segmentation using Space-Time Memory Networks• ICCV2019• Motivation••

Available cues (e.g. video frame(s) with object masks) become richer.
However, the existing methods are unable to fully exploit this rich source of information.

• Contribution•

Propose a novel solution for semi-supervised video object segmentation leveraging memory networks

01 Video Object Segmentation using Space-Time Memory Networks• Overview of the framework• 2 parts :previous frame with mask and current frame• 4 feature maps :Key feature maps(2) + value feature maps(2)01 Video Object Segmentation using Space-Time Memory Networks• Encoder Networks

01 Video Object Segmentation using Space-Time Memory Networks• STM Read
• Soft weights are first computed by measuring the similarities between all pixels of the query key map.
• Extract information from.
• Concatand.

Compare with non-local :01 Video Object Segmentation using Space-Time Memory Networks• Decoder• The read output is first compressed to have 256 channels by a convolutional layer and a residual block.
• A number of refinement modules upscale the feature map by a factor of two.
• The decoder estimates the mask in 1/4 scale of the input image.

01 Video Object Segmentation using Space-Time Memory Networks• 2 Stage Training• 1)Pre-training on images:

A video clip that consists of 3 frames is generated by applying random affine transforms• 2)Main training on videos.

Sample 3 temporally ordered frames from Youtube-VOS or DAVIS-2017. The maximum number of frames tobe skipped is gradually increased from 0 to 25.

• Inference• The first and the previous frame with object masks are the most important.
• For the intermediate frames, we simply save a new memory frame every 5 frames.

01 Video Object Segmentation using Space-Time Memory Networks• The training process is more complicated, need large image data sets
• Multi-object Segmentation need a post-processing step

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• CVPR 2019

• Motivation•

Many of the recent successful methods for video object segmentation (VOS) are overly complicated• Contribution••

Design a simple, fast, end-to-end, strong Network without fine-turning
The network only need one-stage training and can handle Multi-object Segmentation without postprocessing step

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• Overview of the framework• Backbone features• Global matching distance map• Local matching distance map• Previous frame mask

42
3

1

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• 1. Backbone feature map• DeepLab v3+
depthwise separable convolutions (深度可分离卷积)batch normalization
Atrous Spatial Pyramid Pooling

The depthwise separable convolutions divide the original convolution layer into two parts, which can achieve the same
purpose and reduce the number of parameters

𝑁𝑑𝑒𝑝𝑡ℎ𝑤𝑖𝑠𝑒 = 3 ∗ 3 ∗ 3 = 27

𝑁𝑐𝑜𝑛𝑣 = 4 ∗ 3 ∗ 3 ∗ 3 = 108

𝑁𝑑𝑒𝑝𝑡ℎ𝑤𝑖𝑠𝑒 = 1 ∗ 3 ∗ 4 = 12𝑁sum = 27 + 12 = 39

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• 2. Global Matching and Local Matching
Embedding feature vectors
backbone• DeepLab v3+
depthwise separable convolutions (深度可分离卷积)batch normalization
Atrous Spatial Pyramid Pooling• 3x3 conv + 1x1 conv
• Channel = 100

p和q来自不同的输入。根据q的不同，计算两种distance map：
Global distance map和 Local distance map

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• Semantic Embedding.
• Global Matching• Local Previous Frame Matching

A given window size K , we only comprises (𝟐 ∗ 𝑲 + 𝟏)𝟐 elements

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• 3.Previous Frame Predictions

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• Dynamic Segmentation Head

• 4 depthwise separable convolutional layers:a dimensionality of 256, a kernel size of 7 × 7• A ReLU activation function.

02 FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation• Training Details• Backbone
Using weights for DeepLabv3+ which were pre-trained on COCO

• Global Matching
Randomly subsample the pixels from the first frame to contain at most 1024 pixels per object• Local Previous Frame MatchingK = 15• Dataset• 3 frames• DAVIS 2017 training set (60 videos) and YouTube-VOS training set (3471 videos).

• LossBootstrapped cross entropy loss, which only takes into account the 15% hardest pixels for calculating theloss

03 Scale Foreground-Background Integration（ECCV2020）

CFBI：Collaborative Video Object Segmentation by Multi-• Motivation• Background should be equally treated• Contribution• . Global Matching and Local Matching
Forground and background
Pixel-level matching and instance-level embedding• FPN multiscale matching.
• A Atrous Matching (AM) algorithm, which cansignificantly save computation and memory usage ofmatching processes.

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Collaborative Pixel-level Matching

are trainable background bias and foreground bias
• Compared with FEELVOS

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Collaborative Pixel-level Matching• Foreground-Background Global Matching(from t = 0 frame)

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Collaborative Pixel-level Matching• Foreground-Background Multi-Local Matching(from t = T-1 frame)

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Collaborative Instance-level Attention• Get guidance vector
12

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Collaborative Instance-level Attention• Attention mechanism
1.

Concat the guidance vector

2.

a fully-connected (FC) layer

3.

a non-linear activation function

4.

Give each channel a weight

Leverage a full scale of foreground-background information to guide the prediction further

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• An overview of CFBI

① Backbone 的feature② 前一帧的Mask③ Local Matching④ Global Matching03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Multi-scale Matching(CFBI+)

stridesChannel

Window sizes

432{4, 8, 12, 16, 20, 24}864{2, 4, 6, 8, 10, 12}16128{4, 6, 8, 10}

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Atrous Matching (AM)MatchingAtrous Matching (AM)

03 Scale Foreground-Background Integration

CFBI：Collaborative Video Object Segmentation by Multi-• Training details

04 Video Instance Segmentation
ICCV2019• Contribution• 1) A new computer vision task•

The goal of this new task is simultaneous detection, segmentation and tracking of instances invideos• 2) Propose a large-scale benchmark called YouTube-VIS••Based on YouTube-VOS2883 high-resolution videos and 40 common categories.

• 3) Propose a novel algorithm called MaskTrack R-CNN for this task•

A tracking branch to Mask R-CNN to jointly perform the detection, segmentation and tracking
tasks simultaneously.

04 Video Instance Segmentation• An overview of MaskTrack R-CNN

04 Video Instance Segmentation• New Tracking Branch• Two fully connected layers.
The first fully connected layer transforms the 7 × 7 × 256 input feature maps to 1-D 1024 dimensions.
The second fully connected layer also maps its input to 1-D 1024 dimensions.

04 Video Instance Segmentation• Combining Other Cues𝑏𝑖 , 𝑐𝑖 and 𝑠𝑖 denote its bounding box prediction, category label and detection score

04 Video Instance Segmentation• Evaluation Metrics• 1) IoU• 2) AP(average precision)••

AP is averaged over multiple intersection-over-union (IoU) thresholds
IoU thresholds ：10 IoU thresholds from 50% to 95% at step 5%• 3) AR(average recall)•AR is defined as the maximum recall given some fixed number of segmented instances pervideo.