Referring
SegmentationMengxue

ECCV 2016

ContentsContents
Paper List :Segmentation from Natural Language Expressions
Recurrent Multimodal Interaction for Referring Image Segmentation
MAttNet: Modular Attention Network for Referring Expression Comprehension
Referring Expression Object Segmentation with Caption-Aware Consistency
PhraseCut: Language-Based Image Segmentation in the Wild

Datasets
ReferIt | RefCOCOg | RefCOCO | RefCOCO+ | VGPhraseCut

1

ECCV 2016

Word embedding→

Loss

2

ICCV 2017

Segmentation from Natural Language Expressions:· Remember all information → Find the matching region· human：image-sentence-image reading sequence 来回浏览确定目标区域
e.g. “The man on the right wearing blue.”
Step: 1.Find out pixels →“the man”
2.Delete pixels not “on the right”
3.Delete pixels not “wearing blue”→ RMI

CNN+LSTM→RMI

language-only LSTM

RMI

Experiments

3Target object: a red ball·a red ball among 10 black balls ”the red ball” √·a red ball is placed among 3 other red balls“red ball on the right”  location information·a red ball is placed among 100 other red balls“red ball next to the cat” most distinguishing information

MAttNetCVPR 2018

Language Attention Network

Visual Modules

Visual Modules--Subject Module
Attribute Prediction | Phrase-guided Attentional Pooling| Matching Function

Visual Modules--Location Module

Visual Modules--Relationship Module

Loss Function

Results: Referring Expression Comprehension

Precision@0.5 is the percentage of expressions where the IoU of the predicted segmentation and ground-truth is at least 0.5.

4BMVC 2019

Segmentation from Referring Expression▪ Language Encoder

Matt bi-directional LSTM

▪ Visual Encoder

proposal-based Mask R-CNN、ResNet-101▪ Spatial-aware Dynamic Filters▪ Baseline ObjectiveDynamic Filter Network

Filter→Kernel▪ 1. 模型参数(model parameters): 模型参数表示只在训练过程中被更新,在预先被初始化的层参数, 对于所有的样本都是相同的.
▪ 2. 动态产生参数(dynamically generated parameters) :是由样本所决定的, 是动态生成的，不需要初始化. filtergenerating 网络输出动态产生参数, 同时该网络还是有一部分的模型参数.

Spatial-aware Dynamic Filter• Consider the entire image/only be able tocatch the global structure but ignore spatially distributed objects→spatial-aware dynamic filters: including up,down,left,right,horizontal and vertical
middle regionssix additional fully connected layers

Loss:binary cross-entropy loss Lres → R & ground-truth object mask

Loss

▪ Lroi include: classification loss, bounding box loss, mask loss(defined in Mask R-CNN)▪ Lres (in Spatial-aware Dynamic Filter)

A Joint Framework
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention[23]▪ Caption-aware Consistency▪ Overall Objective

location

segmentation

5CVPR 2020▪ a modular approach for combining visual cues related to categories, attributes, andrelationships▪ a systematic approach to improving the performance on rare categories and attributes byleveraging predictions on more frequent ones

Review▪ Dataset▪ Segmentation from Natural Language Expressions▪ Recurrent Multimodal Interaction for Referring Image Segmentation▪ MAttNet: Modular Attention Network for Referring Expression Comprehension▪ Referring Expression Object Segmentation with Caption-Aware Consistency▪ PhraseCut: Language-Based Image Segmentation in the Wild

Thank you