CVPR 2022

Referring-related Tasks
MengxueIntroduction▪ What is Referring? → 指代 → 用文本描述指代特定目标
▪ Referring Image Segmentation: given a sentence, segment corresponding instance.

Introduction
Referring Image Grounding

Detection

Referring Image Segmentation

Segmentation

Referring Video Segmentation

Video Segmentation

Unsupervised Referring Grounding

Unsupervised

Referring Image Matting

Matting

Language-driven Other Visual Tasks

Image Editing/Generation…

1/ DETR-like frameworks▪ Referring Image Grounding▪ Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning▪ Referring Image Segmentation▪ CRIS: CLIP-driven Referring Image Segmentation▪ Referring Video Segmentation▪ Language as Queries for Referring Video Object Segmentation

DETR-like frameworks▪ DETR▪ Vision Encoder▪ Language Encoder▪ V-L Interaction▪ Detection Regression
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding, ICCV21

Referring Image Grounding▪ Vision Encoder▪ Language Encoder▪ V-L Interaction▪ Detection RegressionReferring Image GroundingReferring Image GroundingReferring Image Segmentation▪ Naïve: directly finetune CLIP

▪ {gt_mask, Sigmoid(pixel_feat·text_feat)}Referring Image Segmentation

Referring Image Segmentation

Referring Image Segmentation

Referring Image Segmentation

Referring Video Segmentation▪ Bottom-up▪ Early V-L merge + FCN-like segment▪ Top-down▪ Instance segment + Late V-L merge▪ Pick out the instance mask

▪ Languages as queries▪ Early V-L merge in DETR-like modelReferring Video Segmentation

Referring Video Segmentation

Referring Video Segmentation

Region similarity (J) / Contour accuracy (F)2/ Swin-Transformer-combined frameworks▪ Referring Image Segmentation▪ (CVPR22) LAVT: Image-Aware Vision Transformers for Referring Image Segmentation▪ Referring Image Grounding▪ (ICCV21) TransVG: End-to-End Visual Grounding with Transformers▪ (CVPR22) Shifting More Attention to Visual Backbone: Query-modulated Refinement
Network for End-to-End Visual GroundingReferring Image Segmentation

Swin-Transformer

Referring Image Grounding+Referring Image Grounding3/ Other Referring-related Tasks▪ Unsupervised VG▪ (CVPR22) Generating Pseudo Language Queries for Visual Grounding▪ Referring Image Matting▪ (NIPS22) Referring Image MattingUnsupervised Referring Grounding(c) Unsupervised VG

▪ Nouns: off-the-shelf object detector▪ Attribute Classifier▪ Bottom-up and top-down attention for image captioning and visual question answering.

▪ Relationships▪ 1. Horizontal(left, middle and right) 2. Vertical(top and bottom) 3. depth(front and behind)Referring Image MattingReferring Image Matting▪ Pre-processing and filtering▪ AM-2k, P3M-10k, and AIM-500▪ Annotate the category names of entities▪ Mask RCNN to detect and manually check and correct▪ Adopt WordNet to generate synonyms and manually check and replace with more reasonable
ones▪ Annotate the attributes of entities▪ Color: cluster all the pixel values of the foreground image, find out the most frequent value,
and match it with the specific color in webcolors▪ Gender, age: Age and gender classification using convolutional neural networks. 2015 (CVPRW)▪ Clothes type: Mmfashion: An open-source toolbox for visual fashion analysis. In ACM

Multimedia 2021, Open Source Software Competition, 2021

▪ Salient/Transparent or not: Deep automatic natural image matting. IJCAI-21

Referring Image MattingReferring Image Matting

Summary▪ There is already good performance on classical tasks:▪ Referring Image Grounding▪ Referring Image Segmentation▪ Referring Video Segmentation▪ Model frameworks tend to be similar▪ Explore other settings like unsupervised▪ Explore language-driven other vision tasks like matting/editing/generation

Examples from “Blended Diffusion for Text-Driven Editing of Natural Images”

Thanks