CVPR22 – Open-Set韩坤洋Papers• VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning• Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language KnowledgeDistillation• Decoupling Zero-Shot Semantic Segmentation• Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling• ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via
Exploiting CLIP Cues
• Distinguishing Unseen from Seen for Generalized Zero-shot Learning

Open-set• Opposite to close-set, which train on class set A, test on A.
• Anomaly Detection• Identify unexpected patterns. (K class → K+1 class)• Zero-shot• Identify class of unseen object• Train with auxiliary information• Open-world• Identify unknown object → human annotation → train with new class• Open-vocabulary• Identify class of unseen object• Align embedding space between vision and text

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Dataset• AWA2, 30_475 images, 50 animal classes, 40 seen
• 85 numeric attribute values for each class• CUB, 11_788 images, 200 bird classes, 150 seen
• 312 Binary Attributes• SUN, 14_340 images, 717 scene classes, 645 seen
• 102 attributes

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Motivation• Semantic attributes annotation is labor-intensive. Recently, many works use
word embedding instead.
• Word embedding not always reflect visual similarities• Propose to discover semantic embeddings containing discriminative visualproperties

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Patch Clustering• Watershed segmentation algorithm.
• Split into around 9 parts per image.

• Cluster layer• Output k-dim vector, k clusters• Pretext task
• Patch_i and its neighbors should in same cluster (L2 neighbor)• Avoid all image patches assigned to same cluster

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Class discrimination• Cluster to class layer• CE loss, Seen classes• Semantic relatedness• Map cluster to semantic space• word2vec

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Seen semantic embedding• Get image embedding by averaging all patch• Get semantic embedding for class y_n by averaging all image belong to y_n

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Class Relation Module• Formulate the similarity between seen and unseen• Use word2vec as external knowledge• Solution• (1) directly averaging semantic embedding• (2) optimizing a similarity matrix

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Weighted Average• Retrieve nearest class neighbors in seen class• L2 distance over w2v embedding space• Semantic embedding:VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning• Similarity Matrix Optimization• Learn a similarity matrix• y_m, unseen class• r_i, similarity between y_m and i-th seen class*lower bound is 0 or -1

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning

VGSE: Visually-Grounded Semantic Embeddings
for Zero-Shot Learning

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation• Motivation• One-stage detector is more efficient• Absence of class-agnostic object proposals → Find unseen objects• Open-vocabulary one-stage detection

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation• Network• Based on ATSS, which based on FCOS
• Per-pixel prediction• Class subnet, (hid_dim, cls_num) → (hid_dim, emb_dim)

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation• IKD• Select positive samples• Crop image based on box pred, data augmentation• L1 loss supervision

CLIP

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation• IKD• Select positive samples• Crop image based on box pred, data augmentation• L1 loss supervision

CLIP

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation• GKD• Whole caption is sent into CLIP text encoder• Full image feature (5 FPN layers,• Query: text, Key: vision• Caption-Image score• <Text, Output>• N*N, Contrastive loss

𝐻𝑊patches), pool into vector
𝑁𝑁Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation• GKD• Whole caption is sent into CLIP text encoder• Full image feature (5 FPN layers,• Query: text, Key: vision• Caption-Image score• <Text, Output>• N*N, Contrastive loss

𝐻𝑊patches), pool into vector
𝑁𝑁Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation• Negative samples• Sampling negative samples will boost the performance• Identify novel foreground region as background• Sampling 10% negative samples

Open-Vocabulary One-Stage Detection with HierarchicalVisual-Language Knowledge Distillation

Decoupling Zero-Shot Semantic Segmentation• Motivation• Pixel-level zero-shot classification problem ❌
• Limited capability to integrate vision-language models• Decoupling Zero-Shot Segmentation ✅
• Class-agnostic grouping task
• Zero-shot classification task

Decoupling Zero-Shot Semantic Segmentation

Decoupling Zero-Shot Semantic Segmentation

Instance segmentation, Open-vocabulary, Weak supervision

Open-Vocabulary Instance Segmentation
via Robust Cross-Modal Pseudo-Labeling• Motivation• Aims at segment novel classes without mask annotations• Caption can’t provide details required in pixel-wise segmentation• Propose a cross-modal pseudo-labeling framework

Open-Vocabulary Instance Segmentation
via Robust Cross-Modal Pseudo-Labeling• Framework

Open-Vocabulary Instance Segmentation
via Robust Cross-Modal Pseudo-Labeling• Input• Region proposals (ROI Align)• Nouns from image caption• Target• Mask• Class (Embedding similarity)*background embedding is full of 0

Open-Vocabulary Instance Segmentation
via Robust Cross-Modal Pseudo-Labeling• Mask supervision• Simple BCE, bad teacher prediction• Assumption: pseudo mask is corrupted by a Gaussian noise, var can be estimated*Noise is per-pixel predicted*Noise is predicted based on object region

ProposalCLIP: Unsupervised Open-Category Object
Proposal Generation via Exploiting CLIP Cues
• Motivation• Require a large number of bounding box annotations• Can only generate proposals for limited object categories

ProposalCLIP: Unsupervised Open-Category Object
Proposal Generation via Exploiting CLIP Cues

ProposalCLIP: Unsupervised Open-Category Object
Proposal Generation via Exploiting CLIP Cues
• Select 60% low-similarity-entropy initial proposal

ProposalCLIP: Unsupervised Open-Category Object
Proposal Generation via Exploiting CLIP Cues
• Merging• Semi-supervise• intersection between top 1% low-entropy proposals and top 5% high-initialscore proposals

Zero-shot, GANDistinguishing Unseen from Seenfor Generalized Zero-shot Learning