Vision & Learning Seminar
Instance Segmentation - CVPR2022
Feng Zhu

UTS CRICOS 00099F

Instance Segmentation
Segmentation task: instance segmentation, semantic segmentation and panoptic segmentation.
Instance segmentation is the task of detecting
and delineating each distinct object of interest
appearing in an image.

Semantic segmentation, or image segmentation, isthe task of clustering parts of an image together which
belong to the same object class.

Extend on object detection, need to predict masksof each object instead of bounding box.

1

Instance Segmentation - CVPR2022
Trends:1. Less fully-supervised work, more weakly-supervised and self-supervised work.
2. More video segmentation work.

3. More panoptic segmentation work and 3D instance segmentation work.
4. Focused on boundary refinement and speed-up.
5. Open world and open vocabulary instance segmentation, with language (e.g. CLIP).

2

Instance Segmentation - CVPR2022
Paper sharing:
1. AdaMixer: A Fast-Converging Query-Based Object Detector. (oral, detailed)
2. Sparse Instance Activation for Real-Time Instance Segmentation. (detailed)

3. Mask Transfiner for High-Quality Instance Segmentation. (detailed)
4. Pointly-Supervised Instance Segmentation. (oral, brief)
5. FreeSOLO: Learning to Segment Objects without Annotations. (brief)

3

AdaMixer: A Fast-Converging Query-Based Object Detector (oral)
Ziteng Gao1 Limin Wang1 Bing Han2 Sheng Guo2
1State Key Laboratory for Novel Software Technology, Nanjing University, China
2MYbank, Ant Group, China
Motivation: The recent query-based object detectors still
suffers from slow convergence, limited
performance, and design complexity of extra networks
between backbone and decoder.

The paper finds the key to these issues is the adaptabilityof decoders for casting queries to varying objects.

4AdaMixer: A Fast-Converging Query-Based Object Detector (oral)
Revisit previous query-based method
1. DETR and Deformable DETR can not be adaptive to decode content, linear projection of features;
2. Sparse RCNN use adaptive bbox proposal and dynamic interactive head to decode content, but still
restricted to bbox and specific feature level of FPN.

Ideas:
1. The decoder should adaptively decide which features to sample regarding the query, and adpative inboth spaital and scales dimension.
2. How to adaptively decode the features? To capture correlation in spatial and channel dimension.
5

AdaMixer: A Fast-Converging Query-Based Object Detector (oral)

Method:1. 3D feature sampling: treat multi-scale feature maps as 3D feature space, make query able tohandle both location and scale variations. (group sampling)
2. Adaptive mixing: channel and spatial mixing to the sampled features with dynamic kernels.
6

AdaMixer: A Fast-Converging Query-Based Object Detector (oral)
Experiments

Fast convergence: 1 X training scheme outperforms SOTA.
7

AdaMixer: A Fast-Converging Query-Based Object Detector (oral)Experiments
Ablation Study8

AdaMixer: A Fast-Converging Query-Based Object Detector (oral)
Experiments
SOTA comparison: bbox AP 51.3

9

Sparse Instance Activation for Real-Time Instance Segmentation
Tianheng Cheng 1, 2 Xinggang Wang 1† Shaoyu Chen 1, 2 Wenqiang Zhang 1
Qian Zhang 2 Chang Huang 2 Zhaoxiang Zhang 3 Wenyu Liu 1
1 School of EIC, Huazhong University of Science & Technology 2 Horizon Robotics
3 Institute of Automation, Chinese Academy of Sciences (CASIA)

Motivation: Previous instance segmentation methods
heavily rely on object detection and perform mask prediction
based on bounding boxes or dense centers, which is quite
time-consuming. Moreover, the post-processing operationsuch as NMS takes much time.
This paper aims to design a new segmentation paradigm forreal-time instance segmentation

10

Sparse Instance Activation for Real-Time Instance Segmentation
Method:The paper exploits a sparse set of instance activation maps (IAM) to highlight informative object
regions, which is motivated by CAM widely used in weakly-supervised object localization.

11

Sparse Instance Activation for Real-Time Instance Segmentation
Method:12

Sparse Instance Activation for Real-Time Instance Segmentation
Method:Advantages of using IAM:
(1) it highlights discriminative instance pixels, suppresses obstructive pixels, and conceptually
avoids the incorrect instance feature localization problems in center-/region-based methods;(2) it aggregates instance features from the whole image and offers more contexts;(3) computing instance features with activation maps is rather simple without extra operation like
RoI-Align.
Additional methods:(1) Location-Sensitive Features: similar to CoordConv;(2) IOU-aware Objectness.
13

Sparse Instance Activation for Real-Time Instance Segmentation
Experiments
Comprision with SOTA on COCO

14

Sparse Instance Activation for Real-Time Instance Segmentation
Experiments
Ablation study
Different network of Fiam

15

Sparse Instance Activation for Real-Time Instance Segmentation
Experiments
Visualization of IAM

16

Mask Transfiner for High-Quality Instance Segmentation
Lei Ke1,2 Martin Danelljan1 Xia Li1 Yu-Wing Tai3 Chi-Keung Tang2 Fisher Yu1
1ETH Zurich ¨ 2HKUST 3Kuaishou Technology

Motivation: Two-stage and query-based instancesegmentation methods have achieved remarkable results.
However, their segmented masks are still very coarse.
Large gap between bbox AP and Mask AP.

Accurate mask prediction is highly challenging, due to theneed for high-resolution deep features, which demands
large computational and memory costs.

17

Mask Transfiner for High-Quality Instance Segmentation
Method:1. Identify incoherent regions (error-prone regions),
defined by the loss of information when downsamplingmask. This paper builds a hierarchical quadtree torepresent and process the incoherent image pixels at
multiple scales.
2. To refine the mask labels of the incoherent quadtree
nodes, they design a refinement network based on
the transformer instead of convolutional networks
because they require operating on uniform grids.

18

Mask Transfiner for High-Quality Instance Segmentation
Method:Properties of Incoherent Regions: occupying 43% of all wrongly predicted pixels, while only taking
14% to the corresponding bounding box areas.

19

Mask Transfiner for High-Quality Instance Segmentation
Method:20

Mask Transfiner for High-Quality Instance Segmentation
Experiments
Ablation study21

Mask Transfiner for High-Quality Instance Segmentation
Experiments
Ablation study

22

Mask Transfiner for High-Quality Instance Segmentation
Experiments
Compare with SOTA on COCO23

Mask Transfiner for High-Quality Instance Segmentation
Lei Ke1,2 Martin Danelljan1 Xia Li1 Yu-Wing Tai3 Chi-Keung Tang2 Fisher Yu1
1ETH Zurich ¨ 2HKUST 3Kuaishou Technology

Motivation: Two-stage and query-based instancesegmentation methods have achieved remarkable results.
However, their segmented masks are still very coarse.
Large gap between bbox AP and Mask AP.

Accurate mask prediction is highly challenging, due to theneed for high-resolution deep features, which demands
large computational and memory costs.

24

Pointly-Supervised Instance Segmentation (oral)
Bowen Chen1 Omkar Parkhi2 Alexander Kirillov2
1UIUC 2Facebook AI

Motivation: Manual annotation of object masks fortraining is very complex and time-consuming. For
example, it takes on average 79.2 seconds to create
an object mask in COCO, whereas a bounding box canbe annotated ∼11 times faster in only 7 seconds.
Is object mask training data necessary to get closer
to the fully supervised performance? And is there aneasier to collect annotation form for the instancesegmentation task?

25

Pointly-Supervised Instance Segmentation (oral)
Bowen Chen1 Omkar Parkhi2 Alexander Kirillov21UIUC 2Facebook AI

Method: randomly sample 10 points inside bbox.
Performance: Mask R-CNN trainedon COCO, PASCAL VOC, Cityscapes, andLVIS achieves 94%–98% of its fully-supervised
performance.

26

FreeSOLO: Learning to Segment Objects without Annotations
Xinlong Wang1, Zhiding Yu2, Shalini De Mello2, Jan Kautz2, Anima Anandkumar2,3, Chunhua
Shen4,Jose M. Alvarez2
1The unversity of Adelaide 2NVIDIA 3Caltech 4Zhejiang University

Motivation: Instance segmentation requires costly annotations such as bounding boxes andsegmentation masks for learning.
This paper presents FreeSOLO to explore learning class-agnostic instance segmentation
without any annotations.

27

FreeSOLO: Learning to Segment Objects without Annotations
Method:1.propose the Free Mask approach, which leverages the specific design of SOLO to effectively
extract coarse object masks and semantic embeddings in an unsupervised manner.
2. further propose Self-Supervised SOLO, which takes the coarse masks and semantic
embeddings from Free Mask and trains the SOLO model, with several novel design elements
to overcome label noise in the coarse masks.

28

FreeSOLO: Learning to Segment Objects without Annotations
Performance:FreeSOLO achieves 9.8% AP50 on the challenging COCO dataset, which even outperforms
several segmentation proposal methods that use manual annotations.

29