Interactive Object Segmentation with Inside-Outside Guidance
Shiyin Zhang1, Jun Hao Liew2, Yunchao Wei3, Shikui Wei1, Yao Zhao1
1Beijing Jiaotong University 2National University of Singapore 3University of Technology Sydney• Manual data annotation is costly and time-consuming
Segmentation target

Polygon≈60s per instanceInteractive Object Segmentation

Sparse clicks

Scribbles

Bounding box• Extraction of target object given some user inputs (e.g. points, scribbles,bounding box)

Densecut: Densely connected crfs for realtime grabcut.
Comput. Graph. Forum.2015

Error-tolerant Scribbles Based Interactive Image
Segmentation.cvpr2014

Interactive Image Segmentation via Backpropagating
Refinement Scheme.cvpr2020

Interactive Image Segmentation with First
Click Attention.cvpr2020

Deep Interactive Object Selection.cvpr2016

Existing State-of-the-Art Method• DEXTR (Deep Extreme Cut)
• Take 4 extreme points (top, bottom, leftmost and rightmost pixels) as inputs• Problems:• Confusing annotation:• Multiple extreme points appear at similar location• Unrelated object lying inside the target object

Figure credit: Maninis et al. “Deep Extreme Cut: From Extreme Points to Object Segmentation”, CVPR 2018.

Inside-Outside Guidance (IOG)• Inside-Outside Guidance (3 clicks)• Inside guidance (1 click)• Interior point located roughly at the object center• Disambiguate the segmentation target• Outside guidance (2 clicks)• 2 corner clicks of a box enclosing the object• Indicate the background region• The remaining 2 corners can be inferred automaticallyClicking Paradigm• Steps:(1) Click on a corner point(2) Click on the symmetrical corner(3) Click on the object center

The vertical and horizontal guided lines
are used to make the box visible

Clicking Paradigm• Steps:(1) Click on a corner point(2) Click on the symmetrical corner(3) Click on the object center

The vertical and horizontal guided lines
are used to make the box visible

Clicking Paradigm• Steps:(1) Click on a corner point(2) Click on the symmetrical corner(3) Click on the object centerClicking Paradigm• Steps:(1) Click on a corner point(2) Click on the symmetrical corner(3) Click on the object centerClicking Paradigm• Steps:(1) Click on a corner point(2) Click on the symmetrical corner(3) Click on the object center

Input Representation• Follow the practice of DEXTR• Enlarge the bounding box by 10 pixels to include context• Crop and resize the inputs to 512x512• Input representation• 2 separate Gaussian heatmaps for the inside and outside clicks

Crop&Resize

heatmaps

Cropped
sub-region(a) CoarseNet(b) FineNet

Segmentation Network

Beyond Three Clicks• Our IOG naturally supports interactive adding of new clicks• Add a lightweight branch to accept additional inputs• Train with iterative training strategy(c) Refinement(a) CoarseNet(b) FineNet

(c) Refinement(a) CoarseNet(b) FineNetSegmentation Network• Segmentation errors mostly occur around the object boundaries• Use a coarse-to-fine network structure

The coarse-to-fine structure is similar to : Yilun Chen et al. “Cascaded pyramid network for multi-person pose estimation”, CVPR 2018.

(c) Refinement(a) CoarseNet(b) FineNet

(c) Refinement(a) CoarseNet(b) FineNet

(c) Refinement(a) CoarseNet(b) FineNet

IOG vs. Extreme Clicks• Observation:(1) IOG is more effective than extreme points across different backbone

IOG vs. Extreme Clicks• Observation:(1) IOG is more effective than extreme points across different backbone(2) Using a coarse-to-fine network structure further improves the performance

Comparison with SOTA

Generalization capability
on unseen classes and across different datasets• 1. In domain• 2. Cross domain• Object categories• Stuff categories

1. In domain• on unseen categories

2. cross domain——Object categories

Cityscapes

Agriculture-Vision

RooftopssTEM

2. cross domain——Stuff categories

Extension (Automated Mode)• Without user interaction, our IOG can still harvest high quality masks
from off-the-shelf datasets with box annotations (e.g. ImageNet)(a) CoarseNet(b) FineNet

Extension (Automated Mode)• Solution: Two-stage Training:(S1) Train a network that takes box as inputs to produce segmentation(S2) Infer interior clicks from the masks produced in S1 and apply IOG(a) CoarseNet(b) FineNet(a) CoarseNet(b) FineNet

PIXEL- IM GENET
https://github.com/shiyinzhang/Pixel-ImageNet

Characteristics• #Classes: 1000
• #Instance: >600KPossible Applications•Image classification•Instance segmentation•Semantic segmentation•Salient object detection•…. and moreConclusions• Propose IOG:
• Requires only three points (an inside point and two outside points)• Supports additional points for further correction• Performs well across different datasets and domains• Contribute Pixel-ImageNet:• A large volumes of high-quality pixel-level dataset• Offer unparalleled opportunities to researchers in the computer visioncommunity