Instance Segmentation

韩坤洋Papers• Mask R-CNN• PointRend: Image Segmentation as Rendering• YOLACT: Real-time Instance Segmentation• PolarMask: single shot instance segmentation with polar
representation• SOLO: segmenting objects by locations

Mask R-CNN• Based on faster rcnn
• Output a binary mask for each RoI

Mask R-CNNRoIAlign• RoIPool first quantizes a floating-number RoI to the discrete
granularity of the feature map• Then subdivided into spatial bins which are themselves quantized• Cause misalignments

Mask R-CNNLoss• Define a multi-task loss on each sampled RoI• Mask branch has a K x m x m output for each RoI• m*m resolution, k classes• per-pixel sigmoid• average binary cross-entropy loss
• for RoI with ground-truth class k,

is only defined on the k-th mask

PointRend: Image Segmentation as RenderingPointRend

Point Selection• Inference: adaptive subdivision (in rendering• Train: to select N points• Over generation: select kN points randomly• Importance sampling: select most uncertain βN points (β ∈ [0, 1])• Coverage: (1 - β)N points sampled from a uniform distribution

We use the distance between 0.5 and theprobability of the ground truth class interpolated
from the coarse prediction as the point-wise
uncertainty measure.

PointRend

Point-wise Representation• Fine-grained features: CNN feature maps
• fine segmentation details• bilinear interpolation• single or multiple feature map (res2 or res2 to res5 in a ResNet• Coarse prediction features• region-specific information• contextual and semantic information

PointRend• Point head: MLP• shares weights across all points• binary cross-entropy• Train:• sample 14^2 points (ROI_MASK_HEAD.POOLER_RESOLUTION)• One time (does not improve the baseline Mask R-CNN• Bilinear GT• Inference• Top-k• CascadeYOLACTYOLACTPrototype Generation (protonet• FCN structure,• k channel out (k prototypes• No extra loss

• Attached to p3 in FPN• deeper backbone features, more robust• higher resolution prototypes, higher quality masks

YOLACTMask Coefficients• Three branch• c class confidences• 4 bounding box regressors• k mask coefficients (k prototypes• Nonlinearity• tanh to the k mask coefficients

YOLACTMask Assembly• Linear combination (matrix multiplication and sigmoid• P , prototype masks, h × w × k• C , mask coefficients, n × k• n instances, surviving NMS and score thresholding• Cropping Masks• Evaluation, predicted bounding box• Training, ground truth bounding box

YOLACT

Prototype Behavior

PolarMask : Single Shot InstanceSegmentation with Polar Representation

FCOS

PolarMask

FCOSPolarMask• Polar Representation• pre-define angle interval, predict length of the ray• Center Samples• (x, y), falls into areas around the mass-center, about 9∼16 pixels• Avoid imbalance ..., Mass-center may not be the best center• Distance Regression• multiple intersection, one with the maximum length• does not have intersection, set to 1e-6

• Mask Assembling• Top-k, threshold, NMS• Calculate point position, connect them one by one

PolarMask

Center-ness• Poor performance,• low-quality bounding boxes, far away from the center of an object• Center-ness branch• Single-layer branch• Range from 0 to 1, use BCE loss

• Polar Center-ness

PolarMask

Polar IoU Loss• Converts segmentation into regression• Polar IoU• Polar IOU LossSOLO: Segmenting Objects by Locations• Divides input image into a uniform grids (S×S
• predicting the semantic category• segmenting that object instanceSOLO• Semantic Category• Output S x S x C, C classes

SOLO

Instance Mask

• One-to-one correspondence• Channel k, grid (i, j), where k = i ·S + j• Spatially variant, position sensitive• CoordConv