Self-supervised difference detection for weakly supervised segmentation
Wataru Shimoda

Keiji Yanai The University of Electro-Communications, Tokyo, Japan

Objective

Self-supervised difference detection(SSDD)

Weakly-supervised segmentation- Use only image-level annotation and generate segmentation masks

The definition of difference detection
Estimate difference between knowledge and advice
Difference region: 𝑀𝐾, 𝐴𝐾,𝐴 1
𝑀𝑢 ൝0Input

ifif

Self-supervised
supervision
Knowledge𝐾𝐴𝑚𝑢 = 𝑚𝑢
𝑚𝑢𝐾 ≠ 𝑚𝑢𝐴

SSDD training/inference-Training
Train a difference detection model using the difference of the Knowledge and advice-Inference
Integrate a pair of the mask using DD-net outputs

AdviceBackbone
network

Difference
detection

Knowledge

Backbone
network

Input image
Training

Feature maps

𝑳𝒐𝒔𝒔

𝑳𝒐𝒔𝒔
Segmentation
Before CRF

Difference

Difference Detection
before CRF

Integrater

Integrated result

𝑳𝒐𝒔𝒔

DD-NetDD-NetSegmentation
after CRF

Difference detection network (DD-net)- Input: mask , features (𝑒 ℎ 𝑥 , 𝑒 𝑙 𝑥 )Recent weakly-supervised segmentation methods generate pseudo - Output: Probability map
labels in advance and train a segmentation model with them.
ℎ𝑙
𝐻×𝑊- 𝑑 = 𝐷𝐷𝑛𝑒𝑡(𝑒 𝑥 , 𝑒 𝑥 , 𝑚)
ෝ ∈ℝ-𝑚ෝ is the vectorized mask

DD-NetSegmentation
before CRF

How to improve pseudo labels?

Inference

Feature maps

DD-NetDD-netInput image

Difference Detection
after CRF

Segmentation
After CRF

Static region refinement [A]Knowledge : PSA[1]Advice : PSA[1] + CRF

Vectorized mask

Define the border of the good advice
Segmentation
modelOut

𝑳𝒐𝒔𝒔with difference
pseudo label

We can not evaluate the pseudo labels in weakly supervised setting
Most previous approaches based on heuristic knowledge.
There are noise in the mapped results
Mapping functions
CRF, re-training, etc

The high score indicates good advice
Easy But the threshold is ambiguous
It should be different in each sample
HardKnowledge?

Dynamic region refinement [B]Key idea

w/o difference

We denote the inputs of the mapping functions as knowledge, For a flexible measurement, we also estimate the difference from advice.
We consider the supervision containig the noise as advice.

Knowledge : Pseudo labels
Advice : The outputs of the segmentation model
We attempt to adapt the re-training scheme[2] to SSDD module

We use the distance of the outputs of DDnet for the measurement

AdviceEasy

Problem:- It is unclear which advice is useful- We want to detect good advice
Idea- Different opinions from the adviser are important- Predict the important advice in advance- Use the prediction for detecting good advice

Hard- Dataset: Pascal VOC 2012 dataset- Evaluation metric: mean IoU
The comparison with WSS methods w/o additional supervision.

w/o difference

Mask integration
The computation of the confidence scores<:Easily predictable difference≒ bad advice

𝐷𝐾𝐴
m = 𝑆𝑆𝐷𝐷 𝑒 𝑥; 𝜃𝑒 , 𝑚 , 𝑚 ; 𝜃𝑑

AE-PSL(CVPR2017)

Saliency mask

Hong et al. CVPR2017

Web videos

DSRG (CVPR2018)

Saliency mask

Shen et al. (CVPR2018)

Web images

SeeNet(NIPS2018)

Saliency mask

AISI(ECCV2018)

Instance saliency mask

SSDD(proposed)

-40.6
51.2
55.7
58.7
63.2
63.9
62.8
64.5
65.5

chair

Cow

TableDog

HorseMotor

Pers
onPlant

Sheep

Sofa

Train

TvmIoUThe detailed results on PASCAL VOC 2012 val set.
Catif 𝑤𝑢 ≥ 0if 𝑤𝑢 < 0Saliency mask + Web images

42.0
49.8
55.0
58.1
61.4
63.0
63.1
63.6
64.9

Car𝐾
𝐷 𝑚𝑢𝑚𝑢 ൝ 𝐴𝑚𝑢

We denote this integration process as SSDD module

Bad advice

Good advice

If:

Easily predictable difference≒ good advice

STC (PAMI2017)

Test set

Bus>

If𝐾𝐴
We integrate 𝑚 and 𝑚 using the confidence scores

MIL-seg(CVPR2015)

Val setBottleBad advice

Easy

Additional information
Saliency mask + Imagenet
imagesBoat

Good advice𝐾𝐴wu = 𝑑𝑢 −𝑑𝑢 + 𝑏𝑖𝑎𝑠𝑢

Methods

Test set
24.9
35.6
39.6
45.1
48.0
51.7
53.8
53.7
63.7
65.5

BirdAssumption
The number of the training sample is related to the degreeof difficulty of the inference in difference detection

Methods
Val setFCN-MIL(ICLR2015)
25.7
CCNN(ICCV2015)
35.3
EM-Adapt(ICCV2015)
38.2
DCSM(ECCV2016)
44.1
BFBP(ECCV2016)
46.6
SEC (ECCV2016)
50.7
TPL(ICCV2016)
53.1
CBTS(CVPR2017)
52.8
PSA(CVPR2018)
61.7
SSDD(proposed)
64.9

The comparison with WSS methods w/additional supervision.

Bike

AdviceKnowledge

Aero

Knowledge

with difference

BG

Input (A)-> Label (3)Input (B)-> Label (1)Input (C)-> Label (2)Input (D)-> Label (5)Methods

Input (A)-> Label (3)Input (B)-> Label (0)Input (C)-> Label (5)Input (D)-> Label (5)PSA[1]

88.2

68.2

30.6

81.1

49.6

61.0

77.8

66.1

75.1

29.0

66.0

40.2

80.4

62.0

70.4

73.7

42.5

70.7

42.6

68.1

51.6

61.7

SSDD

89.0

62.5

28.9

83.7

52.9

59.5

77.6

73.7

87.0

34.0

83.7

47.6

84.1

77.0

73.9

69.6

29.8

84.0

43.2

68.0

53.4

64.9

Gain+0.8-5.7-1.7+2.6+3.3-1.5-0.2+7.6+11.9+5.0+17.7+7.4+3.7+15.0+3.5-4.1-12.7+13.3+0.6-0.1+1.8+3.2

References and source codes[1] Pixel-level semantic affinity, Ahn et al., CVPR 2018, arXiv:1803.10464[2] STC: A Simple to Complex Framework, Wei et al., TPAMI 2016, arXiv:1509.03150
Source codes: https:github.com/shimoda-uec/ssdd