Wataru Shimoda Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

Objective

Evaluation of input masks for integrations- Use only image-level annotation and generate segmentation masks

Modeling the decision process about whether to use the adviser's opinion
Decision taking into consideration theopinions of other advisors

Decision by own principles
Make low confidence on
pixels predicted by large
differences between the input
feature maps andsegmentation masks

Make low confidence on
pixels predicted bylearning similar training
samples-In both of decisions, high value (highlighted by white) indicates low confidence
Mask integration
based on the confidence scores

Calculation of confidence scores- Bias 𝑏෠ is for making gap between decisions- Bias 𝑏𝑚 is for missing class labels in results
wubefore =Motivation- CRF is a good refinement method but it often degrades the results
due to unstable unary terms in weakly supervised segmentation- Our motivation is to use CRF results as not teacher but adviser- In our situation, we suppose advisers give us noisy information
Picking𝑢−

𝑏𝑒𝑓𝑜𝑟𝑒𝑑𝑢+ 𝑏𝑚𝑏𝑒𝑓𝑜𝑟𝑒𝑢+ 𝑏෠

𝑎𝑓𝑡𝑒𝑟
𝑚𝑢

𝑏𝑒𝑓𝑜𝑟𝑒,𝑎𝑓𝑡𝑒𝑟if 𝑤𝑢≥0𝑏𝑒𝑓𝑜𝑟𝑒,𝑎𝑓𝑡𝑒𝑟𝑤𝑢<0if

We denote this integration process
as SSDD module by a below equation𝑏෠ = 0.4
𝑎𝑓𝑡𝑒𝑟𝑏𝑒𝑓𝑜𝑟𝑒
𝑏𝑚𝑢 = 1.0 if (𝑁𝑐/𝑁𝑐>0.5, mu = c)
N is number of pixels which belongs to category 𝑐 ∈ 𝑙𝑎𝑏𝑒𝑙 in an image.

m𝑟𝑒𝑓𝑖𝑛𝑒 = 𝑆𝑆𝐷𝐷 𝑒 𝑥; 𝜃𝑒 , 𝑚𝑏𝑒𝑓𝑜𝑟𝑒 , 𝑚𝑎𝑓𝑡𝑒𝑟 ; 𝜃𝑑Static region refinement
Loss for difference detection in PSA and CRF
𝐿𝑐ℎ𝑎𝑛𝑔𝑒 =・×× is ○○𝑎𝑓𝑡𝑒𝑟𝑑𝑢+ 𝑏𝑚𝑎𝑓𝑡𝑒𝑟𝑟𝑒𝑓𝑖𝑛𝑒𝑚𝑢ቐmbeforeu1σ𝑢∈𝑆(𝐽 𝑀𝑝𝑠𝑎,𝑐𝑟𝑓0 , 𝑑 𝑝𝑠𝑎 , 𝑢|𝑆|+ 𝐽(𝑀𝑝𝑠𝑎,𝑐𝑟𝑓0 , 𝑑𝑐𝑟𝑓0 , 𝑢))・△△ is ☆☆

Loss for segmentation network for obtaining good
representation from backbone network

・■■ is ▼ ▼𝐿𝑠𝑒𝑔 = 𝐿𝑚𝑎𝑠𝑘 𝑚𝑝𝑠𝑎 ; 𝜃𝑠𝑒𝑔 ,L𝑚𝑎𝑠k(mmask ; 𝜃) =1
෍σ𝑘∈𝐶 𝑆 𝑚𝑎𝑠𝑘log(ℎ𝑢𝑘 (𝑥; 𝜃))෍𝑘∈𝐶𝑢 𝑢∈𝑆 𝑚𝑎𝑠𝑘𝑘Final loss

To utilize information from advisers-We consider that many people first determine whether it isagainst their principles, and utilize opinions of other advisors forproblems that are difficult to judge-We model this scheme by difference detection task

Overview

𝐿𝑠𝑡𝑎𝑡𝑖𝑐 = 𝐿𝑠𝑒𝑔 + 𝐿𝑐ℎ𝑎𝑛𝑔𝑒

Dynamic region refinement
Losses for difference detection1
𝐿𝑑𝑑−𝑐𝑟𝑓 = σ𝑢∈𝑆(𝐽 𝑀 𝑠𝑒𝑔,𝑐𝑟𝑓1 , 𝑑 𝑠𝑒𝑔 , 𝑢|𝑆|+ 𝐽(𝑀 𝑠𝑒𝑔,𝑐𝑟𝑓1 , 𝑑𝑐𝑟𝑓1 , 𝑢))

𝐿𝑑𝑑−𝑠𝑒𝑒𝑑 =1σ𝑢∈𝑆(𝐽 𝑀 𝑠𝑠𝑑𝑑0,𝑠𝑢𝑏 , 𝑑 𝑠𝑠𝑑𝑑0 , 𝑢 + 𝐽(𝑀 𝑠𝑠𝑑𝑑1,𝑠𝑢𝑏 , 𝑑 𝑠𝑠𝑑𝑑1 , 𝑢))|𝑆|

Overview of Difference Detection Network(DD-Net)-Training
Train a difference detection model using difference regions between raw
segmentation masks generated by PSA[1] and its CRF results-InferenceIntegrate a pair of mask using DD-net outputs

Losses for segmentation network
𝐿𝑠𝑒𝑔−𝑚𝑎𝑖𝑛 = 𝐿𝑚𝑎𝑠𝑘 𝑚 𝑠𝑠𝑑𝑑2 ; 𝜃𝑠0
𝐿𝑠𝑒𝑔−𝑠𝑢𝑏 = 𝛼𝐿𝑠𝑠𝑑𝑑0 + 1 − 𝛼 𝐿𝑠𝑠𝑑𝑑2
𝐿𝑠𝑠𝑑𝑑0 = 𝐿𝑚𝑎𝑠𝑘 𝑚 𝑠𝑠𝑑𝑑0 ; 𝜃𝑠1 , 𝐿𝑠𝑠𝑑𝑑2 = 𝐿𝑚𝑎𝑠𝑘 𝑚 𝑠𝑠𝑑𝑑2 ; 𝜃𝑠1

Final loss
𝐿𝑑𝑦𝑛𝑎𝑚𝑖𝑐 = 𝐿𝑑𝑑−𝑐𝑟𝑓 + 𝐿𝑑𝑑−𝑠𝑒𝑒𝑑 + 𝐿𝑠𝑒𝑔−𝑚𝑎𝑖𝑛 + 𝐿𝑠𝑒𝑔−𝑠𝑢𝑏Experiments

Segmentation
after CRF

Difference Detection
after CRF

Difference detection network
Definition of difference detection task- Input: Pair of mask
(𝑚𝑏𝑒𝑓𝑜𝑟𝑒 , 𝑚𝑎𝑓𝑡𝑒𝑟 ) 𝑀𝑢𝑏𝑒𝑓𝑜𝑟𝑒,𝑎𝑓𝑡𝑒𝑟 ቐ10- Output: Difference region
𝑏𝑒𝑓𝑜𝑟𝑒,𝑎𝑓𝑡𝑒𝑟
𝑀

if
if

Definition of difference detection network (DD-net)
𝑏𝑒𝑓𝑜𝑟𝑒𝑎𝑓𝑡𝑒𝑟- Input: One of the pair of mask (𝑚or 𝑚) andℎ𝑙feature maps (𝑒 𝑥 , 𝑒 𝑥 )- Output: Probability map(𝑑 𝑏𝑒𝑓𝑜𝑟𝑒 , 𝑑 𝑎𝑓𝑡𝑒𝑟 ) ∈ ℝ𝐻×𝑊
beforedℎ𝑙= 𝐷𝐷𝑛𝑒𝑡(𝑒 𝑥 , 𝑒 𝑥

dafter = 𝐷𝐷𝑛𝑒𝑡(𝑒 ℎ 𝑥 , 𝑒 𝑙 𝑥

𝑏𝑒𝑓𝑜𝑟𝑒, 𝑚𝑢)
𝑏𝑒𝑓𝑜𝑟𝑒, 𝑚𝑢)

𝑏𝑒𝑓𝑜𝑟𝑒
𝑎𝑓𝑡𝑒𝑟
𝑚𝑢
= 𝑚𝑢
𝑏𝑒𝑓𝑜𝑟𝑒𝑎𝑓𝑡𝑒𝑟𝑚𝑢
≠ 𝑚𝑢

Cow

66.1

75.1

29.0

66.0

SSDD

89.0

62.5

28.9

83.7

52.9

59.5

77.673.7

87.0

34.0

83.7

Gain+0.8-5.7-1.7+2.6+3.3

-1.5

-0.2

+7.6

+11.9

+5.0

+17.7

mIoU

chair

77.8

Tv

61.4

Cat

53.4

61.0

Train

Static region refinement

Car

59.0

49.6

Sofa

48.0

Bus

PSA+CRF(re-implementation)

Bottle

58.4

Boat

52.5

Bird

PSA (re-implementation)

81.1

Sheep

Integrated result

DD-NetSegmentation
After CRF

Integrater

30.6

Plant

𝑳𝒐𝒔𝒔

DD-NetDifference Detection
before CRF

68.2

PersonSegmentation
Before CRF

88.2

Motor

Segmentation
before CRF

Raw seed(Train set)

MethodDD-NetBike

𝑳𝒐𝒔𝒔DD-Net

Trainedseg model(Val set)

PSA[1]

Horse

Comparison with WSS methods w/o additional supervision.

Aero

Inference

Dog

Feature maps

Input image

Results on PASCAL VOC 2012 val set.
BG

Training- Dataset: Pascal VOC 2012 dataset- Evaluation metric: mean IoU

Table

Feature maps

Input image

Backbone
network

Methods

Backbone
network

PSA[1]

40.2

80.4

62.0

70.4

73.7

42.5

70.7

42.6

68.1

51.6

61.7

SSDD

47.6

84.1

77.0

73.9

69.6

29.8

84.0

43.2

68.0

53.4

64.9

Gain+7.4

+3.7

+15.0

+3.5

-4.1-12.7 +13.3

+0.6

-0.1

+1.8+3.2

Comparison with WSS methods w/o additional supervision.
Methods
Val set Test set
Comparison with WSS methods w/additional supervision.
FCN-MIL(ICLR2015)
25.7
24.9
Methods
Additional information
Val setTest set
CCNN(ICCV2015)
35.3
35.6
Saliency mask + Imagenet images
MIL-seg(CVPR2015)
42.0
40.6
EM-Adapt(ICCV2015)
38.2
39.6
Saliency mask + Web images
STC (PAMI2017)
49.8
51.2
DCSM(ECCV2016)
44.1
45.1
Saliency mask
AE-PSL(CVPR2017)
55.0
55.7
BFBP(ECCV2016)
46.6
48.0
Web videos
Hong et al. CVPR2017
58.1
58.7
SEC (ECCV2016)
50.7
51.7
Saliency mask
DSRG (CVPR2018)
61.4
63.2
TPL(ICCV2016)
53.1
53.8
Web images
Shen et al. (CVPR2018)
63.0
63.9
CBTS(CVPR2017)
52.8
53.7
Saliency mask
SeeNet(NIPS2018)
63.1
62.8
PSA(CVPR2018)
61.7
63.7
Instance saliency mask
AISI(ECCV2018)
63.6
64.5
Static region refinement
61.4
SSDD(proposed)
64.9
65.5
Dynamic region refinement
64.9
65.5

For each row, from the left, (a) input images, (b) Raw PSA segmentation masks,(c) Difference detection maps of (b), (d) CRF masks of (b), (e) Difference
detection maps of (d), (f) Refined segmentation masks by the proposed methodand (g) Ground truth masks. Two bottom rows show failure cases.

References[1] Jiwoon Ahn, Suha Kwak : Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018