Weakly-Supervised Segmentation by Combining CNN Feature
Maps and Object Saliency Maps
Wataru Shimoda Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

Objective

Backward-base saliency maps𝑐
The class score derivative 𝑣𝑖 of the i-th layer is- Use only image-level annotation
Fully supervisedannotation

Weakly supervisedannotation

the derivative of class score 𝑆𝑐 with respect tothe layer Li at the point (activation signal) Li

𝜕𝑆𝑐
𝑐𝑣𝑖 =│L
𝜕𝐿𝑖

Personi

Each class saliency maps 𝑀𝑖𝑐 ∈ ℝ𝑚×𝑛 is
calculated by:horseCar𝑐𝑐
𝑀𝑖,𝑥,𝑦= max |𝑤𝑖,ℎ| ,𝑖 𝑥,𝑦,𝑘- Improved the method by Simonyan et al. [1]- Combine BP-based map with CNN class maps forweakly supervised segmentaion- Achieved comparerable result in weakly-supervised
segmentation with PASCAL VOC 2012𝑘𝑖

where ℎ𝑖 (𝑥, 𝑦, 𝑘) is
the index of the element of 𝑤𝑖𝑐

Combine BP-based maps with featre maps
Zoom out feature

BP-based Visualization
Visualize class-specific saliency maps
based on the derivatives of the class scoreswith respect to the input image- proposed by K. Simonyan et al. at ICLR 2014 [1]- Visualize contributed pixels on CNN classification- Use derivatives obtained by back-propagation
The class score derivative vi of the i-th layer
is the derivative of class score Sc withrespect to the layer Li at the point (activationsignal) Li𝜕𝑆𝑐𝑐𝑤𝑖 =│I𝜕𝑤
0𝑖

ForwardFCFCi

Recognitioni

ZOF for BG
Feature maps
Input

Segmentaion result

Input
Saliency maps for BG

Steps

1. Recognize an image by forwarding
2. Back-propagation for each of thedetected classes
3. Preparing Zoom-out feature by featuremaps of eacy layer
4. Caluculate class probability maps by miSVM with feature maps
5. Unify the class maps and BP-basedsaliency maps by super-pixel-based CRF

Steps

1. Recognize an image by forwarding
2. Back-propagation for each of thedetected classes
3. Obtaine class probability maps fromoutput feature maps
4. Unify the class maps and BP-basedsaliency maps by ILP and thresholding

Training

Training

1. Training multi class recognition CNN

Experiments

Snake

Back-propagation[K. Simonyan+, ICLR 2014]

White region means high derivative values
which corresponds to the important pixels
to enhance the given class score. (In the above fig. “Snake”)

CNN Architecture- Fully Convolutional Net- Guided back propagation [2]- Use the derivatives of multiple intermediate layers

Segmentaion result

BP based maps

1. Training multi class recognition CNN
2. Training Mi-svm for CNN features

Backward

FC FC

Fully convolutional network

Results on PASCAL VOC 2012
-A means using addtinal images for training
MethodMIL-FCN (ICLR 2015)

A-Validation
25.7

test
24.9

EM-Adapt(ICCV 2015)

-38.2

39.6

CCNN (ICCV 2015)

-34.5

35.5

MIL-seg (CVPR2015)

✓42.0

40.6

STC (arXiv:1509.03150)

✓49.8

51.2

Ours (ZOF+GBP)
Ours (FCN+GBP)
Ours (FCN+GBP)

-38.1
33.8
41.4

37.7
33.1
40.7
ZOF + GBPFCN + MCG

FCN + GBP

Ground truth

Compare similar approach methods- Training with global pooling- Enhance FCN output with low-level objectness map

We back-propagate expected class scores generated by
setting 1 for one of the top N-classes and 0 for the others.
𝑤𝑖𝑐 represents up-sampled i-th layer derivative which isobtained by propagating class scores from the top layer.
.

MethodMIL-FCN

Enhance-Mean IU
25.7

MIL-sppxl

Super-pixel refinement

36.6

MIL-bbBB-proposal-based objectness map

37.8

MIL-seg

MCG-based objectness map

42.0

FCN + MCG
FCN + GBP

MCG-based objectness map
BP-based saliency map

33.8
41.4

GBP

MCG

References[1] K. Simonyan et al. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. ICLR, 2014.
[2] J. Springenberg et al. Striving for Simplicity: The All Convolutional Net. ICLR, 2015.
[3] B. Hariharan et al. Semantic Contours from Inverse Detectors. ICCV, 2011.