P-2B-30 Distinct

Class-specific Saliency Maps
for Weakly-supervised Semantic Segmentation
Wataru Shimoda Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

Subtraction of class-specific derivatives

Objective
Weakly supervised segmentation- Use only image-level annotation

Fully supervisedannotation

Weakly supervisedannotation

PersonhorseCar

For multi-class images- Only small differences were observed
among the derivatives of the different classes
Assumption- Raw saliency maps are affected by bothclass-specific saliency and generic object-ness- The degree of class saliency factors should belarger than the generic object-ness factor.
- Background regions do not respond.

Contributions- Improved the method by Simonyan et al. [1] greatly- Achieved state-of-the-art in weakly-supervised
segmentation with PASCAL VOC 2012

- proposed by K. Simonyan et al. at ICLR 2014 [1]- Visualize contributed pixels on CNN classification- Use derivatives obtained by back-propagation

Forward
FCFCi

Recognition

Backward

FC FCi[K. Simonyan+, ICLR 2014]

𝜕𝑆𝑐𝑐𝑣𝑖 =│L0𝜕𝐿𝑖iWhite region means high derivative values
which corresponds to the important pixels
to enhance the given class score. (In the above fig. “Snake”)

CNN Architecture
Improved points (each point contributes 1~3pt improvement)- Fully Convolutional Net - Guided back propagation [2]- Use the derivatives of multiple intermediate layers- Aggregate multi-scale class saliency maps (3 scales)

Train

i

GMP

Personi

GMP

Train ‐ Person

Train

GMP

PersonGMPi

Guided Back-propagation

Proposed Method
Steps
1. Multi-class recognition
2. Back-propagation for eachof the detected classes
3. Subtracting the class maps
among the Top-N classes
4. Unify the class maps byFC-CRF (dense CRF)①②

We regarded the classes theoutput scores from the multi-class
CNN are more that 0.5 as thecandidate classes.

③④Experiments

SnakeBack-propagation

GMPi

Guided Back-propagation

BP-based Visualization
Visualize class-specific saliency maps
based on the derivatives of the class scoreswith respect to the input image

GMP

i𝑐 ∈𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠

Person
Train

FCN
i

Subtraction- Subtract the derivatives among the different classes.
- Interestingly, in most of the cases, we obtainedmuch clearer class maps than raw maps.
- The improved class saliency maps 𝑀𝑖𝑐with respect to class 𝑐 is computed by𝑐𝑐𝑐
𝑀𝑖,𝑥,𝑦=𝑀𝑖,𝑥,𝑦− 𝑀𝑖,𝑥,𝑦, 0 𝑐 ≠ 𝑐′ ,Dataset : Pascal VOC 2012 + trainaug [3]Comparison with Simonayn et al. [1]- Our gradient maps visualize class regions clearly.
- We applied FC-CRF to saliency maps obtainedby Simonyan et al.[1] in the same way.
- The margin was more than 10 %.

MethodSim et al. + CRF
Ours

Mean IOU
33.8
44.2

Effect of subtraction- Subtracting among the top-N classes- N=0 means no subtraction.
- N=4 achieved the best score.
012Class N
Mean IU 38.2 42.2 43.5

345

Car
Buswithout subtraction

1044.1 44.2 44.0 43.7

CarBuswith subtraction

Comparison with state-of-the-arts- A means using additional images.
- B means using additional supervision.
MethodOne point (ECCV 2016)

B

Mean IOU
46.1

MIL-FCN (ICLR 2015)- ✓- ✓- -EM-Adapt(ICCV 2015)--38.2CCNN (ICCV 2015)

--

34.5

MIL-seg (CVPR2015)

✓-

42.0

STC (arXiv:1509.03150)✓-49.8--50.740.5
44.2

We back-propagate expected class scores generated bysetting 1 for one of the top N-classes and 0 for the others.
𝑤𝑖𝑐 represents up-sampled i-th layer derivative which isobtained by propagating class scores from the top layer.
Each class saliency maps 𝑀𝑖𝑐 ∈ ℝ𝑚×𝑛 is calculated by:𝑐𝑐
𝑀𝑖,𝑥,𝑦 = max |𝑤𝑖,ℎ𝑖 𝑥,𝑦,𝑘 | ,Check Mask (ECCV2016)𝑐where ℎ𝑖 (𝑥, 𝑦, 𝑘) is the index of the element of 𝑤𝑖 .

𝑘𝑖A- Fine-tune full-conv VGG-16 network with Sigmoid
SEC (ECCV 2016)
cross entropy loss with random-resized images (300~700px) Ours w/o CRF- Sum up the derivatives of Conv3, Conv4, and Conv5.
Ours w/ CRF- Aggregate the class maps of 400*400, 500*500, and 600*600.

51.5
25.7

Project page
http://mm.cs.uec.ac.jp/shimoda-k/space/dcsm/
Source code
https://github.com/shimoda-uec/dcsm
Caffe-based implementation which takes 0.3 [s]with GPU for an one-time forward-backward pass.

References[1] K. Simonyan et al. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. ICLR, 2014.

Conv1

Conv2

Conv３

Conv4

Conv5

[2] J. Springenberg et al. Striving for Simplicity: The All Convolutional Net. ICLR, 2015.
[3] B. Hariharan et al. Semantic Contours from Inverse Detectors. ICCV, 2011.