Self-supervised Difference Detection for Refinement CRF and Seed Interpolation
Wataru Shimoda Keiji Yanai
Department of Informatics, The University of Electro-Communications, Tokyo

2. Related Works

Abstract

Region refinement for WSS results using CRF CRF [11]can refine the ambiguous outlines using low-level features
such as a color of pixels. Chen et al. [13] and Pathak et
al. [14] adopted CRF as a post-processing method for region refinement and demonstrated the effectiveness of the
CRF for WSS. Kolesnikov et al. [10] proposed to use CRF
during training of a semantic segmentation model. Ahn
et al. [1] proposed a method to learn pixel-level similarity
from CRF results, and apply random walk based region refinement, which achieved the state-of-the-art on the Pascal
VOC 2012 dataset. We focus on preventing a segmentation
mask from being degraded by applying CRF.
Generating pixel-level labels during training of a fullysupervised semantic segmentation (FSS) model Constrained Convolutional Neural Network (CCNN) [14] andEM-adopt [13] generated pixel-level labels during training
using class labels and outputs of the segmentation model.
Wei et al. [25] proposed an online Prohibitive Segmentation Learning (PSL). that utilizes classification score andthe output of the segmentation model for generating mask
during training. Huang et al. [7] proposed Deep Seeded Region Growing (DSRG), which is a method to expand theseed region during training. The authors prepared pixellevel seed labels before training that have unlabeled regions
for unconsidered pixels. In this work, we proposed new
constraint for the generating pixel-level labels during training of a FSS model.

To minimize annotation costs associated with training of
semantic segmentation models, weakly-supervised segmentation approaches have been extensively studied. In thispaper, we propose a novel method: Self-Supervised Difference Detection (SSDD) module which evaluates confidenceof each of the pixels of segmentation masks and integrate
highly confident pixels of two candidate masks.

1. Introduction
In this paper, we focus on avoiding performance degradation by CRF. In the proposed method, we estimate confidence maps on segmentation masks before and after applying CRF, and integrate them into a final mask by selecting
more confident pixels comparing both confident maps. To
estimate confidence maps of segmentation masks, we propose a Self-Supervised Difference Detection (SSDD) module. SSDD contains a Difference Detection network (DDnet), which estimates differences of the regions Since thedifference of segmentation masks can be obtained automatically without any human supervision, we can train a DDnet without any additional information. Thus, training of
DD-net can be considered as self-supervised learning.
It is hard to estimate complicate parts even if an input
region mask contains the regions to be estimated, while itis easy to estimate simple parts. Then we regard an output
of a DD-net as confidence maps of difference between themasks before or after a refinement process. By using theconfidence maps on mask difference, we pick up the pixels
having higher confidence from both the masks, and generate
re-refined segmentation masks.
In this paper, we demonstrate that the propose SSDD
module can be used in both the seed generation stage andthe training stage of fully-supervised segmentation as a “rerefinement module”. In the seed generation stage, we refine
the CRF results of PSA [1] by a SSDD module. In the training stage, we introduce two SSDD modules inside training
loop of a fully-supervised segmentation network. In the experiments, we demonstrate the effectiveness of SSDD modules in both stages. Especially, SSDD modules boosted theperformance of weakly-supervised semantic segmentationon the PASCAL VOC 2012 dataset greatly and, achieved
new state-of-the-art.

3. Method for “Re-refinement” of Refinement
In this paper, we focus on avoiding performance degradation by refinement methods of segmentation masks suchas CRF. To do that, we propose a Self-Supervised Difference Detection (SSDD) module. Fig.1 shows the basic ideaof the processing in SSDD.
Difference Detection Network We denote one pair of segmentation masks as (mbefore , mafter ). Basically, we consider
mbefore is a raw segmentation mask and mafter is a processedmask. We denote their difference regions by a binary mask
M before,after . The difference regions defined by:(
1 if (mbefore= mafterbefore,after
u
u )
Mu
=
,
(1)
before0 if (mu 6= mafteru )where u ∈ {1, 2, .., n} indicates a location of pixels, and1ferent regions in the training phase.
We define confidence score w based on the difference
detection results between (dbefore and dafter ). We calculate theconfidence score w for each pixel u as follows:
before,after,afterbeforewubefore,after = (dafter+ bbefore) + b̂, (2)u + bmafter ) − (dumbeforeu

Figure 1. Basic idea of “re-refinement” by Self-Supervised Difference Detection. (a) Segmentation masks before/after applying
CRF. (b) Confidence maps obtained by DD-Net. (c) Re-refinedmask which is integration of both masks based on confidence
maps.

u

,afterwhere b̂ is a bias for a pair of masks, bbefore
is a bias for acclass (c, c ∈ C ), and C is a set of image-level label. We add,afterfor missing classes between mbefore andthe class bias bbeforecafter
m . In practice, we select missing classes after applying
CRF based on change ratio of pixels. We denote a number of pixels belonging to class c on mbefore as Ncbefore and anumber of pixels belonging to class c on mafter as Ncafter . Thechange ratio of pixels belonging to class c is calculated by:δcbefore,after = Ncafter /Ncbefore .

(3)

The class bias bc is represented using the change
ratio of pixels δcbefore,after as follows: if δcbefore,after >,after,after=bCL else bbefore=0, where bCL is a bias
th then bbeforecc
for missing classes and we simply set the threshold th to
0.5.
Finally, we obtain the refined mask mrefine using the confident score w by:(
if (wubefore,after > 0)
mbeforerefine
u
mu =
(4)
after
mu
if (wubefore,after < 0)
Figure 2. Self-Supervised Difference Detection (SSDD) module.

For simplicity, we denote this region refinement processing
by mrefine =SSDD(e(x; θe ), mbefore , mafter ; θd ).

4. Introducing SSDD modules into the processing flow of WSS

Figure 3. Difference Detection Network (DD-Net).

n is the number of pixels. We use high level featureseh (x; θe ) and low level features el (x; θe ) x is an input
image and e is an embedding function parameterized byθe . As shown in Fig.2, the confidence map of the input
mask, d, is generated by a Difference Detection network,
DDnet(eh (x; θe ), el (x; θe ), m̂; θd ), d ∈ RH×W , where m̂
is a binarized mask with the same number of channels to thetarget class number. The architecture of DD-Net is shownin Fig.3.
Self-Supervised Difference Detection Module In this section, we describe the detail of a SSDD module shown in
Fig.2. We suppose that an input mask which is difficult topredict difference regions is relatively reliable. This insightis based on an assumption that the overall processed masks,
mafter , have enough correct labels against error labels in dif-

Seed mask generation stage with static region refinement Pixel-level Semantic Affinity (PSA) [1] is a method
to propagate label responses to nearby areas which belong
to the same semantic entity. In this section, we refine theoutputs of CRF in PSA by the proposed Self-Supervised
Difference Detection (SSDD) module.
We denote an input image as x and the probability maps obtained by PSA as ppsa =P SA(x; θP )and its CRF results as pcrf0 .
We convert the probability maps (ppsa ,pcrf0 ) to segmentation masks by
crf0
crf0
(mpsa = arg max ppsa,
wherek , m = arg max pk )k∈C y ∪C bg

k∈C y ∪C bg

bg indicates a background class.
The difference detection result for target M psa,crf0 is computedbydpsa =f (eh (x; θe0 ), el (x; θe0 ), m̂psa ; θd0 ),crf0h
d =f (e (x; θe0 ), el (x; θe0 ), m̂crf0 ; θd0 ). From the difference detection results, we obtain a target refinedmask mssdd0 by the proposed SSDD module as follows:
mssdd0 =SSDD(e(x; θe0 ), mpsa , mcrf0 ; θd0 ).
Suppose that σ() is sigmoid function and S is a set of
locations. The loss function for a Difference Detection net-

The difference detection network (DD-net) between mseg
and mcrf1 is optimized by a below loss:

wort (DD-Net) is given by:
Lchange (dpsa , dcrf0 , M psa,crf0 ) =1 X(J(M psa,crf0 , dpsa , u)|S|u∈S

+J(M psa,crf0 , dcrf0 , u)),
(5)
J(M
J(M

psa,crf0
psa,crf0

psa

psa,crf0

log σ(du ) + (1 − Mu

psa,crf0

log σ(du ) + (1 − Mu

, d , u) = Mu
,d

crf0

, u) = Mu

psa

psa,crf0

crf0

psa,crf0

psa crf0

1 X(J(M seg,crf1 , dseg , u)
|S|
(9)
u∈S
+J(M seg,crf1 , dcrf1 , u)),psa

) log(1 − σ(du )),crf0) log(1 − σ(du ))Note that M psa,crf0 corresponds to M before,after in Eq.(1).
The proposed method is not effective in the cases that either
or both of the segmentation masks has no correct labels.
We exclude the bad training samples by simple processing.
and define it by the change ratio of pixels δ. We calculate the change ratio δcpsa,crf0 in the same manner to Eq.(3)and obtain the decisions γcpsa,crf0 for the bad training samples by: if δc , > 0.5 then γc , =1 else γc , =0.
The inX,| >terpolated loss function Lchange as follows: if |γkpsa crf0

Ldd-crf (dseg , dcrf1 , M seg,crf1 ) =psa crf0

psa crf0k∈C y

0 then Lchange =0 else Lchange =Lchange . C is a set of class and C

y∈C
is a set of category C for image-level label y.
We also train embedding function θe0 by training a segmentation network with mpsa as follows:
X X
1log(hku (x; θs0 )),Lseg = − X psa(6)|Sk | k∈C u∈S psakk∈C

where Skpsa is a set of locations that belong to class k on themask mpsa . Final loss function for static region refinement
using difference detection is given by:
Lstatic = Lseg + Lchange(7)

Training stage of a fully-supervised segmentation modelwith dynamic region refinement In this work, we propose a novel approach to constrain interpolation of the seedlabels during training of a segmentation model. The ideaof the constraint is to limit interpolation of seed labels to
only predictable regions of the difference be-tween newly
generated pixel-level labels and seed labels.
In the first step, for an input image x, we obtain outputsof a segmentation model pseg =g(x; θs1 ) and its CRF outputs pcrf1 , where g is a function of the segmentation model.
We convert them to segmentation masks (m = arg max pk ,
seg

seg

J(M
J(M

seg,crf1

seg,crf1

seg

seg,crf1

log σ(du ) + (1 − Mu

crf1

seg,crf1

log σ(du ) + (1 − Mu

, d , u) = Mu,d, u) = Mu

seg

seg,crf1

crf1

seg,crf1

seg) log(1 − σ(du )),crf1) log(1 − σ(du )).

In the similar manner to Sec.4, in the second stage, we
also exclude bad samples based on the change ratio of pixels. We define the interpolated loss function Ldd-crf for thepair Pof masks (mseg , mcrf1 ) by applying below processing.
If | γkseg,crf1 | > 0 then Ldd-crf = 0 else Ldd-crf = Ldd-crf .
We explain about the training of the DD-net for (mssdd1 ,ssdd0
m ). The difference detection between mssdd1 and mssdd0 is
an easy task because the seed masks mssdd0 are always constant during training. To avoid the problem, we add a new
segmentation branch to a segmentation network, which isalso trained with the pixel-level seed labels mssdd0 . We obtain segmentation probability maps from this branch psub andconvert the map to mask by msub = arg max psub . We traink∈C y ∪C bg

the difference detection network, DD-net, to predict the difference of the pair of masks (mssdd0 , msub ) and (msub , mssdd1 ).
This replacing of the mask makes it hard to predict the difference between mssdd0 and mssdd1 because the situation oftraining and inference become different. We denote the loss
function for the difference detection in this case as follows:
Ldd-seed (dssdd0 , dssdd1 , M ssdd0,ssdd1 ) =1 X(J(M ssdd0,sub , dssdd0 , u)|S|u∈S

+J(M ssdd1,sub , dssdd1 , u)),
(10)
J(M
J(M

ssdd0,subssdd1,subssdd0,sub

,d

ssdd0

, u) = Mu

,d

ssdd1

, u) = Mu

ssdd1,sub

ssdd0

ssdd0,sub

ssdd1

ssdd1,sublog σ(du )+(1−Mulog σ(du )+(1−Mussdd0) log(1−σ(du ))ssdd1) log(1−σ(du )).

The parameter θs10 of the new segmentation branch istrained by:k∈C y ∪C bg

Lseg-sub = αLssdd0 + (1 − α)Lssdd2 ,mcrf1 = arg max pcrf1
k ). Then, we obtain refined pixel-level labelsk∈C y ∪C bg

ssdd1

m
by applying the proposed refinement method as follows: m =SSDD(e(x; θe1 ), m , m ; θd1 ). In the second step,
we apply the proposed method to the obtained mask mssdd1
and seed labels mssdd0 . The further refined mask mssdd2 isobtained by mssdd2 =SSDD(e(x; θe1 ), mssdd1 , mssdd0 ; θd2 ). We
generate the mask mssdd2 in each iteration and train the segmentation model using the generated mask mssdd2 .
We train the semantic segmentation model with the generated mask mssdd2 by:
X X
1Lseg-main = − Xlog(hku (x; θs1 )). (8)|Skssdd2 | k∈C u∈S ssdd2
ssdd1

seg

crf1

k

k∈C

L

ssdd0

=−X

1

X X
ssdd0

|Sk | k∈Cu∈S ssdd0

ssdd2

=−X
k∈C

k

log(hu (x; θs10 ))

k

k∈C

L

(11)

1

X X
ssdd2

|Sk | k∈Cu∈S ssdd2klog(hu (x; θs10 )).

k

α is a hyper parameter of the mixing ratio of Lssdd0 and Lssdd2 .
The final loss function of the proposed dynamic region
refinement method is calculated as below:
Ldynamic = Ldd-crf + Ldd-seed + Lseg-main + Lseg-sub .

(12)

Method
FCN-MIL [15]ICLR2015
CCNN [14]ICCV2015
EM-Adapt [13]ICCV2015
DCSM [22]ECCV2016
BFBP [19]ECCV2016
SEC [10]ECCV2016
CBTS [18]CVPR2017
TPL [9]ICCV2017
MEFF [4]CVPR2018
PSA [1]CVPR2018
PSA (Re-implementation)
SSDD (Static)
SSDD (Dynamic)

Val
25.7
35.3
38.2
44.1
46.6
50.7
52.8
53.1
61.7
59.0
61.4
64.9

Test
24.9
35.6
39.6
45.1
48.0
51.7
53.7
53.8
55.6
63.7
65.5

MethodMIL-seg [16]CVPR2015
MCNN [23]ICCV2015
AFF [17]ECCV2016
STC [26]PAMI2017
Oh et al. [20]CVPR2017
AE-PSL [25]CVPR2017
Hong et al. [5]CVPR2017
WebS-i2 [8]CVPR2017
DCSP [2]BMVC2017
GAIN [12]CVPR2018
MDC [27]CVPR2018
MCOF [24]CVPR2018
DSRG [7]CVPR2018
Shen et al. [21]CVPR2018
SeeNet [6]NIPS2018AISI [3]ECCV2018
SSDD (Dynamic)

Additional supervision
Saliency mask + Imagenet images

Web videos
Saliency mask
Saliency mask + Web images

Saliency mask
Saliency mask
Web videos
Web images
Saliency mask
Saliency mask
Saliency mask
Saliency mask
Saliency mask
Web images
Saliency mask
Instance saliency mask
-

Val
42.0
38.1
54.3
49.8
55.7
55.0
58.1
53.4
60.8
55.3
60.4
60.3
61.4
63.0
63.1
63.6
64.9

(a)

(b)

(c)

(d)

train

sofa

sheep

plant

horse

motor

mIoU

methods w/o additional supervision.

dog

tv

Table 3. Comparison with WSS methods w/ addiTable 2. Comparison with WSS tional supervision.

table

cow

chair

cat

car

bus

bottle

boat

bird

bike

61.7
64.9
+3.2

aero

88.2 68.2 30.6 81.1 49.6 61.0 77.8 66.1 75.1 29.0 66.0 40.2 80.4 62.0 70.4 73.7 42.5 70.7 42.6 68.1 51.6
89.0 62.5 28.9 83.7 52.9 59.5 77.6 73.7 87.0 34.0 83.7 47.6 84.1 77.0 73.9 69.6 29.8 84.0 43.2 68.0 53.4
+0.8 -5.7 -1.7 +2.6 +3.3 -1.5 -0.2 +7.6 +11.9 +5.0 +17.7 +7.4 +3.7 +15.0 +3.5 -4.1 -12.7 +13.3 +0.6 -0.1 +1.8

bg

methods
PSA [1]
SSDD
Gain

personTable 1. Results on PASCAL VOC 2012 val set.

(e)

(f)

(g)

Test
40.6
39.8
55.5
51.2
56.7
55.7
58.7
55.3
61.9
56.8
60.8
61.2
63.2
63.9
62.8
64.5
65.5

5. Experiments
We evaluated the proposed methods using the PASCAL
VOC 2012 data. For calculating the mean IoU on val andtest sets, We used the official evaluation server.
Implementation details Our experiments are heavily basedon previous work [1]. For the generating results of PSA, weused implementations provided by the authors that are publicly available. We followed the paper [1] and set hyperparameters that achieved the best performance in their paper.
As segmentation model we used a ResNet-38 model, whichis almost the same to the architecture used in [1] except for
upsampling rate. We explore good hyperparameters in the
proposed method by grid search.

Figure 4. For each row, from the left, (a) input images, (b) Raw
PSA segmentation masks, (c) Difference detection maps of (b), (d)CRF masks of (b), (e) Difference detection maps of (d), (f) Refinedsegmentation masks by the proposed method and (g) Ground truth
masks. Two bottom rows show failure cases.
InputsPSA(Re-implementation)(59.0%)SSDD(Static)(61.4%)SSDD(Dynamic)(64.9%)

Ground truth

Comparison Table 1 shows the comparison of dynamic region refinement method with PSA. We denote the dynamic
region refinement as “SSDD” in all the tables. We observethat the proposed method outperforms PSA with over 3.2
point margin. In Table 1, we show the gains between the
proposed method and PSA for detailed analysis as well.
Table 2 shows the results of the proposed method and recent weakly supervised segmentation methods that use no
additional supervisions on PASCAL VOC 2012 validation
data and PASCAL VOC 2012 test data. We observe that
our method achieves the highest score compared with all
the existing methods, which use the same types of supervision [14, 13, 22, 19, 10, 9, 18, 4, 1].
Table 3 shows the comparison of the proposed methodwith some weakly supervised segmentation methods, which
employ relatively cheap additional information. Thoughcompletely fair comparisons for them are difficult becauseof the difference of network model, augmentation technique, the number of iteration epochs and so on, theproposed method demonstrates comparable or better performance without any additional information for training.
Fig.4 shows the examples of difference detection results andFig.5 shows the examples of semantic segmentation results.

Figure 5. Segmentation examples of results on Pascal VOC 2012.

6. Conclusion
In this paper, we proposed a novel method to refine a segmentation mask from a pair of segmentation masks before
and after refinement process such as CRF by the proposed
SSDD module. We demonstrated the proposed method canbe used effectively in two stages: static region refinement in
the seed generation stage, and dynamic region refinement in
the training stage.
Acknowledgments This work was supported by JSPS
KAKENHI Grant Number 17J10261, 15H05915,
17H01745, 19H04929 and 17H06100.

References[1] J. Ahn and S. Kwak. Learning pixel-level semantic
affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR, 2018. 1,2, 4[2] A. Chaudhry, K. P. Dokania, and P. H. Torr. Discovering class-specific pixels for weakly-supervised semantic segmentation. In BMVC, 2017. 4[3] R. Fan, Q. Hou, M. M. Cheng, G. Yu, R. R. Martin, and S. M. Hu. Associating inter-image salient instances for weakly supervised semantic segmentation.
In ECCV, 2018. 4[4] W. Ge, S. Yang, and Y. Yu. Multi-evidence filteringand fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning. In CVPR, 2018. 4[5] S. Hong, D. Yeo, S. Kwak, H. Lee, and B. Han.
Weakly supervised semantic segmentation using webcrawled videos. In CVPR, 2017. 4[6] Q. Hou, P. T. Jiang, Y. Wei, and M. M. Cheng.
Self-erasing network for integral object attention. In
Advances in Neural Information Processing Systems,
2018. 4[7] Z. Huang, W. Xinggang, J. Wang, W. Liu, and W. Jingdong. Weakly-supervised semantic segmentation network with deep seeded region growing. In CVPR,
2018. 1, 4
[8] B. Jin, M. Segovia, and S. Susstrunk. Webly supervised semantic segmentation. In CVPR, 2018. 4[9] D. Kim, D. Cho, D. Yoo, and I. Kweon. Two-phaselearning for weakly supervised object localization. In
ICCV, 2017. 4[10] A. Kolesnikov and C. H.Lampert. Seed, expand andconstrain: Three principles for weakly-supervised image segmentation. In ECCV, 2016. 1, 4
[11] P. Krahenbuhl and V. Koltun. Efficient inference infully connected crfs with gaussian edge potentials. In
Advances in Neural Information Processing Systems,
2011. 1[12] K. Li, Z. Wu, K.-C. Peng, J. Ernest, and Y. Fu. Tell me
where to look: Guided attention inference network. In
CVPR, 2018. 4[13] G. Papandreou, L.-C. Chen, K. Murphy, and A. L.
Yuille. Weakly-and semi-supervised learning of adcnn for semantic image segmentation. In ICCV,2015. 1, 4
[14] D. Pathak, P. Krahenbuhl, and T. Darrell. Constrained
convolutional neural networks for weakly supervised
segmentation. In ICCV, 2015. 1, 4
[15] D. Pathak, E. Shelhamer, J. Long, and T. Darrell. Fully
convolutional multi-class multiple instance learning.
In ICLR, 2015. 4[16] P. Pinheiro and R. Collobert. From image-level topixel-level labeling with convolutional networks. In
CVPR, 2015. 4[17] X. Qi, Z. Liu, J. Shi, H. Zhao, and J. Jia. Augmented
feedback in semantic segmentation under image levelsupervision. In ECCV, 2016. 4[18] A. Roy and S. Todorovic. Combining bottom-up, topdown, and smoothness cues for weakly supervised image segmentation. In CVPR, 2017. 4[19] F. Saleh, M. Akbarian, M. Salzmann, L. Petersson, S. Gould, and J. M.Alvares. Built-in foreground/background prior for weakly-supervised semantic segmentation. In ECCV, 2016. 4[20] O. Seong, B. Rodrigo, K. Anna, A. Zeynep, F. Mario,
and S. Bernt. Exploiting saliency for object segmentation from image level labels. In CVPR, 2017. 4[21] T. Shen, G. Lin, C. Shen, and R. Ian. Bootstrapping
the performance of webly supervised semantic segmentation. In CVPR, 2018. 4[22] W. Shimoda and K. Yanai. Distinct class saliency
maps for weakly supervised semantic segmentation.
In ECCV, 2016. 4[23] P. Tokmakov, K. Alahari, and C. Schmid. Weaklysupervised semantic segmentation using motion cues.
In ECCV, 2016. 4[24] X. Wang, S. You, X. Li, and H. Ma. Weaklysupervised semantic segmentation by iteratively mining common object features. In CVPR, 2018. 4[25] Y. Wei, J. Feng, X. Liang, M. Cheng, Y. Zhao, andS. Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation
approach. In CVPR, 2017. 1, 4[26] Y. Wei, X. Liang, Y. Chen, X. Shen, M. Cheng,
Y. Zhao, and S. Yan. STC: A simple to complex
framework for weakly-supervised semantic segmentation. In IEEE Trans. on PAMI, 2017. 4[27] Y. Wei, H. Xiao, H. Shi, Z. Jie, J. Feng, and T. Huang.
Revisiting dilated convolution: A simple approach forweakly- and semisupervised semantic segmentation.
In CVPR, 2018. 4