Supplement Material
SCARF: A Semantic Constrained Attention Refinement Networkfor Semantic Segmentation
Xiaofeng Ding1 , Chaomin Shen2 , Zhengping Che3 , Tieyong Zeng4 , Yaxin Peng1
1
Department of Mathematics, School of Science, Shanghai University
2
School of Computer Science and Shanghai Key Laboratory of Multidimensional Information Processing,East China Normal University 3 Didi Chuxing 4 The Chinese University of Hong Kong
{dxfeng, yaxin.peng}@shu.edu.cn, cmshen@cs.ecnu.edu.cn
chezhengping@didiglobal.com, zeng@math.cuhk.edu.hk

A. Supplement Material

self-attention
CAefficient-CABalance

Visualization of SCARF model. We visualize the segmentation results in Fig. 1. The CA block improves theclassification accuracy of the foreground categories since
it eliminates the local noise by aggregating the categoryprior contextual information. In addition, the BCA block
further improves the segmentation results of backgroundcategory, implying that the BCA block overcomes the contextual information confusion problem and enhances thedifferences between foreground and background categories.

Memory(MB)2000

We report some additional experimental results in thesupplement material due to the page limit in the main submission. This supplement material consists of experiments
on Pascal VOC 2012 dataset.

1500
1000
500032x32

64x6496x96128x12896x96

128x128

Image size HxW

1e 4

self-attentionCAefficient-CABalance

17.5

Time(s)

15.0
12.5
10.0
7.5
5.0
2.5

(a) image(b) GT(c) baseline(d) CA(e) BCA

Figure 1. Visualization of segmentation results on PASCAL VOC
2012 validation set: (a) raw image, (b) ground truth, (c) baselinemodel, (d) SCARF with CA, (e) SCARF with BCA.

Ablation studies for computation cost. In the manuscript,we have conducted the ablation studies for computation cost(memory and time) of our proposed model (full network).
In this section, we further design one simple yet effective
experiment to directly evaluate the computation cost of different contextual information aggregation methods (only

32x32

64x64

Image size HxWFigure 2. Comparison of computational costs (memory and time)
for self-attention, CA, efficient CA and BCA methods.

module). Suppose the input feature channel number D is128, and the category number C is 21. The image size
HW varies from 32 × 32 to 128 × 128. Fig. 2 illustrates
the computational cost of different methods, including self-Method

mIoU

MemoryTime

Parameters

PSPNetDANet
EncNet

82.6
82.6
82.9

2300M
2111M
1976M

0.101s
0.107s0.102s513,965kb
520,785kb
473,140kb

Baseline
SCARF

82.6
85.0

1880M
1904M

0.092s
0.099s459,636kb
470,534kb

Table 1. Cost comparisons on the PASCAL VOC 2012 test set.
Deepsupervisionattention, CA, efficient CA and balance methods. The computation cost of self-attention and CA methods highly increases with the image size due to the generation of the attention A with computational complexity O(HW × HW ).
Our proposed efficient CA method highly reduces both thememory and time costs of the CA method with computational complexity O(HW ). In addition, our methods (efficient CA and balance) achieve extremely low computational costs over self-attention method for feature fusion ofcontextual information.
Furthermore, we provide the comparisons of computation cost with some recent methods, including PSPNet,DANet, and EncNet. All methods are tested with thesame experiment environment, preprocessing, and encoderarchitecture. We report the segmentation accuracy on thePASCAL VOC test set, the computation cost (memoryand time) of single scale inference, and the model parameters. Table 1 shows that SCARF achieves the superior
performance over other methods with low computation andparameter costs. Specifically, our model do not lead tohigh computation and parameter costs since we adopt smallfeature channel C = 128 for all decoder layers. Most ofthe computation and parameter costs are from the encoderarchitecture.
Methods

DS

Non-dilated

DilatedSCARF
SCARF√

80.6
80.7

81.0
81.6

Table 2. Ablation results for deep supervision to SCARF network
on PASCAL VOC 2012 validation set (mIoU (%))

Ablation studies of deep supervision to SCARF network. As shown in Table 2, deep supervision improves
the performance of SCARF network from 80.6%/81.0%
to 80.7%/81.6%, indicating the significant effect of deepsupervision to the SCARF network. Furthermore, weshow the segmentation of all layers in SCARF networkwith/without deep supervision on the PASCAL VOC 2012validation set. As shown in Fig. 3, SCARF network has arough trend to learn the segmentation even without deep supervision, implying the consistency of the SCARF networkand deep supervision. SCARF network without deep supervision, however, leads to segmentation confusion shown inDeepsupervision(a) image(b) L4(c) L3(d) L2(e) L1Figure 3. Visualization of segmentation results and attention maps
without (with) deep supervision for different layers on PASCAL
VOC 2012 validation set. SCARF network without deep supervision (first and second rows) leads to the segmentation confusion ofhigh level layer L4 . In addition, SCARF network tends to learn thesegmentation iteratively from high level L4 to low level L1 bothwith and without deep supervision.

.

the first (second) row second column of Fig. 3, decreasing
both the effect of attention process, which to some extent
explains the influence of deep supervision to SCARF network.