SCARF: A Semantic Constrained Attention Refinement Network
for Semantic Segmentation
Xiaofeng Ding1 , Chaomin Shen2 , Zhengping Che3 , Tieyong Zeng4 , Yaxin Peng1
1
Department of Mathematics, School of Science, Shanghai University
2
School of Computer Science and Shanghai Key Laboratory of Multidimensional Information Processing,East China Normal University 3 Didi Chuxing 4 The Chinese University of Hong Kong
{dxfeng, yaxin.peng}@shu.edu.cn, cmshen@cs.ecnu.edu.cn
chezhengping@didiglobal.com, zeng@math.cuhk.edu.hk

Abstract
Semantic segmentation has achieved great progress by
exploiting the contextual dependencies. In this paper, we
propose an end-to-end Semantic Constrained Attention ReFinement (SCARF) network, based on semantic constrained
contextual dependencies, to fully utilize the semantic information across different layers. Our novelties lie in the following aspects: Firstly, we present a general framework for
capturing the non-local contextual dependencies. Secondly,
within the framework, we introduce an efficient Category
Attention (CA) block to capture semantic-related context
by using the category constraint from coarse segmentation,
which reduces the computational complexity from O(n2 )to O(n) for image with n pixels. Thirdly, we overcome
the contextual information confusion problem by balancing
the non-local contextual dependencies and the local consistency adaptively using a category-wise learning weight. Finally, we fully utilize the multi-scale semantic-related contextual information by refining the segmentation iteratively
across layers with semantic constraint. Extensive evaluations demonstrate that our SCARF network significantly
improves the segmentation results and achieves superior
performance 85.0% mIoU on PASCAL VOC 2012, 55.0%
mIoU on PASCAL Context, and 82.1% mIoU on Cityscapes.

1. Introduction
Semantic segmentation, aiming to assign pixel-wise category labels for a given image, has been widely applied tovarious real-world applications such as autonomous driving[8, 13] and medical diagnosing [21]. Being a fundamental task in computer vision, it achieves great success. Nevertheless, challenges still remain. For instance, Fig. 1(c)(a) Raw image(b) Ground truth(c) Baseline(d) SCARF (Ours)

Figure 1. Example of a segmentation result from the Cityscapes
validation set. Compared with the baseline model, our network
called SCARF enhances the consistency of the intra-class features.

illustrates one problem of inaccurate segmentation led by
objects of different categories (bus and truck) have similarfeature representations. This paper attempts to provide asolution to this issue by aggregating features of same category to enhance the intra-class feature representation, as theresult shown in Fig. 1(d).
Pioneer work for semantic segmentation includes the
Fully Convolutional Network (FCN) [26]. Nevertheless,due to the short-range receptive field caused by its fixed
structure, FCN inherently does not make full use of thecontextual information, leaving room for improvement. Recently, the contextual information has shown its capability
for semantic segmentation. Specifically, it is aggregatedwith multi-scale dilation convolutions via the Atrous Spatial Pyramid Pooling (ASPP) module in DeepLab methods[6, 5], and extracted with pyramid pooling module in PSPNet [46]. The contextual information, however, is aggregated non-adaptively in these methods, not satisfying thedifferent contextual dependencies to different pixels.
To aggregate the contextual information adaptively, theattention-based methods are extensively studied. Point-wiseattention based methods [47, 38] connect all pixels in thefeature maps through the predicted attention map. Meanwhile, self-attention [32], a non-local contextual aggregation method, is employed by segmentation [36, 12, 45, 19]
to aggregate the contextual information for each position.
These methods, however, have two problems. First, theylead to the high computational complexity O(HW ×HW ),with HW the image size. Second, the attention mechanism is not clear with decreased pair-wise relationship fromstructure reasoning, since the attention map is completely
learned by back-propagation without prior information.
To address the above two problems, we propose a novel
network, called Semantic Constrained Attention ReFinement (SCARF), based on the semantic constrained contextual dependencies. We first present a general framework
for capturing the non-local contextual information. Within
this framework, we introduce a Category Attention (CA)block to capture the semantic context by using the category
constraint from coarse segmentation. More specifically, we
utilize the covariance matrix of coarse segmentation to generate the attention map, since the inner product of category
probability vectors inherently represents the pairwise relationship of different pixels, enhancing the structure reasoning of the attention block. With the help of the associative law, we provide an efficeint version of CA, called theefficient CA, reducing the computational complexity from
O(HW × HW ) to O(HW ). In addition, given different
contextual dependencies to different categories, we adaptively balance the non-local contextual dependencies andthe local consistency by introducing a category-wise learning weight, overcoming the contextual information confusion problem. Finally, our model refines the segmentation
iteratively across layers, fully utilizing the semantic information. In summary, our main contributions are as follows:• We introduce an efficient CA block to capture
semantic-related context by using the category constraint from coarse segmentation, reducing the computational complexity from O(HW × HW ) to O(HW ).
• Our model adaptively balances the non-local contextual dependencies and the local consistency by introducing a category-wise learning weight, overcoming
the contextual information confusion problem.
• Our model refines the segmentation iteratively acrosslayers, fully using coarse segmentation information.

progress by exploiting the feature context, including
single-branch and multi-branch methods. For single-branchmethods, SegNet [2], U-Net [29], RefineNet [25] adopt
encoder-decoder structure to fuse multi-level features;
Deeplab [6, 5] and PSPNet [46] collect the multi-scale
context by designing the pyramid modules; CCNet [20],CFNet [45] and DANet [12] employ the self-attention
method [36] to aggregate long-range spatial information;
Conditional random field methods [22, 48, 33, 4] are also
employed to harvest the feature context. For multi-branchmethods, DFN [40], UPerNet [37] and GSCNN [31] improve the segmentation branch by introducing other visualconcepts, such as boundary, textual and shape branches;
FuseNet [14], RTFNet [30], DFM-RTFNet [35] and SNERoadSeg+ [34] learn informative features with data-fusion,
which improves segmentation by using addition visual
modalities, such as depth and thermal images.
Attention mechanism. A rich body of literature investigates approaches for attaching attention mechanism to the
Deep Neural Networks (DNNs). Inspired by SENet [18],which attaches attention on the channel to select desiredfeature maps, BiSeNet [39], DFN [40] and EncNet [44]utilize channel-wise attention for semantic segmentation.
Meanwhile, [36] models pair-wise relationships by calculating the correlation matrix of feature maps as the attention
map, and it matches the self-attention mechanism [32]. Following this, DANet [12] takes both channel-wise attention
and self-attention into account, and CCNet [20] replaces
the self-attention block with criss-cross attention to reduce
computational complexity. Similarly, PSANet [47] and
CPNet [38] obtain the pair-wise attention map adaptively
via a single convolution layer. In addition, AA [10]integrates different types of attention modules and proposes
an attention aggregation framework.
Coarse-to-fine methods. Coarse-to-fine is a hierarchicalcontext mining mechanism widely used in applications suchas face detection [11], shape detection [1], and opticalflow [3]. Recently, a lot of deep network based segmentation methods [25, 43, 50] adopt coarse-to-fine strategy. RefineNet [25] proposes a multi-path refinement network to
enable high-resolution prediction using long-range residual
connections. ACFNet [43] utilizes the coarse segmentation
to calculate the class center of each category on fine stage.
CiSS-Net [50] introduces reinforce learning to realize thecoarse-to-fine strategy by treating the coarse segmentation
as environment. In this paper, we propose a simple yet effective coarse-to-fine segmentation framework by refining
the segmentation with semantic constraint.

2. Related Work
Semantic segmentation. As a pioneer, FCN [26] first introduces the pretrained network for semantic segmentation.
Following FCN, various nice works have made remarkable

3. Methodology
We first introduce a general definition of non-local operation for contextual information aggregation in Sec. 3.1,

softmax(a) category attention(a) point-wise attention(b) self-attention

Figure 2. Diagrams of two typical attention methods for contextual
information aggregation. The output signal Y is aggregated by theattention map A.

then present an efficient CA block in Sec. 3.2. Next, we
adaptively balance the non-local and local information by acategory-wise learning weight in Sec. 3.3. Finally, we construct the SCARF network in Sec. 3.4.

3.1. General framework of non-local block
The general definition of the non-local operation for contextual information aggregation is:Xyi = N (aij · g(xj )),(1)jwith xj ∈ RD the input signal and yi ∈ RD the output signal. The unary function g computes the embedding of input
signal. aij is a scalar representing the pairwise relationshipbetween positions i and j. N is a normalization operator.
The general definition (1) includes two special cases oftypical attention based methods, i.e., the point-wise attention and self-attention methods:• For point-wise attention methods [47, 38] shown in
Fig. 2(a), the unary function g is the identity embedding: g(xj ) = xj , and the attention map is A = f (X),
where f is a series of convolution layers with batchnormalization and activation function.
• For self-attention methods [36, 45, 12] shown in
Fig. 2(b), the unary function g(xj ) = Wg xj is a linear embedding, where Wg is a weight matrix to beTlearned. The attention map aij = eθ(xi ) φ(xj ) , whereθ(xi ) = Wθ xi and φ(xj ) = Wφ xj are embeddings.
P
The output signal is normalized by a factor j aij .
Here X = [x1 , x2 , · · · , xHW ]T is the input signal matrix,
and Y = [y1 , y2 , · · · , yHW ]T is the output signal matrix,
with HW the image size and D the channel dimension.
A = (aij )HW ×HW is the spatial attention map.
The non-local block is significant for semantic segmentation by capturing the long-range contextual dependencies.

(b) efficient category attention

Figure 3. Our proposed non-local blocks with category prior: (a)Category attention block with computation complexity O(HW ×
HW ), (b) Efficient category attention block with computation
complexity O(HW ).

However, it has two flaws. First, the relationship representation of the attention mechanism is less clear due to the lackof context prior on the attention map. Second, the non-local
block needs to generate the spatial attention map to measure the impact between different positions, leading to highcomputational complexity O(HW × HW ).
To overcome these two flaws, we apply the coarse-tofine strategy with supervision to the coarse segmentation.
To make the attention mechanism clear, we represent the
pairwise relationship by the similarity of coarse segmentation probability vectors. In addition, the main computationand time cost of non-local block is from the attention map.
We change the calculation order of non-local block to reduce the computation cost.

3.2. Efficient category attention
Attention with coarse segmentation. Here we first propose a category prior non-local block, called CA block, as
follows. We set the unary function g to the identity embedding g(xj ) = xj . As shown in Fig. 3(a), we construct the
non-local block by representing the definition (1) as
yi = P

X
1(pTi pj )xj ,
T
p
p
j
i
j
j

(2)

where aij = pTi pj is the pair-wise relationship, pi ∈ RC
is the coarse segmentation probability vector in position i,with C the category number, and [p1 , p2 , · · · , pHW ]T constitutes the coarse segmentation map P . The inner product of category probability vectors inherently represents thesimilarity of two vectors. Then, the contextual aggregationfrom input to output signal is:
Y = N AX = N P P T X.

(3)

Here N is the normalization matrix


1
P
0
0
pT
p
j
 1 j

..

N =
.
0
0


0
0 pT 1P pj
HW

j

(4)

HW ×HW

which is efficiently generated and applied by the elementwise calculation.
Reduction of computational complexity. We improve thestructure of non-local block from CA block to efficient CA
block by changing the calculation order. In fact, the attention map A shown in Fig. 3(a) is the covariance matrix ofcoarse segmentation A = P P T , which needs to be calculated generally. But in our case, it is not necessary due tothe associative law. As depicted in Fig. 3(b), we employ thefollowing calculation order
Y = N P (P T X)

(5)

to avoid calculating the covariance matrix. Specifically, forCA block, the generation of covariance matrix A = P P T
and the aggregation process (P P T )X result in the computational cost O(HW ×HW ×C) and O(HW ×HW ×D),
respectively. The CA block thus has quadratic complexity in the image size HW , causing high computational cost
since HW usually is very large. Differently, for efficient
CA block, the two matrix multiplication process P T X and
P (P T X) lead to computational cost O(HW × D × C) andO(HW × D × C), respectively. Therefore, the efficient
CA block reduces the computational cost from quadratic tolinear complexity in the number of HW , highly increasing
the computational efficiency.

3.3. Non-local and local information
Balance of non-local and local information. The nonlocal operation (2) reveals that the output signal yi is the
expectation of input signal xj related to the category similarity pTi pj , eliminating the local noise by aggregating the
global contextual information. This aggregation method,
however, is not appropriate for some categories whose feature information varies in a wide range. Aggregation ofsuch category may lead to the contextual information confusion, which means the method does not capture the useful class representation, and even provides the unreasonable
information to mislead the feature representation. For instance, if the background category contains both the sky
and building area, the aggregation process will result in thecontextual information confusion since their feature information would be totally different. Therefore, we propose abalance operation as:zi = λi · yi + (1 − λi ) · xi .

(6)Here zi is the fusion of non-local signal yi and local signal
xi . λ = [λ1 , λ2 , · · · , λHW ]T is an adaptively learning
weight to balance the non-local and local information
with λi ∈ [0, 1] the weight in position i. Equation (6)degenerates to (2) when the entry λi = 1 for all i, whichmeans only the non-local term contributes to the output.
Category-wise adaptively learning weight. In order toovercome the contextual information confusion caused by
the imbalance within category, we set the weight
λi = pTi w,(7)where w = [w1 , w2 , · · · , wC ]T is a category-wise weight
and the entry wc ∈ [0, 1] is the non-local weight for category c, which is adaptively updated by the deep network.
We employ the sigmoid function to restrict all entries of
the category-wise weight w from 0 to 1 to ensure λi ∈ [0, 1]
for all i, and initialize all entries close to 1 to enhance theeffect of non-local contextual information. By learning theweight w through back-propagation, our model adaptively
balances the non-local and local information, overcoming
the contextual information confusion problem. Then weconstruct the balance block as:
Z = ΛY + (I − Λ)X,
where I is the identity matrix, and T
p1 w 0
0

.
..
Λ= 0
0
0

0

pTHW w

(8)





(9)

HW ×HWis the weight matrix efficiently generated and applied by
the element-wise calculation.
Balance Category Attention block. By balancing the nonlocal and local information (8), we construct the Balance
Category Attention (BCA) block as:
Z = ΛN P P T X + (I − Λ)X.

(10)

The flowchart of BCA block (10) is shown in Fig. 4.

3.4. Semantic constrained attention refinement
Category attention residual module. Based on the BCA
block, we propose a category prior coarse-to-fine module,
called the Category Attention Residual (CAR) module.
As shown in Fig. 4, CAR takes the coarse logit Lcoarse(last feature map before the coarse segmentation P ) as
the input to generate the fine logit Lf ine . Specifically, we
first employ BCA block (10) to aggregate the long-range
contextual information of input signal X with the coarse
segmentation P . The output feature maps are given by
concatenating the output signal Z of BCA block with

proportion between coefficients of adjacent scales 12 .
BCA

softmax

4. Experiments

conv

CAR module

In this section, we introduce the implementation details
and show a series of evaluation on PASCAL VOC 2012 [9],PASCAL Context [27] and Cityscapes [8] datasets.

4.1. Implementation details

Efficient CA block

Balance block
BCA blockFigure 4. The Category Attention Residual (CAR) module.

the input signal X to avoid the information loss. Finally,
several convolution layers are utilized to aggregate thefeature information and generate the residual of coarse andfine logits.
Semantic constrained attention refinement network.
Based on the CAR module, we construct a multi-scalecoarse-to-fine network, called SCARF network. The overallarchitecture of our network is depicted in Fig. 5. We employ
the pretrained ResNet-101 [17] backbone with decoder layers as our base network, and employ the CAR module torefine the segmentation iteratively across layers:
Ls = CAR(Ls+1 , X s ),

(11)

s

where L represents the logit of scale s ∈ [1, 4], and CAR
is the CAR module. For each scale s, the feature map X s
is obtained by combining high level feature X s+1 and low
level feature F s , using concatenation and two convolution
layers. The CAR module takes the coarse logit Ls+1 andthe feature map X s as the input to obtain the fine logit
Ls , realizing a coarse-to-fine process from high to lowlevel. By iteratively refining the logits, we get the output
segmentation prediction L1 .
Loss function. Cross-entropy (CE) is utilized as the loss
function. Besides, we apply deep supervision [23] to all
logits to constrain the predicted segmentation probability
maps. To minimize the difference of the network output tothe ground truth, we weight the loss of different scales toa certain proportion. Suppose {αs }4s=1 are coefficients tobalance the segmentation losses for different scales. The
final loss is computed byl=4X

αs · CE(Ls , GT),

(12)

s=1

where GT is the ground truth, and the coefficient αs decreases with the scale s. We empirically set α1 = 1 and the

Network and loss. We employ ResNet-101 [17] pretrainedon ImageNet with ASPP [6] as the baseline backbone, andfollow [6, 44, 12] to apply the dilated strategy to the last
two ResNet blocks. Synchronized batch normalization [44]is utilized in the training phase. In addition, we adopt deepsupervision [23] to enhance the gradient flow of multi-scalemodels. Since deep supervision improves the performanceof all the models, we employ it for all the experiments(including baseline with decoders). All the predictions are
up-sampled by the bilinear interpolation to compute thesegmentation loss.
Optimization. Following [6, 44, 38], we employ a polyiter 0.9) , with γ0 thelearning rate scheduling γ = γ0 ·(1− NNtotal
base learning rate, Niter the current iteration number, andNtotal the total iteration number. We set the base learning
rate to 0.001 for all datasets. Momentum and weightdecay coefficients of Stochastic Gradient Descent (SGD)
optimizer are set to 0.9 and 0.0001, respectively. Besides,
the batch size is set to 8 for Cityscapes and 16 for otherdatasets. The training time is set to 80 epochs for PASCAL
VOC augmented set, 50 epochs for fine-tuning on PASCAL
VOC train+validation set, 80 epochs for PASCAL Context,
and 240 epochs for Cityscapes. For Cityscapes dataset, we
adopt the warm-up training strategy [17] with 5 epochs,
the Online Hard Example Mining (OHEM) [42], and ahierarchy of grids (multi-grid) [7, 12] of different sizes (4,8, 16) in the last ResNet block.
Data augmentation. In the training phase, we apply
random horizontal flip to the input images, and then scale
the images to the ratio of 0.5 to 2.0. Finally, we randomly
crop the images into the training size (480 × 480 for
PASCAL VOC 2012, 544 × 544 for PASCAL Context,
768 × 768 for Cityscapes).
Inference. We conduct comparison experiments with stateof-the-art algorithms on PASCAL VOC 2012, PASCAL
Context and Cityscapes datasets. During inference, we crop
images into training image size and feed them into the network for Cityscapes dataset. For other datasets, we get theprediction map by feeding the full image into the network.
In addition, multi-scale inputs and left-right flip are employed during evaluation. We follow [44] to average thesegmentation probability maps from multi scales for inference, where the scales is set to {0.75, 1.0, 1.25, 1.5, 1.75,

Backbone

Decoder

Res-4

Input

high level

Res-3

CAR

Res-2

CAR
Groud truth

CAR

Res-1

CAR

Deep
supervision

low level

Output

CAR module

Figure 5. An overview of the SCARF network, with {F s }4s=1 features of ResNet backbone, {X s }4s=1 the decoder features and {Ls }4s=1
output logits (segmentation) of different scales. The model iteratively refines the segmentation to generate the output segmentation L1 .

2.0} for Cityscapes and {0.5, 0.75, 1.0, 1.25, 1.5, 1.75} forother datasets. We adopt mean intersection of union (mIoU)
as the evaluation metric for PASCAL Context dataset. For
PASCAL VOC 2012 and Cityscapes datasets, we utilize thepublic server for evaluation.

4.2. Evaluations on PASCAL VOC 2012 dataset
Dataset description. PASCAL VOC 2012 dataset is one of
the gold-standard benchmarks for semantic segmentation,
including 10,582 augmented, 1,464 train, 1,449 validation,and 1,456 test images. The performance is measured interms of the mIOU across 21 classes.
Ablation studies of the CAR module to different layers.
In order to validate the effectiveness of the CAR module,
we apply it to the baseline model iteratively across layers.
Besides, both non-dilated and dilated cases (dilated strategy
for the backbone) are conducted for ablation studies. For
fair comparisons, decoder layers are applied to both thebaseline model and the SCARF network from high to
low levels (level 4 to 1). As shown in Table 1, SCARF
outperforms the baseline model on both non-dilated anddilated cases. In addition, the multi-scale refinement of
CAR module highly improves the performance.
MethodBaseline
Baseline
Baseline
Baseline
SCARF
SCARF
SCARF

4
√
√
√
√

3
√
√
√

√
√
√

√
√
√

2
√
√
√
√

1

√

√

Non-dilated

Dilated

73.4
78.7
79.1
78.8

80.0
80.0
80.2
80.4

79.3
79.7
80.7

81.1
81.2
81.6

Table 1. Ablation results for CAR module to different layers on
PASCAL VOC 2012 validation set (mIoU (%))

Ablation studies of BCA block. We also study the effectof the BCA block, as shown in Table 2. We apply CA
block and balance block to the baseline model with fullencoder layers (4321), respectively. SCARF network with
CA block (3) achieves 0.8%/1.0% improvement of themodel performance, and the network with both the CA andbalance blocks (10) makes 1.9%/1.2% improvement.
MethodCA

Baseline
SCARF
SCARF

√
√

BalanceNon-dilated

Dilated

√

78.8
79.6
80.7

80.4
81.4
81.6

Table 2. Ablation results for BCA block on PASCAL VOC 2012validation set (mIoU (%))

Variation of balance weight w for different categories.
The category-wise weight w balances the non-local andlocal information. In our experiments, we employ sigmoid
function to restrict the category-wise weight w from 0 to
1, and initialize all entries close to 1 to enhance the effectof non-local contextual information. Table 3 displays thevalue of weight w for all categories after training. As we
assumed, the non-local term plays the main role of thebalance block for almost all categories except backgroundcategory. For background category, the balance weightdecreases to 0.00%, which means the network adaptively
switches the contextual information from non-local to local
term, verifying the contextual information confusion of
background category and the importance of local term.
Ablation studies of computation cost. We study thecomputation cost of the BCA block by applying it tothe baseline model. We report the model performance,
memory and time cost in the inference stage with the batch
size 1. As shown in Table 4, the first and second rows
illustrate the results of the baseline model without and

BG
0.00

plane
99.81

bike
99.75

bird
99.92

boat
99.83

bottle
99.91

bus
99.91

car
99.75

cat
99.88

chair
99.95

cow
99.86

table
99.91

dog
99.86

horse
99.92

motor person plant
99.80 99.94 99.85

sheep
99.91

sofa
99.86

train
99.81

TV
99.91

Table 3. Value of balance weight w after training, with the weight
w initialized close to 1 for all categories. BG is the backgroundcategory.

Methods

Reference

Backbone mIoU (%)

PSPNet [46]DFN [40]
EncNet [44]DANet [12]CFNet [45]APCNet [16]DMNet [15]SANet [49]SpyGR [24]CVPR 2017
CVPR 2018
CVPR 2018
CVPR 2019
CVPR 2019
CVPR 2019ICCV 2019
CVPR 2020
CVPR 2020

Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101

82.6
82.7
82.9
82.6
84.2
84.2
84.4
83.2
84.2

Res101

85.0

SCARF

with decoder layers, respectively. For further comparison
with typical non-local methods, we add the self-attention
method [32] shown in the third row. Both the CA blockand self-attention block lead to high memory cost 2062M.
The efficient CA block highly reduces the memory cost of
CA block to 24M, overcoming the high computation cost
shortcoming of attention methods. In addition, the BCA
block further improves the model performance from 81.4%
to 81.6% without extra computation cost, which validates
the effectiveness of the balance block.
MethodBaseline (4)Baseline (1234)
Self-attention [32]CA
Efficient CA
BCA

Memory

Time

mIoU

1880M
1902M (+22)
3942M (+2062)
3942M (+2062)
1904M (+24)
1904M (+24)

0.092s
0.097s
0.112s
0.108s
0.098s
0.099s

80.0%
80.4%
81.2%
81.4%
81.4%
81.6%

Table 4. Ablation of computation cost on PASCAL VOC 2012 validation set.

Comparison with state-of-the-art. We evaluate the performance of SCARF network on the PASCAL VOC 2012segmentation dataset. The comparison results are shown in
Table 5. SCARF achieves 85.0% mIoU on the test set, outperforming previous works without COCO pretraining.

4.3. Evaluations on Cityscapes dataset
Dataset description.
Cityscapes dataset is a highresolution city street parsing dataset. 2,975, 500 and 1,525
fine annotated images captured from 50 different cities
are provided for training, validation and testing. In our
experiments, we only utilize the fine annotations including
19 categories for evaluation.
Visualization of the category attention maps. To understand the category contextual dependencies, we visualize
the pairwise similarity aij between a given pixel i and otherpixels for all j in the attention map A. As depicted in Fig. 6,
CA block aggregates the contextual information from pixels
with similar class information, enhancing both consistencyof intra-class features and differences of inter-class features.

Table 5. Quantitative evaluations on PASCAL VOC 2012 test set
without pretraining on COCO dataset.

(a) image(b) ground truth (c) segmentation (d) attention map

Figure 6. Pairwise similarity visualization of all pixels to a given
pixel. The selected pixel i is marked as + in (a) raw image, (b)ground truth, and (c) segmentation. (d) Attention map shows the
pairwise similarity Aij of all pixels for all j to selected pixel i.

.

Visualizations of SCARF model. For further understanding of our model, we visualize the segmentation
results of the baseline model and SCARF on Cityscapes
datasets shown in Fig. 7. The SCARF model improves theclassification accuracy since it eliminates the local noiseby aggregating the category-prior contextual information,
overcoming the contextual information confusion problem
and enhancing the differences between different categories.
Comparison with state-of-the-art. We compare SCARF
network with the existing methods on Cityscapes test set.
As illustrated in Table 6, the SCARF network achieves
82.1% mIoU on the Cityscapes test set, outperforming othermethods. Among previous works, ACFNet [43] utilizes
the coarse segmentation to calculate the center features ofdifferent classes. OCR [41] calculates the relationship between pixel and category with the coarse segmentation. Different from them, our SCARF network exploits the coarse
segmentation by efficiently capturing the pairwise relationship of different pixels and obtain better performance.

(a) image(b) ground truth(c) baseline

void

road

sidewalkbuilding

wall

fence

pole

terrainsky

person

rider

car

truck

bus

(d) SCARF

traffic light traffic sign vegetation
train

motorcycle

bicybleFigure 7. Visualization results on Cityscapes validation set.

MethodReference

Backbone mIoU (%)

MethodReference

Backbone mIoU (%)

RefineNet [25]PSPNet [46]DFN [40]
PSANet [47]
CFNet [45]SeENet [28]DANet [12]ACFNet [43]CPNet [38]SpyGR [24]
OCR [41]

CVPR 2017
CVPR 2017
CVPR 2018
ECCV 2018
CVPR 2019ICCV 2019
CVPR 2019ICCV 2019
CVPR 2020
CVPR 2020
ECCV 2020

Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101

73.6
78.4
79.3
80.1
79.6
81.2
81.5
81.8
81.3
81.6
81.8

RefineNet [25]PSPNet [46]EncNet [44]DANet [12]CFNet [45]APCNet [16]SpyGR [24]CPNet [38]SANet [49]
OCR [41]

CVPR 2017
CVPR 2017
CVPR 2018
CVPR 2019
CVPR 2019ICCV 2019
CVPR 2020
CVPR 2020
CVPR 2020
ECCV 2020

Res152
Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101
Res101

Res101

82.1

SCARF

Table 6. Quantitative evaluations on Cityscapes test set. We list
the methods training with the fine data.

4.4. Evaluations on PASCAL Context dataset
Dataset description. PASCAL Context dataset is a scene
parsing dataset, including 4,998 training and 5,105 testing
images. We conduct our experiments on the most frequent
59 classes with background class (60 classes in total).
Comparison with state-of-the-art. We conduct comparisons with state-of-the-art methods [44, 12] on PASCAL
Context dataset. As shown in Table 7, SCARF network
achieve 55.0% mIoU, highly outperforming other methods.

5. Conclusion
We have proposed the SCARF network. We first introduce the semantic constrained attention mechanism tocapture contextual information from coarse segmentation.
Then, we present the efficient CA block to capture seman-

47.3
47.8
51.7
52.6
54.0
54.7
52.8
53.9
54.4
54.8

SCARF
Res101
55.0
Table 7. Quantitative evaluations on PASCAL Context validation
set. (Note: mIoU on 60 classes w/ background.)tic constrained context. Besides, we adaptively balance thenon-local and local information by introducing a categorywise attention weight. Finally, our model refines the segmentation iteratively across layers with semantic constraint.
Extensive evaluations demonstrate that our model can efficiently capture the long-range contextual information withsemantic constraint layer-by-layer, enhancing the structure
reasoning of the model.

Acknowledgement
This work was supported in part by the National Key
R&D Program of China (2021YFE0203700), the National Natural Science Foundation of China under Grant
11771276, Grant NSFC/RGC (N CUHK 415/19), Grants
RGC (14300219, 14302920, 14301121), and CUHK Direct
Grant for Research (4053405, 4053460), Shanghai Scienceand Technology Innovation Action Plan (18441909000,
21S31901000).

References[1] Yali Amit, Donald Geman, and Xiaodong Fan. A coarseto-fine strategy for multiclass shape detection.
IEEE
Transactions on Pattern Analysis and Machine Intelligence,
26(12):1606–1621, 2004.
[2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.
SegNet: A deep convolutional encoder-decoder architecture
for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2481–2495, 2017.
[3] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim
Weickert. High accuracy optical flow estimation based on atheory for warping. In European Conference on Computer
Vision, pages 25–36, 2004.
[4] Siddhartha Chandra, Nicolas Usunier, and Iasonas Kokkinos. Dense and low-rank Gaussian CRFs using deep embeddings. In Proceedings of the IEEE International Conference
on Computer Vision, pages 5103–5112, 2017.
[5] Liangchieh Chen, Yukun Zhu, George Papandreou, FlorianSchroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
European Conference on Computer Vision, pages 833–851,
2018.
[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 40(4):834–848,
2017.
[7] Liang-Chieh Chen, George Papandreou, Florian Schroff, andHartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587,
2017.
[8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo
Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe
Franke, Stefan Roth, and Bernt Schiele. The Cityscapes
dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 3213–3223, 2016.
[9] Mark Everingham, Luc Van Gool, Christopher KI Williams,
John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) challenge. International Journal of Computer Vision, 88(2):303–338, 2010.
[10] Rui Fan, Hengli Wang, Mohammud J Bocus, and Ming Liu.
We learn better road pothole detection: from attention aggregation to adversarial domain adaptation. In European Conference on Computer Vision, pages 285–300, 2020.
[11] Francois Fleuret and Donald Geman. Coarse-to-fine facedetection. International Journal of Computer Vision, 41(12):85–107, 2001.
[12] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network forscene segmentation. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 3146–
3154, 2019.
[13] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we
ready for autonomous driving? the KITTI vision benchmark
suite. In Proceedings of the IEEE Conference on Computer

Vision and Pattern Recognition, pages 3354–3361. IEEE,
2012.
[14] Caner Hazirbas, Lingni Ma, Csaba Domokos, and DanielCremers. Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture. In Asian Conference on Computer Vision, pages 213–228, 2016.
[15] Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multiscale filters for semantic segmentation. In Proceedings of theIEEE International Conference on Computer Vision, pages
3562–3572, 2019.
[16] Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu
Qiao. Adaptive pyramid context network for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 7519–7528, 2019.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 770–778, 2016.
[18] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 7132–7141, 2018.
[19] Ping Hu, Fabian Caba, Oliver Wang, Zhe Lin, Stan Sclaroff,and Federico Perazzi. Temporally distributed networks forfast video semantic segmentation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 8818–8827, 2020.
[20] Zilong Huang, Xinggang Wang, Lichao Huang, Chang
Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-crossattention for semantic segmentation. In Proceedings of theIEEE International Conference on Computer Vision, pages
603–612, 2019.
[21] Ma Jun, Wang Yixin, An Xingle, Ge Cheng, Yu Ziqi,
Chen Jianan, Zhu Qiongjie, Dong Guoqiang, He Jian, He
Zhiqiang, Ni Ziwei, and Yang Xiaoping. Towards efficient
COVID-19 CT annotation: A benchmark for lung and infection segmentation. arXiv preprint arXiv:2004.12537, 2020.
[22] Philipp Krähenbühl and Vladlen Koltun. Efficient inferencein fully connected CRFs with Gaussian edge potentials. In
Advances in Neural Information Processing Systems, pages
109–117, 2011.
[23] Chenyu Lee, Saining Xie, Patrick W Gallagher, Zhengyou
Zhang, and Zhuowen Tu. Deeply-supervised nets. In International Conference on Artificial Intelligence and Statistics,
pages 562–570, 2015.
[24] Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen
Lin, and Hong Liu. Spatial pyramid based graph reasoning
for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
8950–8959, 2020.
[25] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian
Reid. Refinenet: Multi-path refinement networks for highresolution semantic segmentation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 1925–1934, 2017.
[26] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 3431–3440, 2015.

[27] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu
Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, andAlan Yuille. The role of context for object detection andsemantic segmentation in the wild. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 891–898, 2014.
[28] Yanwei Pang, Yazhao Li, Jianbing Shen, and Ling Shao. Towards bridging semantic gap to improve semantic segmentation. In Proceedings of the IEEE International Conference
on Computer Vision, pages 4230–4239, 2019.
[29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. UNet: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pages 234–241,
2015.
[30] Yuxiang Sun, Weixun Zuo, and Ming Liu. RTFNet: RGBthermal fusion network for semantic segmentation of urban
scenes. IEEE Robotics and Automation Letters, 4(3):2576–
2583, 2019.
[31] Towaki Takikawa, David Acuna, Varun Jampani, and Sanja
Fidler. Gated-scnn: Gated shape cnns for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5229–5238, 2019.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In Advances in Neural
Information Processing Systems, pages 5998–6008, 2017.
[33] Raviteja Vemulapalli, Oncel Tuzel, Ming-Yu Liu, and Rama
Chellapa. Gaussian conditional random field network forsemantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
3224–3233, 2016.
[34] Hengli Wang, Rui Fan, Peide Cai, and Ming Liu.
Sne-roadseg+: Rethinking depth-normal translation anddeep supervision for freespace detection. arXiv preprint
arXiv:2107.14599, 2021.
[35] Hengli Wang, Rui Fan, Yuxiang Sun, and Ming Liu. Dynamic fusion module evolves drivable area and road anomaly
detection: A benchmark and algorithms. IEEE Transactions
on Cybernetics, 2021.
[36] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018.
[37] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, andJian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418–434, 2018.
[38] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu,
Chunhua Shen, and Nong Sang. Context prior for scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12416–12425,
2020.
[39] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao,
Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation
network for real-time semantic segmentation. In Proceedings of the European Conference on Computer Vision, pages
325–341, 2018.

[40] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao,
Gang Yu, and Nong Sang. Learning a discriminative feature
network for semantic segmentation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 1857–1866, 2018.
[41] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Objectcontextual representations for semantic segmentation. Proceedings of the European Conference on Computer Vision,
2020.
[42] Yuhui Yuan and Jingdong Wang. Ocnet: Object context network for scene parsing. arXiv preprint arXiv:1809.00916,
2018.
[43] Fan Zhang, Yanqin Chen, Zhihang Li, Zhibin Hong, Jingtuo
Liu, Feifei Ma, Junyu Han, and Errui Ding. Acfnet: Attentional class feature network for semantic segmentation. In
Proceedings of the IEEE International Conference on Computer Vision, pages 6798–6807, 2019.
[44] Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang,
Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In Proceedingsof the IEEE Conference on Computer Vision and Pattern
Recognition, pages 7151–7160, 2018.
[45] Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan
Xie. Co-occurrent features in semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 548–557, 2019.
[46] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang
Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2881–2890, 2017.
[47] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen
Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wisespatial attention network for scene parsing. In Proceedingsof the European Conference on Computer Vision, pages 267–
283, 2018.
[48] Shuai Zheng, Sadeep Jayasumana, Bernardino RomeraParedes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang
Huang, and Philip HS Torr. Conditional random fields asrecurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1529–1537,
2015.
[49] Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu,
Ibrahim Ben Daya, Zhifeng Li, Wei-Shi Zheng, Jonathan Li,
and Alexander Wong. Squeeze-and-attention networks forsemantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
13065–13074, 2020.
[50] Yizhou Zhou, Xiaoyan Sun, Zheng-Jun Zha, and Wenjun
Zeng. Context-reinforced semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 4046–4055, 2019.