E-CRF: Embedded Conditional Random Field for Boundary-caused Class Weights Confusion in Semantic Segmentation
Authors: Jie Zhu
Huabin Huang
Banghuai Li
Leye Wang
Venue: ICLR 2023 poster (2023)

Abstract: Modern semantic segmentation methods devote much effect to adjusting image feature representations to improve the segmentation performance in various ways, such as architecture design, attention mechnism, etc. However, almost all those methods neglect the particularity of class weights (in the classification layer) in segmentation models. In this paper, we notice that the class weights of categories that tend to share many adjacent boundary pixels lack discrimination, thereby limiting the performance. We call this issue Boundary-caused Class Weights Confusion (BCWC). We try to focus on this problem and propose a novel method named Embedded Conditional Random Field (E-CRF) to alleviate it. E-CRF innovatively fuses the CRF into the CNN network as an organic whole for more effective end-to-end optimization. The reasons are two folds. It utilizes CRF to guide the message passing between pixels in high-level features to purify the feature representation of boundary pixels, with the help of inner pixels belonging to the same object. More importantly, it enables optimizing class weights from both scale and direction during backpropagation. We make detailed theoretical analysis to prove it. Besides, superpixel is integrated into E-CRF and served as an auxiliary to exploit the local object prior for more reliable message passing. Finally, our proposed method yields impressive results on ADE20K, Cityscapes, and Pascal Context datasets.

Published as a conference paper at ICLR 2023
E-CRF: EMBEDDED CONDITIONAL RANDOM FIELD FOR BOUNDARY-CAUSEN CLASS WEIGHTS CONFUSION
IN SEMANTIC SEGMENTATION
Jie Zhu Huabin Huang* Banghuai Li Leye Wang1.2+
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China
School of Computer Science, Peking University, Beijing, China? 2
MEGVII Technology
zhujie @ stu. wwu.pku..du.cm (huanghuabin libanghuai] J @gmail.com, leyewang @ pku.edu.cn
libanghaaj@@gmail.coomm
ABSTRACT
Modern semantic segmentation methods devote much effect to adjusting image feature representations to improve the segmentation performance various ways, such as architecture design, attention mechnism, etc. However, almost all those methods neglect the particularity of class weights (in the classification layer) in segmentation models. In this paper, notice that the class weights of categories that tend to share many adjacent boundary pixels lack discrimination, thereby limiting the performance. We call this issue Boundary-caused Class Weights Confusion (BCWC). We try to focus on this problem and propose a novel method named Embedded Conditional Random Field (E-CRF) to alleviate it. B-CRF innovatively fuses the CRF into the CNN network as an organic whole for more effective end-to-end optimization. The reasons to guide the message passing between pixels in high-level features to purify the feature representation of boundary pixels, with the help of inner pixels belonging to the same object. More importantly, it enables optimizing class weights from both scale and direction during backpropagation. We make detailed theoretical analysis to prove it. Besides, superpixel is integrated into E-CRF and served as an auxiliary to exploit the local object prior for more reliable message passing. Finally, our proposed method yields impressive results on ADE20K, Cityscapes, and Pascal Context datasets.
two folds. It utilizes CRF
INTRODUCTION
Semantic segmentation plays an important role in practical applications such as autonomous driving. image editing, etc. Nowadays, numerous CNN-based methods (Chen al.) 2014 Fu et al. 2019 Ding et al. 2019) have been proposed. They attempt to adjust the image feature representation of the model itself to recognize each pixel correctly. However, almost all those methods neglect the particularity of class weights (in the classification layer) that play an important role in distinguishing pixel categories in segmentation models. Hence, it is critical to keep class weights discriminative. Unfortunately, CNN models have the natural defect for this. Generally speaking, most discriminative higher layers in the CNN network always have the larger receptive field, thus pixels around the boundary may obtain confusing features from both sides. As result, these ambiguous boundary pixels will mislead the optimization direction of the model and make the class weights of such categories that tend to share adjacent pixels indistinguishable. For the convenience of illustration, we call this issue as Boundary-causer Class Weights Confusion (BCWC). We take DeeplabV3+ (Chen) al. 20) as example to train ADE20K (Zhou et al.] (017)) dataset. Then, count the number of adjacent pixels for each class pair and find a corresponding category that has the most adjacent pixels for each class. Fig aa) shows the similarity of the class weight between these pairs in descending order according to the number of adjacent pixels. It clear that if two categories share more adjacent pixels, their class weights tend be more similar, which actually indicates that BCWC makes class representations lack discrimination and damages the overall segmentation performance.
"Corresponding author.

---

Published as a conference paper at ICLR 2023
Previous works mainly aim improve boundary pixel segmentation, but they seldom explicitly take class weights confusion i..., BCWC, into consideration
Painwbe
F
FP
baseline
0.20
ours
class wengiiitimmmtt
0.33
F
F
Adjacent pioels numberidescanddgggnrrod order)
massage
(8)
(b)
Figure (a) Observations on ADE20K. We find a corresponding category that shares the most adjacent pixels for each class and calculate the similarity of their class weights. X-axis stands for the number of adjacent pixels for each class pair in descending order, and Y-axis represents the similarity of their class weights. Blue line denotes baseline model while orange line denotes E-CRF. Apparently, two categories that share more adjacent pixels are inclined to have more similar class weights, while E-CRF effectively decreases the similarity between adjacent categories and makes their class weights more discriminative. (b) Message passing procedure of E-CRF.
the original feature maps of the CNN network. E-CRF utilizes pairwise module vi and auxiliary superpixel-based module vi! F to obtain refined feature maps FP and F respectively. Then F, FPP and F are fused as F to further segment the image.
Considering the inherent drawback of CNN networks mentioned before, delving into the relationship between raw pixels becomes a potential alternative to eliminate the BCWC problem, and Conditional Random Field (CRF) (Chen al., 2014) stands out. generally known that pixels of the same object tend to share similar characteristies in the local area. Intuitively, CRF utilizes the local consistency between original image pixels to refine the boundary segmentation results with the help of inner pixels of the same object. CRF makes some boundary pixels that are misclassified by the
CNN network quite easy to be recognized correctly. But these CRF-based methods (Chen et al. [2014] Zhen et al. 2020a) only adopt CRF as offline post-processing module, call it Vanilla- CRFF to refine the final segmentation results. They are incapable relieving BCWC problem as CRF and the CNN network are treated as totally separate modules.
Based on Chen et al. 2014 2017a), Lin et al. (2015); Arnab et al. (0066): Zheng et al. (2015) go step further unity the segmentation model and CRF in single pipeline for end-to-ent training. We call it Joins-CRi for simplicity. Same as Vanilla-CRFF Coi inclines to rectify those misclassified boundary pixels via increasing the prediction score of the associated category, which means it still operates on the object class probabiliries. But it can alleviate the BCWC problem some extent as the probability score refined by CRF directly involves in the model backpropagation. Afterwards, the disturbing gradients caused by those pixels will be relieved, which will promote the class representation learning. However, as shown Fig 3 the effectiveness of Joint-CRF is restricted as it only optimizes the scale of the gradient and lacks the ability to optimize class representations effectively due to the defective design. More theoretical analysis can be found in Sec. 3.3
To overcome the aforementioned drawbacks, in this paper, present novel approach named Embedded CRF (E-CRF) to address the BCWC problem more effectively. The superiority of E-CRF lies in main aspects. On the one hand, by fusing CRF mechanism into the segmentation model, E-CRF utilizes the local consistency among original image pixels to guide the message passing of high-level features. Each pixel pair that comes from the same object tends obtain higher message passing weights. Therefore, the feature representation of the boundary pixels can be purified by the corresponding inner pixels from the same object. In turn, those pixels will further contribute to the discriminative class representation learning. On the other hand, it extends the fashion of optimizing class weights from one perspective (i.e., scale) two (i.e., scale and direction) during
'These methods improve boundary segemetation and may have effect class weights. But they are not. explicit and lack theoretical analysis. We show great henifit of explicitly considering BCWC issue. See A.4.1
2

---

Published as a conference paper at ICLR 2023
Backbona
Class weight
Joint CRF
L-CRF
weight
Superpixels
Figure 2: Illustration of Joint-CRF and E-CRF. The first row is the simplified structure of Joint-CRF, which unifies the CNN network and CRF in a single pipeline for end-to-end training. However, CRF only serves post-processing module. The second row is the overview of our E-CRF, which fuses CRF into CNN network as an organic whole to eliminate BCWC problem.
backpropagation. In Sec. 3.3 we prove theoretically that E-CRF outperforms other CRF-based methods eliminating the BCWC problem by optimizing both direction and scale of the disturbing gradient of class weights. However, during this process, the noise information can also have direct influence on the class weights (likely to hinder the optimization for the BCWC problem). In addition, E-CRF adopts superpixel (Ren & Malik 2003) as an auxiliary and leverage its local prior to suppress the noise and further strengthen the reliability of the message passing the boundary pixels. Superpixel groups adjacent pixels that share similar characteristics to form block. prone to achieve clear and smooth boundaries and increases the potential for higher segmentation performance. In E-CRF, we average the deep feature representation of all inner pixels in the same superpixel block and then add this local object prior to each pixel back to enhance the representation of boundary pixels.
In this work, we explicitly propose the BCWC problem in semantic segmentation and an effective approach to alleviate it. We conduct extensive experiments on three challenging semantic seg- mentation benchmarks, iee. ADE20K (Zhou et al., 2017), Cityscapes (Cordts al., (016), and Pascal Context (Mottaghi et al., 2014). and yeild impressive results. For example, E-CRF outper- forms baselines (DeeplabV3+ CCen et al. 2018 with ResNet-101 (He et al. 2016)) by 1.42% mloU on ADE20K and 0.92% mloU on Cityscapes in single scale. In addition, we make ex- haustive theoretical analysis in Sec, 3.3 to prove the effectiveness E-CRF. Code is available at https://qithu.. http:///htuu.cmmmmeeeteee wwt.o////eee///ooo
2 RELATED WORK
Semantic Segmentation. Fully convolutional network (FCN) (Lnng al. 2013) based methods have made great progress in semantic segmentation by leveraging the powerful convolutional features of classification networks (He et al.) 2016 Huang al. 2017) pre-trained sakovsky et al.) 2015). There are several model variants proposed to enhance contextual aggregation. For example, De (Chen al. 2017a) and V3 (Chen et al. 2017b) take advantage of the astrous spatial pyramid pooling (ASPP) to embed contextual information, which consists of parallel dilated convolutions with different dilated rates to broaden the receptive field. Inspired by the encoder-decode structures (Ronneberger et al. 2015 Ding et al. 2018), Deeplab (Chen al. 2018a) adds decoder upon DeeplabV3 to refine the segmentation results especially along object boundaries. With the success of self-attention mechanism in natural language processing, Non-local (Wang al. 2018) first adopts self-attention mechanism tasks, such video classification, object detection and instance segmentation. AeNet (Chen et al. 2018b) proposes the double attention block to distribute and gather informative global features from the entire spatio-temporal space of the images.
large-scale data (Rus
module for computer vision
Conditional Random Fields. Fully connected CRFs have been used for semantic image labeling in (Payet & Todorovic, 2010; Toyoda & Hasegawa, 2008), but inference complexity in fully connected models has restricted their application to sets of hundreds of image regions or fewer. To address this issue, densely connected pairwise potentials (Krähenbühl & Koltun 2011) facilitate interactions

---

Published as a conference paper at ICLR 2023
between all pairs of image pixels based mean field approximation to the CRF distribution. Chen et al. (2014) show further improvements by post- processocsssing the results of CNN with a CRF. Subsequent works (Lin al. |015|Amab Arnab al. 2016: Zheng et al. 2015) have taken this idea further by incorporating a CRF as layers within a deep network and then learning parameters of both the CRF and CNN together via backpropagation. In terms of enhancements conventional CRF models, Ladický et al. (2010) propose using an off-the-shelf object detector to provide additional cues for semantic segmentation.
Superpixel. Superpixel (Ren & Malik 2003) is pixels with similar characteristics that are grouped together to form a large block. Since its introduction in 2003, there have been many mature algorithms (Achanta et al. 2012; Weikersdorfer et al.) 2013) 3: Van den Bergh et al., 2012). Owing to
their representational and computational efficincc,, superpixels are widely-used computer vision algorithms such as target detection (Shu et al. 2013 Yan et al. 2015), semantic segmentation (Gould
al. 2008 [Sharma et al., 2014 Gadde et al. 2016). and saliency estimation (He al. 2013 Perazzi
ct al. 2012). Yan et al. (2015) convert object detection problem into superpixel labeling problem
and conducts an energy function considering appearance, spatial context and numbers of labels.
Gadde et al. (2166) use superpixels to change how information is stored in the higher level of a CNN.
In (He et al. 2013), superpixels are taken as input and contextual information is recovered among superpixels, which enables large context be involved in analysis.
We give detailed discussion about the difference between E-CRF and three highly related works including PCGrad eu al. 20206), OCNet Yuan & Wang 2018), and SegFix (Yuan et al. 2020b) in Appendix A.5
3 METHOD
3.1 REVISITING CONDITIONAL RANDOM FIELD (CRF)
CRF is typical diseriminative model suitable for prediction tasks where contextual information or the state of the neighbors affects the current prediction. Nowadays. it is widely adopted in the semantic segmentation field (Krähenbühl & Koltun| [011]] Chen et al., 2014). CRF utilizes the corrllation between original image pixels to refine the segmentation results by modeling this problem as the maximum a posteriori (MAP) inference in a conditional random field (CRF), defined over original image pixels. In practice, the most common way is to approximate CRF as message passing procedure among pixels and can be formulated as:
Y Z, (va(1) + Σ (o(1)))
where Y; and Y," are defined as the classification scores of CNN model and CRF respectively for pixel Z, the normalization factor known as the partition function, and (ia a unary function which often adopts Y; as the default value. G is the associated pixel set with pixel i. For example, DenseCRF (Krähenbühl & Koltun| (01T)) takes all other pixels except pixel i itself as the set G. Moreover, the pairwise function (p())) defined to measure the message passing weight from pixel j to pixel i. It is formulated as:
(p((.)) p(i.)) Σ (m)) (mm) f,)
(2)
k(f.f.))
where (i(i,) is a label compatibility function that introduces the co-occurrent probability for specific label pair assignment at pixel and j. while k(f,, f,) is a set of hand-designed Gaussian kernels, f; and are feature vectors of pixel i and j in any arbitrary feature space, such as RGB images. (m) is the corresponding linear combination weight for each Gaussian kernel. When dealing with multi-class image segmentation, M=2 is a common setting. Then, kff, fy) is carefully designed sontrist-sensitvve two-kernel potentials, defined in terms of color vectors (J,, I5) and
4

---

Published as a conference paper at ICLR 2023
position coordinates (pt. Pÿ) for pixel i and respectively:
k(f,, f,) = w1) exp
Pt ps
11
I 203
+ (2) exp
Pi pj
20
202
appearonce
kernel
smoothness
kernel
The appearance kernel is inspired by the observation that nearby pixels with similar colors are more likely to share the same class. 8a and fg scale factors to control the degree of these two elements, ie., similarity and distance between two pixels. Apart from this, the smoothness kernel further removes the influence of some small isolated regions (Krähenbühl & Koltun 2011) and 0, is the associated scale factor. Notably, all these parameters are learnable during the model training.
Unfortunately, current CRF-based methods (Chen al., 2014 2017a| Lin al., 2015 Liu al.) 2015) for semantic segmentation always adopt CRF as post-processing module. For example, VanillaCRRF (Chen et al.
2014, 2017a) utilizes CRF to refine segmentation scores offline, which has no impacts on BCW since the CNN network and CRF are treated as two separate modules. Joint-CRF (Lin et al., 2015 Liu al. 2015] Lin et al., 2016) works in a similar way although CRF involves in the backpropagation of CNN networks, restricting its ability to relieve BCWC.
3.2 EMBEDDED CRF
To solve the BCWC problem in a more intrinsical way, we propose novel method named Embedded CRF (E-CRF) to tackle the tough problem via fusing the CRF mechanism into the CNN network as an organic whole for more effective end and-to-end training. An overview of E-CRF can be found in Fig 2 and formulate its core function based on Eq (1 as:
F
1 Z,
+ Σ (( j
Specifically, the first two terms are analogous to Eq T but we perform CRF mechanism on the high-level features. F; stands for the original output of feature extractors for pixel i, vi) and dy (i: play the same role as they do in Eq (1). (i) takes F, as the default value. addition, we reformulate vnf (i, to perform message passing between pixel pairs in the high-level feature:
(i.)) = rt (f),
worth noting that k(f., f)) is no longer hand-designed Gaussian kernels as Eq I but simple convolution operators instead make the whole model more flexible for end-to-ens traming and optimization. Experiments in Sec. prove this modification a more suitable choice:
(ff,, f2) = f;
conv((:; conv((U;
where [ar, y] denotes the concatenation operator. Different from Eq 33) we normalize the input image 7 into the range [o, to eliminate the scale variance between pixels and we replace original absolute position coordinates p with cosine position embeddings (Vaswani et al.) (017) to make it more compatible with CNN networks. E-CRF encodes the appearance and position pixels into more discriminative tokens via the flexible convolution operation, then the dot product is adopted to measure the similarity between pixel pairs. As indicated in Eq (6). E-CRF intends to make nearby pixel pairs that share same appearance to achieve higher k(f,,, fy). Its intention is the same Eq 33) Correspondingly, we also adjust μd (i,j) as the feature compatibility to measure the co-occurrent probability of F; and Fy:
nd (i, = sigmooid(comm/! [F,, F2))
Another component in Eq 4 is F.S. relies on the superpixel algorithm Renn & Malik 2003 Weikersdorfer al., 2013 Van den Bergh et al. 2012 Gadde al. (016) divide the whole image into several overovergg blocks. Pixels in the same superpixel block tend to share the same characteristics. Thus, we adopt this local object prior achieve the more effective message passing between pixels in the high-level feature space. Concretely, we design F, as:
Q
FSS = Σ
(((())( Σ F
(8)
5

---

Published as a conference paper at ICLR 2023
WW
R
19
VW
8 WW
0g
VW
W
8
W.
(a)
(b)
(c)
Figure 3: Different optimization effects for baseline, Joint-CRF and E-CRF. W1 and W2 are two class weight vectors that share adjacent pixels. VW is the gradient variation for W1 and W is the new class weight after gradient descent. F5 is a sample boundary pixel whose ground-truth label keeps consistent with Wi but contains confusing features from both sides. measures the distance between W₂ and W (a) VW tends to push W1 towards W₂ due to the confusing features from both classes. (b) Joint- eases the disturbing gradients and reduces the scale of VW. Obviously, 02 is larger than 0,. (c) E-CRF aims to enhance the feature representation of F3 via the inner pixels like Fj from the same object. adjusts both scale and direction of VW to make fg 02 > 01.
Q is the associated superpixel block that contains pixel i and v (l) devotes the re- re-weighting factor for the deep features of pixel in Q. We adopt // (4) = 1 and is the total number of pixels in supplement in Eq (4) to add the local object prior each pixel back, which
Q. F.S. serves increases the reliability of message passing in E-CRF. What's more, superpixel (Gould et al. 2008 Sharma al.) 2014 Gadde al. 2016) always tends generate clearer and smoother boundary segmentation results than traditional CNN networks or CRFs do, which also increases the potential for more accurate segmentation results. Detailed experiments can be found in Sec.
3.3 HOw E-CRF RELIEVES BCWC
In this section, without loss of generality, we take multi-class segmentation problem an example to dive into the principle of E-CRF from the perspective of gradient descent. Suppose F3 is the feature vector of a foreground boundary pixel l whose class label is E [0, n 1] and its prediction probability is Pt. Then, considering the label yk is one-hot form, the typical cross-ents cross-entropy loss Lx can be defined as:
i<n
La =
Ue In P = -In PE,
(9)
1=0
Ph = softmax
Σ et")) ) ee and = C-W.. Fk. møe
(10)
where W. is the class weight c-th category. Y is calculated by other class weights and unrelated with W. and Y,S Below the gradient variation VW, can be formulated as:
OL OL aP OYS
VW
(11)
aW. OP OY. aW.
Through Eq 9.. Eq 10 and Eq class weight in the next iteration will be updated
2
W W. VW, (1 Fi.
(12)
As shown in Eq 122 the direction of the gradient descent keeps the same as F& while the magnitude of the gradient is decided by Pg. What happens if we integrate the CRF into the segmentation pipeline? As we have discussed in Sec. [ Vanilla- CRF has nothing do with the optimization process of CNN networks, while if we adopt Joint-CRF, VW, can be reformulated as:
1
-W, =1 P Fx = (1
W.P PE) FF
(13)
Zx
jEG
scale
"The detailed derivation process be found in our Appendix
A.I

---

Published as a conference paper at ICLR 2023
where Pf is the refined score by CRF, w is the message passing weight from pixel j to pixel & and Pÿ is the original score of pixel j. In general, boundary pixel &: is hard to classify correctly due to the confusing features from both sides. Thus the original probability PE is always small. In contrast, other inner pixels of the same object are easy to recognize and tend to achieve a higher probability. Consequently, PE is usually larger than PE and disturbing gradients caused by boundary pixel will be relieved some extent, which makes inter-class distance further shown in Fig b)). However, Eq (13) only adjusts the scale of the gradient descent while the direction still keeps the same as Fk. which weakens its effects for better representation learning. When comes to our proposed E-CRF, VW, can be further defined as:
-V = (1 (1- Pk
F (k-Pk)
Σ
(14)
Zk
scate
JEG
direction
P = softmau
Σ jEG
w W + +X)
(15)
where FE the refined feature representations by E-CRE, and PE is the refined score which is analogous to PE in Eq (13). Comparing with Joint-CRF, it is clear that E-CRF not only changes the scale of the gradient descent but also adjusts its optimization direction. The optimization process is directly applied to the class weight matrix (in the final layer), which opens up room for more discriminative class weights. In other words, we can adjust the class weight from both the scale and direction to make the class weights more discriminative to decrease the class weights similarity (or class weights confusion). As depicted in Fig ce), assume Wi the class weight vector that a pixel belongs to, while W2 is the other one which has higher co-occurrent probability with W1 in the same image. E-CRF designs an effective message passing procedure to purify the feature representation of boundary pixels assisted by inner pixels from the same object (F2 in Fig c)). In this way, relieves the influence of disturbing gradients and makes the inter-class distance between W1 (WT) and W2 further, which means more discriminative feature representations.
4 EXPERIMENT
4.1
IMPLEMENTATION DETAILS
We follow the previous works Chen et al. 2014 He et al. 2019b Chen et al.
2018a)
and perform (Zhou et al.
experiments on three challenging semantic segmentation benchmarks, Lee,, A DE20K
2017). Cityscapes (Cordts et al. 2016) and Pascal Context (Mottaghi et al. space limit, detailed description of these three datasets can be found in our Appendix A. We adopt V3+ (Chen et al. 2018a) with ResNet (He et al. (016)) pretrained on ImageNet (Russakovsky al. 2015) as our baseline implement E-CRF. The detailed information follows standard settings in (Chen et al. 2014 8a) and we add it into our Appendix A. Specially, we employ SLII (Achanta al. 2012). common superpixel segmentation algorithm, to divide each image of ADE20K, Cityscapes and Pascal Context into 200, 600, and 200 blocks respectively. Note that the superpixel is generated offline. To verify the effectiveness of our approach for semantic segmentation, we adopt two common metrics in our experiments, i.e., class-wise mloU to measure the overall segmentation performance and 1 lipiell boundary F-score (Takikawa et al. 2019 Tan et al. 2023) to measure the boundary segmentation performance.
2014). Due to the
4.2
ABLATION STUDY
4.2. 1
COMPARISONS WITH RELATED METHIOSS
As shown in Table 1 we compare our proposed E-CRF with other traditional CRF-based methods, ie.. Vanilla-CRF and Joint-CRE, First of all, it is clear that all the CRF-based methods outperform the baseline model by large margin, which well verifies the main claim in (Chen et al. 2014 70 Lin et al. 2015 Liu et al. 2013 Lin et al. 2016) that CRF is beneficial to boundary segmentation (F-score). What's more, E- CRF achieves the best result among all those methods, which surpasses the baseline model with up to 1.48 % mloU and 2.20% F-score improvements. E-CRF fuses the CRF
mechanism into the CNN network organic whole. It relieves the disturbing gradients caused
7

---

Published as a conference paper at ICLR 2023
Table I: Comparisons with baseline, Vanilla-CRF and Joint-CRF on ADE20K val dataset. 'It stands for Deeplab V3+ followed by DenseCRF. "An end-to-end manner Vanilla-CRe similar to Zheng
et al. 2015).
ResNet-50
ResNet-101
Method
F-score (%)
mIoU (%)
F-score (%)
mloU (%)
DeeplabV3+
14.25
42.72
16.15
44.60
Vanilla-CRF
16.26
43.18 (40.46)
17.89
45.14 (+0.54)
Joint-CRF
16.32
43.69 (+0.96)
18.03
45.61 (+1.01)
E-CRF (Ours)
16.45
44.20 (+1.48)
18.32
46.02 (+1.42)
by the BCWC problem and adjusts the feature representations boost the overall segmentation performance and the boundary segmentation. (a) also proves that E-CRF can decrease the inter- class similarity consistently which results in discriminative feature representations. Experiments on Cityscapes dataset can be found in our Appendix A.
4.2.2 ABLATION ON MESSAGE PASSING STRATEGIES
As we have discussed in Sec. 3.2 two message passing components, i.e., pairwise module vhy and superpixel-bar -biieeellbbaedddd module v/d play vital roles in our proposed E-CRF. Table shows that vi and ve can boost the overall segmentation performance on ADE20K val dataset with up to 1.19% % mloU and 1.25% mIoU gains when integrated into the baseline model respectively. Moreover, if we fuse them as whole into E-CRF, they can further promote the segmentation performance by up to 1.48% mloU improvements. We also compare with Non-local (Wang et al., (2018), ) another famous attention- based message passing method, into our experiments for comprehensive comparisons even though it actually has different design concepts from ours. Unfortunately, find that although Non-local achieves improvements over the baseline, it is still inferior to our E-CRF.
Table 2: Comparisons between message passing strategies, and ablation studies for different message passing components E-CRE, pairwise vif. and auxiliary superpixel-based w.
mloU((%)
Method
ResNet-50
ResNet-101
DecplabV3+
42.72
44.60
Non-local
43.52 (f 0.80)
45.34 0.74)
43,91 (f 1.19)
45.47 (1 0.87)
E-CRF
43.83 (f 1.11) 44.20 (f 1.48)
45.85 (f 1,25)
46.02 (1 1.42)
4.2.3 ABLATION OF SUPERPIXEL NUMBERS
We follow standard settings in our paper and take DeeplabV3+ based on ResNet-50 as the baseline model to present the performance of E-CRF under different superpixel numbers. De- tailed comparisons on ADE20K dataset are reported in Table 3 and SP denotes SuperPixel. As shown Table 3. different numbers of superpixels indeed affect the performance of E-CRF Intuitively, when the number of superpixels is 200, E-CRF acquires the best performance it achieves a better trade-off between the su- perpixel purity and the long-range dependency. Moreover, it is worth noting that when the pair-
Table 3: Comparisons with different superpixel numbers on ADE20K val dataset.
num
mloU w\ u.f (%)
mloU woo b/ (%)
wise message passing strategy (iee., vly/) is also adopted in E-CRF, it becomes more robust to the different numbers of superpixels that may introduce noise, ing mechanism (including vi and wh() can be compatible with the variance.
No
43.91
42.72
100
44.02
43.43
200
44,20
43.83
adaptive message pass-
300
44.13
43.56
400
43.96
43.22
More ablation studies including comparison of different boundary refinement and computational cost presented in Appendix A.4
8

---

Published as a conference paper at ICLR 2023
Table Comparisons with other state-of-the-ee methods on ADE20K val dataset, Cityscapes val and Test, and Pascal Context val dataset.
backbone
mloU(%)
Method
ADE-val
City-val
City-test 8T.9
Pas-Con
CCNet (Huang et al.
2019
ResNer101
45.22
81.3
ANL (Zhu ct al.
2019)
ResNet101
45.24
52.8
GFFNet [ et
2020c
ResNet101
45.33
81.8
82.3
54.2
APCNet He et al.
20199b
ResNet101
45.38
54.7
DMNet Te
2019a
ResNet101
45.50
54.4
SpyGR Li
al.
2020a
ResNet101
80.5
81.6
52.8
RecoNet (Chnn cf al. al. 2020
ResNet101
45.54
81.6
82.3
54,8
SPNet Hou
2020)
ResNet101
45.60
54.5
DNL Yin et al. 2020)
ResNet101
45.82
55.3
RANel (Sen el al.
2020
ResNet101
81.9
82.4
$4.9
ACNet Fu et al.
2019)
ResNet101
45.90
82.0
82.3
54.1
HANet (C hot et al.
2020
ResNet101
82.05
82.1
RPCNet Zhen
2020b
ResNet101
82.1
81.8
CaCNet Liu
2020
ResNet101
46.12
CPNet Yu al.
2020a)
ResNet101
46.27
53,9
STLNet 7hu ct al.
2021
ResNet101
46.48
82.3
82.3
55.6
E-CRF (Ours)
ResNet101
46.83
82.74
82.5
56.1
4.3
COMPARISONS WITH SOTA METHODS
In this research, we mainly focus on the Boundary-caused Class Weight Confusion (BCWC) in CNN models. Hence, in this section, choose CNN-based methods for fair comparisons)
ADE20K. We first compare our E-CRF (ResNet101 as backbone) with existing methods on the ADE20K val set. We follow standard settings in (Huang et al. 2019 Yuan al.) 2020a Zhu et al.
2021 to adopt multi-scale testing and left-right flipping strategies. Results are presented Table
It is shown that E-CRF outperforms existing approaches. Segmentation visualization is presented in our Appendix
Cityscapes. To verify the generalization of our method, we perform detailed comparisons with other SOTA methods on Cityscapes val and test set. Multi-scale testing and left-right flipping strategies are also adopted. The results with ResNet101 as backbone are reported in Table 4 Remarkably, our algorithm achieves 82.74% mIoU in val set and outperforms previous methods by large margin.
Pascal Context. To further verify the generalization of E-CRF (ResNet101 backbone), we compare our method with other SOTA method on Pascal Context dataset as shown in Table 4 We adopt multi-scale testing and left-right flipping strategies as well. The result suggests the superiority of our method.
CONCLUSION AND FUTURE WORKS
In this paper, we focus on the particularity of class weights semantic segmentation and explicitly consider an important issue, named Boundary-caused Class Weights Confission (BCWC). We dive deep into it and propose a novel method, E-CRF, via combining CNN network with CRF as an organic whole to alleviate BCWC from two aspects (i.e., scale and direction). In addition, make an exhaustive theoretical analysis to prove the effectiveness of E-CRF. Eventually, our proposed method achieves new results ADE20K, Cityscapes, and Pascal Context datasets.
There are two important directions for future research. this work, we use SLIC, a common cluster-based algorithm for fast implementation. There exist many other superpixel algorithms such as graphical-basse (Felzenszwalb & Huttenlocher] (0044 and CNN-based (Jampani et al. (018)) that may give better boundary results for objects. Therefore how these different methods influence the performance in our framework is interesting. Besides, We find that transformer- based networks suffer from BCWC issue as well and make a preliminary exploration. More works are expected focus on this issue.
'e also conduct experiments based on SegFormer Xie et al. 2021 to make preliminary exploration in Appendix [ as found that BCWC issue also exists in transformer-han hased models.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their constructive comments. We also sincerely thank Tiancai Wang for useful discussion. This work is supported by the NSFC Grants no. 61972008.
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the art superpixel methods. IEEE transactions pattern analysis and machine intelligence, 34(11) (4)224 2282, 2012.
Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, and Philip HS Torr. Higher order conditional random fields in deep neural networks. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part Il 14, pp. 524 520-540. Springer, 2016.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint 0rXiv:1412.70633 2014.
Liang-Chie Slang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4))8344 848, 2017a.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 05587, 2017b.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), pp. 801-814 2018a.
Wanli Chen, Xinge Zhu, Ruoqi Sun, Junjun He, Ruiyu Li, Xiaoyong Shen, and Bei Yu. Tensor low-rank reconstruction for semantic segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 52-69, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58520-4.
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A^2-nets: Double attention networks. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018b.
Sungha Choi, Joanne T. Kim, and Jaegul Choo. Cars can't 't fly up in the sky: Improving urban-scene segmentation via height-driven attention networks. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang. Context contrasted feature and gated multi-scale aggregation for scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2393-2402, 2018.
Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary- aware feature propagation for scene segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6819-6829, 2019.
10

---

Published as a conference paper at ICLR 2023
Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-baset image segmentation. Inter- national journal of computer vision, 59:167-181, 2004.
Jun Fu, Jing Liu, Yuhang Wang. Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu. Adaptive context network for scene parsing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6748 6757, 2019.
Raghudeep Gadde, Varun Jampani, Martin Kiefel, Daniel Kappler, and Peter V Gehler. Superpixel convolutional networks using bilateral inceptions. In European conference computer vision, pp. 597-613. Springer, 2016.
Stephen Gould, Jim Rodgers, David Cohen, Gal Elidan, and Daphne Koller. Multi-class segmentation with relative location prior. International Journal of Computer Vision, 80(3):300-314 2008.
Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multi-scale filters for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3562-3572, 2019a.
Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7519-7528, 2019b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Shengfeng He, Rynson WH Lau, Wenxi Liu, Zhe Huang, and Qingxiong Yang. Superenn: A superpixelwise convolutional neural network for salient object detection. International journal of computer vision, (5(3))330--3 2015.
Qibin Hou, Li Zhang. Ming-Ming Cheng, and Jiashi Feng. Strip pooling: Rethinking spatial pooling for scene parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4003 4012, 2020.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4705 2017.
Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 603-612, 2019.
Varun Jampani, Deqing Sun, Ming-Yo Yu Liu, Ming-Hsual Yang, and Jan Kautz. Superpixel sampling networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 3683 2018.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems, 24:109-11
2011. 2011.
Lubor Ladicky, Paul Sturgess, Karteck Alahari, Chris Russell, and Philip H. S. Torr. What, where and how many? combining object detectors and erfs. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios (eds.). Computer Vision ECCV 2010, pp. 424 437, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.
Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen Lin, and Hong Liu. Spatial pyramid based graph reasoning for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020a.
Xiangtai Li. Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, and Yunhai Tong. Improving semantic segmentation via decoupled body and edge supervision. In Computer Vision-ECCo ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pp. 435-452, Springer, 2020b.

---

Published as a conference paper at ICLR 2023
Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Shaohua Tan, and Kuiyuan Yang. Gated fully fusion for semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11418-11425 2020c.
Guosheng Lin, Chunhua Shen, lan Reid, and Anton van den Hengel. Deeply learning the messages in message passing inference. Advances Neural Information Processing Systems, 28, 2015.
Guosheng Lin, Chunhua Shen, Anton Van Den Hengel, and lan Reid. Efficient piecewise training of deep structured models for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3194-3203, 2016.
Jianbo Liu, Junjun He, Yu Qiao, Jimmy S. Ren, and Hongsheng Li. Learning to predict context- adaptive convolution for semantic segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, pp. 769-786, Cham, 2020. Springer International Publishing. ISBN 978-3-030-55595-2.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision. pp. 10012-10022, 2021.
Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Semantic image segmentation via deep parsing network. In Proceedings of the IEEE international conference on computer vision, pp. 1377-1385, 2015.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3431-3440, 2015.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-W Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 891-898, 2014.
Nadia Payet and Sinisa Todorovic. 2-random forest random field. Advances Neural Information Processing Systems, 23, 2010.
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181-6189, 2018
Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Hornung. Saliency filters: Contrast based filtering for salient region detection. In 2012 IEEE conference computer vision and pattern recognition, pp. 733-740. IEEE, 2012.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12179-12188, 2021.
Xiaofeng Ren and Jitendra Malik. Learning classification model for segmentation. In Computer Vision, IEEE International Conference on, volume 2, pp. 10-10. IEEE Computer Society, 2003.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-22 2015.
Abhishek Sharma, Oncel Tuzel, and Ming- Yu Liu. Recursive context propagation network for semantic scene labeling. In NIPS, volume 1, pp. 2014.
12

---

Published as a conference paper at ICLR 2023
Dingguo Shen, Yuanfeng Ji, Ping Li, Yi Wang, and Di Lin. Ranet: Region attention network for semantic segmentation. Advances in Neural Information Processing Systems, 33:13927-13938, 2020.
Guang Shu, Afshin Dehghan, and Mubarak Shah. Improving an object detector and extracting regions using superpixels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3721-3727, 2013.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7262-7272, 2021.
Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-senn: Gated shape cnns for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5229-5238, 2019.
Haoru Tan, Sitong Wu, and Jimin Pi. Semantic diffusion network for semantic segmentation. arXiv preprint arXiv:2302.02055 2023.
Takahiro Toyoda and Osamu Hasegawa. Random field model for integration of local information and global information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(8): 1483 1489,1499,440 2008.
Michael Van den Bergh, Xavier Boix, Gemma Roig. Benjamin de Capitani, and Luc Van Gool. Seeds: Superpixels extracted via energy-driven sampling. European conference on computer vision, pp. 13-26. Springer, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chi Wang, Yunke Zhang, Miaomiao Cui, Peiran Ren, Yin Yang, Xuansong Xie, Xian-Sheng Hua, Hujun Bao, and Weiwei Xu. Active boundary loss for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2397-2405, 2022.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
David Weikersdorfer, Alexander Schick, and Daniel Cremers. Depth-adaptive supervoxels for rgb-d video segmentation. In 2013 IEEE International Conference on Image Processing, pp. 2708-2712.
IEEE, 2013.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077-12090, 2021.
Junjie Yan, Yinan Yu, Xiangyu Zhu, Zhen Lei, and Stan Z Li. Object detection by labeling superpixels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5107- 5116, 2015.
Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled non-local neural networks. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 191-207, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58555-66
Changqian Yu, Jingbo Wang. Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang. Context prior for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12416-12423 2020a.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020b.
13

---

Published as a conference paper at ICLR 2023
Yuhui Yuan and Jingdong Wang. Ocnet: Object context network for scene parsing, arXiv preprint arXiv:1809.0091 2018.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt VI 16, pp. 173-190. Springer, 2020a.
Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refin-- ment for segmentation. European Conference on Computer Vision, pp. 489-506. Springer, 2020b.
Mingmin Zhen, Shiwei Li, Lei Zhou, Jiaxiang Shang, Haoan Feng, Tian Fang, and Long Quan. Learning discriminative feature with erf for unsupervised video object segmentation. In European Conference Computer Vision, pp. 445 462. Springer, 2020a.
Mingmin Zhen, Jinglu Wang, Lei Zhou, Shiwei Li, Tianwei Shen, Jiaxiang Shang, Tian Fang, and Long Quan. Joint semantic segmentation and boundary detection using iterative pyramid contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18666-136755 2020b.
Shuai Zheng. Sadeep Jayasumana, Bernardino Komera-Paredee Vibhav Vincet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE international conference computer vision, pp. 1529-11 12377153, 2015.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequenee perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6881 6881-6890 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Lanyun Zhu, Deyi Ji, Shiping Zhu, Weihao Gan, Wei Wu, and Junjie Yan. Learning statistical texture for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12537-12546, 2021.
Zhen Zhu, Mengde Xu. Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 593-602, 2019.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 FORMULA DERIVATION
Firstly, we give the gradient equation of VW. mentioned in this paper:
0L
0L.
aPk. aYg
VW
(16)
ow.
O
dYg aw.
According to it, we present the derivative of each term respectively:
aL aPk
(17)
R
(E c + e) .e Xx-
OPE σYe
moc
k
e
2
(18)
(E vL +e 6)2
Σ
+ 3
Σ myfo
th
+ c
myfc
myfe
and
OYE
FN.
(19)
OW.
Further, it is worth noting that PE is given by:
PE
(20)
Σ
e k + cx
myte
To simplify Eq (18). we take Eq
20
into account. Thus, Eq 188 is formulated as:
P
Ph
Pk
PE = PE (1 P-)-
(21)
So after integrating Eq
17 Eq
(1
and
Eq
19p. Eq 16) can be formulated as:
0L
1
VW.
PE (1
F (1- PE) Fu.
(22)
aw.
PE
Finally, the class weights W in the next iteration will be updated:
W W. VW. W. + (1 Pg) Fk.
(23)
A.2 EXPERIMENT SETUP
ADE20K ADE20K Zhou et al., 2017 is one of the most challenging benchmarks, containing 150 fine-grainet semantic concepts and a variety of scenes with 1,038 image-level labels. There 20210 images in training set and 2000 images validation set.
Cityscapes Cityscapes Coorts et al. 2016) has 5,000 images captured from 50 different cities. Each image has 2048 x 1024 pixels, which have high quality pixel-level labels of 19 semantic classes. There are 2,975 images in training set, 500 images in validation and 1,525 images in test set. We do not use coarse data in our experiments.
Pascal Context PASCAL Context (Mottaghi et al. 2014) is challenging scene understanding dataset, which provides the semantic labels for the images. There are 4, 998 images for training and 105 images for validation on PASCAL Context dataset. In our experiment, the 59 most frequent categories are used for training.
Implementation Details. The initial learning rate is set as 0.01 for both datasets. We employ poly learning rate strategy where the initial learning rate multiplied by (1 - iter Mker/totaliter) 0.9 after each iteration. We set training time to 80000 iterations for ADE20K and Pascal Context, and 180 epochs for Cityscapes. Momentum and weight decay coefficients are set as 0.9 and 0.0005, respectively. For data augmentation, we apply the common scale (0.5 to 2.0), cropping and flipping of the image to augment the training data. Input size for ADE20K dataset is set to 512 x 512, and 480 480 is for Pascal Context while input size for Cityscapes dataset is set to 832 x 832. The syncBN (Peng et al.) 2018) is adopted in all experiments, and batch size on ADE20K and Pascal Context set to 16 and is set to for Cityscapes.
15

---

Published as a conference paper at ICLR 2023
A.3 COMPARISONS WITH RELATED METHODS ON CITYSCAPES
To further evaluate our proposed method, we thoroughly compare our approach with baseline and other traditional CRF-based methods, i.e., Vanilla-CRF and Joint-CRF on Cityscapes dataset. As shown in Table 5 E-CRF achieves the best result among all those methods, which outperforms the baseline model by 0.92% % in mloU and 3.81 in F-score respectively. Obviously, our method is more effective than both Vanilla-CRF and Joint-CRE.
ResNet-50
10s
Method
F-score (%)
mloU (%)
F-score (%)
mloU (%)
V
60.48
79.54
61.94
80.85
Vanilla-CRF
62.38
79, (+0.11)
63.46
80.92 (+0.07)
Joint-CRF
63.44
79.78 (+0.24)
64.43
81.05 (+0,20)
E-CRF (Ours)
64.29
80.35 (+0.81)
65.57
81.77 (+0.92)
Table 5: Comparisons with baseline, Valina-CRF and Joint-CRF on Cityscapes val dataset.
A.4 MORE ABLATION STUDIES
A.4.1 DIFFERENT BOUNDARY REFINEMENT
We consider three typical methods inlcuding Segfix Yuan al. al.]| (2020b), DecoupleSegNet Li et al.) (2020b), and ABL 6 Wang et al. 2022). They refine boundary segmentation via post-processing, improving boundary representation, and adding boundary allignment loss respectively. But all of them ignore the existence of BCWC issue, which may restrict their capability. In Table 4, we use DeeplabV3+ with ResNet101 as our baseline method. For Segfix, we use the official code (It uses HRNet as backbone) to boost the performance of DeeplabV3+. For DecoupleSegNet which is constructed based on Deeplab V3+, we also use the official code (It uses ResNet101 as backbone). All the models trained on ADE20K for 80K iterations with batch size set to 16. When testing, we adopt the single-scale testing strategy (i.e., raw image) because using a single scale (i.c., raw images) when comparing with baselines performing ablation studies is traditional default setting in the semantic segmentation field. The goal is to eliminate the effect other elements (e.g.. image augmentation).
It is observed that all the methods enhance the performance and E-CRF achieves the highest mloU and F-score. We speculate that this is because E-CRF has explicitly considered the BCWC problem and opti- mizes the class weights from both scale and direction aspects while refining bound- ary representation. It also indicates the im- portance of obtaining distinguishable class weights in semantic segmentation.
Table 6: Comparisons with other boundary refining methods on ADE20K val dataset. Method
mloU (%)
F-score (%)
DeeplabV3+ (Chen et al. 2018a)
44.60
16.15
SegFix YYaan et al 22020b
45.62
18. 14
DecoupteSegNer (1. al. 2020h
45.73
18.02
ABI. (Wang al. 2022)
45.38
E-CRF
46.02
18.32
A.4.2 COMPARISONS ON COMPUTATIONAL COSTS
We take DeeplabV3+ based ResNet101 as the baseline model perform the training time comparisons. Image size is set to 512 x 512 and all the experiments are conducted on 8 GeForce RTX 2080Ti GPUs with two images per GPU. The FLOPs, parameter size, and inference FPS are also reported in Table 7 We can find that our proposed E-CRF brings negligible extra costs over the baseline model. The cost difference between E-CRF and Joint-CRF marginal. We also measure the time consuming of superpixel method (5ms), which is much smaller than that of inference (55ms).
TThe authors do not provide the code in their paper. Hence, just report the result based on OCRNet Yuan 2020a) in their paper.
16

---

Published as a conference paper at ICLR 2023
Table 7: Comparisons Computational costs on ADE20K dataset.
Method
Backbone
Training Time(s)
GLOPs(G)
Parameters(M)
FPS
V3++labV3+
ResNet101
0.71
254.8
60.1
19.75
Vanilla-CRR CRR
ResNet101
0.71
254.8
60.1
1.86
Joint-CRF
ResNet101
0.73
254.9
60.2
19.04
E-CRF
ResNet101
0.74
255.0
60.2
18.32
A.5 DISCUSSION
In this section, we discuss the difference between E-CRF and three related works including PC- Grad (Yu et al. 2020b), OCNet (Yuan & Wang 2018), and SegFix (Yuan et al. (020b).
Difference between Projecting Conflicting Gradients (PCGrad) and E-CRF: PCGrad and E-CRF are both gradient-based methods that focus on adjusting the gradient properly to optimize the learning process more effectively and efficiently. PCGrad is designed mitigate key optimization issue in multi-task learning caused by conflicting gradients, where gradients for different tasks point away from one another measured by negative inner product. If two gradients are conflicting, PCGrad alters the gradients by projecting each onto the normal plane of the other, preventing the interfering components of the gradient from being applied to the network. The idea behind PCGrad is simple and the method is effective. PCGrad is a task-level gradient optimization method, mainly focusing on conflicting gradients caused by multiple tasks during training (e.g.. in semantic segmentation and depth estimation). E-CRF is finer-graines pixel-level gradient optimization method. E-CRF mainly aims at mitigating the boundary- ccaused class weights confusion in semantic segmentation via adjusting class weights from both scale and direction.
Difference between OCNet and E-CRF: OCNet uses self-attention implement the object context pooling module. The object context pooling estimates the context representation of each pixeliby aggregating the representations of the selected subset of pixels based on the estimated dense relation matrix. Further, OCNet combines context pooling module with the conventional multi-scale context schemes including PPM and ASPP. In E-CRF, we follow the idea behind Conditional Random Filed and embed it from logit space deep-feature space. We instance the unary function and reformulate the pairwise function with a convolutional based kernel. The kernel takes raw image RGB value and relative position embeddings inputs (See Eq (6)) which is different from self-attention that takes extracted deep features as inputs. We also maintain one special term in CRF called label compatibility and transfer it to feature compatibility (See Eq 7 Such is missing in self-attention. Besides, do not measure the similarity between superpixel center and boundary pixels. We simply leverage the local prior in superpixel and use guide deep feature averaging. The motivation to suppress noise information. Finally, the motivation between OCNet and E-CRF is different. OCNet mainly focuses on integrating as much object context as possible while E-CRF explicitly targets on the BCWC problem and optimizes class weights from both scale and direction.
Difference between SegFix and E-CRF: SegFix first encodes input image and predicts a boundary map and direction map. Then SegFix uses the predicted boundary map and offset map derived from the direction map to correct the wrongly classified boundary pixels via internal points with high confidence. SegFix is beneficial for refining boundary segmentation and is served as post- processing method, thus lacking the capability to alleviate BCWC problem. Our method is derived from traditional CRF (a post-processing method) and can be regarded as plug-ano and-play module. It can be easily integrated with other methods. Besides, our method extends the optimization flexibility for BCWC problem. Empirically,
have compared Segfix and E-CRF based on Deeplab in Table 6 (Please see J...J). E-CRF produces 46. 46.02% for mloU on ADE20K, outperforning SegFix (45.62%) by 0.4%. This indicates the importance of alleviating BCWC problem.
A.6 PAIRWISE MESSAGE PASSING VISUALIZATION
E-CRF takes an equivalent transformation to replace hand-designed Gaussian kernels in Vanilla- CRF with simple convolution operators for more flexible end-to-end optimization. The convolution operation involves two aspects. One is the appearance similarity and the other one is the relative position between pixels. As depicted in Fig 4 a), we take pixel k in the stool for example and
17

---

Published as a conference paper at ICLR 2023
show its relationship with other pixels in the image. Fig (b) and Fig 4[c) show its appearance similarity and cosine position embedding with other pixels respectively. It clear that pixels share similar colors close to the target pixel & tend to be highlighted. Subsequently, in Fig d), directly visualize the results of our pairwise message passing module v/d defined in (q. We can find that di becomes concentrated on the most relevant pixels compared with the pixel k, which verifies the reliability of our pairwise message passing design.
Figure 4: Visualization of pairwise message passing module ey in E-CRF. (a) A target pixel & of the stool in the image. (b) The appearance similarity between other pixels and k. Pixels share the similar colors with k tend to be highlighted. (c) The visualization of the relative position between and other pixels. Pixels close k achieve higher values. (d) The visualization of eff in E-CRE, eh focuses more on most relevant pixels compared to k.
bbi
(d)
Figure 5: Visualization comparisons between our method and baseline ADE20K validation set. (a) Images from ADE20K dataset. (b) Segmentation output from D3eplabV3+. (c) Segmentation output from method. Obviously, compared with baseline, the results are segmented well by E-CRE. (d) Image labels.
18

---

Published as a conference paper at ICLR 2023
B BCWC IN TRANSFORMER
Are transformer-base models also suffering from Boundary-caused Class Weight Confusion? Is our method effective to transformer-basee models? To answer these questions, we make a preliminary exploration in this section.
B. OBSERVATIONS ADE20K
Following the same idea in Fig. I(a)
baseline
of this paper, we take Segformer (Xie et al., 202T) (a transformer-be based seg- mentation model) as an example to train on ADE20K (Zhou al. 2017)
0.15
Class weight similarity
dataset. We count the number of ad- jacent pixels for each class pair and find a corresponding category that has the most adjacent pixels for each class. Then,
0.10
calculate the simi-
larity of their class weights and de- piet it in Fig 6 X-axis stands for the number of adjacent pixels for each class pair in descending order, and Y- axis represents the similarity of their class weights. Blue line denotes Seg- former while orange line denotes E- CRF based on Segformer. As shown in Fig 6 two categories that share more adjacent pixels are inclined to have more similar class weights, while E-CRF effectively decreases similarity between adjacent categories and makes their class weights more discriminative. These observations on transformer-bas bbsed model are quite similar to previous results CNN- based models. Apparently, transformer. based models are also suffering from Boundary-cante causeed Class Weight Confusion.
0.05
Adjacent pixels number[ order)
Figure 6: Class weight similarity on transformer- based model
B.2 EFFECTIVENESS ON TRANSFORMER
To evaluate the effectiveness of our method, we take Segformer (Xie al. 202I) (based on MiT-B5) as our transformer baseline and incoporate E-CRF into it. Experiments are conducted on ADE20K and Cityscapes datasets. Similarly. also compare methods, i.e., Vanilla-CRF and Joint-CRF As shown in Table E-CRF achieves the best result among all those methods, which surpasses the baseline model with up 1.01 % mloU and 3.81 % F-score improvements. By E-CRF relieves the disturbing gradients caused by the BCWC problem boost the overall segmentation performance and the boundary segmentation. F proves that E-CRF can decrease the inter-class similarity consistently which results in more discriminative feature representations.
method with other traditinal CRF-base based
ADE20K
Cityscapes
Method
F-score (%)
mIoU (%)
F-score (%)
mIoU (%)
Segformer
18.53
49.13
62.42
82,25
Vanilla-CRF
21.72
49.36 (+0.23)
64.06
82.31 (+0.06)
Joint-CRF
21.91
49.55 (+0.42)
64.93
82.41 (+0.16 16)
E-CRF (Ours)
22.34
50.14 (+1.01)
66.05
83.07 (+0.82)
Table 8: Comparisons with baseline, Valina-CRF, and Joint-CRE on ADE20K and Cityscapes val
datasets.
Note that they also based Segformer
19

---

Published as a conference paper at ICLR 2023
B.3 COMPARISONS WITH SOTA METHODS
To further verify the effectiveness, we compare our methods with other transformer-based SOTA methods with similar number of parameters (except SETR) for fair comparisons in both ADE20K and Cityscapes datasets. Multi-scale testing and left-right flipping strategies are adopted. As shown in Table our method achieves the best results among all the SOTA methods in both ADE20K and Cityscapes datasets. Besides, our method also has the smallest number of parameters.
Table 9: Comparisons with other transformer-bas baasod SOTA methods on ADE20K val dataset and Cityscapes val and test dataset.
mloU(%)
Method
Backbone
ADE-val City-val City-text Params (M)
SETR (Zheng et al
2021
ViT-L (307M)
50.20
82.15
82.2
310M
UperNet
Swin-B
(Liu ct al.
2021
(88M)
49.65
121M
UperNet
Twins-L
Chu
2021
(99M)
50.20
133M
SegFormer (iie UperNet
2021
MrT-B5
MM)
50,22
83.48
82.2
85M
XCYT-M24 (Chu et al.
2021
(84M)
48.40
109M
DPT RRantll et al. 2021
VTT-B (86M)
48.34
112M
Segmentor (Strudel ct al.
2021
DeiT-B (86M)
50.08
80.60
86M
E-CRF (Ours)
MiT-B5 (81M)
51.28
83.7
82.5
85M
20


Reviews:
[{"review_id": "uGiD_DGUrQ9", "reviewer": "Reviewer_aDCi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the Boundary-caused Weights Confusion problem in semantic segmentation. The authors propose an E-CRF model by integrating CRF into CNN and pursuing gradient descent not only from scale but direction aspects. The method improves baseline for about 1% mIoU.", "summary_of_the_review": "This paper focuses on addressing BCWC problem in CRF, which could have been more generic in class weighting problem of semantic segmentation. The analysis is insightful and interesting, but the proposed solution is not novel and elegant, commonly used and taking additional cost. I consider this paper could be better if it extends more, but currently it lacks novelty in solution.", "strength_and_weaknesses": "Strength:\n+ This paper is well-written and provides insights, e.g., the illustration and interpretation of optimization of different CRF schemes is insightful.\n+ The proposed method extends the gradient descent from scale aspect to both scale and direction aspects, making the optimization more effective.\n\n\nWeakness\n\n- The discussed solution on CRF is a little narrow and out-of-date. Since class-weight problem is general for all semantic segmentation models, it could be better to extend the method to other models.  I also look forward to a discussion part for how to extend this method to Transformer-based segmentation models.\n- Although the discussion of BCWC is meaningful, the proposed solution is not novel. The method of discriminating boundary pixels from local representative pixels (e.g., super-pixel center) by measuring their similarities (attention) is commonly used. Similar ideas are in the below papers\n  - OCNet: Object Context for Semantic Segmentation, \n  - SegFix: Model-Agnostic Boundary Refinement for Segmentation\n- The improvement from Joint-CRF is marginal, about 0.5% mIoU.\n- The design of using super-pixel to extract pixel representation is not elegant. It takes additional cost, and the segmentation performance highly depends on the quality of super-pixel. Is there any way to embed this process into the model, to make the partitioned representation learnable?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m1TIYfjlrC", "reviewer": "Reviewer_JKCJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors observe the Boundary-caused Class Weights Confusion (BCWC) problem in semantic segmentation. To solve this problem, they propose E-CRF to fuse the CRF into CNN networks for more effective optimization. The E-CRF owns two advantages: use CRF to purify the feature representation of boundary pixels; enables optimizing class weights from both scale and direction.  Moreover, they add the superpixel as an auxiliary to boost the performance. Experiment results on popular datasets like ADE20K and Cityscapes are provided.  ", "summary_of_the_review": "This paper studied the boundary problem in semantic segmentation from a new perspective and observe the BCWC problem. They propose E-CRF and superpixel to solve the BCWC problem. However, the motivation of superpixel is unclear and the explanation for E-CRF is weak. Even worse, there are some unfair comparisons in the experiment, which makes the effectiveness of proposed method unconvincing. Based on these reasons, I think this paper is below the acceptance threshold. \nUpdate: From the response, the authors have addressed my key concern, and I would like to raise my rating.", "strength_and_weaknesses": "Strength:\n1. Although it is well known that boundary pixels are difficult to classify, this paper views this problem from a new perspective – class weight. They identify this as BCWC problem and make statistics to prove that there is a positive relationship between class weight similarity and adjacent pixel number.  \n2. The authors use both theoretical derivation and figure to prove that the proposed E-CRF can adjust both the scale and direction of optimization, which makes the presentation clearer and convincing. \n3. The ablation study is good, fully studying the effects of various components like E-CRF and superpixel. \n\nWeaknesses:\n1. It is strange to use superpixel in this paper, which has limited relation to the BCWC problem (the motivation). The superpixel is more like an auxiliary module to boost the performance, since E-CRF alone is not enough for SOTA performance.   \n2. In the ablation study 4.2.1, there is no obvious advantages of E-CRF over joint-CRF, and the F-score gain is less than 0.3%, and mIoU gain is around 0.5%. Moreover, without superpixel. E-CRF is even worse than joint-CRF with ResNet101 backbone (45.47% vs 45.61%). \n\n3. In the Table 4, the comparison of different boundary refinement methods might be unfair. The backbone and test augmentation (e.g. multi-scale testing) of various methods are not specified. For example, in the ABL paper, they can achieve 52.40 with Swin-B and multi-scale testing. The Table 6 has the same problem.    \n4. The section 3.3 can only prove that E-CRF can adjust both the scale and direction of optimization but cannot prove that E-CRF can relief BCWC. There lacks a connection between optimization flexibility and BCWC problem.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kDtZCdrt0O8", "reviewer": "Reviewer_qXNf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The goal of this paper is to improve the semantic segmentation performance through addressing the boundary-caused class weights confusion issue. The two semantic classes that share more adjacent pixels tend to have more similar class weights in the state-of-the-art semantic segmentation solution such as DeepLabv3+. The proposed approach tries to ensure that the pixel-wise labeling are less depends on the adjacency.\n\nThe main contribution of this paper is the introduction of the gradient-based method to the CNN-CRF-based semantic segmentation. The proposed approach works both the levels of directional and scale, while the previous works only focus on using the gradient-based method from scale perspective. The claimed benefits of doing this is to bring better model that helps brining the boundary-caused class weights confusion down.", "summary_of_the_review": "On the positive side, this paper brings the new gradient-based method perspective into the semantic segmentation, which is interesting. The paper has also shown good results on various popular benchmarks.\n\nHowever, the paper has not provided sufficient experiments to back up the claims. The improvements of the paper on those benchmarks could be due to the better hyper-parameter search instead of the things claimed in the paper. It is also a bit disappointed to see that the paper does not refer and discuss the difference between the proposed approach with the prior work PCGrad.", "strength_and_weaknesses": "Pros\nThe main contribution of the proposed E-CRF is to introducing the perspective of gradient descent and optimizes the class weights using both scale and direction using back propagation. This seems to be novel to me as I did not find any other similar semantic segmentation works touch this perspective. Recent works either focus on updating the backbone architectures, or on how to do better than the CRF message passing. This paper brings the new perspective of different gradient-based method.\n \nThe proposed approach has backed the claims with impressive results on various challenging semantic segmentation datasets including ADE20K, Pascal Context, and Cityscapes. The improvements are relative small, but the gain is built on top of the previous state-of-the-art methods.\n\nCons\nI think the main weakness is that the paper does not really refer the good literature, making it appears that the gradient-based method is introduced the first time and the authors apply that to the semantic segmentation. In particularly, I would consider the proposed approach is very similar to the other works such as PCGrad (Gradient Surgery for Multi-Task Learning, https://arxiv.org/abs/2001.06782, NeurIPS 2020), while the paper does not cite that one.\n\nAnother issue is that the proposed approach does not provide sufficient experiments to support the claim. I would like to see the fine-grained experiments to show that the proposed approach get real improvements on the classes that are sharing the most adjacent pixels. This will be like a slice-and-dice results from the aggregated results reported by the current experiments.\n\nThird, I am not fully understand how the proposed approach generalize the formulation to the multi-class classification cases. The current formulation presented in the paper only covers the binary case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UmcPtUF9sy1", "reviewer": "Reviewer_VZu5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focus on the particularity of class weights in semantic segmentation and explicitly consider an important issue , named as Boundary-caused Class Weights Confusion (BCWC).  The authors propose a novel method, E-CRF, via combining CNN network with CRF as an organic whole to alleviate BCWC from two aspects (i.e., scale and direction). In addition, the authors make an exhaustive theoretical analysis to prove the effectiveness of E-CRF. Eventually, the proposed method achieves new results on ADE20K, Cityscapes, and Pascal Context datasets.", "summary_of_the_review": "Though this paper idea is good, but we cannot see the experiment results are convincing enough. As i put in the *Strength And Weaknesses, my main concern is that the experimental results between E-CRF and joint-crf.\n", "strength_and_weaknesses": "Strength:\n1. The paper is well-written and easy to understand.\n2. The idea Embedded CRF (E-CRF) is used to address the BCWC problem is good.\n3. The experiments shows the proposed methods are effective and verified.\n\n\nWeakness\n1. The proposed E-CRF is very similar to Joint-CRF. Though the authors claims that joint-crf  lacks the ability to optimize class representations effectively due to the defective design and E-CRF can solve it well, the experimental results can not show E-CRF improve a lot.  In table 1, E-crf has only no more than 1% improvement than joint-crf. This improvement is not so obvious.\n2. E-CRF adopts superpixel  as an auxiliary to further strengthen the reliability of the message passing to the boundary pixels. Compared with joint-crf, it need more computation for superpixel generation. That is to say, E-CRF use more computation resourses.\n3. There is no model size comparison and time cost comparison.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster