Semantic Diffusion Network for Semantic Segmentation
Authors: Haoru Tan
Sitong Wu
Jimin Pi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Diffusion Process
Semantic Segmentation
Deep Learning

Abstract: Precise and accurate predictions over boundary areas are essential for semantic segmentation. However, the commonly used convolutional operators tend to smooth and blur local detail cues, making it difficult for deep models to generate accurate boundary predictions. In this paper, we introduce an operator-level approach to enhance semantic boundary awareness, so as to improve the prediction of the deep semantic segmentation model. Specifically, we formulate the boundary feature enhancement process as an anisotropic diffusion process. 
We propose a novel learnable approach called semantic diffusion network (SDN) for approximating the diffusion process, which contains a parameterized semantic difference convolution operator followed by a feature fusion module and constructs a differentiable mapping from original backbone features to advanced boundary-aware features. The proposed SDN is an efficient and flexible module that can be plugged into existing encoder-decoder segmentation models. Extensive experiments show that our approach can achieve consistent improvements over several typical state-of-the-art segmentation baseline models on challenging public benchmarks. 

Semantic Diffusion Network for Semantic Segmentation
Haoru Tan'
Sitong Wu*
Jimin Pi
Baidu Research
tanhr20140163. com, wusitong980gmail.. com jpi@connect. ust.hk
Abstract
Precise and accurate predictions over boundary areas are essential for semantic seg- mentation. However, the commonly-used convolutional operators tend to smooth and blur local detail cues, making it difficult for deep models to generate accurate boundary predictions. In this paper, we introduce enhance semantic boundary awareness, so as to improve the prediction of the deep semantic segmentation model. Specifically, enhancement as an anisotropic diffusion process. We then propose novel Icarnable approach called semantic diffusion network (SDN) to approximate the diffusion process, which contains a parameterized semantic difference convolution operator followed by a feature fusion module. Our SDN aims to construct a differentiable mapping from the original feature to the inter-class boundary-enhanced enhanced feature. The proposed SDN is an efficient and flexible module that can easily plugged into existing encoder-decoder segmentation models. Extensive experiments show that our approach can achieve consistent improvements over several typical and state-of-the-art segmentation baseline models on challenging public benchmarks.
operator-level approach
first formulate the boundary feature
Introduction
Semantic segmentation aims at assigning pixel-wise semantic labels to an image. It plays an important role various applications such autonomous driving 9 [3] medical diagnostics [I] and virtual
reality 4 A key challenge in semantic segmentation is how to improve the boundary quality of
segmentation results (Figure []] In general, people have been working on this problem three
different ways: (i) adopting a post-processing module to refine the boundary [26] 23 43 39 [58] (ii) jointly learned the edge detection and semantic segmentation task for boundary-awarenee
600
and (iii) making training more focused on the boundary by designing a boundary-awar loss with 45)
higher sensitivity the boundary changes
Nevertheless, all these methods explored the boundary information via either post-processing refine- ment or additional supervision, rather than directly address the intrinsic reason for fuzzy boundary and detail degradation, that is, vanilla convolution (24] naturally tends to smooth the feature (47] [3]]
[56], making it difficult capture the boundary cues (see Figure (p)). Although traditional gradient operators [19] [2] and difference convolutions 56 41 have the ability to perceive boundary, they are sensitive to all the boundaries including intra-class texture edges. which makes not suitable for semantic segmentation. This motivates us to explore a gegnentation-rrienee operator with only inter-class boundary awareness improve the boundary ambiguity in semantic segmentation.
"Contribute equally
I "orresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: Qualitative analysis of the segmentation error maps. The base- line denotes OCRNet [57], and the non-black pixels indicate the wrong predictions. It can be seen that seg- mentation error is particularly obvi- ous in the boundary region. Our method achieves significant bound- ary improvements over the baseline model. The examples are selected and cropped from Cityscapes-val (
(a) Image
(b) Ground-Truth
(c) Baseline
(d) Baseline+Ours
Our method. In this paper, we are committed to improving the boundary quality of semantic segmentation from the operator level. Inspired by the anisotropic diffusion [33 in the signal processing field, we formulate the inter-class boundary enhancement as a modified anisotropic diffusion process guided by semantics, which can be solved by the finite difference method. While such a traditional solver can hardly be integrated into deep segmentation networks due to the hyper-parameter sensitivity, numerical instability, and high computational complexity caused by its iterative formulation. To solve this problem, we propose a learnable semantic diffusion network (SDN) approximate the diffusion process, which parameterizes the traditional solver and only requires only one forward instead of multiple iterations. The SDN aims to construct a differentiable mapping from the original feature to the inter-class boundary-enhaned semantic difference convolution and a feature fusion operation. Our proposed SDN is an efficient and flexible module that can be easily plugged into the existing encoder-decoder segmentation models. To evaluate its effectiveness, we integrate our SDN into multiple baseline segmentation models and conduct experiments on two challenging benchmarks (i.e., ADE20K (22 and Cityscapes μ). Extensive experiments show that our approach can bring consistent improvements for existing segmentation methods with few additional computational costs. In particular, our SDN can bring +2.57% and +1 +1.43% mloU improvements for Segmenter [40] with ViT-B [1] backhone on ADE20K
feature, which consists of
62 and Cityscapes [. respectively. Qualitative visualization verifies the obvious boundary quality improvement of our approach.
2 Related Work
first proposed to use a fully convolutional network to generate dense predictions, which is regarded as a milestone for semantic segmentation. Since then, lots of works have been exploring larger receptive fields for better scene understanding based on CNN architecture. For example, PSPNet (15) proposed a general pyramid pooling module to fuse features at different scales. DeepLab family (26 27
Semantic Segmentation. In 2015, FCN 188
25 28 used atrous convolution to enlarge the receptive fields while keeping the resolution. Recent works focused on how to capture more and better boundary
precise context 57
29 52
58 3
Considering that the attention
operation [46] has a large receptive field and can adaptively aggregate features, [12] 16
63 2
employed the iitention-basedd modules on the top of CNN segmentation networks adaptively combine local features with their global dependencies. Inspired by the success of Transformer [44] in
NLP tasks, SETR 61 and DPT (35) attempted to use Vision Transformer [1] as the backbone. They demonstrated the effectiveness of global receptive fields for semantic segmentation tasks. Inspired by [5] Tran2Seg (51) explored a hybrid encoder and prototype-basee transformer decoder. FTN 49 and Segmenter [40] further designed fully transformer networks. SegFormer (50) proved that lightweight MLP decoder can also achieve satisfactory results under a strong transformer encoder. Recently, MaskFormer [8] proposed simple but effective mask classification scheme to predict set of binary masks, which unified the semantic, instance, and panoptic segmentation.
Boundary-aware Segmentation. Methods for promoting boundary quality in semantic segmentation can be divided into three categories: (1) Adopting the post-prucessing module. Most methods applied a post-processing module on the top of coarse segmentation results to refine the boundary and recover the details (26 23 (8). DenseCRF model 23 the most well-known post-processing method, which achieves fine boundary results by encouraging assigning neighboring pixels with similar colors

---

to the same semantic label. SegFix [58] proposed model-agnostic boundary refinement model, which aims replace the ambiguous predictions boundaries with predictions at the intra-class areas.
utilized the multi-task learning strategy by jointly learning boundary detection and semantic segmentation together. For example, RPCNet
(2) Auxiliary branch for boundury-avareness. 660
i 30
[60] introduced the auxiliary branch for boundary enhancement to the semantic segmentation model. (3) Boundary-aware loss. These works guide the model to focus more on boundary accuracy by designing the boundary-aware loss function with high sensitivity to the boundary changes [3 20 45
Inverseform [3] introduced a boundary distance-based measure into the popular segmentation loss functions. Kervadec et.al. [20] proposed a boundary loss taking the form of distance metric on the space of contours for highly unbalanced situations. ABL [45] designed an active boundary loss function for progressively encouraging the predicted boundaries matching the ground-tree truth boundaries. Instead of imposing various boundary-related constraints, we consider the intrinsic reason for fuzzy boundary from the operator level, and design boundary-sensitiv operator to make the network naturally capable of modeling and perceiving boundaries.
Related Operators. The convolution operator, which calculates the kemnel's response to each local patch of the input feature maps, is not good at modeling boundary information. To solve this, people try to combine the traditional image gradient operator with convolution operators. Local binary convolution (LBC) [19] used a series of pre-defined binary filters to replace learnable kernels in convolution. Subsequently, series of difference convolution operators represented by central difference convolution (CDC) [33 [66] used learnable kernels to capture useful boundary information on the central difference map. PDC 4I further extended the flexibility of CDC and achieved excellent results in boundary detection tasks. In addition, the local self-attention (LSA) (34] operator replaces the fixed kernel with the content-adaptiie attentive weights achieve local feature fusion, making it good at capturing contextual dependence. But it is still not good at perceiving the boundary. Different from existing methods, we are committed to designing a novel and efficient operator-level solution with a high sensitivity to the inter-class boundaries rather than all the edges.
3 Preliminaries
3.1 Diffusion Process
Diffusion is a physical model aimed at minimizing the spatial concentration difference [37], which is widely used in image processing 33 process needs to solve the well-known second-order partial differential equation:
6 48, 47]. Given an image tensor U to be smoothed, this
dU
Øt = Div(D VU),
where St
aU is the time derivative of the solution tensor, Div is the divergence operator, D is the diffusivity tensor, which describes the speed of the diffusion process, V is the gradient operator. With increasing time I, the solution U, of this process will correspond to increasingly smoothed versions of the original tensor.
The concrete form of D determines the properties of the PDE. For D = 1, is called homogeneous linear diffusion, where the diffusion velocity of spatial concentration is exactly the same in all directions. For a spatial-dependent D = D(x) (e.g. learnable kernel,, the process is linear and inhomogeneous. However, the two kinds of simple linear diffusion processes not only smooth the noise but also blur the edges. An ideal diffusivity function allows more smoothing parallel to image edges and less smoothing perpendicular to these edges, which is actually anisotropic and nonlinear. For this reason, some classic milestones [3 6 48 47 increase the nonlinear modeling ability by designing complex diffusivity functions D D(U) depending on the input U.
3.2 Rethinking Operator-level Boundary Awareness
Here we discuss the operator- causes of potential CNN failures boundary predictions. Given the input feature map x, the vanilla convolution calculates the output tensor through the inner product between the kernel w and each local feature patch. Without loss of generality, we assume that the channel number of both input and output is 1. The vanilla convolution with kernel w E R over a input feature I E RHX could be written as: Yp = (w, 2,)), where xp is the local patch centering at the position p. Intuitively, the vanilla convolution is just local weighted average
3

---

B
B
A2
(a) Image
(b) Vanilla Conv
(c) CDC
Figure 2: Visualization of the feature maps from (b) vanilla convolution [24] (c) central difference convolution (CDC) [55]. We choose three points A1, A2, B on the image, corresponding to car-body, car-glass and plant, respectively. In (b), due to the local smoothness of the vanilla-cony operator, we can see that there is no significant difference in the feature responses at the three coordinates in the feature map. According to (c), the CDC operator can hardly distinguish the true semantic boundary and the pseudo boundary caused by textures or noise, for example, we can see that the features captured by CDC have cluttered false boundary responses between these two coordinates A1 (car-body) and A2 (car-ghe (car-glass).
operation, which tends smooth the local detailed information (55]. Especially when entries within the kernel are non-negative, the smoothness will be particularly significant as it forms a low-pas. filter. This results in poor capacity for modeling the semantic boundary information, see Figure Z
Recently, people try to combine the traditional gradient operator (edge-detector) with convolution
operators to enhance its edge awareness [19 53-56 41 Local binary convolution (LBC) [19
used a series of pre-defined binary filters to replace learnable kernels convolution. Subsequently, series of difference convolution operators represented by central difference convolution (CDC) (53566 used learnable kernels to capture useful boundary information on the central difference map, that is. Dp (w, (xpp Feenter), )) where Feenter indicates the center entry of the current patch, and the rest symbols are defined in the same way as that of the vanilla convolution. It aggregates the center-oriented gradient within the local region instead of the feature values. Even though this kind of operator has better edge awareness, however, because the gradient operator cannot distinguish between real semantic boundary and pseudo boundary caused by textures noise (see Figure 22), is difficult for us to directly apply them to tasks such as semantic segmentation.
Methodology
This section introduces the semantic diffusion network (SDN) for semantic segmentation tasks to compensate for the detail blur caused by the vanilla convolution operator. It is combined with CNN backbone when used, so as to strengthen semantic boundary information and filter out pseudo boundary information caused by texture or noise.
4.1 Formulation
Considering CNN backbone model with L stacked neural stages mapping image to pixel-wise feature maps X. Let X denote the feature map from the E-th stage. We define the semantic diffusion process over the feature map X as:
au,
Div Div ((
(2)
Øt
8U where ar is the time derivative of the solution tensor, Div is the divergence operator, V is the gradient operator. V is the guidance feature map with the same resolution of U. The diffusivity term is a monotonically decreasing function of the square of the gradient /VV22 for example, g( ( = 1/ 2//2. Ideally, the diffusion close to the semantic boundary areas will be suppressed, while the diffusion far away from the semantic boundary will be accelerated. Therefore, V should contain sufficient semantic information, so the gradient norm /VI22 can be used as reliable semantic boundary indicator to construct the diffusivity. The PDE the non-linear diffusion process defined by Eq. could be approximately solved by performing the finite difference update

---

rule for T steps:
U
E
(((V(s)) - ((,n)))) ( - U U(mm)))
(U(os)
(3)
(m,n)
(m,))N
U
(m,m) = a², U (m,n) + 3
ūit+
(m.n)'
(4)
where the step-index & {O,..., T' 1]. the tuple (i.j) enumerates the index of every entry within the local neighbor region M(m..) of the position (m, n.), and the initial state is U = X In fact, it is obvious that the diffusivity ((V(())) - V=h calculates the similarity between semantic guidance features (i. j) and (m, n.). The solver builds differentiable map from the input feature map X to the boundary-enhancee feature.
Remark. The finite difference based solver can hardly be directly integrated with deep segmentation models for several reasons: (1). the stability of the finite difference method depends on the setting of boundary conditions and the selection of parameters (a, B); (2). the solver requires multiple iterations, which will consume high computational complexity. (3). integrating the iterative solver with the deep networks would result in numerical instability to the training process, such as gradient
explosion
39
4.2 Semantic Diffusion Network
This subsection describes a novel learnable approach called semantic diffusion network (SDN) for approximating the diffusion process, which contains parameterized semantic difference convolution operator followed by a feature fusion module and constructs a differentiable mapping from original backbone features to advanced boundary- features. As learnable PDE approximate solver, the well-trained SDN could approximate the diffusion process in only one forward computation, reducing the computational complexity and improving the stability. Overall, formulate the SDN as Fsnn SDN(N) V), where U is the input feature map to be processed and V is the semantic guidance map.
4.2.1 Semantic Difference Convolution
Inspired by Eq. 3) we develop the semantic difference convolution (SDC) operator. We set the Nm,,. as h x w local region centered at position (m, n). SDC takes the feature map and the semantic guidance V as inputs and calculates the output feature map Y. Specifically, each component of indexed by (m, n) is calculated via
Y ((m,))
Σ (mm.))
W(s) S (V(.)) V(mm)) (U U U(m,)),
(5)
where the tuple (i,j) enumerates the index of every entry within the local neighbor region Nm... In the right hand (5. the first term is the learnable convolutional kernel W, of which (height, width) = (h, w). The kernel introduces more powerful modeling capability than the naive non- non-linear diffusion process in Eq. The second term S (Vi.)) V(m.n)). called the semantic similarity term, is a scalar measuring the similarity between semantic guidance features at (i, j) and (m, n). This term corresponds to the diffusivity ((VV Vi 12) (q. The third term (U)))) U(T (I,,,)) ). called the pixel difference term, calculates the feature difference between features at the position (i,j) and (m, n). Note that Eq. 5) can be implemented efficiently using CUDA. By
default, the kernel size of W is set to 3 x 3.
Figure prrvidee the comparison between SDC with other well- convolution operators. As defined (5,, the proposed SDC operator contains two sequential steps: (i) calculating the semantic similarity, and (ii) calculating the output feature. The first step consumes a complexity of O (HHWhwCy). and the second step consumes a complexity of O HW C..)C.... Thus, the overall complexity is Complexity input feature, and h, w are the height and width the kernel tensor. Cs Co, and Cy are the channel number of input, output, and semantic features, respectively.
O(HWhw hawlW,CC....C (C,C., Cf)), where H, W are the height and width of the
5

---

learnable weight
pixel feature
semamtic similarity between und
du
pixel-wise difference berween and Xy
Wo
M12
Xg
X2
Wo
W2
do.
d224
42.
W2
$1.4
$2.4
do d dza
5
44
W/s
Xg
X5
W3
W4
Mr
dg
d4
ds,a
Wy
We
Ws
X3
$3.4
$4.4
$5.4
X ds, das ds,
Xa
da.
Wa
$5.4
58.4
daa dra
das
(a) Vanilla Convolution
(b) Central Difference Convolation
(c) Semantic Difference Convolution
Figure 3: Comparisons between the vanilla convolution, the central difference convolution, and our semantic difference convolution.
4.2.2 Feature Fusion
This step aims to fuse the semantic difference convolution output Y and the module's input feature map U into the final boundary-enhance feature map Fsdn, This module is actually corresponding to Eq. in the classic finite differential solution. Here, we extend the simple weighted addition
operation in Eq. to learnable and more flexible way, that is, Fsdn Conv (U r It first
concatenates the original input feature U with the semantic difference convolution output Y and then uses simple but effective 1 convolution to achieve channel-wise feature fusion, resulting in the boundary-enhance feature Frsin, Note that both U and Y should have the same shape. Otherwise, we need to perform the bilinear interpolation upsampling operation on the smaller one to align the shapes.
4.3 Segmentation Model with SDN
Here, we introduce how to integrate SDN into the existing secooder-decode-bbaeeee segmentation models. For better compatibility, we treat SDN as a neck part between the encoder and decoder, without breaking the original design of the baseline network. According the feature scales required by the decoder, existing segmentation models can be divided into two categories: (i) encoder with single-scale decoder (i.e.. ViT+Segmenter and ResNet+FCNN; (ii) encoder with single-scale decoder, such as ResNet-SemannicFPN Next, we discussed the two cases separately.
SDN with single-scale decoder. As shown in Figure a), given input image I E R3 the encoder is first applied to it to extract the feature, resulting in F RC² H²' xw" where H', W and C" denote the height, width and channel numbers, respectively. We then incorporate our SDN on the encoder output feature F enhance the boundary details, and the output of SDN Fsda is sent to the decoder. Note that Eesdo has the same size of F. As mentioned in Sec. 4.2.1 SDN requires an additional feature with rich semantics as guidance. To construct such a semantic feature F for SDN, we stack a lightweight layer & on the top of the encoder, where & is instantiated as a 3 x 3 convolution with stride 2. Thus, have gdnn SDN (F, F*).
SDN with multi-scale decoder. As Figure 4b), the input image I R3xHxW is first passed through a backbone with totally T. scales (generally T. = 4), resulting in multi-scale features (F, E RC, xC, 1-=1- H,, W; and C2 denote the height, width, and channel numbers, respectively. We then incorporate the SDN after all the backbone stages to strengthen the boundaries comprehensively, In particular, for the i-th SDN, takes F; and Fü the inputs, and generates the feature: Fr SDN(F, F < i < L, where Fe denotes the semantic feature scale i, which is defined as follows:
F.11 Φ (FL),
1si<L
F
i=L
Similarly, & is also implemented by a 3 x convolution with stride 2. Note that the SDNs applied on different stages have the same architecture but not share the same parameters. Finally, the boundary-enhanced features are fed into the decoder to generate the pixel-wise probability map.

---

Decoder
@ (2Ae
Segmentation
Encoder
SDN
Head
aa)
Encoder SDN
Decoder
airms
F
+ pinta
SDN
F
SDN
Segmentation
SDN
Fade
Head
SDN
padn
(b)
Figure 4: Illustration of how to combine the proposed semantic diffusion network (SDN) with the baseline segmentation model with single-scale decoder and multi-scale decoder in (a) and (b), respectively. The SDN is regarded as a neck part in order not to break the original encoder and decoder design of the baseline model.
Experiments
In this section, we first introduce the datasets and implementation details in Sec. 5.I Then, we evaluate approach over several baseline segmentation models these public benchmarks in Sec. (52] Finally, in Sec. [53] we conduct ablation studies to analyze the effect of key designs and the compatibility of our approach.
5.1 Experimental Setup
Datasets. Experiments are conducted on two widely-used public benchmarks: ADE20K (62] is very challenging benchmark including 150 categories, which is split into 20000 and 2000 images for training and validation. Cityscapes [9] is real-world scene parsing benchmark, which contains over 5000 urban scene images with 19 classes. The number of images for training, validation, and testing are 2975, 500, and 1525, respectively.
Baseline models. To evaluate the effectiveness of our SDN, we integrate it into two typical CNN- based baseline methods (FCN 18 and SemanticFPN 21 and a recent Transformer-bases segmen-
tation model (Segmenter 40
Training. All the experiments implemented with PyTorch and conducted 4 NVIDIA V100 GPUs. For fair comparisons, we apply the same settings as the baseline methods. Specifically, for CNN baselines (i.e., FCN and SemanticFPN), we use the SGD optimizer with momentum 0.9 and weight decay 0.0005. The learning rate is initialized at 0.01 and decays until lc-4 by polynomial strategy with power 0.9. For Transformer baselines (i.e., Segmenter), AdamW used as the optimizer without weight decay. The learning rate is initialized as 6e-5 and decays until 0 via a polynomial strategy with power 1.0. We train the models with 160k and 80k iterations for ADE20K and Cityscapes, respectively. For ADE20K, we train the models with 160k iterations, each of which involves 16 images. For Cityscapes, training contains 80k iterations with a batch size of 8. Synchronized BN [32 is used to synchronize the mean and standard deviation of BN 38 across multiple GPUs. The backbone initialized by the ImageNet-1K K (36 and ImageNet-22K pre-trained weights for ResNet-50 [4 and ViT-B [Л], respectively.
10

---

Table 1: Experimental results ADE20K-val val and Cityscapes-val. "s.s." and "m.s." denote the
single-scale and multi-scale mloU, respectively.
ADE20K
Cityscapes
Method
Encoder
mloU (s.s.)
mloU (m.s.)
mloU (s.s.)
mIoU (m.s.)
FCN [88]
ResNet-50
36.10
38.08
72.64
73.32
FCN+Ours
ResNet-50
38.12 (+2.02)
39.36 (+1.28)
74.75 (+2.11) 75.79 (+2.47)
SeM 21
ResNet-50
37.49
39.09
74.10
75.98
SemanticFPN+Ours
ResNet-50
38.79 (+1.30)
40.27 (+1.18)
75.97 (+1.87) 77.31 (+1.33)
Segmenter 40
ViT-B ViT-B
48.48
50.00
77.97
80.07
Segmenter+Ours
51.05 (+2.57) 52.18 (+2.18) 79.42
(+1.45) 81.38 (+1.31)
Table 2: Boundary F-score comparisons Cityscapes-val.
F-score
Method
Encoder
with SegFix
58
with SDN (ours)
1px
3px
X
65.2
76.7
OCRNet 57
HRNet-48 42
X
66.7 (+1.5)
77.1 (+0.4)
X
V
69.5 (+4.3)
78.2 (+1.5)
Evaluation. We adopt the widely-used mean intersection of union (mloU) to measure the overall segmentation performance. In order to measure the boundary quality of the prediction, compute F-score only on the Ipx and 3px boundary region, which denotes the I pixel and 3 pixels width area along the ground-truth boundary. For the multi-scale inference, follow the previous works (57 8 to perform horizontal flip and average the predictions at multiple scales [0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0].
5.2 Main Results
To verify the effectiveness and generalization of our approach, we apply the proposed SDN to several baseline segmentation models various datasets. Table|]reports the quantitative comparisons.
ADE20K. Table [ shows that our approach brings significant mIoU improvements for the baselines the challenging ADE20K dataset, i.e., improves FCN by +2.02% mloU (s.s.), improves SeN by +1 +1.30% mloU (s.s.), and improves Segmenter by +2.57% mloU (s.s.). The im- provement could be attributed to the fact that explicit boundary information is particularly useful for ADE20K images with complex scenarios. With the HRNet-48, ResNet- 101, and Swin-B backbone, contribute +2.6%, +1.7%, and +1.1% mloU gain over OCRNet and MaskFormers, respectively.
Cityscapes. Table D shows that our approach brings nontrivial mloU improvements for all the baselines, i.e,, improves FCN by +2.47% mloU (m.s.), improves SemanticHPN by +1 +1.87% mloU (s.s.), and improves Segmenter by +1.45% mloU (s.s.). Additionally, with the OCRNet(HRRNet
57 baseline, SDN brings +0.9% mloU improvement to the baseline OCRNet (81.9% to 82.8% mloU) and outperforms the SegFix model by +0.3% mloU (the OCRNet's results are not listed in T..le Moreover, our approach (84.7% mloU) outperforms the OCRNet+ (84.5% mioU) by +0.2% on Cityscapes-lexy, where the detailed results can found the web-link We also compare the suindary-proootig ability of SDN and the classic SegFix (58) model in Table 2 where method brings significant improvement, i.e., +4.3 and +1.5 for OCRNet terms of the Ipx and 3px F-score. Figure provides some typical visualizations to show the boundary improvement effect of our method.
5.3 Further Study
Compare SDC with other operators. For direct comparisons with other related operators, we replace the semantic difference convolution (SDC) SDN with other two related operators, vanilla convolution (24] and CDC (55). Table B(a) compares their boundary performance Cityscapes

---

Input Image
(b) Ground-Trruhh
(c) Baseline
(d) Bascline+
Visualization comparisons of the segmentation results on Cityscapes- where the baseline is OCRNet.
Figure 5:
with OCRNet [57 as the baseline model. It can be found that SDC outperforms its vanilla convo- lution counterpart by +4.3 F-score, and surpasses CDC by a large margin, which demonstrates the irreplaceability of SDC.
Effect of kernel settings of SDC. We conduct experiments under different choices of the kernel size and dilation rate of SDC in Table 3b). It can be found that: (1). Increasing the size of the convolution kernel SDC does not bring significant changes. (2). Excessive dilation rate leads negative effects. In fact, large kernels and dilated convolution are mainly used for enlarging the receptive fields. While, the goal of our SDC is to infer the inter-class boundary cue in a local region, which may not require more contexts brought by large kernels. Similarly, large dilation rate may lead to the lack of local information, where, however, the local information is critical for local boundary exploration.
Compatibility with other boundary promoting methods. Our approach can be incorporated with other boundary improvement methods, for example, using SDN and a post-processing boundary refinement module simultaneously in network. Table presents the good compatibility of our approach with three well-known methods on Cityscapes-vall Specifically, the additional
DenseCRF [23], SegFix [58) and InverseForm 3 further bring +0.2, +0.6 and +1.0 boundary F-score gains, respectively.
worth noting that, our method can directly achieve +4.4 boundary F-score improvement over the baseline without any other methods. Benefiting from the intrinsic boundary exploration, segmentation networks with our SDN can relieve the dependence on other boundary improvement techniques.
Table Further studies on (a) comparison with other operators, where V-Conv denotes the vanilla convolution, (b) the effect of the kernel size and dilation rate settings of the SDC operator in the SDN module, and (c) compatibility with other boundary- prondooii methods in semantic segmentation.
(a)
(b)
(c)
Operator
F-score
Kernel-size
Dilation- rate
F-score
Model
F-score
Baseline
65.2
Baseline
65.2
3 x 3
1
69.5
Ours
69.5
V-Conv
65.2
5 x
69.6
Ours + CRF 23
69.7
CDC 53
60.1
x 3
3
69.1
Ours + Seg Fix (58
70.1
SDC (ours)
69.5
3 x
5
68.8
Ours + IF
70.5

---

Image
Ground- -ruth
Stage-1
Stage-2
Stage-3
Stage-3
SDN-1
SDN-2
SDN-3
SDN-3
Figure 6: Visualization of the feature mapping in SDN. The second shows the output feature of each encoder stage, which is also the imput feature of SDNs. The output features of SDN according to each stage are shown in the third row. Obviously, our SDN enhances inter-class boundaries and suppresses the intra-class pseudo boundaries. The example is selected from Cityscapes.
Conclusion
In this paper, we have summarized the internal reasons why the deep semantic segmentation model is difficult produce accurate boundary prediction, that is, the commonly used convolutional operators tend to smooth and blur local detail cues, making it difficult for deep models to capture precise boundary information. To solve this problem, this paper introduces an operator-level approach to enhance semantic boundary awareness, so as to improve the prediction of the deep semantic segmentation model. Specifically, we formulate the boundary feature enhancement process as an anisotropic diffusion process and propose the novel semantic diffusion network (SDN) to approximate anisotropic diffusion process. The proposed SDN an efficient, stable, and flexible module that can be plugged into existing encoder-decoder segmentation models. Extensive experiments show that our approach can achieve consistent improvement over various bascline semantic segmentation models and significantly improve the boundary quality of segmentation results.
Limitations and Social Impact.
We believe that SDN is general approach that can benefit deep learning models in various popular visual tasks. However, this paper only verifies the effectiveness of SDN on single task (semantic segmentation). In the recent future, we will explore the application of SDN in other 2D/3D tasks, such as instance segmentation, panoptic segmentation, flow estimation, image de-noising, image super-resolution, and so on. SDN has potential value in important applications such autonomous driving and medical image analysis. But it can also deploy inhumane surveillance. The potential negative effects be avoided by implementing strict and secure regulations.
10

---

References
[1] Adam Paszke, Sam Gross, Soumith Chintala, G Chanan, Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Antiga, A Lerer, ct.al:: Automatic differentiation in pytorch. In: Advances in neural information processing systems Workshop (2017)
[2] Bello, I., Zoph, B., Vaswani, A., Shlens, J., Le, Q.V.: Attention augmented convolutional networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3286-3295 (2019)
[3] Borse, S., Wang, Y. Zhang, Y., Porikli, F:: Inverseform: A loss function for structured boundary- aware segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5901-5911 (June 2021)
[4] Brooks, F: What's real about virtual reality? IEEE Computer Graphics and Applications 19(6),
16-27 (1999). https://doi.org org.o/o.oor//1.. 10. (009998.79972..
[5] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers, In: European Conference Computer Vision. pp. 213-229 Springer (2020)
[6] Catté, F., Lions, P.L., Morel, J.M., Coll, T.: Image selective smoothing and edge detection by nonlinear diffusion. SIAM Journal on Numerical Analysis 29)1), 182-193 (1992)
[7] Chen, L.C., Barron, J.T., Papandreou, G., Murphy, K., Yuille, A.L.: Semantic image seg- mentation with task-specific edge detection using cnns and discriminatively trained domain transform. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4545 4554 (2016)
[8] Cheng, B., Schwing, A., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems (2021)
[9] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S.. Schiele, B.: The cityscapes dataset for semantic urban scene understanding, In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
[10] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255. leee (2009)
[11] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly. S., Uszkoreit, J., Houlsby, N.: An image is worth 16x words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021), https://openrevios: net httpa//rwwwr.eeeeeoooooodddodde. ffrrmr?ddiiicbbaNTTy
[12] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang. Z., Lu, H.: Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3146-3154 (2019)
[13] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354 3361 (2012).
[14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778 (2016)
[15] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia: Pyramid scene parsing network. In: Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (2017)
[16) Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W.: Ccnet: Criss-cross attention for semantic segmentation. In: Proceedings of the IEFE/CVF International Conference on Computer Vision. pp. 603-612 (2019)

---

[24] Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE 86(11), 2278-2324 (1998). http://i.orr//1..11/////./.....0 [17] Jin, Z., Liu, B., Chu. Q. Yu, N.: Isnet: Integrate image-level and semantic-level context for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7189-7198 (2021)
[18] Jonathan Long, Evan Shelhamer, Trevor Darrell: Fully convolutional networks for semantic segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (2015)
[19] Juefei-Xu, F., Boddeti, V.N., Savvides, M.: Local binary convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
[20] Kervadec, H., Bouchtiba, J., Desrosiers, C., Granger, E., Dolz, J., Ben Ayed, Boundary loss for highly unbalanced segmentation. Medical Image Analysis 67, 101851 (2021)
[21] Kirillov, A., Girshick, R., He, K., Dollár, P.: Panoptic feature pyramid networks. In: Proceedings of the IEFE/CVF conference on computer vision and pattern recognition. pp. 6399-6408 (2019)
[22] Komulainen, J., Hadid, A., Pietikäänen, M.: Context based face anti-spoofing. In: 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS). pp. 1-8. IEEE (2013)
[23] Krähenbühl, P., Koltun, V: Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems 24, 109-117 (2011)
[25] Liang-Chich Chen, George Papandreou, Florian Schroff, Hartwig Adam: Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv e-prints (2017)
[26] Siang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L. Yuille: Semantic image segmentation with deep convolutional nets and fully connected crfs. In: ICLR (2015)
[27] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(4), (April 2018)
[28] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam: Encoder- decoder with atrous separable convolution for semantic image segmentation. In: European Conference Computer Vision (2018)
[29] Liu, M., Schonfeld, D., Tang, W.: Exploit visual dependency relations for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.
9726-9733 9735 (2021)
[30] Marmanis, D., Schindler, K., Wegner, J., Galliani, S., Datcu, M., Stilla, U.: Classification with an edge: Improving semantic image segmentation with boundary detection. ISPRS Journal of Photogrammetry and Remote Sensing 135. 158-172 (2018)
[31] Olaf Ronneberger, Philipp Fischer, Thomas Brox: U-net: Convolutional networks for biomedi- cal image segmentation. In: Medical Image Computing and Computer-Assisted Intervention (2015)
[32] Peng, C., Xiao, T., Li, Z., Jiang, Y., Zhang, X., Jia, K., Yu, G., Sun, J.: Megdet: A large mini-batch object detector. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6181-6189 (2018)
[33] Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Transac- tions on Pattern Analysis and Machine Intelligence 12(7), 629-639 (1990)
[34] Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., Shlens, J.: Stand-alone self-attention in vision models. In: Advances in Neural Information Processing Systems (2019)
12

---

[35] Ranftl, R., Bochkovskiy, A., Kolum, V. Vision transformers for dense prediction. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12179-12188 (October 2021)
[36] Russakovsky, O., Deng. J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115(3), 211-252 (2015)
[37] Sapiro, G.: Geometric Partial Differential Equations and Image Analysis. Cambridge University Press (2001). http:://oi.org// wtt.://og//1..11/////1111111 1017/CB0978051 1626319
[38] Sergey Ioffe, Christian Szegedy: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37. 448 456 (2015)
[39] Shuai Zheng, Sadeep Jayasumana, Bernardino Komera-Paredee Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, Philip Torr: Conditional random fields as recurrent neural networks. In: International Conference Computer Vision (2015)
[40] Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic segmenta- tion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 7262-7272 (October 2021)
[41] Su, Z.. Liu, W., Yu. Z.. Hu, D., Liao, Q.. Tian, Q., Pictikäinen, M., Liu, L.: Pixel difference networks for efficient edge detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5117-5127 (October 2021)
[42] Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang. X., Liu, W., Wang. J.: High-resolution representations for labeling pixels and regions. ArXiv abs/1994.04514 (2019)
[43] Teichmann, M., Cipolla, R.: Convolutional erfs for semantic segmentation. In: BMVC (2019)
[44] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A.N., Kaiser, t.., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998-6008 (2017)
[45] Wang, C., Zhang, Y., Cui, M., Liu, J., Ren, P., Yang, Y., Xie, X., Hua, X., Bao, H., Xu, W.: Active boundary loss for semantic segmentation. In: AAAI (2022)
[46] Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7794-7803 (2018)
[47] Weickert, J., Romeny, B., Viergever, M.: Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Transactions Image Processing 7(3), 398-410 (1998)
[48] Weickert, J.: Coherence-enhancing diffusion filtering. International Journal of Computer Vision 31, 111-127 (1999)
[49] Wu, S., Wu, T., Lin, F., Tian, S., Guo, G.: Fully transformer networks for semantic image segmentation. arXiv preprint arXiv:2106.0411 04008 (2021)
[50] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34 (2021)
[51] Xie, E., Wang, W., Wang, W., Sun, P., Xu, H., Liang, D., Luo, P.: Segmenting transparent objects in the wild with transformer. In: Zhou, Z.H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1 1194-1200, International Joint Conferences on Artificial Intelligence Orgamizationn (8 2021). 24963/ /ic/aijcaai.ii 2021/ main Track
20211/116 https
[52] Yu, C., Wang J., Gao, C., Yu, G., Shen, C., Sang, N.: Context prior for scene segmentation. In: Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition. pp- 12425-12225 (2020)
13

---

[53] Yu, Z., Qin, Y., Zhao, H., Li. X., Zhao, G.: Dual-cross central difference network for face anti- spoofing, In: Zhou, Z.H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, (.CAI-21. pp. 1281-1287. International Joint Conferences on Artificial Intelligence Organization (8 2021).
https://doo
org/10. 20963/ /aaca///jj...////// 2021/1777
main Track
[54] Yu, Z.. Wan, J., Qin, Y., X., Li, S.Z., Zhao, G.: Nas-fas: Static-dynamic central difference network search for face anti-spoofing. IEEE Transactions on Pattern Analysis and Machine (09/TPAMM..000......
Intelligence 43(9), 3005-3023 (2021). https://ooi.or//10.
[55] Yu, Z., Zhao, C.. Wang, Z., Qin, Y., Su, Z., Li, X., Zhou, F,, Zhao, G.: Searching central difference convolutional networks for face anti-spoofing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
[56] Yu, Z., Zhou, B., Wan, J., Wang, P., Chen, H., Liu, X., Li, S.Z., Zhao, G.: Searching multi-rate and multi-modal temporal enhanced networks for gesture recognition. IEEE Transactions on Image Processing 30, 5626-5640 (2021). https://do..org/11 www../r./.....11.......1 1109/T1P.2021.00877
[57] Yuan, Y., Chen, X., Wang. J.: Object-contextual representations for semantic segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision ECCV 2020. pp. 173-190. Springer International Publishing, Cham (2020)
[58] Yuhui Yuan, Jingyi Xie, Xilin Chen, Jingdong Wang: Segfix: Model-agnostic boundary refine- ment for segmentation. In: ECCV (2020)
[59] Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang. X., Tyagi, A., Agrawal, A.: Context encoding for semantic segmentation. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 7151-7160 (2018)
[60] Zhen, M., Wang, J., Zhou, I.,, Li, S., Shen, T., Shang, J., Fang, T., Quan, L.: Joint semantic segmentation and boundary detection using iterative pyramid contexts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
[61] Zheng, S.. Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., al.: Rethinking semantic segmentation from transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6881-6890 (2021)
perspective with
[62] Zhou, B., Zhao, H.. Puig. X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127(3), 302-321 (2019)
[63] Zhu, Z., Xu, M., Bai, S., Huang, T., Bai, X.: Asymmetric non-local neural networks for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 593-602 (2019)
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
14

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]
15


Reviews:
[{"review_id": "vS9ZHl1gGB", "reviewer": "Reviewer_GLui", "summary": "This paper proposes a novel convolution-based operation/operator that could utilize the semantic features in an efficient and effective way for the segmentation task. The main contributions of this paper are (1) a new operation (semantic difference convolution, SDC) that is effective in the segmentation of target objects with precise boundaries, (2) the related module (SDM) to enhance the boundary information at the feature level, and (3) the proposed network exhibit the state-of-the-art semantic segmentation performance with precise boundaries of the target objects.", "questions": "* Questions & Discussions\n1. The semantic distance (SD in equation 3) should be discussed. The SD is calculated using the Euclidean distance of two semantic features (f) with different positions. Intuitively, SD value by similar feature exhibits small value, whereas SD value exhibits magnified values when calculating boundaries. At this point, the reviewer is curious about the followings:\n\n- 1-1. The authors’ address is that f exhibits semantic feature, and the Euclidian distance between two f exhibits semantic distance. Here, the definition and the mathematical illustration for the semantic feature and its meaning should be discussed, or a reference is required. \n\n- 1-2. The normalization method (even without) can extremely suppress the values of the inner parts of the objects. In general, the color and semantic features significantly exhibit similar values in the same objects. Therefore, the SD value inside the object can be almost 0. Whereas the SD values nearby boundaries could be extremely magnified. Therefore, the reviewers worried that extremely small inner values degrade feature extraction for semantic features related to the target objects and only focus on the boundary-oriented features, and thus the predicted segmentation mask includes big halls inside.\n\n2. Since the convolution operation can be utilized in many applications, the reviewer is curious that the motivation of the proposed operation in the semantic segmentation task. The proposed operation can be used in general applications such as classification and image generation tasks.\n", "limitations": "1. Major issues\n- The limited novelty should be discussed. As illustrated above (Question), the proposed operation exhibits limited novelty. Despite the simple yet effective mathematical illustration of the proposed convolutional operation, the strict mathematical modeling for the operation and mathematical analysis of why the proposed operation exhibits outstanding performance in segmenting objects’ boundaries should be discussed. \n\n- The experiments should be improved. As the reviewer already understands, the evaluation metric of “mean Intersection over Union (mIoU)” can qualitatively measure the object details. However, recent studies [1-3] proposed new evaluation metrics to measure the object detail (especially boundaries of the target objects) quantitatively. Otherwise, the visualization of the feature map could illustrate the novel feature extraction when processing object details. Please refer to the activation maps [4]. To clear the authors’ addresses, in terms of the improved boundary-oriented segmentation, more experimental or mathematical evidence should be justified.\n\n- Since the authors proposed the convolution-based operation, the mathematical modeling of the operation should be discussed with the parameters used for the previous convolution operation. The mathematical modeling can include the rank, dimension, size, and inputs (parameters). For instance, the size of the output feature map can be determined as --- when the padding parameter is “valid” or “same”, or when using dilation parameters. The author should illustrate the mathematical form to calculate the output shape using kernel size, padding, and strides (additionally dilation). Please refer to this page [5] for the mathematical form.\n\n\n[1] Fernandez-Moral, Eduardo, et al. \"A new metric for evaluating semantic segmentation: leveraging global and contour accuracy.\" 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.\n\n[2] Lee, Kyungsu, et al. \"Boundary-oriented binary building segmentation model with two scheme learning for aerial images.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.\n\n[3] Cheng, Bowen, et al. \"Boundary IoU: Improving object-centric image segmentation evaluation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[4] Selvaraju, Ramprasaath R., et al. \"Grad-cam: Visual explanations from deep networks via gradient-based localization.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n[5] https://www.tensorflow.org/api_docs/python/tf/nn/conv2d\n\n2. Minor issues\n- The reviewer recommends reviewing the grammar and typo errors to improve the quality of the manuscript.\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n1. Strengths\n- The reviewer significantly understands the significance and importance of the task proposed in this paper. Segmenting various objects in real-world images is significantly important for many applications. Especially, clear and precise boundaries are required for the precise semantic segmentation task. \n\n- Reproducibility of the manuscript. The manuscript is well organized in terms of exhibiting the hyper-parameters and model architecture. \n\n- The manuscript is well organized and well written.\n\n2. Weakness\n- More detailed descriptions are illustrated in the “Question” and “Limitation” sections. Please see below.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "XO1wjhUFHEj", "reviewer": "Reviewer_GWNK", "summary": "This paper studies the problem of how to improve the semantic segmentation results for existing architectures. The authors propose a semantic difference guided convolution, so-called SDC, to deliberately enhance the representations in the semantic boundary area. Specifically, it enhances current layer’s kernel with the the next layer’s feature differences, instead of using current layer’s differences like previous methods, which means that the next layer’s features could serve as a semantic guidance to the current layer. The proposed module is claimed to be flexible to co-operate with any ‘encoder-decoder’ architectures. And the effectiveness is verified through extensive experiments.", "questions": "Please refer to the Strengths and weakness.", "limitations": "The authors have pointed out the limitations of their work.\n\nSince it is just a basic method to deal with 2D semantic segmentation, it has nothing to do with the potential negative societal impact.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strength ** \n\n+: The paper is well written and easy to follow.\n\n+: The insight of enhancing the semantic boundary through next stage’s differences is interesting and has somewhat novelty for the semantic segmentation. Can this method generalize to other dense estimation task? \n\n+: The authors fully demonstrate the effectiveness of the proposed method on various datasets.\n\n** Weakness **\n\n-: The wording of “intra-class boundary“ is not precise enough. According to my understanding, the semantic difference is provided by the next layer’s feature, not the output layer. And only the output layer corresponds to the class-level information exactly. Especially in the early layer, two approaching pixels with different features do not imply they belong to different classes.\n\n-: There seems to be no normalization to the SD(*). Would this induces unstable training if the SD(*) generates a quite large number? \n\n-: In table 5(b), it would be more convincing if the author organizes the comparisons as “Baseline, Baseline+Ours, Baseline+DenseCRF, Baseline+DenseCRF+Ours, Baseline+SegFix, Baseline+SegFix+Ours, Baseline+InverseForm, Baseline+InverseForm+Ours”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RxJH8D_mN7z", "reviewer": "Reviewer_U8Zf", "summary": "Overall, this is a good work.\n\nIn this paper, a new semantic segmentation pipeline is presented, based on proposed semantic difference convolution. The semantic difference convolution incorporates higher-level semantics into the lower-level convolution and helps learn better features which are sensitive to semantic boundaries. The proposed method is easy to implement on existing feed-forward segmentation baselines. According to a set of experiments on previous segmentation models including DeepLabv3, OCRNet and MaskFormer on 4 public datasets, the effectiveness of the proposed method is clearly demonstrated. Besides, this paper also provides analysis on the computational complexity and various comparisons regarding the configurations, as well as many ablations to show the SOTA results from the proposed method.\n\nThis paper is also well-organized, that I can follow their draft easily. Even though there are a few comments from my side, that I believe it could be improved, this is a good work for NeurIPS.\n\nAfter reading all the comments and authors' feedback, I vote for accept (7) on this submission.", "questions": "1. This paper applies DeepLabv3, why not DeepLabv3+?\n2. In Table 5 (b), why is the baseline performance only 65.1? What is the baseline here? More details are needed. Besides, why not apply a stronger baseline in Table 5?", "limitations": "No. This work is a fundamental research in computer vision.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This paper is well organized. It is overall easy to follow.\n+ This paper proposes a new convolution operation, which is simple to implement. Based on the proposed semantic difference convolutions (SDC) and SDM, more favorable segmentation results can be achieved. To show the effectiveness of the proposed method, the authors show different network backbone as well as segmentation models.\n+ The proposed SDC achieves more improvements compared with previous boundary-aware methods. Further, the SDC is compatible with other techniques, such as DenseCRF, SegFix.\n+ From the analysis and empirical evaluation, the proposed SDC will not increase the computational cost significantly. \n\nWeakness:\n- In section 3.2, it needs to discuss more details of f_{pi}. What is f_{pi}? Even though readers can understand when they see section 4.2 and Figure 4, it is a little confused in section 3.2. More explanation on the semantic information f_{pi} will be helpful.\n- Lack of comparison with more boundary-aware methods, such as Gated-SCNN.\n- In this work, DeepLabv3, OCRNet and MaskFormer are applied as baseline models. Why not apply the proposed method on the current state-of-the-art model to achieve new SOTA result, for example SegFormer?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "47bxz-N6N9O", "reviewer": "Reviewer_XkWA", "summary": "This paper proposes an efficient boundary-aware convolution operator to boost the boundary modeling capacity for semantic segmentation. The proposed operator is sensitive to the inter-class boundary while ignoring the noisy intra-class pseudo-boundaries. Based on the proposed operator, a lightweight module is designed to enhance boundary-related information, which is flexible to be plugged into any existing encoder-decoder segmentation model. Experimental results validate that  the proposed approach can achieve consistent improvements for boundary regions over several baselines.\n\n", "questions": "-\tThe further analyses in Table 5a are appreciated, it will be more convincing if some details are given. How to calculate $\\rou$ in this table? How many percents of more parameters are introduced by the proposed approach for each convolution layer? Does CDC also introduce more parameters to the network?\n-\tAbout the pure improvement of the semantic difference term: If we replace SDC with CDC in the proposed approach, how well does it perform?\n-\tDoes the benefit of the semantic difference term for intra-class boundaries rely on the scale of convolution kernels and the quality of the semantic feature?\n", "limitations": "The authors adequately addressed their work's potential negative social impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n-\tThis paper proposes an efficient boundary-aware convolution operator to boost the boundary modeling capacity for semantic segmentation.\n-\tThe proposed approach shows promising performance for inter-class boundaries.\n-\tComprehensive ablative studies are conducted to show the effectiveness of the proposed approach.\n\nWeakness:\n-\tThe proposed approach may introduce new parameters to the network.\n-\tThe proposed Semantic Difference Convolution (SDC) is similar to Central Difference Convolution (CDC).\n-\tThe effectiveness of the semantic difference term needs more clarification.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept