Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
Authors: Zhiding Yu
Rui Huang
Wonmin Byeon
Sifei Liu
Guilin Liu
Thomas Breuel
Anima Anandkumar
Jan Kautz
Venue: NeurIPS 2021 Poster (2021)
Keywords: Semantic Segmentation
Semantic Edge Detection
Structured Prediction
Propagation Network

Abstract: Image segmentation and edge detection are both central problems in perceptual grouping. It is therefore interesting to study how these two tasks can be coupled to benefit each other. Indeed, segmentation can be easily transformed into contour edges to guide edge learning. However, the converse is nontrivial since general edges may not always form closed contours. In this paper, we propose a principled end-to-end framework for coupled edge and segmentation learning, where edges are leveraged as pairwise similarity cues to guide segmentation. At the core of our framework is a recurrent module termed as dynamic graph propagation (DGP) layer that performs message passing on dynamically constructed graphs. The layer uses learned gating to dynamically select neighbors for message passing using max-pooling. The output from message passing is further gated with an edge signal to refine segmentation. Experiments demonstrate that the proposed framework is able to let both tasks mutually improve each other. On Cityscapes validation, our best model achieves 83.7% mIoU in semantic segmentation and 78.7% maximum F-score in semantic edge detection. Our method also leads to improved zero-shot robustness on Cityscapes with natural corruptions (Cityscapes-C).

Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
Zhiding Yu" Rui Huang"t, Wonmin Byeon, Sifei Liu, Guilin Liu,
Thomas Breuel, Anima Anandkumar, Jan Kautz
NVIDIA
Abstract
Image segmentation and edge detection are both central problems in perceptual grouping. It is therefore interesting study how these two tasks can be coupled benefit each other. Indeed, segmentation can be easily transformed into contour edges guide edge learning. However, the converse is nontrivial since general edges may not always form closed contours. In this paper, we propose principled end-to-end framework for coupled edge and segmentation learning. where edges are leveraged as pairwise similarity cues to guide segmentation. At the core of our framework recurrent module termed as dynamic graph propagation (DGP) layer that performs message passing dynamically constructed graphs. The layer uses learned gating to dynamically select neighbors for message passing using max-pooling. The output from message passing is further gated with edge signal to refine segmentation. Experiments demonstrate that the proposed framework is able let both tasks mutually improve each other. On Cityscapes validation, best model achieves 83.7% mloU in semantic segmentation and 78.7% maximum F-score in semantic edge detection. Our method also leads to improved zero-shot robustness on Cityscapes with natural corruptions (Cityscapes-C).
Introduction
Image segmentation and edge detection have been widely studied important perception problems. The two problems are closely related. In fact, segmentation subsumes edge detection since any segmentation contour makes a closed boundary of region. The converse is however not since general edges do not always form closed contours. Nevertheless, edge detection can serve as an auxiliary task improve segmentation performance since edges provide important pairwise similarity cues for segmentation. Early works tend to focus on the grouping and contrast of pixels from perceptual similarity perspective. Martin et [1] proposed the Berkeley Segmentation Dataset, popular benchmark for segmentation and boundary detection that inspired many impactful works in perceptual grouping [2-5]. The recent surge deep learning renders powerful representations with learned features using convolutional neural networks (CNNs) [6]. This has led to great advances in both areas [7-12], but two tasks are often considered separately.
In light of the status quo, we consider coupled edge and segmentation learning. Our goal is two- fold: (1) Multi-task learning being able produce high quality edge detection and segmentation. (2) Mutual improvement the two tasks can help each other with non-trivial performance gains. Designing a principled framework however nontrivial. The key question is how sparse edge signals can be effectively transformed into dense region-level ones to interact with segmentation. To this end, we propose learnable recurrent message passing layer where semantic edges are considered
"Equal contribution. Correspondence Zhiding Yu <zhidingy@nvidia.co Work partially done during an internship NVIDIA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Input Image
Dynamic Graph Prop (DGP)
Segmentation
Affinity Stream
Edge Stream
(Raw)
Edge (Retined)
Figure 1: Framework Overview. The backbone network provides the encoded features as well as an edge stream that produces a semantic edge map. The DGP layer takes them as inputs and uses learnable message passing produce refined feature map (F*) to predict segmentation and edges.
as explicitly learned gating signals to refine segmentation. An overview of our framework is shown in Fig. 1. Specifically, the dynamic message passing layer uses affinity gates to select the neighbor for message passing using max-pooling. It conducts message passing sweeps in each of the four directions: left to right, right to left, top to bottom and bottom top. The message passing is jointly gated by both the affinity and edge gates, therefore allowing edge cues to naturally influence range dense predictions. As such, our framework presents context module that is clean, compact yet powerful. Our technical contributions can be summarized as follows:
We formulate recurrent message passing as dynamic graph propagation (DGP). We show that such formulation simplifies the required normalization in propagation networks [13]. It also dynamically finds graph structures that encodes pixel affinities and improves dense prediction.
We propose double-gate gate design where message passing jointly gated by both the affinity and edge gates to refine the segmentation feature map. We show that this design together with the dynamic I-way connection DGP better couples segmentation and edge learning.
We obtain state-of-the-ard results on joint semantic segmentation and edge detection. We also show that DGP leads to strong zero-shot robustness to natural corruptions with significant improvement over prior methods on Corrupted Cityscapes (Cityscapes-C),
Multitasking segmentation and edge learning is desirable for several reasons: There many downstream applications where both are needed, such as occlusion reasoning [14], localization [15], proposal generation [16-19] and conditional generation [20]. 2) There are many challenging cases where segmentation quality is poor but edge quality is far more superior, e.g. segmentation tends to be inferior near object boundaries since they are often optimized for loU rather than precision [21].
In these cases, edge learning can potentially capture details missed by segmentation. 3) Implicitly improved model generalization result of the coupled learning [22].
Related Work
Semantic Segmentation. There is a rich set of prior work in semantic segmentation. Long et al., [7] proposed fully convolutional end-to-end training of semantic segmentation and pointed out its connection to recognition [6]. Chen et al. [8] introduced atrous convolution and atrous spatial pyramid pooling (ASPP) to capture multi-scale image contexts. Another contemporary work is the wide ResNet-38 which explores a relatively shallower but wider backbone [23]. has also been shown that context plays important role in segmentation, including context encoding [24, 25], multi-scale context [26-28] as well as relational context [29, 30]. More recently, there has been a surge of interests in segmentation with Transformers [31-34]
Boundary/Edge Detection. Similar to segmentation, boundary/edge detection have been widely studied as perceptual grouping problems in early literature [1-3]. Recent methods tend to resort to CNNs. For example, Bertasius et al. proposed a multi-scale deep network architecture for top-down contour detection, where as Xie et al. [11] further introduced holistically-neste edge detection

---

(HED) for end-to-end edge learning. Besides detecting binary edges, Hariharan et al. [35] proposed the Semantic Boundaries Dataset (SBD) which has become a popular benchmark for semantic edge detection. Compared to binary edge detection, semantic edge detection involves the semantic classification of edge pixels in addition to localization which presents additional challenges to existing frameworks. A series of works including HFL [36], CASENet [12], SEAL [37] and STEAL [38] have followed up [35] and pushed the boundaries.
Multi-task Segmentation and Edge Learning, Multi-task segmentation and edge learning remains under-studied but is not entirely new. Edges have been pairwise similarity cues to improve seg- mentation [3, 39, 18] and superpixels [4]. It was shown that edges can be transformed into dense regions through the Laplacian Eigenmaps of boundary transformed affinity [36, 5, 40]. Despite being robust, these methods are generally slow and not end-to-ene trainable. For more recent CNN based models, end-to-end multi-task learning on shared backbone is natural choice [41]. In addition, proper regularization between segmentation and edge has been shown to improve the performance of both tasks. For example, Takikawa et al. [21] use softmax with temperature to impose consistency between segmentation and semantic edges. Zhen et al. [42] correlate semantic segmentation and edge detection tasks with a consistency loss. Our work goes beyond multi-tasking and let the two task more deeply coupled through dynamic graph propagation.
Structured Dense Prediction. Segmentation is a dense prediction task where structured information can be useful. Structured prediction models such as Markov random fields (MRFs) [43]. conditional random fields (CRFs) [44-46] and energy minimization [47, 48] have widely proved helpful in segmentation problems by imposing contrast- sensitive smoothness. In addition, structured inference can also be unfolded as network layers for end-to-end training [49, 50], therefore combining the advantages from both ends. Recently, there is an increasing trend to directly model the message passing process itself using multi-dimensional RNN [51-53) graph RNN [54, 55] and spatial propagation network (SPN) [13, 56, 57]. A common advantage is that these methods render more context-aware prediction with larger receptive fields while preserving local details similar to CRFs. Their ability to train end-to-end allows more powerful representation of inter-pixel similarities and thus better dense prediction quality. Our work can be broadly categorized into this category. It is worth mentioning that the contrast sensitive smoothness term in CRF and propagation networks [52, 13] is an implicit modeling of edge signals by learning to relax smoothness constraints at high contrast areas. There have also been variants of SPN [58] that take binary edge as gate regularization. However, none of these works have explicitly addressed multi-tasking learning with category-aware edges whereas the proposed DGP framework presents a novel and effective solution to this problem.
3 Multi-task network
As first step towards coupled segmentation and edge learning, we introduce two novel multi-task networks that are able to perform multi-task learning for both segmentation and edge detection. An overview of their architectures is shown in Fig. 2. The rest of the paper follows these notations and covers more details for each module.
Backbone. A CNN giving a semantic feature map (denoted as F) encoding the segmentation information. There could be multiple choices of the backbone networks, ranging from the popular architectures of Deepl.abv2 [8], DeepLabv3 [27] to latest state-of-the-are -fttt--rtss-sst-eeeeeeee such as DeepLabvv++ [28] and Atal-Attention Network [30]. Adopting powerful backbones will surely benefit the sestem-level performance, but the major purpose of this work does not completely lie in achieving state-of-the-are We are more interested in showing the effectiveness of a proposed framework on standard backbones. To this end, we consider both CASENet [12] and ResNet-38 [23] as the standard backbones for benchmarking. Both of them comprehensively covers the feature map resolutions of 1, 1/2, 1/4 and 1/8, with the last resolution being a rule of thumb adopted by many segmentation networks. For both networks, we adopt atrous spatial pyramid pooling layers with dilation rate [6, 12, 18, 24, 30, 36) to capture context information. We set the output channels to be 128 to produce semantic feature map, fowllowed by either direct segmentation prediction (baselines) or the proposed DGP layer.
Affinity stream. Convolution layers that aggregate the cross-layer information of the backbone and produces an affinity feature map (denoted as A) to encode the affinity of pairwise neighboring pixels. This stream only used with the presence of recurrent message passing layer (including its related baselines). The stream aims to model the simialrity of pair-wise neighboring pixels and serve
3

---

DVA,, 2944
128
IMA 260
Cola
T7A
17A
17A
138
(L1, 1241
Tack
133 LD 2C
bene
9323
22-24
12.
32444
25844
JUEEXS
2,.00
Fdge
Fdge
Stramm
Sinsar
Figure 2: An overview the architecture for the proposed multi-task Network. Left: architecture with the CASENet backbone. Right: architecture with the ResNet-38 backbone. Each residual block of the backbone is marked with corresponding output scales (1, 1/2, 1/4, 1/8). In addition, the output feature maps A, F and E are marked with (scale, channel number).
major input to the gating signal recurrent message passing layer. Compared to edge stream, the affinity stream seeks to capture coarser-level inter-pixel affinity with slightly deeper convolutions and higher dimensions. The resulting feature map A is a 256 dimensional tensor concatenated by ASPP and the side convolutional feature maps from the affinity stream.
Edge stream. Dense skip-layer features with abundant details that are combined with edge classifica- tion layer through shared concatenation [12] to produce a semantic edge map (denoted as E). We design an improved edge stream over CASENet [12] better leverage detail information at multiple scales/levels with dense side connections. Unlike CASENet where the bottom blocks only provide 1 I-channel side features, we densely obtain side features from every sub-block residual modules. In CASENet, this gives side features with total number of 31 dimensions. Similar rules apply to ResNet-38, where side feature has a total number of dimensions We also found that applying sigmoid after side features with resolution 1/8 greatly benefits edge training from two aspects: (1) helps stabilize and prevent gradient explosions from dense skip connections. (2) removes the typical aliasing edge issue caused by gpsamplig/decconvluiioo and produces elegant smooth edges. Even though we do not explicitly apply techniques such as edge alignment [37] in this work, the proposed backbone is able produce high quality edge predictions under noisy labels. We also notice that is better to remove sigmoid for side features with higher resolutions. Edge prediction is multi-tasked alongside with ASPP using x convolution top of Res Block5. This returns the K classes coarse edge predictions. Similar to CASENet, we upsample all side features together with the K di-mensioonll edge predictions to full image resolution and apply shared concatenation, where side features are repetitively shared and concatenated with each class for K times, followed by K K-way 3 x group convolution to produce semantic edges.
Dynamic graph propagation. Learnable recurrent message passing layer that takes the above three branches as input to produce refined semantic feature map (denoted as F*). More details regarding this module will be introduced in the next section.
Coupled segmentation and edge learning
Although the multi-task network is able to jointly produce segmentation and edge prediction, we interested in letting these two tasks better coupled to mutually improve each other. To this end, we look into recent spatial propagation networks where edges can be transformed into gating signals that produce long range influence to segmentation. We first give the notations and definitions:
4.1 Notations and Settings
We start with two-dimensional recurrent network architecture with a linear propagation module passing messages (memories) spatially over a feature map. We define 4-way message passing: 1. Left>> Right (Mode I), Right~>Left (Mode R), Top-+Bottom (Mode T) and Bottom->To TTop (Mode B). Each way of message passing will separately generate hidden state map H which can be approximately viewed refined (smoothed) version of F. Finally, we take element-wise max

---

operation to ensemble them where the model automatically selects the optimal direction with highest neural activation for each pixel:
F
H max (H H H² H
(1)
To perform message passing in each way, one often needs to pre-define a graph that encodes the message passing paths. For tractability and and computation issues, such graph is often sparsely defined with locally connections between immediate neighboring pixels. We follow spatial prop- agation network (SPN) [13] by defining three-way local connection, as illustrated on the hand side of Fig. 3. The advantage of such design is obvious: Taking Mode R propagation
11 1
H
FLW
Mode
Mode
H
(8.28 NiL.
Mode
Mode
example, one just needs to initiate the message passing from the right most column, and recurrently pass the message from right to left column by col- umn. In this case, the hidden state of each pixel is directly influenced by the three immediate neighboring pixels on the right column. Details of the propagation under Mode R is illustrated on the left hand side of Fig. 3.
Figure 3: Illustration of spatial propagation. Left: prop- agation under Light=>Left mode. Right: the defined graphs and neighbors of different propagation modes.
Let F E #xxxC be the feature map input to the propagation module, H E # the propagation latent space top of F. In addition, let h and fú, denote the hidden state and feature of the th pixel located recurrence ta on H and F, respectively. We denote (pīelk E N(i, ()} the set of learnable propagation gating weights for hidden state h(i, t) where N(i, t) is the set of neighbors of pixel (i, l). In this case, the spatial propagation in each mode is defined as:
h
Σ p kER(),))
fú,
Σ P.. ha.e-1
REN()))
where @ is element-wise product, and hR the hidden state of pixel (i,t)'s neighbors from the previous recurrence. {pL,1* ik N(i, ()) is expandable an affinity matrix, revealing the global and dense message passing among all the pixels of F.
4.2 Dynamic graph propagation (DGP)
As linear module, the above operation requires careful normalization of among {pt, lk E N(i, {)). In particular, ΣxEN(S.) P., S should be satisfied since the energy of the hidden state signal gets unbounded easily under the recurrence operation. To normalize the gating weights, one may consider linear self-none normalization scheme [13] where the constraint ΣkE((L,) Ip,.l I is imposed to guarantee the stableness of the propagation" via dividing each pt., with ExEN(..) lptal-
But the above formulation also leads to certain limitations. For example, the linear form allows pl to be both positive and negative, which potentially encourages př, to be large towards either positive or negative side rather than being monotonic. addition, when pf, 2 0. ExEN(L.)) pt. = which means that such formulation considers zero unary input from ft,- Therefore, another choice is to constrain pL. with probabilistic output:
exp(p!.)
ph.
ExEN(i.) (p.(p)))) 2 (p(.).
(3)
where p is defined as:
P..-
A
(4)
3 There is one-to-one correspondence between (i, and pixel index (h, w)) But the mapping varies subject the propagation mode. Mode R for example, corresponds to column with ur whereas h = Similar property holds when the same constraint applied each dimension of př, in Equation (2).
5

---

and (-) is the Sigmoid function. Note that P is the raw gating activation without normalization, and is obtained via linear projection on the concatenated affinity stream features.
The above formation satisfies both EkeN(s,)) P S and PL, 2 O, while partially considering the unary imput with the Sigmoid term. Yet the framework is a bit complicated with many non-linear terms. To this end, we take one more step to further sparsifying and simplifying the gating response by considering a softmax- witta---itt--eepratuure formulation and taking the limit of T + 0:
exp(f%,)/7
PL.
lim T- EkEN(L,)) exp(P",)TT (
(5)
Substituting Equation (5) into Equation (2), we have:
h
or (p o fúz + to(p)) @ hi
(6)
where the n-th dimension of p and h are defined as:
k - arg max [n]] ph,[1]] = pf, hi,(n] [m] = 1x-,e-1mm) [n].
(7)
kEN(1,0)
Equation (6) essentially leads to dynamic graph propagation (DGP) framework where one performs mes- sage passing on a dynamic graph structure by picking neighbor with the highest response. Intuitively, DGP captures compact structure that has close relation to directed minimum spanning tree, Thu-Liu/Edmonds al- gorithm [59] and the recently pro- posed tree filters [60, 61]. Such structure presents inductive bias that benefits segmentation by filtering out noisy prediction and following only strong signals. Figure 4 illustrates DGP where we show three example paths on three feature channels. Each channel independently takes different paths depending on its own neighbor affinities.
F
H
Figure 4: Illustration of dynamic graph propagation.
4.3 Coupling edge prediction with segmentation
To deeply couple edge prediction with segmentation, we further incorporate edge signal into the above dynamic graph propagation by proposing double-gate framework:
h = (1 o (pt,) (gg.t)) @ fi.t + (pt,,) O (g..)) @ hi.
(8)
where gt defined as:
g W. Eli - 1
(9)
Note that the edge gating signal is obtained by via a 3 x 3 convolution on the K K-channel edge activation map E, which outputs vector g sharing the same dimension (128) as p'. This way, edges are able to actively influence segmentation via edge-sensitive gating on message passing.
We also hope that the refined segmentation activation after DGP can alternatively serve shape regularizer of the edge prediction. To this end, we output a K-channel edge regularization feature from F,, using a refined edge map E*. This is illustrated in Fig. on the bottom right. One shall see that the above coupled design let the two tasks greatly improve each other, and we term the final framework CSEL.
convolution, followed by another x 3 convolution fuse with E to produce
4.4 Training loss
The training loss for CSEL multi-task combination cross-entropy (BCE) losses for multi-label edge learning and the cross-entropy (CE) loss for segmentation:
L Lseg LEdge LCE (ce((Cn((F")) X(CBCE E*) + LBCE (E))
(10)
where À is the parameter controlling the weight of segmentation loss. For segmentation, we use 3 x 3 convolution to lincarly project the 128-channel activation F" into K- -channeel before the loss.

---

Experiments
5.1 Datasets and metric
Cityscapes. Cityscapes [62] contains 2975 training images, 500 validation images and 1525 private testing images with 19 pre-defined semantic classes. The dataset has been widely adopted as the standard benchmark for both semantic segmentation and semantic edge detection. Following number of previous works [12, 37, 38, 21J, we comprehensively conduct ablation and quantitative studies for both segmentation and edge detection on the validation set.
SBD. The Semantic Boundaries Dataset [35] contains both category-level and instance level semantic segmentation annotations. The dataset contains 1 11355 images (8498 for training and 2857 for testing) from the trainval of PASCAL VOC2011 [63], and follows the 20-class VOC definition.
PASCAL VOC 2012 [64] a semantic segmentation dataset with 1464 (training) and 1449 (val) and 1456 (test) images. We use the augmented dataset with 10582 training images, as in [28]. The dataset contains 20 foreground object classes and background class.
COCO Panoptic [10] contains the mask annotations for both things and stuff, with split setting (118K train 2017 images and 5K validation 2017 images) following the detection community.
Evaluation metrics. For evaluation, We consider intersection-over-uiion (IoU) for segmentation, and the maximum F-score (MF) at optimal dataset scale (ODS) for edge detection We also consider the boundary F-score proposed in [21] for direct comparison on edge detection task.
5.2 Implementation details
Data loading. During training, we unify the training crop size as 1024 X 1024 for Cityscapes, 472 x 472 for SBD and VOC12, and 464 x 464 for COCO Panoptic. All models are trained with 150k iterations on Cityscapes with batch size 8, 30k iterations on SBD and VOC12 with batch size 16, and 220k iterations on COCO Panoptic with batch size 16. We also perform data- data--ugmentation with random mirror, scaling (scale factors in [0.5, 2.0) and color jittering.
Optimization, we apply an SGD optimizer with a weight decay of 5 x 10-1 during training. For baselines and methods that involving CSEL, we additionally apply a second ADAM optimizer to the propagation layers. In our case, we empirically found that ADAM optimizer is more capable of optimizing the propagation layers with better performance.
Learning rate and loss. The base learning rates for methods with ResNet-101/Resst 101/ReeNeteeB33B backbones unified as 3.0 10 1--0/ x 10 across Cityscapes, SBD and VOC12. On COCO Panoptic, the base learning rate is unified as 5.0 x 10 for all comparing methods. Unless indicated, the segmentation weight À in Eq (7) is empirically set it to 0.5 to balance LSeg and Edgee We found method not sensitive to X, and setting À 0.5 works excellently for all backbones and datasets.
Using Mapillary Vistas. On Cityscapes, a number of literature consider large-scale pretraining on Mapillary Vistas (MV) [65], which shown to considerably benefit the segmentation performance. Unless indicated, our method does not adopt Mapillary Vistas pre-training for fair comparison.
5.3 Experiments on Cityscapes
Ablation study (semantic segmentation). Table shows the ablation studies on semantic segmentation where consider apple-to-apple compared methods that are trained following the same backbones and training protocols: I) ST: Single task segmentation network without propagation layer but keeping the 128-channel feature map F. 2) MT: The same ST but naively multi-tasking segmentation with edge detection. 3) SPN [13]: Implementation of SPN with the 3-way propagation on F, where gating signal is computed using the affinity stream A. 4) SPN+Edge: Coupling SPN with edge learning following exactly the same double gate design. 5) DGP: Baseline which
Table I: Ablation study segmentation Cityscapes validation set.
method
ResNet- 101
ResNet-38
ST
77.9
78.55
MT
78.4
79.43
SPN
80.0
SPN+
80.4
DGP
81.3
CSEL-
80.9
CSEL
82.8
82.8
CSEL-Ms
83.7
83.4
we follow [37] by applying exactly the parameters and settings.
7

---

Table 2: Main results semantic segmentation and semantic edge learning on Cityscapes.
(a) Main semantic segmentation resulls
(b) Main semantic segmentation results
the Cityscapes validation set.
on the Citycaaes test set.
(c) Main results of semantic edge detection on the
Method
Backbone
MV
mlol
Method
Backbene
MV
mioli
Cityscapes validation set. Results measured by
PSPNer [26]
ResNee-
PSPNet [25]
ResNet-101
78.4
Maximum F-Score (MF) at optimal data scale.
Dee [38
ResNes-101
PSANer (69)
ResNer-101
80.1
CCNa (66)
ResNa-101
80.5
ASPP [27]
ResNet-101
80.1
Methed
Backhoon
MF (1S)
MF (Nas-15)
RPCNee (42)
GSCNN]211
DANe (50g
ResNer- 10]
ResNet-101
ResNes101
82.1
DFP [58]
MM-Net| [30]
ResNect-10 101
ResNet-101
81.4
81.5
CASUNNc (12)
SEAL
ResNer- 101
101
89:1
CSFI.
ReaNes-001
OCR [29]
ResNet-101
81.8
Ros
CCNet [66]
ResNer-101
81.9
RPCNet (42)
ResNee- 101
78.2
Viteo-P (67)
GSCNN! [211]
SGIN (57)
ResNet 38
RecNer-18
1e
90.8
80.9
CSEL
RPCNel [12]
ResNect-11
ResNet- 101
82.1
CSEL CSEL
ResNes 101
ResNet 38
78.1
73.3
Axial DespLab [65]
AxiilRes XL.
$1.1
GSCNN
ResNet
82.9
CSEL.
ReaNet-38
83.4
CSBL
ResNet-38
83.5
contains the proposed DGP layer without the double gate design. 6) CSEL Baseline which is the same as CSEL except for removing the edge loss. 7) CSEL: Our full method with single scale inference. 8) CSEL-MS: CSEL with multiscale inference at {0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0]. We also consider both instance-sensitive (IS) and non-instance-sennitive (Non-IS) settings where the edge training/evaluation labels are with/without instance-edges.
Discussions. We make sure that the comparisons are apples to apples and fair, by using the same backbones and training recipes (such as learning rate, crop size, batch size, and number of iterations) for all comparing methods.Thm DGP baseline can be considered as an apples to apples counterpart of SPN, whereas the CSEL baseline is a further study to understand whether the improvement in CSEL purely comes from the enriched representation with edge features. One could observe several trends from the results: 1) The proposed multi-tas talti-task network consistently outperforms its single task counterpart. 2) CSEL outperforms both SPN and SPN+Edge with the dynamic graph propagation and edge gating. 3) The dynamic graph design in DGP alone helps it to achieve 81.3% mloU, outperforming both SPN and Sdg+Edge by 1.3% and 0.9%, respectively. 4) The incorporation of edge guidance with the double gate design further leads to another non non-trivial 1.5% improvement. Simply adding the edge feature does not improve the segmentation quality. In fact, CSEL is even slightly lower than DGP where no edge features are involved. This reflects the importance of edge signal as structural guidance than making the representation more expressive.
Comparison state-of-the-art (semantic segmentation), We also compare CSEL to state-of-thr the art semantic segmentation models and list the results in Table 2a and 2b. For clarity, we divide to table into two parts with models using the same type of backbone putting together. Among the comparing methods, CSEL obtains the best results using both ResNet-101 and ResNet-38 backbones.
Ablation study (edge detection). We additionally con- duct experiments on edge detection. We first show an ablation study in Table 3, where ST indicates single-task edge detection network with the proposed edge stream.
Table 3: Ablation study adge detection the Cityscapes validation set. Results measured by MF and scaled by %.
Method
Rex1Ol (1S)
Rest0l (Cls)
Res (Cls)
ST
74.66
74.87
75:42
MT indicates naive multi-task training of both segmenta- tion and edge. MT leads to slight performance degrada- tion on the edge detection task compared single-task
MT
74.48
74.62
75.38
CSEL
77.58
77.67
78.19
CSEL-Ms
78.09
78.32
78.73
training on edge detection. However, the degradation is marginal compared to the significant im- provement of the edge quality introduced by deep coupling of the two tasks in CSEL, Again, this shows the considerable benefit of coupled learning with both segmentation and edge detection.
Comparison to state-of-the-art (edge detection). We also compare to statc-of-the-art semantic edge detec- tion methods in Table 2c where we present the train- ing/evaluation protocols in two categories (IS and Non- IS) following SEAL. Note that we evaluate the results with the edge thinning protocol. One could see that our method achieves the best performance in both settings with significant performance gain from the coupled edge and segmentation learning. as well as the
Table 4: Results of boundary evaluation Cityscapes vali- dation set following [2I]. Results measured by boundary F- score and sealed by %. 1 indicates results reported in [21].
Method
Backboas
12px
9ps
Spx
3px
ResNet-38
80.1
74.7
69.7
GSCNN 1211
ResNet-38
81.8
80.7
77.6
73.6
CSEL
ResNet-38
87.0
86.0
83.4
79.5
CSEL
ResNet-101
87.2
86.2
83,5
79.6
benefit from the dynamic graph propagation module. Besides the evaluation protocol from SEAL,, we also follow separate evaluation protocol from GSCNN [21] in Table 4, where we use the original
8

---

evaluation code base from GSCNN to evaluate the boundary quality of semantic segmentation. One can observe that all different thresholds, CSEL with both ResNet- 101 and ResNet-38 backbones outperform Deepl Lab ++ and GSCNN by significant margins in terms of boundary quality.
5.4 Experiments on SBD
Table S: Semantic edge detection on the re-annorated SBD
We also evaluate CSEL on the SBD dataset and compare with previous state-of-the-art semantic edge detection models. The main results are presented in Table 5. Note that we used the re-annotated SBD test set from [37] to pursue more precise semantic edge evaluation. From the table, one could see CSEL outperforms other ofate-of- the-art methods in both IS and Non-IS settings.
test set. Results messured by MF and scaled by %. 1 indi- cates results reported 138).
Method
Backbone
MF (IS)
MF (Non-IS)
CASENet [12]
ResNet-101
63.6
63.51
SEAL 1371
ResNet-101
67.0
66.8
STEAL
ResNet-101
65.8
68.2
DEF
ResNet-101
68.0
CSEL
ResNet-101
73.1
73.2
5.5 Experiments on PASCAL VOC12
We evaluate CSEL on the PASCAL VOC12 dataset and report the main results of semantic segmentation in Table 6. The performance is evaluated as the mean IoU over the 20 PAS- CAL VOC classes plus the background class. We compare CSEL with previous competitive methods that state-of-the- art (DeepLabv3+) or considerably related (SPN/DFN). CSEL achieves considerable improvement these methods.
Table Results VOC12 validation
Method
Backbone
mloU
DT DT-EegeNee [52]
ResNet-101
69.96
SPN (3-way) [13]
ResNet-101
75.28
DFN [41]
ResNet-101
80.60
Dbeplaabe3+ [28]
ResNet-101
80.57
CSEL
ResNet-101
83.30
5.6 Experiments on COCO Panoptic
We further conduct experiments the full COCO Panoptic dataset. Several earlier methods have reported results on COCO- Stuff 10K which an earlier version of COCO with much less data. However, this does not lead to significant differences with VOC12 in both the size and the diversity of data. We therefore adopt the full COCO Panoptic dataset which is sig- nificantly larger and is widely accepted by the detection and instance/panoptic segmentation communities. We hope that our work present solid baselines that inspires subsequent research on this benchmark.
Table 7: Results COCO Panoptic val2017.
Method
Backbone
mloU
ST
ResNet- 101
53.95
MT
ResNet-101
54.14
SPN
ResNet-101
56.49
SPN-Edge
ResNet-101
56.51
CSEL.
ResNet-101
57.03
Specifically, we convert the instance segmentation annotations to semantic segmentation ones, and compare segmentation methods with or without the proposed CSEL module. All comparing methods adopt the same CASENet (ResNet- 101) architecture. Other hyperparameters are kept exactly the same. Models trained on the train2017 split and tested val2017 with single scale inference.
From the results in Table 7, one could see that adding CSEL leads to consistently improved results over the naïve segmentation basclines. In addition, multi-task learning with both segmentation and edge losses also slightly outperforms the single segmentation. Finally, CSEL slightly outperforms SPN with reduced gain compared to results on Cityscapes. We hypothesize that the reduced gain is partly caused by the noisier mask annotations on COCO Panoptic which makes edge learning more challenging and decreases the edge gate quality.
5.7 Robustness Against Natural Corruptions
We hypothesize that CSEL also brings robustness to the segmentation model. We show that this the case and empirically verify on Cityscapes-C in Table 8. Specifially, we follow the standard corruption package provided by [70] to corrupt the Cityscapes validation images. This expands the Cityscapes validation set with 16 types of algorithmically generated corruptions from 4 major categories: "Noise". "Blur", "Weather" and "Digital". Each corruption type also contains 5 severity levels, leading to 2500 evaluation images for each type alone
Our result shows that CSEL overall improves the robustness significantly, especially compared to ResNet-38 and GSCNN [21] which share the same backbone. On the other hand, the method does
'We only consider level 1-3 for the "Noise" category following [71].

---

still show some vulnerability to certain type of corruptions, particularly those belonging to the "Noise" category. However, this trend aligned with ResNet-38 and GSCNN and we hypothesize that may be partly related to the specific design of ResNet-38 based on this pattern.
Table 8: Main results on Cityscapes-C. "DIv3+", "MBv2", "R" and "X" refer Deepl.aby3+, MobileNetv2, ResNet and Xception. The mloUs of compared methods reported from [71]]
Arch.
Clean
Motion
Detoc
Blur Glass
Gauss
Gauss
Impol
Noise
Shot
Speck
Bright
Coptr
Digital
Satur
JPEG
Snow
Weather Spart
Fog
Frost
DLv3+ (MBv2)
72.0
53.5
49.0
45.3
49.1
7.0
6.6
16.6
51.7
46.7
32.4
27.2
13.7
38,9
47.4
17.3
DL.3+ (R50)
76.6
58.5
56.6
47.2
57.7
7.2
10.0
31.1
58.2
54.7
41.3
27.4
12.0
42.0
55.9
22.8
DLv3+ (R101)
77.1
59.1
56.3
47.7
57.3
13.2
13.9
16.3
36.9
59.2
54.5
41.5
37.4
11.9
47.8
55.1
22.7
DLv3+ (X41)
77.8
61.6
54.9
51.0
54.7
17.0
17.3
21.6
43.7
63.6
56.9
51.7
38.5
18.2
46.6
57.6
20.6
DLv3+ (X65)
78.4
63.9
59.1
52.8
59.2
15.0
10.6
19.8
42.4
65.9
59.1
46.1
31.4
19.3
50.7
63.6
23.8
DLv3+ (X71)
78.6
64.1
60.9
52.0
60.4
14.9
10.8
19.4
41.2
68.0
58.7
47.1
40.2
18.8
50.4
64.1
20.2
ICNet
65.9
45.8
44.6
47.4
44.7
8.4
8.4
10.6
27.9
41.0
33.1
27.5
34.0
6.3
30.5
27.3
11.0
FCN8s
66.7
42.7
37.0
34.1
5.7
24.9
53.3
39.0
36.0
21.2
31.6
37.6
19.7
DilatedNeet
68.6
44.4
36.3
32.5
38.4
15.6
14.0
18.4
32.7
$2.7
32.6
38.1
29.1
12.5
32.3
34.7
19.2
ResNet-38
77.5
54.6
45.1
43.3
47.2
13.7
16.0
18.2
38.3
60.0
50.6
46.9
14.7
13.5
45.9
52.9
22.2
PSPNet
78.8
59.8
53,2
44.4
53.9
11.0
15.4
15.4
34.2
60.4
51.8
30.6
21.4
8.4
42.7
34,4
16.2
GSCNN
80.9
58.9
58.4
41.9
60.1
2.6
6.8
24.7
75.9
61.9
70.7
12.0
12.4
47.3
67.9
32.6
CSEL (R38)
82.8
63.7
66.4
44.3
66.4
15.1
6.4
17.5
40.4
79.7
66.0
78.3
17.1
19.5
41.4
78.5
33.0
6
Conclusion
We proposed unified framework for coupled segmention and edge learning. Our work revisits the two long-standing and important perceptual grouping problems semantic segmentation and edge detection. Our method (CSEL) includes a novel end-to-end multi-task network, recurrent dynamic graph propagation layer, as well as deep coupling of the two tasks on top on top of them. Finally, results show that careful coupling of the tasks leads to significant improvement both of them.
Broader Impact
Our method coupled two long-standing important vision problems, semantic segmentation and edge detection, under unified framework. Besides the various practical benefits from deeply coupling these two tasks, we expect the research to inspire considerable insights, interests and revisits on perceptual grouping, mid-level representations and structured prediction. The result of the research is likely to find diverse scene understanding applications such as autonomous driving and robot navigation. Like many other discriminative recognition models, our method inevitably faces challenges from input data quality and underlying data biases. The model behavior is subject to various factors such as data distributions, domain gaps, label quality and fairness. We encourage researchers to focus the "in the wild" robustness when the above challenges are present.
Acknowledgement
We thank the NVIDIA GPU Cloud (NGC) team for the computing support of this work. We also thank the anonymous reviewers and the other NVIDIA colleagues who helped to improve this work with discussions and constructive suggestions.
References
David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik, al. A database of human segmented natural images and its application to evaluating segmentation algorithms measuring ecological statistics. In ICCV, 2001. 1, 2
David R Martin, Charless C Fowlkes, and Jitendra Malik. Learning detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. PAMI, 2004.
[] Pablo Arhelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Trans. PAMI, 2010. 2,3
[4] Piotr Dollár and C Lawrence Zitnick. Fast edge detection using structured forests. IEEE Trans. PAMI. 2014. 3
10

---

[5] Jordi Tuset, Pablo Arbelaez, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans. PAMI, 2016.
1,3
[6] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012. 1,2
[7] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015, 1,2
[8] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. PAMI, 2017. 3,3
Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. /CCV, 2017.
[10] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. 1.Xiv:8801.00000 2018."
[11] Saining Xie and Zhuowen Tu. Holisticall-nnested edge detection. In ICCV, 2015.
[12] Zhiding Yu, Chen Feng, Ming- Liu, and Srikumar Ramalingam. Casenet: Deep category-aware semantic edge detection. In CVPR, 2017. 1,3, 4, 7,8,
[13] Sifci Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz. Learning affinity via spatial propagation networks. In NewrlPS, 2017. 2, 7,
[14] Patrik Sundberg, Thomas Brox, Michael Maire, Pablo Arbeláez, and Jitendra Malik. Occlusion boundary detection and figure/ground assignment from optical flow. In CVPR, 2011. 2
[15] Xin Yu, Sagar Chaturvedi, Chen Feng, Yuichi Taguchi, Lee, Clinton Fermandes, and Srikumar Rama- lingam. Vlase: Vehicle localization by aggregating semantic edges. In IROS, 2018. 2
[16] C Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. ECCV, 2014.
[17] Pablo Arbeláez, Jordi Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial groppppng In CVPR, 2014.
[18] Kevis-Kokiet Maninis, Jordi Pont-Tuset, Pablo Arbeláez, and Luc Van Gool. Convolutional oriented boundaries. In ECCV, 2016.
[19] Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Arheláez, and Luc Van Gool, Convolutional oriented boundaries: From image segmentation high-level tasks. IEEE Trans. PAMI, 2018.
[20] Ting-Chin Wang, Ming-Yos Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. Advances Neural Information Processing Systems (NeurIPS), 2018.
[21] Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-senn: Gated shape cnns for
semantic segmentation. In ICCV, 2019. 2, 2,3,7, 3,
[22] Lin Yen-Chen, Andy Zeng, Shuran Song, Phillip Isola, and Tsung- Yi Lin, Learning to see before learning to act: Visual pre-training for manipulation. In ICRA, 2020. 2
[23] Zifeng Wu, Chunhua Shen, and Anton Van Den Hengel. Wider deeper: Revisiting the resnet model for visual recognition. In Pattern Recognition, 2019. 2,
[24] Hexiang Hu, Zhiwei Deng, Guang-Tong Zhou, Fei Sha, and Greg Mori. Labelbank: Revisiting global perspectives semantic segmentation. 17Xivv1703.09891 2017. 2
[25] Hang Zhang. Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang. Ambrish Tyagi. and Amit Agrawal. Context encoding for semantic segmentation. In CVPR, 2018. 2
[26] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. 2,
[27] Liang- Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05530 05587, 2017. 3,8
Liang-Chieh

---

[28] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decodlr with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 2, 3. 7, 8,
Liang-Chieh
[29] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. arXiv:I909.10 1906, 2019. 2,
[30] Jun Fu, Jing Liu, Haijie Tian, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation.
In CVPR, 2019. 2, 3,8
[31] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yahiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequeeeee perspective with transformers. In CVPR. 2021. 2
[32] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbonc for dense prediction without convolutions. arXiv preprint arXiv: 12122, 2021.
[33] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2103. 14030, 2021.
[34] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. arXiv preprint urXiv:2104. 13840, 2021. 2
[35] Bharath Hariharan, Pablo Arbeláez. Lubomir Bourdev, Subhransu Maji. and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011. 3,
[36] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. High-for- low and low-for -iigh Efficient boundary detection from deep object features and its applications high-level vision. In ICCV, 2015. 3
[37] Zhiding Yu, Weiyang Liu, Yang Zou, Chen Feng, Srikumar Ramalingam, BVK Vijaya Kumar, and Jan Kautz. Simultancous edge alignment and learning. In ECCV, 2018. 3, 4, 7, 8,9
[38] David Acuna, Amlan Kar, and Sanja Fidler. Devil in the edges: Learning semantic boundaries from noisy annotations. In CVPR, 2019. 3, 7, 8,9
[39] Xuming He, Richard S Zemel, and Miguel Á Carreira-Perpinan. Multiscale conditional random fields for image laheling. In CVPR, 2004. 3
[40] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural fields. In CVPR, 2016. 3
[41] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Learning discrimina- tive feature network for semantic segmentation. In CVPR, 2018. 3,9 9
[42] Mingmin Zhen, Jinglu Wang, Lei Zhou, Shiwei Li, Tianwei Shen, Jiaxiang Shang, Tian Fang, and Long Quan. Joint semantic segmentation and boundary detection using iterative pyramid contexts. In CVPR, 2020. 3,8 8
[43] Charles A Bouman, Ken Sauer, and Suhail Saquib. Markov random fields and stochastic image models. In 1995 IEEE International Conference Image Processing. 1995. 3
[44] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, 2001.
[45] Sanjiv Kumar and Martial Hebert. Discriminative random fields. IJCV, 2006.
[46] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. In NeurlPS, 2011. 3
[47] Yuri Boykoy, Olga Veksler, and Ramin Zabih, Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI, 2001. 3
[48] Yuri Boykov and Gareth Funka-Lea. Graph cuts and efficient nd image segmentation. IJC, 2006. 3
[49] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera- Comera-Paredee Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks, In ICCV, 2015.
12

---

[50] Guosheng Lin, Chunhua Shen, Anton Van Den Hengel, and Ian Reid. Efficiint piecewise training of deep structured models for semantic segmentation. In CVPR, 2016. 3
[51] Wonmin Byeon, Thomas M Breuel, Federico Raue, and Marcus Liwicki. Scene labeling with Istm recurrent neural networks. CVPR, 2015. 3
[52] Liang-Chine Chen, Jonathan T Barron, George Papandreou, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with task-specific edge detection using cnns and discriminatively trained domain transform. In CVPR, 2016. 3,9
[53] Bing Shuai, Zhen Zuo, Bing Wang, and Gang Wang. Scene segmentation with dag recurrent neural networks. IEEE transactions on pattern analysis and machine intelligence, 40(6):1480 1493, 2017.
[54] Xiaodan I Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, and Shuicheng Yan. Semantic object parsing with graph lstm. In ECCV, 2016. 3
[55] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural- Deep learning on spatio- temporal graphs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5308-5317, 2016. 3
[56] Xiangyu Xu, Deqing Sun, Sifei Liu, Wengi Ren, Yu-Jin Zhang, Ming- Hsuan Yang, and Jian Sun. Rendering portraitures from monocular camera and beyond. In ECCV, 2018. 3
[57] Sifei Liu, Xueting Li, Varun Jampani, Shalini De Mello, and Jan Kautz. Learning propagation for urbitrarll--structued data. In Proceedings of the IEEE International Conference on Computer Vision, pages 662-661,27 2019. 8,8
[58] Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary-aware feature propagation for scene segmentation. In ICCV, 2019. 3.8 8
[59] Yoeng-Jin Chu. On the shortest arborescence of a directed graph. Scientia Sinica, 14:1396-1400, 1965.
[60] Lin Song, Yanwei Li, Zeming Li, Gang Yu. Hongbin Sun, Jian Sun, and Nanning Zheng. Learnalle tree filter for structure-preserne feature transform. In NewrlPS, 2019. 6
[61] Lin Song. Yanwei Li, Zhengkai Jiang, Zeming Li, Xiangyu Zhang, Hongbin Sun, Jian Sun, and Nanning Zheng. Rethinking learnable tree filter for generic feature transform. In NeurIPS, 2020. 6
[62] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 7
[63] M. Everingham, I. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. PASCAL Visual Object Classes Challenge 2011 (VOC2011) Results. http://www.pascal- http://nggeee////o//11///1//
The
[64] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-11 136, 2015. 7
[65] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017. 7
[66] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pages 603-612, 2019. 8
[67] Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In CVPR, 2019. 8
[68] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh decplab: Stand-alone axtal-attention for panoptic segmentation. In European Conference on Computer Vision, pages 108-126, Springer, 2020. 8
Chich Chen. Axial-
[69] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 267-283, 2018. 8
13

---

[70] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann. Alexander S. Ecker, Matthis Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint
2019.
[71] Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmentation models. In CVPR, 2020, 9,
7 Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see result analysis on Cityscapes-C.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see Broader Impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] Going through legal approval process at this moment. Will release source code upon approval.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to implementation details in both the main paper and the supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We observed stable mloU results for semantic segmenta- tion models.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer the supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [Yes] The data used in our work is open source and can be used for adademic research.
(c) Did you include any new assets either in the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A] The data used in our work is open source and can be used for adademic research.
(c) Did you discuss whether the data you are using/curating contains personally identi- fiable information or offensive content? [N/A] The data does not contain personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "vRpr1ZtoUOG", "reviewer": "Reviewer_YztE", "summary": "This paper proposed a unified framework for detecting edges and segmenting objects. They conduct experiments on Cityscapes, SBD and Pascal VOC. The edge detection results are competitive to previous SOTA methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses the problem of joint predicting the segmentation results and the edge. By the proposed DGP layer, they achieve good results among several benchmarks. They also show robustness by experimenting on Cityscapes-C. There are some minor concerns, which need more clarification:\n1. Do the PASCAL VOC and SBD train together? Are the method in Table 5 and Table 6 also trained under multi-task settings?\n\n2. I am wondering about the efficiency of the proposed DGP. It will be better if the computational cost and inference speed could be compared and discussed.\n\n3. Will edge detection help the semantic segmentation? It will be better if the single edge detection and single segmentation results could be reported in Table 2.\n\n4. There are some typos, for example, Line 78...", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "eeUPFPFMwwd", "reviewer": "Reviewer_vms1", "summary": "The paper proposes coupling edge prediction and semantic segmentation using a multi-task network dubbed CSEL. Some of their goals include the mutual improvement of performance on the two tasks when learned together. The proposed model (CSEL) extends spatial propagation network SPN  with a dynamic graph propagation (DGP) scheme. DGP is a gated recurrent graph message passing that is responsible for selecting neighboring pixels with the highest response. In DPG, the semantic edges are considered as gating signals to refine semantic segmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is nontrivial and provides some additional gains over SOAT. The paper also contains an insightful ablation study, which is positive. In many experiments, the performance gains are not very expressive, which makes me feel lukewarm about this approach. A major issue to me seems to be the lack of clarity in the explanations and figures. I feel it is hard for a reader to understand the concepts/architecture and, replicate the work for future research. I would not recommend this paper for publication in this current state, however, I will keep an open mind for post rebuttal remarks. Some more detailed comments are presented below:\n\n1) I would appreciate hearing from the authors on why the edges only serve segmentation only as gating signals?  Could it not be helpful to enrich the segmentation features as well? Actually, I believe this is one point where clarification could be improved. The message passing figures seem a bit confusing to me and the legends are not informative.\n\n2) Figure 1, again the legend could be improved. There are elements in the figure that are not explained in the legend, which forces the reader to look for them in the text. In the same figure, it does not show how the message passing is happening in DGP and which features are involved, which makes it harder to understand the overall framework.\n\n3) Clearer comparison with SPN, since the model seems to build over SPN, I was expecting a closer analysis on the difference between them plus, the exact difference in terms of technical components.\n\n4) I was expecting some qualitative results of CSEL in the mains paper (there are some in suppl. which is ok). However, I could not find any qualitative comparison of CSEL and SPN, and I was curious about how the performance gains of CSEL translate visually against SPN+edge, for example.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2eyf40xfKB_", "reviewer": "Reviewer_qLf1", "summary": "This paper proposes a dynamic (i.e. recurrent) graph propagation module to improve the results of a feedforward semantic segmentation algorithm on the Cityscapes dataset. In addition to outputting the typical feature map where each pixel is assigned to one of 19 categories (i.e. the semantic segmentation map), the proposed network produces supervised predictions of semantic _edges_ (the boundaries between classes) and a set of local pixel-pixel affinities. These two novel outputs are used as gates in a recurrent message-passing algorithm, which refines the semantic segmentation map (as well as the edge map.) The authors show that the refined map gives improved semantic segmentation scores over the unadorned baseline and a number of other state-of-the-art algorithms. Finally, they find that the dynamic graph propagation module gives substantial improvements on \"corrupted\" versions of the Cityscape dataset (at least for some types of corruption.)", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors' approach is intuitive and clearly explained, as is the logic for why information about edges should be useful for recurrently refining a feedforward feature map. They thoroughly test their model and ablations on Cityscapes, and I buy that adding their module yields an improvement. \n\nThe performance improvements on some types of image corruptions are especially interesting. I wish the authors had included a bunch of qualitative examples and tried to analyze why their module is helpful in dealing with some types of corruption but not others. Indeed, the results in this section are strong enough that it's arguably their most important contribution, since the Cityscapes semantic segmentation task is so close to performance ceiling anyway (and adding a few percentage points to mIoU may not mean detecting new objects that were being missed before, etc.) \n\nOther than this, my main suggestion is that the authors evaluate their model on at least one, and ideally several other datasets and tasks. Cityscapes semantic segmentation is peculiar in that there are not too many categories and the images are pretty homogeneous. If their module significantly improved results on a more diverse dataset (e.g. COCO) I'd be more inclined to think that they had really identified a very useful novel computation for the segmentation task. Likewise, it seems that their approach could be adopted for performing or improving instance segmentation (rather than semantic segmentation), which is a task in much greater need of algorithmic improvement than the latter. If the authors could offer more than this one example (Cityscapes semantic segmentation) of how Dynamic Graph Propagation with joint edge learning improves results, it would much more strongly support their initial motivation for introducing this architecture.\n\n---- Updated ----", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the clear and intuitive presentation of the proposed graph propagation strategy. The author responses addressed well the requests for clarification and additional experiments made by reviewers, including results on COCO, the discussion of computational cost, and the details of performance gains over DGP. All reviewers recommend acceptance. The ACs concur with this recommendation.

Decision: Accept (Poster)