Coupled Segmentation and Edge Learning via Dynamic Graph Propagation
Year: 2021

Abstract:
Image segmentation and edge detection are both central problems in perceptual grouping. It is therefore interesting to study how these two tasks can be coupled to benefit each other. Indeed, segmentation can be easily transformed into contour edges to guide edge learning. However, the converse is nontrivial since general edges may not always form closed contours. In this paper, we propose a principled end-to-end framework for coupled edge and segmentation learning, where edges are leveraged as pairwise similarity cues to guide segmentation. At the core of our framework is a recurrent module termed as dynamic graph propagation (DGP) layer that performs message passing on dynamically constructed graphs. The layer uses learned gating to dynamically select neighbors for message passing using max-pooling. The output from message passing is further gated with an edge signal to refine segmentation. Experiments demonstrate that the proposed framework is able to let both tasks mutually improve each other. On Cityscapes validation, our best model achieves 83.7% mIoU in semantic segmentation and 78.7% maximum F-score in semantic edge detection. Our method also leads to improved zero-shot robustness on Cityscapes with natural corruptions (Cityscapes-C).

heading: 1 Introduction
text: Image segmentation and edge detection have been widely studied as important perception problems. The two problems are closely related. In fact, segmentation subsumes edge detection since any segmentation contour makes a closed boundary of a region. The converse is however not true since general edges do not always form closed contours. Nevertheless, edge detection can serve as an auxiliary task to improve segmentation performance since edges provide important pairwise similarity cues for segmentation. Early works tend to focus on the grouping and contrast of pixels from a perceptual similarity perspective. Martin et al. [1] proposed the Berkeley Segmentation Dataset, a popular benchmark for segmentation and boundary detection that inspired many impactful works in perceptual grouping [2–5]. The recent surge of deep learning renders powerful representations with learned features using convolutional neural networks (CNNs) [6]. This has led to great advances in both areas [7–12], but the two tasks are often considered separately.
In light of the status quo, we consider coupled edge and segmentation learning. Our goal is twofold: (1) Multi-task learning - being able to produce high quality edge detection and segmentation. (2) Mutual improvement - the two tasks can help each other with non-trivial performance gains. Designing a principled framework is however nontrivial. The key question is how sparse edge signals can be effectively transformed into dense region-level ones to interact with segmentation. To this end, we propose a learnable recurrent message passing layer where semantic edges are considered ∗Equal contribution. Correspondence to Zhiding Yu <zhidingy@nvidia.com>. †Work partially done during an internship at NVIDIA.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
as explicitly learned gating signals to refine segmentation. An overview of our framework is shown in Fig. 1. Specifically, the dynamic message passing layer uses affinity gates to select the neighbor for message passing using max-pooling. It conducts message passing sweeps in each of the four directions: left to right, right to left, top to bottom and bottom to top. The message passing is jointly gated by both the affinity and edge gates, therefore allowing edge cues to naturally influence long-range dense predictions. As such, our framework presents a context module that is clean, compact yet powerful. Our technical contributions can be summarized as follows:
• We formulate recurrent message passing as dynamic graph propagation (DGP). We show that such a formulation simplifies the required normalization in propagation networks [13]. It also dynamically finds graph structures that encodes pixel affinities and improves dense prediction.
• We propose a double-gate design where message passing is jointly gated by both the affinity and edge gates to refine the segmentation feature map. We show that this design together with the dynamic 1-way connection in DGP better couples segmentation and edge learning.
• We obtain state-of-the-art results on joint semantic segmentation and edge detection. We also show that DGP leads to strong zero-shot robustness to natural corruptions with significant improvement over prior methods on Corrupted Cityscapes (Cityscapes-C).
Multitasking segmentation and edge learning is desirable for several reasons: 1) There are many downstream applications where both are needed, such as occlusion reasoning [14], localization [15], proposal generation [16–19] and conditional generation [20]. 2) There are many challenging cases where segmentation quality is poor but edge quality is far more superior, e.g. segmentation tends to be inferior near object boundaries since they are often optimized for IoU rather than precision [21]. In these cases, edge learning can potentially capture details missed by segmentation. 3) Implicitly improved model generalization as a result of the coupled learning [22].

heading: 2 Related Work
text: Semantic Segmentation. There is a rich set of prior work in semantic segmentation. Long et al., [7] proposed fully convolutional end-to-end training of semantic segmentation and pointed out its connection to recognition [6]. Chen et al. [8] introduced atrous convolution and atrous spatial pyramid pooling (ASPP) to capture multi-scale image contexts. Another contemporary work is the wide ResNet-38 which explores a relatively shallower but wider backbone [23]. It has also been shown that context plays an important role in segmentation, including context encoding [24, 25], multi-scale context [26–28] as well as relational context [29, 30]. More recently, there has been a surge of interests in segmentation with Transformers [31–34]
Boundary/Edge Detection. Similar to segmentation, boundary/edge detection have been widely studied as perceptual grouping problems in early literature [1–3]. Recent methods tend to resort to CNNs. For example, Bertasius et al. proposed a multi-scale deep network architecture for top-down contour detection, where as Xie et al. [11] further introduced holistically-nested edge detection
(HED) for end-to-end edge learning. Besides detecting binary edges, Hariharan et al. [35] proposed the Semantic Boundaries Dataset (SBD) which has become a popular benchmark for semantic edge detection. Compared to binary edge detection, semantic edge detection involves the semantic classification of edge pixels in addition to localization which presents additional challenges to existing frameworks. A series of works including HFL [36], CASENet [12], SEAL [37] and STEAL [38] have followed up [35] and pushed the boundaries.
Multi-task Segmentation and Edge Learning. Multi-task segmentation and edge learning remains under-studied but is not entirely new. Edges have been pairwise similarity cues to improve segmentation [3, 39, 18] and superpixels [4]. It was shown that edges can be transformed into dense regions through the Laplacian Eigenmaps of boundary transformed affinity [36, 5, 40]. Despite being robust, these methods are generally slow and not end-to-end trainable. For more recent CNN based models, end-to-end multi-task learning on a shared backbone is a natural choice [41]. In addition, proper regularization between segmentation and edge has been shown to improve the performance of both tasks. For example, Takikawa et al. [21] use softmax with temperature to impose consistency between segmentation and semantic edges. Zhen et al. [42] correlate semantic segmentation and edge detection tasks with a consistency loss. Our work goes beyond multi-tasking and let the two task more deeply coupled through dynamic graph propagation.
Structured Dense Prediction. Segmentation is a dense prediction task where structured information can be useful. Structured prediction models such as Markov random fields (MRFs) [43], conditional random fields (CRFs) [44–46] and energy minimization [47, 48] have widely proved helpful in segmentation problems by imposing contrast-sensitive smoothness. In addition, structured inference can also be unfolded as network layers for end-to-end training [49, 50], therefore combining the advantages from both ends. Recently, there is an increasing trend to directly model the message passing process itself using multi-dimensional RNN [51–53], graph RNN [54, 55] and spatial propagation network (SPN) [13, 56, 57]. A common advantage is that these methods render more context-aware prediction with larger receptive fields while preserving local details similar to CRFs. Their ability to train end-to-end allows more powerful representation of inter-pixel similarities and thus better dense prediction quality. Our work can be broadly categorized into this category. It is worth mentioning that the contrast sensitive smoothness term in CRF and propagation networks [52, 13] is an implicit modeling of edge signals by learning to relax smoothness constraints at high contrast areas. There have also been variants of SPN [58] that take binary edge as gate regularization. However, none of these works have explicitly addressed multi-tasking learning with category-aware edges whereas the proposed DGP framework presents a novel and effective solution to this problem.

heading: 3 Multi-task network
text: As a first step towards coupled segmentation and edge learning, we introduce two novel multi-task networks that are able to perform multi-task learning for both segmentation and edge detection. An overview of their architectures is shown in Fig. 2. The rest of the paper follows these notations and covers more details for each module.
Backbone. A CNN giving a semantic feature map (denoted as F ) encoding the segmentation information. There could be multiple choices of the backbone networks, ranging from the popular architectures of DeepLabv2 [8], DeepLabv3 [27] to latest state-of-the-arts such as DeepLabv3+ [28] and Dual-Attention Network [30]. Adopting powerful backbones will surely benefit the system-level performance, but the major purpose of this work does not completely lie in achieving state-of-the-arts. We are more interested in showing the effectiveness of a proposed framework on standard backbones. To this end, we consider both CASENet [12] and ResNet-38 [23] as the standard backbones for benchmarking. Both of them comprehensively covers the feature map resolutions of 1, 1/2, 1/4 and 1/8, with the last resolution being a rule of thumb adopted by many segmentation networks. For both networks, we adopt atrous spatial pyramid pooling layers with dilation rate [6, 12, 18, 24, 30, 36] to capture context information. We set the output channels to be 128 to produce a semantic feature map, fowllowed by either direct segmentation prediction (baselines) or the proposed DGP layer.
Affinity stream. Convolution layers that aggregate the cross-layer information of the backbone and produces an affinity feature map (denoted as A) to encode the affinity of pairwise neighboring pixels. This stream is only used with the presence of recurrent message passing layer (including its related baselines). The stream aims to model the simialrity of pair-wise neighboring pixels and serve as a
major input to the gating signal in recurrent message passing layer. Compared to edge stream, the affinity stream seeks to capture coarser-level inter-pixel affinity with slightly deeper convolutions and higher dimensions. The resulting feature map A is a 256 dimensional tensor concatenated by ASPP and the side convolutional feature maps from the affinity stream.
Edge stream. Dense skip-layer features with abundant details that are combined with edge classification layer through shared concatenation [12] to produce a semantic edge map (denoted as E). We design an improved edge stream over CASENet [12] to better leverage detail information at multiple scales/levels with dense side connections. Unlike CASENet where the bottom blocks only provide 1-channel side features, we densely obtain side features from every sub-block residual modules. In CASENet, this gives side features with a total number of 31 dimensions. Similar rules apply to ResNet-38, where the side feature has a total number of 17 dimensions We also found that applying sigmoid after side features with resolution 1/8 greatly benefits edge training from two aspects: (1) It helps to stabilize and prevent gradient explosions from dense skip connections. (2) It removes the typical aliasing edge issue caused by upsampling/deconvolution, and produces elegant smooth edges. Even though we do not explicitly apply techniques such as edge alignment [37] in this work, the proposed backbone is able to produce high quality edge predictions under noisy labels. We also notice that it is better to remove sigmoid for side features with higher resolutions. Edge prediction is multi-tasked alongside with ASPP using 1× 1 convolution on top of Res Block5. This returns the K classes coarse edge predictions. Similar to CASENet, we upsample all side features together with the K-dimensional edge predictions to full image resolution and apply shared concatenation, where side features are repetitively shared and concatenated with each class for K times, followed by a K-way 3× 3 group convolution to produce semantic edges.
Dynamic graph propagation. Learnable recurrent message passing layer that takes the above three branches as input to produce a refined semantic feature map (denoted as F ∗). More details regarding this module will be introduced in the next section.

heading: 4 Coupled segmentation and edge learning
text: Although the multi-task network is able to jointly produce segmentation and edge prediction, we are interested in letting these two tasks better coupled to mutually improve each other. To this end, we look into recent spatial propagation networks where edges can be transformed into gating signals that produce long range influence to segmentation. We first give the notations and definitions:

heading: 4.1 Notations and Settings
text: We start with a two-dimensional recurrent network architecture with a linear propagation module passing messages (memories) spatially over a feature map. We define a 4-way message passing: 1. Left→ Right (Mode L), Right→Left (Mode R), Top→Bottom (Mode T) and Bottom→Top (Mode B). Each way of message passing will separately generate a hidden state map H which can be approximately viewed as a refined (smoothed) version of F . Finally, we take an element-wise max
operation to ensemble them where the model automatically selects the optimal direction with highest neural activation for each pixel:
F ∗ ←H , max (HL, HR, HT , HB) (1)
To perform message passing in each way, one often needs to pre-define a graph that encodes the message passing paths. For tractability and and computation issues, such graph is often sparsely defined with locally connections between immediate neighboring pixels. We follow spatial propagation network (SPN) [13] by defining a three-way local connection, as illustrated on the hand side of Fig. 3. The advantage of such design is obvious: Taking Mode R propagation as an example, one just needs to initiate the message passing from the right most column, and recurrently pass the message from right to left column by column. In this case, the hidden state of each
pixel is directly influenced by the three immediate neighboring pixels on the right column. Details of the propagation under Mode R is illustrated on the left hand side of Fig. 3.
Let F ∈ RH×W×C be the feature map input to the propagation module, H ∈ RH×W×C the propagation latent space on top of F . In addition, let hi,t and fi,t denote the hidden state and feature of the i th pixel located at recurrence t3 on H and F , respectively. We denote {pki,t|k ∈ N(i, t)} the set of learnable propagation gating weights for hidden state h(i, t) where N(i, t) is the set of neighbors of pixel (i, t). In this case, the spatial propagation in each mode is defined as:
hi,t = ( 1− ∑ k∈N(i,t) pki,t ) fi,t + ∑ k∈N(i,t) pki,t hk,t−1 (2)
where is element-wise product, and hk,t−1 is the hidden state of pixel (i, t)’s neighbors from the previous recurrence. {pki,t|k ∈ N(i, t)} is expandable to an affinity matrix, revealing the global and dense message passing among all the pixels of F .

heading: 4.2 Dynamic graph propagation (DGP)
text: As a linear module, the above operation requires careful normalization of among {pki,t|k ∈ N(i, t)}. In particular, ∑ k∈N(i,t) p k i,t ≤ 1 should be satisfied since the energy of the hidden state signal gets unbounded easily under the recurrence operation. To normalize the gating weights, one may consider a linear self-normalization scheme [13] where the constraint ∑ k∈N(i,t) |pki,t| ≤ 1 is imposed to
guarantee the stableness of the propagation4 via dividing each pki,t with ∑ k∈N(i,t) |pki,t|.
But the above formulation also leads to certain limitations. For example, the linear form allows pki,t to be both positive and negative, which potentially encourages pki,t to be large towards either positive or negative side rather than being monotonic. In addition, when pki,t ≥ 0, ∑ k∈N(i,t) p k i,t = 1 which means that such formulation considers zero unary input from fi,t. Therefore, another choice is to constrain pki,t with a probabilistic output:
pki,t = exp(p̂ki,t)∑
k∈N(i,t) exp(p̂ k i,t) σ(p̂ki,t), (3)
where p̂ki,t is defined as:
p̂ki,t = W > a [Ai,t; Ak,t−1], (4)
3There is a one-to-one correspondence between (i, t) and pixel index (h,w). But the mapping varies subject to the propagation mode. In Mode R for example, t corresponds to column with w = W − t whereas h = i+1.
4Similar property holds when the same constraint is applied to each dimension of pki,t in Equation (2).
and σ(·) is the Sigmoid function. Note that p̂ki,t is the raw gating activation without normalization, and is obtained via a linear projection on the concatenated affinity stream features.
The above formation satisfies both ∑
k∈N(i,t) p k i,t ≤ 1 and pki,t ≥ 0, while partially considering the
unary input with the Sigmoid term. Yet the framework is a bit complicated with many non-linear terms. To this end, we take one more step to further sparsifying and simplifying the gating response by considering a softmax-with-temperature formulation and taking the limit of T → 0:
pki,t = lim T→0 exp(p̂ki,t)/T∑ k∈N(i,t) exp(p̂ k i,t)/T σ(p̂ki,t) (5)
Substituting Equation (5) into Equation (2), we have: hi,t = ( 1− σ(p∗i,t) ) fi,t + σ(p∗i,t) h∗i,t, (6)
where the n-th dimension of p∗i,t and h ∗ i,t are defined as:
k∗ , argmax k∈N(i,t) {p̂ki,t[n]} p∗i,t[n] = p̂k ∗ i,t [n] h ∗ i,t[n] = hk∗,t−1[n]. (7)
Equation (6) essentially leads to a dynamic graph propagation (DGP) framework where one performs message passing on a dynamic graph structure by picking neighbor with the highest response. Intuitively, DGP captures a compact structure that has close relation to directed minimum spanning tree, Chu–Liu/Edmonds’ algorithm [59] and the recently pro-
posed tree filters [60, 61]. Such structure presents an inductive bias that benefits segmentation by filtering out noisy prediction and following only strong signals. Figure 4 illustrates DGP where we show three example paths on three feature channels. Each channel independently takes different paths depending on its own neighbor affinities.

heading: 4.3 Coupling edge prediction with segmentation
text: To deeply couple edge prediction with segmentation, we further incorporate edge signal into the above dynamic graph propagation by proposing a double-gate framework:
hi,t = ( 1− σ(p∗i,t) σ(gi,t) ) fi,t + σ(p∗i,t) σ(gi,t) h∗i,t, (8)
where gi,t is defined as:
gi,t , We ∗E[i− 1 : i+ 1, t− 1 : t+ 1] (9) Note that the edge gating signal is obtained by via a 3 × 3 convolution on the K-channel edge activation map E, which outputs a vector g sharing the same dimension (128) as p∗. This way, edges are able to actively influence segmentation via edge-sensitive gating on message passing.
We also hope that the refined segmentation activation after DGP can alternatively serve as a shape regularizer of the edge prediction. To this end, we output a K-channel edge regularization feature from Fm using 1× 1 convolution, followed by another 3× 3 convolution to fuse with E to produce a refined edge map E∗. This is illustrated in Fig. 1 on the bottom right. One shall see that the above coupled design let the two tasks greatly improve each other, and we term the final framework CSEL.

heading: 4.4 Training loss
text: The training loss for CSEL is a multi-task combination of cross-entropy (BCE) losses for multi-label edge learning and the cross-entropy (CE) loss for segmentation:
L = LSeg + LEdge = LCE(Conv(F ∗)) + λ(LBCE(E∗) + LBCE(E)) (10) where λ is the parameter controlling the weight of segmentation loss. For segmentation, we use a 3× 3 convolution to linearly project the 128-channel activation F ∗ into a K-channel before the loss.

heading: 5 Experiments

heading: 5.1 Datasets and metric
text: Cityscapes. Cityscapes [62] contains 2975 training images, 500 validation images and 1525 private testing images with 19 pre-defined semantic classes. The dataset has been widely adopted as the standard benchmark for both semantic segmentation and semantic edge detection. Following a number of previous works [12, 37, 38, 21], we comprehensively conduct ablation and quantitative studies for both segmentation and edge detection on the validation set.
SBD. The Semantic Boundaries Dataset [35] contains both category-level and instance level semantic segmentation annotations. The dataset contains 11355 images (8498 for training and 2857 for testing) from the trainval set of PASCAL VOC2011 [63], and follows the 20-class VOC definition.
PASCAL VOC 2012 [64] is a semantic segmentation dataset with 1464 (training) and 1449 (val) and 1456 (test) images. We use the augmented dataset with 10582 training images, as in [28]. The dataset contains 20 foreground object classes and 1 background class.
COCO Panoptic [10] contains the mask annotations for both things and stuff, with a split setting (118K train 2017 images and 5K validation 2017 images) following the detection community.
Evaluation metrics. For evaluation, We consider intersection-over-union (IoU) for segmentation, and the maximum F-score (MF) at optimal dataset scale (ODS) for edge detection5. We also consider the boundary F-score proposed in [21] for direct comparison on edge detection task.

heading: 5.2 Implementation details
text: Data loading. During training, we unify the training crop size as 1024 × 1024 for Cityscapes, 472 × 472 for SBD and VOC12, and 464 × 464 for COCO Panoptic. All models are trained with 150k iterations on Cityscapes with batch size 8, 30k iterations on SBD and VOC12 with batch size 16, and 220k iterations on COCO Panoptic with batch size 16. We also perform data-augmentation with random mirror, scaling (scale factors in [0.5, 2.0]) and color jittering.
Optimization. we apply an SGD optimizer with a weight decay of 5 × 10−4 during training. For baselines and methods that involving CSEL, we additionally apply a second ADAM optimizer to the propagation layers. In our case, we empirically found that ADAM optimizer is more capable of optimizing the propagation layers with better performance.
Learning rate and loss. The base learning rates for methods with ResNet-101/ResNet-38 backbones are unified as 3.0 × 10−8/7.0 × 10−8 across Cityscapes, SBD and VOC12. On COCO Panoptic, the base learning rate is unified as 5.0 × 10−8 for all comparing methods. Unless indicated, the segmentation weight λ in Eq (7) is empirically set it to 0.5 to balance LSeg and LEdge. We found our method not sensitive to λ, and setting λ = 0.5 works excellently for all backbones and datasets.
Using Mapillary Vistas. On Cityscapes, a number of literature consider large-scale pretraining on Mapillary Vistas (MV) [65], which is shown to considerably benefit the segmentation performance. Unless indicated, our method does not adopt Mapillary Vistas pre-training for fair comparison.
5.3 Experiments on Cityscapes
Table 1: Ablation study of Semantic segmentation on Cityscapes validation set.
method ResNet-101 ResNet-38
ST 77.9 78.55 MT 78.4 79.43 SPN 80.0 - SPN+Edge 80.4 -
DGP 81.3 - CSEL− 80.9 - CSEL 82.8 82.8 CSEL-Ms 83.7 83.4
Ablation study (semantic segmentation). Table 1 shows the ablation studies on semantic segmentation where we consider apple-to-apple compared methods that are trained following the same backbones and training protocols: 1) ST: Single task segmentation network without propagation layer but keeping the 128-channel feature map F . 2) MT: The same as ST but naively multi-tasking segmentation with edge detection. 3) SPN [13]: Implementation of SPN with the 3-way propagation on F , where gating signal is computed using the affinity stream A. 4) SPN+Edge: Coupling SPN with edge learning following exactly the same double gate design. 5) DGP: Baseline which
5we follow [37] by applying exactly the same parameters and settings.
Table 2: Main results of semantic segmentation and semantic edge learning on Cityscapes.
(a) Main semantic segmentation results on the Cityscapes validation set.
Method Backbone MV mIoU
PSPNet [26] ResNet-101 78.8 DeepLabV3+ [28] ResNet-101 78.8 CCNet [66] ResNet-101 80.5 GSCNN [21] ResNet-101 74.7 DANet [30] ResNet-101 81.5 RPCNet [42] ResNet-101 82.1 CSEL ResNet-101 83.7
SGPN [57] ResNet-38 80.9 GSCNN† [21] ResNet-38 80.8 VRec-JP (LR) [67] ResNet-38 81.4 Axial-DeepLab [68] AxiaiRes-XL 81.1 CSEL ResNet-38 83.4
(b) Main semantic segmentation results on the Cityscapes test set.
Method Backbone MV mIoU
PSPNet [26] ResNet-101 78.4 PSANet [69] ResNet-101 80.1 ASPP [27] ResNet-101 80.1 BFP [58] ResNet-101 81.4 DA-Net [30] ResNet-101 81.5 OCR [29] ResNet-101 81.8 CCNet [66] ResNet-101 81.9 RPCNet [42] ResNet-101 81.8 CSEL ResNet-101 82.1
GSCNN ResNet-38 X 82.9 CSEL ResNet-38 X 83.5
(c) Main results of semantic edge detection on the Cityscapes validation set. Results measured by Maximum F-Score (MF) at optimal data scale.
Method Backbone MF (IS) MF (Non-IS)
CASENet [12] ResNet-101 68.1 68.9 SEAL [37] ResNet-101 69.1 - STEAL [38] ResNet-101 69.7 71.4 RPCNet [42] ResNet-101 - 78.2
CSEL ResNet-101 78.1 78.3 CSEL ResNet-38 - 78.7
contains the proposed DGP layer without the double gate design. 6) CSEL−: Baseline which is the same as CSEL except for removing the edge loss. 7) CSEL: Our full method with single scale inference. 8) CSEL-MS: CSEL with multiscale inference at {0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0}. We also consider both instance-sensitive (IS) and non-instance-sensitive (Non-IS) settings where the edge training/evaluation labels are with/without instance-edges.
Discussions. We make sure that the comparisons are apples to apples and fair, by using the same backbones and training recipes (such as learning rate, crop size, batch size, and number of iterations) for all comparing methods.The DGP baseline can be considered as an apples to apples counterpart of SPN, whereas the CSEL− baseline is a further study to understand whether the improvement in CSEL purely comes from the enriched representation with edge features. One could observe several trends from the results: 1) The proposed multi-task network consistently outperforms its single task counterpart. 2) CSEL outperforms both SPN and SPN+Edge with the dynamic graph propagation and edge gating. 3) The dynamic graph design in DGP alone helps it to achieve 81.3% mIoU, outperforming both SPN and SPN+Edge by 1.3% and 0.9%, respectively. 4) The incorporation of edge guidance with the double gate design further leads to another non-trivial 1.5% improvement. Simply adding the edge feature does not improve the segmentation quality. In fact, CSEL− is even slightly lower than DGP where no edge features are involved. This reflects the importance of edge signal as a structural guidance than making the representation more expressive.
Comparison to state-of-the-art (semantic segmentation). We also compare CSEL to state-of-the art semantic segmentation models and list the results in Table 2a and 2b. For clarity, we divide to table into two parts with models using the same type of backbone putting together. Among the comparing methods, CSEL obtains the best results using both ResNet-101 and ResNet-38 backbones.
Ablation study (edge detection). We additionally conduct experiments on edge detection. We first show an ablation study in Table 3, where ST indicates single-task edge detection network with the proposed edge stream. MT indicates naive multi-task training of both segmentation and edge. MT leads to slight performance degradation on the edge detection task compared to single-task
training on edge detection. However, the degradation is marginal compared to the significant improvement of the edge quality introduced by deep coupling of the two tasks in CSEL. Again, this shows the considerable benefit of coupled learning with both segmentation and edge detection.
Comparison to state-of-the-art (edge detection). We also compare to state-of-the-art semantic edge detection methods in Table 2c where we present the training/evaluation protocols in two categories (IS and NonIS) following SEAL. Note that we evaluate the results with the edge thinning protocol. One could see that our method achieves the best performance in both settings
with significant performance gain from the coupled edge and segmentation learning, as well as the benefit from the dynamic graph propagation module. Besides the evaluation protocol from SEAL, we also follow a separate evaluation protocol from GSCNN [21] in Table 4, where we use the original
evaluation code base from GSCNN to evaluate the boundary quality of semantic segmentation. One can observe that at all different thresholds, CSEL with both ResNet-101 and ResNet-38 backbones outperform DeepLabV3+ and GSCNN by significant margins in terms of boundary quality.
5.4 Experiments on SBD
We also evaluate CSEL on the SBD dataset and compare with previous state-of-the-art semantic edge detection models. The main results are presented in Table 5. Note that we used the re-annotated SBD test set from [37] to pursue more precise semantic edge evaluation. From the table, one could see CSEL outperforms other state-ofthe-art methods in both IS and Non-IS settings.
5.5 Experiments on PASCAL VOC12
We evaluate CSEL on the PASCAL VOC12 dataset and report the main results of semantic segmentation in Table 6. The performance is evaluated as the mean IoU over the 20 PASCAL VOC classes plus the background class. We compare CSEL with previous competitive methods that are state-of-theart (DeepLabv3+) or considerably related (SPN/DFN). CSEL achieves considerable improvement these methods.
5.6 Experiments on COCO Panoptic
We further conduct experiments on the full COCO Panoptic dataset. Several earlier methods have reported results on COCOStuff 10K which is an earlier version of COCO with much less data. However, this does not lead to significant differences with VOC12 in both the size and the diversity of data. We therefore adopt the full COCO Panoptic dataset which is significantly larger and is widely accepted by the detection and instance/panoptic segmentation communities. We hope that our
work present solid baselines that inspires subsequent research on this benchmark.
Specifically, we convert the instance segmentation annotations to semantic segmentation ones, and compare segmentation methods with or without the proposed CSEL module. All comparing methods adopt the same CASENet (ResNet-101) architecture. Other hyperparameters are kept exactly the same. Models are trained on the train2017 split and tested on val2017 with single scale inference.
From the results in Table 7, one could see that adding CSEL leads to consistently improved results over the naïve segmentation baselines. In addition, multi-task learning with both segmentation and edge losses also slightly outperforms the single segmentation. Finally, CSEL slightly outperforms SPN with reduced gain compared to results on Cityscapes. We hypothesize that the reduced gain is partly caused by the noisier mask annotations on COCO Panoptic which makes edge learning more challenging and decreases the edge gate quality.

heading: 5.7 Robustness Against Natural Corruptions
text: We hypothesize that CSEL also brings robustness to the segmentation model. We show that this is the case and empirically verify on Cityscapes-C in Table 8. Specifially, we follow the standard corruption package provided by [70] to corrupt the Cityscapes validation images. This expands the Cityscapes validation set with 16 types of algorithmically generated corruptions from 4 major categories: “Noise”, “Blur”, “Weather” and “Digital”. Each corruption type also contains 5 severity levels, leading to 2500 evaluation images for each type alone6.
Our result shows that CSEL overall improves the robustness significantly, especially compared to ResNet-38 and GSCNN [21] which share the same backbone. On the other hand, the method does
6We only consider level 1-3 for the “Noise” category following [71].
still show some vulnerability to certain type of corruptions, particularly those belonging to the “Noise” category. However, this trend is aligned with ResNet-38 and GSCNN and we hypothesize that it may be partly related to the specific design of ResNet-38 based on this pattern.

heading: 6 Conclusion
text: We proposed a unified framework for coupled segmention and edge learning. Our work revisits the two long-standing and important perceptual grouping problems - semantic segmentation and edge detection. Our method (CSEL) includes a novel end-to-end multi-task network, a recurrent dynamic graph propagation layer, as well as deep coupling of the two tasks on top on top of them. Finally, results show that careful coupling of the tasks leads to significant improvement on both of them.
Broader Impact
Our method coupled two long-standing important vision problems, semantic segmentation and edge detection, under a unified framework. Besides the various practical benefits from deeply coupling these two tasks, we expect the research to inspire considerable insights, interests and revisits on perceptual grouping, mid-level representations and structured prediction. The result of the research is likely to find diverse scene understanding applications such as autonomous driving and robot navigation. Like many other discriminative recognition models, our method inevitably faces challenges from input data quality and underlying data biases. The model behavior is subject to various factors such as data distributions, domain gaps, label quality and fairness. We encourage researchers to focus on the “in the wild” robustness when the above challenges are present.

heading: Acknowledgement
text: We thank the NVIDIA GPU Cloud (NGC) team for the computing support of this work. We also thank the anonymous reviewers and the other NVIDIA colleagues who helped to improve this work with discussions and constructive suggestions.

heading: 7 Checklist
text: 1. For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] Please see result analysis on
Cityscapes-C. (c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see
Broader Impact. (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes] 2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)? [No] Going through legal approval process at this moment. Will release source code upon approval.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to implementation details in both the main paper and the supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] We observed stable mIoU results for semantic segmentation models.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to the supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets? [Yes] The data used in our work is open
source and can be used for adademic research. (c) Did you include any new assets either in the supplemental material or as a URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [N/A] The data used in our work is open source and can be used for adademic research.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The data does not contain personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [N/A]

Review summaries:
This paper proposes a dynamic (i.e. recurrent) graph propagation module to improve the results of a feedforward semantic segmentation algorithm on the Cityscapes dataset. In addition to outputting the typical feature map where each pixel is assigned to one of 19 categories (i.e. the semantic segmentation map), the proposed network produces supervised predictions of semantic _edges_ (the boundaries between classes) and a set of local pixel-pixel affinities. These two novel outputs are used as gates in a recurrent message-passing algorithm, which refines the semantic segmentation map (as well as the edge map.) The authors show that the refined map gives improved semantic segmentation scores over the unadorned baseline and a number of other state-of-the-art algorithms. Finally, they find that the dynamic graph propagation module gives substantial improvements on "corrupted" versions of the Cityscape dataset (at least for some types of corruption.)

The paper proposes coupling edge prediction and semantic segmentation using a multi-task network dubbed CSEL. Some of their goals include the mutual improvement of performance on the two tasks when learned together. The proposed model (CSEL) extends spatial propagation network SPN  with a dynamic graph propagation (DGP) scheme. DGP is a gated recurrent graph message passing that is responsible for selecting neighboring pixels with the highest response. In DPG, the semantic edges are considered as gating signals to refine semantic segmentation.

This paper proposed a unified framework for detecting edges and segmenting objects. They conduct experiments on Cityscapes, SBD and Pascal VOC. The edge detection results are competitive to previous SOTA methods.