Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a dynamic (i.e. recurrent) graph propagation module to improve the results of a feedforward semantic segmentation algorithm on the Cityscapes dataset. In addition to outputting the typical feature map where each pixel is assigned to one of 19 categories (i.e. the semantic segmentation map), the proposed network produces supervised predictions of semantic _edges_ (the boundaries between classes) and a set of local pixel-pixel affinities. These two novel outputs are used as gates in a recurrent message-passing algorithm, which refines the semantic segmentation map (as well as the edge map.) The authors show that the refined map gives improved semantic segmentation scores over the unadorned baseline and a number of other state-of-the-art algorithms. Finally, they find that the dynamic graph propagation module gives substantial improvements on "corrupted" versions of the Cityscape dataset (at least for some types of corruption.)
main_review: The authors' approach is intuitive and clearly explained, as is the logic for why information about edges should be useful for recurrently refining a feedforward feature map. They thoroughly test their model and ablations on Cityscapes, and I buy that adding their module yields an improvement. 

The performance improvements on some types of image corruptions are especially interesting. I wish the authors had included a bunch of qualitative examples and tried to analyze why their module is helpful in dealing with some types of corruption but not others. Indeed, the results in this section are strong enough that it's arguably their most important contribution, since the Cityscapes semantic segmentation task is so close to performance ceiling anyway (and adding a few percentage points to mIoU may not mean detecting new objects that were being missed before, etc.) 

Other than this, my main suggestion is that the authors evaluate their model on at least one, and ideally several other datasets and tasks. Cityscapes semantic segmentation is peculiar in that there are not too many categories and the images are pretty homogeneous. If their module significantly improved results on a more diverse dataset (e.g. COCO) I'd be more inclined to think that they had really identified a very useful novel computation for the segmentation task. Likewise, it seems that their approach could be adopted for performing or improving instance segmentation (rather than semantic segmentation), which is a task in much greater need of algorithmic improvement than the latter. If the authors could offer more than this one example (Cityscapes semantic segmentation) of how Dynamic Graph Propagation with joint edge learning improves results, it would much more strongly support their initial motivation for introducing this architecture.

---- Updated ----
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a unified framework for detecting edges and segmenting objects. They conduct experiments on Cityscapes, SBD and Pascal VOC. The edge detection results are competitive to previous SOTA methods. 
main_review: This paper addresses the problem of joint predicting the segmentation results and the edge. By the proposed DGP layer, they achieve good results among several benchmarks. They also show robustness by experimenting on Cityscapes-C. There are some minor concerns, which need more clarification:
1. Do the PASCAL VOC and SBD train together? Are the method in Table 5 and Table 6 also trained under multi-task settings?

2. I am wondering about the efficiency of the proposed DGP. It will be better if the computational cost and inference speed could be compared and discussed.

3. Will edge detection help the semantic segmentation? It will be better if the single edge detection and single segmentation results could be reported in Table 2.

4. There are some typos, for example, Line 78...
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes coupling edge prediction and semantic segmentation using a multi-task network dubbed CSEL. Some of their goals include the mutual improvement of performance on the two tasks when learned together. The proposed model (CSEL) extends spatial propagation network SPN  with a dynamic graph propagation (DGP) scheme. DGP is a gated recurrent graph message passing that is responsible for selecting neighboring pixels with the highest response. In DPG, the semantic edges are considered as gating signals to refine semantic segmentation.
main_review: The proposed method is nontrivial and provides some additional gains over SOAT. The paper also contains an insightful ablation study, which is positive. In many experiments, the performance gains are not very expressive, which makes me feel lukewarm about this approach. A major issue to me seems to be the lack of clarity in the explanations and figures. I feel it is hard for a reader to understand the concepts/architecture and, replicate the work for future research. I would not recommend this paper for publication in this current state, however, I will keep an open mind for post rebuttal remarks. Some more detailed comments are presented below:

1) I would appreciate hearing from the authors on why the edges only serve segmentation only as gating signals?  Could it not be helpful to enrich the segmentation features as well? Actually, I believe this is one point where clarification could be improved. The message passing figures seem a bit confusing to me and the legends are not informative.

2) Figure 1, again the legend could be improved. There are elements in the figure that are not explained in the legend, which forces the reader to look for them in the text. In the same figure, it does not show how the message passing is happening in DGP and which features are involved, which makes it harder to understand the overall framework.

3) Clearer comparison with SPN, since the model seems to build over SPN, I was expecting a closer analysis on the difference between them plus, the exact difference in terms of technical components.

4) I was expecting some qualitative results of CSEL in the mains paper (there are some in suppl. which is ok). However, I could not find any qualitative comparison of CSEL and SPN, and I was curious about how the performance gains of CSEL translate visually against SPN+edge, for example.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The reviewers appreciated the clear and intuitive presentation of the proposed graph propagation strategy. The author responses addressed well the requests for clarification and additional experiments made by reviewers, including results on COCO, the discussion of computational cost, and the details of performance gains over DGP. All reviewers recommend acceptance. The ACs concur with this recommendation.

Decision: Accept (Poster)