Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel convolution-based operation/operator that could utilize the semantic features in an efficient and effective way for the segmentation task. The main contributions of this paper are (1) a new operation (semantic difference convolution, SDC) that is effective in the segmentation of target objects with precise boundaries, (2) the related module (SDM) to enhance the boundary information at the feature level, and (3) the proposed network exhibit the state-of-the-art semantic segmentation performance with precise boundaries of the target objects.
strengths_and_weaknesses: 
1. Strengths
- The reviewer significantly understands the significance and importance of the task proposed in this paper. Segmenting various objects in real-world images is significantly important for many applications. Especially, clear and precise boundaries are required for the precise semantic segmentation task. 

- Reproducibility of the manuscript. The manuscript is well organized in terms of exhibiting the hyper-parameters and model architecture. 

- The manuscript is well organized and well written.

2. Weakness
- More detailed descriptions are illustrated in the “Question” and “Limitation” sections. Please see below.


questions: * Questions & Discussions
1. The semantic distance (SD in equation 3) should be discussed. The SD is calculated using the Euclidean distance of two semantic features (f) with different positions. Intuitively, SD value by similar feature exhibits small value, whereas SD value exhibits magnified values when calculating boundaries. At this point, the reviewer is curious about the followings:

- 1-1. The authors’ address is that f exhibits semantic feature, and the Euclidian distance between two f exhibits semantic distance. Here, the definition and the mathematical illustration for the semantic feature and its meaning should be discussed, or a reference is required. 

- 1-2. The normalization method (even without) can extremely suppress the values of the inner parts of the objects. In general, the color and semantic features significantly exhibit similar values in the same objects. Therefore, the SD value inside the object can be almost 0. Whereas the SD values nearby boundaries could be extremely magnified. Therefore, the reviewers worried that extremely small inner values degrade feature extraction for semantic features related to the target objects and only focus on the boundary-oriented features, and thus the predicted segmentation mask includes big halls inside.

2. Since the convolution operation can be utilized in many applications, the reviewer is curious that the motivation of the proposed operation in the semantic segmentation task. The proposed operation can be used in general applications such as classification and image generation tasks.

limitations: 1. Major issues
- The limited novelty should be discussed. As illustrated above (Question), the proposed operation exhibits limited novelty. Despite the simple yet effective mathematical illustration of the proposed convolutional operation, the strict mathematical modeling for the operation and mathematical analysis of why the proposed operation exhibits outstanding performance in segmenting objects’ boundaries should be discussed. 

- The experiments should be improved. As the reviewer already understands, the evaluation metric of “mean Intersection over Union (mIoU)” can qualitatively measure the object details. However, recent studies [1-3] proposed new evaluation metrics to measure the object detail (especially boundaries of the target objects) quantitatively. Otherwise, the visualization of the feature map could illustrate the novel feature extraction when processing object details. Please refer to the activation maps [4]. To clear the authors’ addresses, in terms of the improved boundary-oriented segmentation, more experimental or mathematical evidence should be justified.

- Since the authors proposed the convolution-based operation, the mathematical modeling of the operation should be discussed with the parameters used for the previous convolution operation. The mathematical modeling can include the rank, dimension, size, and inputs (parameters). For instance, the size of the output feature map can be determined as --- when the padding parameter is “valid” or “same”, or when using dilation parameters. The author should illustrate the mathematical form to calculate the output shape using kernel size, padding, and strides (additionally dilation). Please refer to this page [5] for the mathematical form.


[1] Fernandez-Moral, Eduardo, et al. "A new metric for evaluating semantic segmentation: leveraging global and contour accuracy." 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.

[2] Lee, Kyungsu, et al. "Boundary-oriented binary building segmentation model with two scheme learning for aerial images." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.

[3] Cheng, Bowen, et al. "Boundary IoU: Improving object-centric image segmentation evaluation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[4] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEE international conference on computer vision. 2017.

[5] https://www.tensorflow.org/api_docs/python/tf/nn/conv2d

2. Minor issues
- The reviewer recommends reviewing the grammar and typo errors to improve the quality of the manuscript.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an efficient boundary-aware convolution operator to boost the boundary modeling capacity for semantic segmentation. The proposed operator is sensitive to the inter-class boundary while ignoring the noisy intra-class pseudo-boundaries. Based on the proposed operator, a lightweight module is designed to enhance boundary-related information, which is flexible to be plugged into any existing encoder-decoder segmentation model. Experimental results validate that  the proposed approach can achieve consistent improvements for boundary regions over several baselines.


strengths_and_weaknesses: Strength:
-	This paper proposes an efficient boundary-aware convolution operator to boost the boundary modeling capacity for semantic segmentation.
-	The proposed approach shows promising performance for inter-class boundaries.
-	Comprehensive ablative studies are conducted to show the effectiveness of the proposed approach.

Weakness:
-	The proposed approach may introduce new parameters to the network.
-	The proposed Semantic Difference Convolution (SDC) is similar to Central Difference Convolution (CDC).
-	The effectiveness of the semantic difference term needs more clarification.

questions: -	The further analyses in Table 5a are appreciated, it will be more convincing if some details are given. How to calculate $\rou$ in this table? How many percents of more parameters are introduced by the proposed approach for each convolution layer? Does CDC also introduce more parameters to the network?
-	About the pure improvement of the semantic difference term: If we replace SDC with CDC in the proposed approach, how well does it perform?
-	Does the benefit of the semantic difference term for intra-class boundaries rely on the scale of convolution kernels and the quality of the semantic feature?

limitations: The authors adequately addressed their work's potential negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of how to improve the semantic segmentation results for existing architectures. The authors propose a semantic difference guided convolution, so-called SDC, to deliberately enhance the representations in the semantic boundary area. Specifically, it enhances current layer’s kernel with the the next layer’s feature differences, instead of using current layer’s differences like previous methods, which means that the next layer’s features could serve as a semantic guidance to the current layer. The proposed module is claimed to be flexible to co-operate with any ‘encoder-decoder’ architectures. And the effectiveness is verified through extensive experiments.
strengths_and_weaknesses: ** Strength ** 

+: The paper is well written and easy to follow.

+: The insight of enhancing the semantic boundary through next stage’s differences is interesting and has somewhat novelty for the semantic segmentation. Can this method generalize to other dense estimation task? 

+: The authors fully demonstrate the effectiveness of the proposed method on various datasets.

** Weakness **

-: The wording of “intra-class boundary“ is not precise enough. According to my understanding, the semantic difference is provided by the next layer’s feature, not the output layer. And only the output layer corresponds to the class-level information exactly. Especially in the early layer, two approaching pixels with different features do not imply they belong to different classes.

-: There seems to be no normalization to the SD(*). Would this induces unstable training if the SD(*) generates a quite large number? 

-: In table 5(b), it would be more convincing if the author organizes the comparisons as “Baseline, Baseline+Ours, Baseline+DenseCRF, Baseline+DenseCRF+Ours, Baseline+SegFix, Baseline+SegFix+Ours, Baseline+InverseForm, Baseline+InverseForm+Ours”.

questions: Please refer to the Strengths and weakness.
limitations: The authors have pointed out the limitations of their work.

Since it is just a basic method to deal with 2D semantic segmentation, it has nothing to do with the potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Overall, this is a good work.

In this paper, a new semantic segmentation pipeline is presented, based on proposed semantic difference convolution. The semantic difference convolution incorporates higher-level semantics into the lower-level convolution and helps learn better features which are sensitive to semantic boundaries. The proposed method is easy to implement on existing feed-forward segmentation baselines. According to a set of experiments on previous segmentation models including DeepLabv3, OCRNet and MaskFormer on 4 public datasets, the effectiveness of the proposed method is clearly demonstrated. Besides, this paper also provides analysis on the computational complexity and various comparisons regarding the configurations, as well as many ablations to show the SOTA results from the proposed method.

This paper is also well-organized, that I can follow their draft easily. Even though there are a few comments from my side, that I believe it could be improved, this is a good work for NeurIPS.

After reading all the comments and authors' feedback, I vote for accept (7) on this submission.
strengths_and_weaknesses: Strengths:
+ This paper is well organized. It is overall easy to follow.
+ This paper proposes a new convolution operation, which is simple to implement. Based on the proposed semantic difference convolutions (SDC) and SDM, more favorable segmentation results can be achieved. To show the effectiveness of the proposed method, the authors show different network backbone as well as segmentation models.
+ The proposed SDC achieves more improvements compared with previous boundary-aware methods. Further, the SDC is compatible with other techniques, such as DenseCRF, SegFix.
+ From the analysis and empirical evaluation, the proposed SDC will not increase the computational cost significantly. 

Weakness:
- In section 3.2, it needs to discuss more details of f_{pi}. What is f_{pi}? Even though readers can understand when they see section 4.2 and Figure 4, it is a little confused in section 3.2. More explanation on the semantic information f_{pi} will be helpful.
- Lack of comparison with more boundary-aware methods, such as Gated-SCNN.
- In this work, DeepLabv3, OCRNet and MaskFormer are applied as baseline models. Why not apply the proposed method on the current state-of-the-art model to achieve new SOTA result, for example SegFormer?
questions: 1. This paper applies DeepLabv3, why not DeepLabv3+?
2. In Table 5 (b), why is the baseline performance only 65.1? What is the baseline here? More details are needed. Besides, why not apply a stronger baseline in Table 5?
limitations: No. This work is a fundamental research in computer vision.
review_title: null

Meta-review: metareview: This submission got a mixed rating: 1 borderline reject, 2 week accept and 1 accept. 

Most of the concerns lie in the explanations on the details and experimental comparison with certain baselines/variants. The authors addressed them well by providing additional experiment results in their response. 

The remained concern from the reviewers giving borderline reject lies in theoretical justification of the proposed operation. The authors managed to provide a theoretical interpretation from the viewpoint of diffusion process, which partially addresses the reviewer's question. 

Overall, all the reviewers agree that this submission introduces a simple and effective method for the segmentation field. The effectiveness of the proposed method has been validated via extensive experiments. The performance improvement is significant. The manuscript is written clearly. The contribution is sufficient. 

Based on the above considerations, AC recommends accept for this submission.  
confidence: Certain
Decision: Accept