Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors observe the Boundary-caused Class Weights Confusion (BCWC) problem in semantic segmentation. To solve this problem, they propose E-CRF to fuse the CRF into CNN networks for more effective optimization. The E-CRF owns two advantages: use CRF to purify the feature representation of boundary pixels; enables optimizing class weights from both scale and direction.  Moreover, they add the superpixel as an auxiliary to boost the performance. Experiment results on popular datasets like ADE20K and Cityscapes are provided.  
strength_and_weaknesses: Strength:
1. Although it is well known that boundary pixels are difficult to classify, this paper views this problem from a new perspective – class weight. They identify this as BCWC problem and make statistics to prove that there is a positive relationship between class weight similarity and adjacent pixel number.  
2. The authors use both theoretical derivation and figure to prove that the proposed E-CRF can adjust both the scale and direction of optimization, which makes the presentation clearer and convincing. 
3. The ablation study is good, fully studying the effects of various components like E-CRF and superpixel. 

Weaknesses:
1. It is strange to use superpixel in this paper, which has limited relation to the BCWC problem (the motivation). The superpixel is more like an auxiliary module to boost the performance, since E-CRF alone is not enough for SOTA performance.   
2. In the ablation study 4.2.1, there is no obvious advantages of E-CRF over joint-CRF, and the F-score gain is less than 0.3%, and mIoU gain is around 0.5%. Moreover, without superpixel. E-CRF is even worse than joint-CRF with ResNet101 backbone (45.47% vs 45.61%). 

3. In the Table 4, the comparison of different boundary refinement methods might be unfair. The backbone and test augmentation (e.g. multi-scale testing) of various methods are not specified. For example, in the ABL paper, they can achieve 52.40 with Swin-B and multi-scale testing. The Table 6 has the same problem.    
4. The section 3.3 can only prove that E-CRF can adjust both the scale and direction of optimization but cannot prove that E-CRF can relief BCWC. There lacks a connection between optimization flexibility and BCWC problem.   

clarity,_quality,_novelty_and_reproducibility: 1. Most content of this paper is clear, and the graph is good for understanding. However, the motivation of superpixel is not clear, and why E-CRF can relief the BCWC problem is not well explained. 
2. The novelty is good but not outstanding. Both boundary problem and CRF are already studied for a long time in semantic segmentation. 
3. Not sure about reproducibility. 

summary_of_the_review: This paper studied the boundary problem in semantic segmentation from a new perspective and observe the BCWC problem. They propose E-CRF and superpixel to solve the BCWC problem. However, the motivation of superpixel is unclear and the explanation for E-CRF is weak. Even worse, there are some unfair comparisons in the experiment, which makes the effectiveness of proposed method unconvincing. Based on these reasons, I think this paper is below the acceptance threshold. 
Update: From the response, the authors have addressed my key concern, and I would like to raise my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The goal of this paper is to improve the semantic segmentation performance through addressing the boundary-caused class weights confusion issue. The two semantic classes that share more adjacent pixels tend to have more similar class weights in the state-of-the-art semantic segmentation solution such as DeepLabv3+. The proposed approach tries to ensure that the pixel-wise labeling are less depends on the adjacency.

The main contribution of this paper is the introduction of the gradient-based method to the CNN-CRF-based semantic segmentation. The proposed approach works both the levels of directional and scale, while the previous works only focus on using the gradient-based method from scale perspective. The claimed benefits of doing this is to bring better model that helps brining the boundary-caused class weights confusion down.
strength_and_weaknesses: Pros
The main contribution of the proposed E-CRF is to introducing the perspective of gradient descent and optimizes the class weights using both scale and direction using back propagation. This seems to be novel to me as I did not find any other similar semantic segmentation works touch this perspective. Recent works either focus on updating the backbone architectures, or on how to do better than the CRF message passing. This paper brings the new perspective of different gradient-based method.
 
The proposed approach has backed the claims with impressive results on various challenging semantic segmentation datasets including ADE20K, Pascal Context, and Cityscapes. The improvements are relative small, but the gain is built on top of the previous state-of-the-art methods.

Cons
I think the main weakness is that the paper does not really refer the good literature, making it appears that the gradient-based method is introduced the first time and the authors apply that to the semantic segmentation. In particularly, I would consider the proposed approach is very similar to the other works such as PCGrad (Gradient Surgery for Multi-Task Learning, https://arxiv.org/abs/2001.06782, NeurIPS 2020), while the paper does not cite that one.

Another issue is that the proposed approach does not provide sufficient experiments to support the claim. I would like to see the fine-grained experiments to show that the proposed approach get real improvements on the classes that are sharing the most adjacent pixels. This will be like a slice-and-dice results from the aggregated results reported by the current experiments.

Third, I am not fully understand how the proposed approach generalize the formulation to the multi-class classification cases. The current formulation presented in the paper only covers the binary case.
clarity,_quality,_novelty_and_reproducibility: 
Clarity 
I think the paper has not fully made itself clear about how to generalize the proposed approach for addressing the multi-class classification case. The current formulation only touches the binary case. This needs to be clarified.

Quality
The paper is presented in a good quality. However, I think a more slice and dice results need to be presented to back up the claims in the contribution of the paper.


Novelty
I think the proposed approach is novel, though it needs to ensure that the previous literatures are properly referred and discussed.


Reproducibility
The proposed approach seems to be easy to understand and could be reproduced with the third parties.


summary_of_the_review: On the positive side, this paper brings the new gradient-based method perspective into the semantic segmentation, which is interesting. The paper has also shown good results on various popular benchmarks.

However, the paper has not provided sufficient experiments to back up the claims. The improvements of the paper on those benchmarks could be due to the better hyper-parameter search instead of the things claimed in the paper. It is also a bit disappointed to see that the paper does not refer and discuss the difference between the proposed approach with the prior work PCGrad.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the Boundary-caused Weights Confusion problem in semantic segmentation. The authors propose an E-CRF model by integrating CRF into CNN and pursuing gradient descent not only from scale but direction aspects. The method improves baseline for about 1% mIoU.
strength_and_weaknesses: Strength:
+ This paper is well-written and provides insights, e.g., the illustration and interpretation of optimization of different CRF schemes is insightful.
+ The proposed method extends the gradient descent from scale aspect to both scale and direction aspects, making the optimization more effective.


Weakness

- The discussed solution on CRF is a little narrow and out-of-date. Since class-weight problem is general for all semantic segmentation models, it could be better to extend the method to other models.  I also look forward to a discussion part for how to extend this method to Transformer-based segmentation models.
- Although the discussion of BCWC is meaningful, the proposed solution is not novel. The method of discriminating boundary pixels from local representative pixels (e.g., super-pixel center) by measuring their similarities (attention) is commonly used. Similar ideas are in the below papers
  - OCNet: Object Context for Semantic Segmentation, 
  - SegFix: Model-Agnostic Boundary Refinement for Segmentation
- The improvement from Joint-CRF is marginal, about 0.5% mIoU.
- The design of using super-pixel to extract pixel representation is not elegant. It takes additional cost, and the segmentation performance highly depends on the quality of super-pixel. Is there any way to embed this process into the model, to make the partitioned representation learnable?


clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
This paper is well written with clear definition and explanation.

Novelty:
The proposed solution of dealing with BCWC is not novel, but the theoretical analysis is insightful.

Reproducibility:
It is clear to reproduce the method.

summary_of_the_review: This paper focuses on addressing BCWC problem in CRF, which could have been more generic in class weighting problem of semantic segmentation. The analysis is insightful and interesting, but the proposed solution is not novel and elegant, commonly used and taking additional cost. I consider this paper could be better if it extends more, but currently it lacks novelty in solution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focus on the particularity of class weights in semantic segmentation and explicitly consider an important issue , named as Boundary-caused Class Weights Confusion (BCWC).  The authors propose a novel method, E-CRF, via combining CNN network with CRF as an organic whole to alleviate BCWC from two aspects (i.e., scale and direction). In addition, the authors make an exhaustive theoretical analysis to prove the effectiveness of E-CRF. Eventually, the proposed method achieves new results on ADE20K, Cityscapes, and Pascal Context datasets.
strength_and_weaknesses: Strength:
1. The paper is well-written and easy to understand.
2. The idea Embedded CRF (E-CRF) is used to address the BCWC problem is good.
3. The experiments shows the proposed methods are effective and verified.


Weakness
1. The proposed E-CRF is very similar to Joint-CRF. Though the authors claims that joint-crf  lacks the ability to optimize class representations effectively due to the defective design and E-CRF can solve it well, the experimental results can not show E-CRF improve a lot.  In table 1, E-crf has only no more than 1% improvement than joint-crf. This improvement is not so obvious.
2. E-CRF adopts superpixel  as an auxiliary to further strengthen the reliability of the message passing to the boundary pixels. Compared with joint-crf, it need more computation for superpixel generation. That is to say, E-CRF use more computation resourses.
3. There is no model size comparison and time cost comparison.

clarity,_quality,_novelty_and_reproducibility: yes. 
summary_of_the_review: Though this paper idea is good, but we cannot see the experiment results are convincing enough. As i put in the *Strength And Weaknesses, my main concern is that the experimental results between E-CRF and joint-crf.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Four experts reviewed the paper. Three reviewers recommended it "marginally above the acceptance threshold" with high confidence, and one rated it "marginally below the acceptance threshold" with low confidence. Hence, the decision is to recommend the paper for acceptance. However, the reviewers did raise some concerns about the experiments and motivation. The authors are encouraged to make the necessary changes to the paper to the best of their ability. 
justification_for_why_not_higher_score: The reviewers were positive but not excited about the paper. Three reviewers recommended it "marginally above the acceptance threshold" with high confidence, and one rated it "marginally below the acceptance threshold" with low confidence.
justification_for_why_not_lower_score: The reviewers were positive but not excited about the paper. Three reviewers recommended it "marginally above the acceptance threshold" with high confidence, and one rated it "marginally below the acceptance threshold" with low confidence.
Decision: Accept: poster