Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel convolution-based architecture that could utilize convolutional attention in an effective way to encode contextual information for the segmentation task. The main contributions of this paper are (1) a new tailored network architecture (SegNeXt) that envokes spatial attention via multi-scale convolution features, (2) the illustration that the encoder with simple and cheap convolutions can exhibit improved performance than the vision transformer, and (3) the proposed network exhibit the state-of-the-art semantic segmentation performance.
strengths_and_weaknesses: 
1. Strengths
- The reviewer significantly understands the significance and importance of the task proposed in this paper. Segmenting various objects in real-world images is significantly important for many applications. Recent studies have researched the Vision-Transformer to improve the segmentation performance in vision fields, but it exhibits many limitations such as requirements a large number of images and datasets, and even heavy cost. However, this paper address that still the convolution and attention-based operations could exhibit significantly improved performance at a low cost. 

- Reproducibility of the manuscript. The manuscript is well organized in terms of exhibiting the hyper-parameters and model architecture. 

- The manuscript is well organized and well written.

2. Weakness
- More detailed descriptions are illustrated in the “Question” and “Limitation” sections. Please see below.


questions: * Questions & Discussions
1. The novelty and the contribution of this paper should be more discussed. In recent years, the new methods related to vision transformer and ConvNext are popularly studied, and they showed extremely improved accuracy in terms of classification and segmentation. At the first gland, this paper just addresses the application of the ConvNext to the segmentation task. At this point, the reviewer is curious that the combination of the ConvNext to the segmentation task could be a novel contribution to this Neurips society. 

2. Effectiveness of the proposed operation. The authors addressed in the abstract that “convolutional attention is a more efficient and effective way to encode contextual information than the self-attention mechanism in transformers”. However, the manuscript could not effectively exhibit this property which is the main contribution of this paper. The mathematical proof/illustration or experimental analysis for contextual information should be discussed to improve the quality of the manuscript.

limitations: 1. Major issues
- The limited novelty should be discussed. As illustrated above (Question), the proposed operation exhibits limited novelty. The reviewer expects that the limitations could be improved in the discussion periods.

- The experiments should be improved. As the authors commented in the contribution 2, the proposed module can perform better than vision transformers, “especially when processing object details”. As the reviewer already understand, the evaluation metric of “mean Intersection over Union (mIoU)” can qualitatively measure the object details. However, recent studies [1-3] proposed new evaluation metrics to measure the object detail (especially boundaries of the target objects) quantitatively. Otherwise, the visualization of the feature map could illustrate the novel feature extraction when processing object details. Please refer the activation maps [4]. To clear the authors’ addresses, more experimental or mathematical evidence should be justified.


[1] Fernandez-Moral, Eduardo, et al. "A new metric for evaluating semantic segmentation: leveraging global and contour accuracy." 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.

[2] Lee, Kyungsu, et al. "Boundary-oriented binary building segmentation model with two scheme learning for aerial images." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.

[3] Cheng, Bowen, et al. "Boundary IoU: Improving object-centric image segmentation evaluation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[4] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." Proceedings of the IEEE international conference on computer vision. 2017.


2. Minor issues
- Definition of the contextual information

- The detailed description should be improved. For instance, what does the arrow indicate in figure 3? The reviewer recommends authors to review the figures to improve the explanation details for all readers.

- The reviewer recommends reviewing the grammar and typo errors to improve the quality of the manuscript.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents SegNeXt, a new convolution-based model for semantic segmentation. The authors first propose 4 properties that constitute a successful semantic segmentation model: 1) A strong encoder backbone, 2) Multi-scale information interaction, 3) Spatial attention, and 4) low computational complexity. Based on these requirements, the authors propose a convolutional encoder based on a new multi-scale convolutional attention module (MSCA). This module uses multi-branch convolutions of different sizes to generate an attention mask, which is then used to weight the input (via element-wise multiplication) to generate the output of the module.

As a decoder, the authors propose to concatenate the convolutional features from the last input stages and feed them as input to a Hamburger module.

The authors evaluate their proposed architecture on several datasets like ADE20K, Cityscapes, Pascal VOC, Pascal context, COCO-Stuff and iSAID. The results of the experiments show that SegNeXt matches or outperforms other models with similar number of parameters while requiring less computation.
strengths_and_weaknesses: ## Strengths
+ The paper is well written and the new concepts and ideas are explained well and with enough detail. The paper is well structured and the content is presented in a clear and logical way.

+ The paper proposes an extensive evaluation to show the performance of the proposed model and how it compares against previous state-of-the art approaches. Additionally, the authors perform some ablation experiments to backup their choices of multi-scale in the MSCA module, Hamburger as a decoder and the decoder structure.

+ The proposed model outperforms previous state of the art approaches while also requiring less computation.

## Weaknesses
- A weakness of this paper is that it doesn’t justify enough the four properties argued to be necessary for a successful semantic segmentation model. The authors mention that by looking at previous work (DeepLabV3+, HRNet, SETR and SegFormer) they are able to conclude that a good model needs these four properties (strong backbone, multi-scale, attention and low complexity), but that might just indicate recent trends, not necessarily important properties. For example, a stronger justification to claim that the model needs to use spatial attention would be to design an experiment (or a citation to another paper) that shows why this property is needed, instead of assuming that because recent papers used attention, this property becomes necessary.

- The authors explicitly mention that 
> “taking the aforementioned **analysis** into account, … we propose an efficient yet effective encoder-decoder architecture of semantic segmentation”. 
  
  However, as I mentioned before, a more detailed analysis of the different properties of recent models would make the claims more significant.

- Just so the authors are aware, there is already another model called SegNext [1].

To summarize, the paper proposes a novel architecture for semantic segmentation which outperforms previous approaches. The authors perform a thorough evalution to empirically show the superior performance. However, some of the justifications are a bit weak or not supported enough with data.

[1] T. Forbes and C. Poullis, "Deep Autoencoders with Aggregated Residual Transformations for Urban Reconstruction from Remote Sensing Data," 2018 15th Conference on Computer and Robot Vision (CRV), 2018, pp. 23-30, doi: 10.1109/CRV.2018.00014.

questions: 1) Why does the model need to use spatial attention? What is the property of spatial attention that makes it a key component for semantic segmentation models? (Be it self-attention or computed attention masks like in this paper)
2) In Section 1, Table 1, the authors mention that “Strong encoder denotes strong backbones, and adopts the advanced training strategy”. What is the “advanced training strategy”? Does this paper use it?
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents SegNeXt, a simple convolutional network architecture for semantic segmentation. The main contribution of this paper is replacing the standard convolutions and self-attention with the spatial attention proposed in this paper. Experimental results demonstrate that SegNeXt surpasses current state-of-the-art transformer-based methods by a considerable margin.
strengths_and_weaknesses: **Strengths**: 

1) This paper is well written, and I can easily understand the overall flowchart of this paper.

2) The experimental results of this paper are good.

**Weaknesses**: 

Although the idea of this work seems interesting, the good performance is not sufficiently documented.  

Firstly, the motivation of this paper is not clearly explained. In Line.112 to Line.137, the authors directly tell us that the pipeline proposed MSCA, while they do not tell us why to design such architecture.  This makes me confused that would multi-scale convolutional attention be really better than the self-attention mechanism?

Secondly, the authors do not provide any visualization results to show the shows the superiority of MSCA compared with self-attention.

In conclusion, I think the main drawback of this paper is the authors do not tell us that what defects of self-attention are solved by MSCA?
questions: Please see the weaknesses.
limitations: This work has described some limitations and potential negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper proposes a simple convolution network SegNeXt for semantic segmentation which include three important properties : strong encoder, multiscale interaction and spatial attention. It shows convolutional attention is a more efficient and effective way to encode contextual information than the self-attention mechanism in transformers. SegNeXt improves the performance of previous state-of-the-art methods on popular benchmarks, including ADE20K, Cityscapes, COCO-Stuff, Pascal VOC, Pascal Context, and iSAID.

strengths_and_weaknesses: 
Strength:
1, The paper is well written and easy to follow for both method and experiment part. The summary of Table.1 is interesting. 

2. The proposed approach achieve new state-of-the-art results on iSAID, ADE-20k, COCO-stuff and Pascal Context dataset compared with recent transformer based approach. The result in Figure.1 looks good. 

3, The proposed approach: convolutional attention is simple yet effective. MSCA is an improved version of large kernel design for segmentation. 

4, The results on Pascal VOC dataset rank the first. 

Weakness:

1, The technical novelty is very limited. For example, despite the MSCA is simple, it follows the large kernel design which has been proposed in previous works. [1] [2]

Moreover. The decoder adopts multi-scale feature fusion via Hamburger which is a previous work. 

Thus the technical novelty makes the submission looks like a report. 

2, Could not find new insights. The ablation studies are not good which lacks design analysis of component. A better comparison with HRFormer or HRnet is needed to show the effectiveness of spatial attention.


3, In Tab10. Why not list the FPS of other method for better comparison. 


4, Missing ablation on MSCA design on both ImageNet or other segmentation datasets.



[1]  Large kernel matters–improve semantic segmentation by global convolutional network. CVPR-2017. 

[2] Convnext -CVPR-2022

[3] Is attention better than matrix decomposition? ICLR-2021.

questions: See the weakness part. 
limitations: None
review_title: null

Meta-review: metareview: Four knowledgeable referees reviewed this submission. The reviews raised concerns about the novelty of the proposed approach (rY1T, S4w5), the motivation of the model design and properties (mij9, r2Bq), and the empirical evidence to support some of the effectiveness and efficiency claims (rY1T, r2Bq, S4w5). The rebuttal addresses the reviewers' concerns by (1) highlighting the differences of the proposed approach with ConvNext, (2) providing additional comparisons with state-of-the-art methods as suggested by the reviewers, (3) performing ablations of the MSCA design which empirically emphasize its advantages, and (4) partially clarifying the motivation. The authors engage in discussion with the reviewers and provide additional clarifications (e.g. what will be introduced in the main body of the paper, and whether the code will be released). During the discussion phase, the reviewers show some hesitations wrt novelty of the proposed approach which is perceived as incremental wrt ConvNext. However, the reviewers agree that the paper is well written, the approach is simple and appears effective, and the experimental evidence is extensive and supports the claims made in the manuscript. The reviewers appreciate the benchmarking efforts of this work and lean towards acceptance. The AC agrees with the reviewers' assessment that the strength of this paper lies in its extensive experimental validation, and recommends to accept.
confidence: Less certain
Decision: Accept