Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposed an efficient Transformer-based semantic segmentation model called SegFormer. In essence, it consists of a hierarchical Transformer encoder and a lightweight MLP decoder. Compared to ViT, the transformer encoder employs overlapped patch merging, self-attention with reduced (key, value) for efficient computation, and a depthwise convolution inserted between two MLP layers in FFN to replace position embedding. The authors also conducted an analysis based on effective receptive field and revealed that transformer's encoder with large receptive field enables relative lightweight decoder design. Extensive experiments demonstrate that SegFormer achieves strong speed-accuracy trade-off and better robustness.
main_review: - Overall writing is clear and easy to follow. As for the method, the architectural design is simple yet effective as proven by extensive experiments and analysis. As a result, this could serve as a new baseline for semantic segmentation task. 
- In L165, it was stated that "CPVT [54] uses 3x3 Conv together with PE to implement a data-driven PE". To my best understanding, [54] does not require PE. Instead, similar to this work, a 2D Conv with zero padding is employed to replace PE.  
- Swin Transformers [9] is missing from the comparison with state-of-the-arts. 
- The results in ablation experiment (Tab.1(b)(d)) does not match the final performance in Tab.1(a). For example, in Tab.1(d), the model variant MiT-B3 (S1-4) achieves 48.6 mIoU on ADE20K whereas MiT-B3 in Tab.1(a) attains 49.4/50.0 mIoU on the same ADE20K dataset despite having the same FLOPs and number of parameters. Why is this so?
___________
UPDATE AFTER REBUTTAL

I appreciate the authors' detailed rebuttal. Here are my responses after reading other reviewers' comments and the authors' response:

I have to admit that when I first read this paper, I had the same concern as other reviewers that this work shares high similarity with some recent vision transformer works, especially PvT [8] in terms of its hierarchical representation structure and efficient self-attention design. However, the efficient self-attention design was not claimed as the contribution of this work. In fact, the authors admitted in L155 that such design was taken from [8]. Nevertheless, I agree with reviewer GD12 that some of the presented components (e.g. overlapped patch merging, hierarchical representation) are intermixed with techniques from prior works. 

The main reason for me to recommend an accept are two-fold: 1) the effectiveness of the proposed Mix-FFN for addressing train-test resolution mismatch; 2) the simplicity and efficiency of the overall design. The former is similar to LocalViT [57] which employs a 3x3 depthwise conv between two consecutive MLPs in FFN. Nevertheless, [57] retains the use of position encoding. On the other hand, this work demonstrates their Mix-FFN not only removes the need of positional encoding, but also fixes train-test resolution disparity while a fixed shape positional encoding fails to. I believe this finding is important for dense prediction task such as semantic segmentation where the testing resolution might differ from the training ones.

Secondly, the simplicity and efficiency of the design makes it serve as a strong baseline for semantic segmentation. SETR has previously shown that transformer backbones work well on semantic segmentation. Differently, this work demonstrates a compact and efficient solution for the task while performing comparably or even better. Although each component may not be entirely novel or inspiring, combining them to offer a simple yet effective solution is beneficial to the community, similar to the role of DeepLab series is playing in the segmentation community.

After reading other reviewers' comments and authors' response, I think most of the concerns were well addressed or at least to a certain extent (e.g. different combinations of encoder and decoder, comparison with more real-time segmentation models, variance of different runs, experiments on PASCAL Context dataset etc.). I agree with reviewer n5uX that the technical contribution may be insufficient for both vision and semantic segmentation community. However, as explained earlier, although each component may not be entirely novel, I think offering a feasible compact solution that could serve as a strong baseline for semantic segmentation would be beneficial to the community. After taking all these into accounts, I have decided to lower my rating to 7.
limitations_and_societal_impact: Please refer to "Main Review" section above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perceptron (MLP) decoders. The proposed SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The extensive experiments show that the proposed method achieves the state-of-the-art performance on semantic segmentation task.
main_review: 1. The paper presents a transformer based network for semantic segmentation task. Different from the last transformer based network SETR, the proposed SegFormer uses  positional-encoding-free,  hierarchical Transformer encoder and lightweight All-MLP decoder, which is more efficient and flexible than SETR.
2. My concerns and suggestions about this paper are as follows:
         1). The author claims that the segformer is novel, but the used overlapped patch embedding, efficient attention, the multi-stage structure are basically from PVT, PVTv2.  There is no any novel module or techniques proposed in this paper. So i think the novelty is limited.
         2). The author mainly uses the SETR as baseline method, but actually recent Swin and PVT (cited in the paper) also show the SOTA performance, which are not compared in the experimental report.
         3). The lightweight sseg models are compared in the experimental part, but only mobilenet based networks are listed,   ESPNet, ESPNetv2, BiSeNet, ICNet and other lightweight models are not reported and compared. i recommend the author can give more detailed comparison.
        4). In addition, UNet-like network is also popular network in semantic segmentation. Does it work for the transformer based network? it's better to give experiments about this.




limitations_and_societal_impact: the suggestions are presented in Main Review part
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the SegFormer model, a Transformer-based model for semantic segmentation (i.e. dense pixel classification) in images. The model uses prior architectural innovations such as a pyramidal structure (progressive downsampling) and a factorized version of self-attention for computational efficiency. The main novelties over prior work on Transformer-based semantic segmentation models and image classification models are: 1) the use of convolutional filters to propagate position information from the image boundaries (as opposed to using explicit positional encodings) and 2) skip-connections between individual layers directly to the output layer, using spatial upsampling. Experimental evaluation demonstrates improved mIoU scores on default semantic image classification benchmarks compared to a baseline SETR (Segmentation Transformer) model, while achieving computational benefits thanks to the adoption of a more efficient Transformer architecture.
main_review: The paper is overall well-written, well-structured and easy to follow, although clarity could be improved in several aspects (see below). The architectural choices are well-motivated and experimental results demonstrate clear advantages over prior approaches. The code / model implementation looks clean, and providing this implementation is a big plus and will allow for easy reproduction. Transformer architectures for applications in visual processing are very popular right now, and so the paper should be relevant for a NeurIPS audience, although the presented scope of the method (semantic pixel classification) is rather narrow. 

Overall, despite the positive points mentioned above, I am arguing for rejecting this paper in its current form. I lay out the reasons for this decision in the following.

The presented novelties (1. use of convolutional filters to propagate position information from the image boundaries; 2. skip-connections between individual layers directly to the output layer) are intermixed with techniques from prior work, such as using a pyramidal Transformer architecture (as in [1]) and the use of an efficient, factorized form of self-attention (also from [1]). In the experimental section, it is unclear whether the performance benefits stem from utilizing these components from prior work [1] or whether the benefits stem from the novel contributions 1. and 2. -- the computational efficiency gains appear to be largely due to the use of the architecture from [1], which does not merit a separate conference paper (swapping out an image classification with a pixel classification target is of limited novelty).

The effective receptive field analysis and the analysis of robustness to natural corruptions are interesting, but they do not compare to the more recent SETR baseline, but instead to a weaker ConvNet-based baseline published in 2018. They thus unfortunately add little to convince the reader about the benefits of the proposed novelties over the current state of the art. The showcased qualitative improvements on Cityscapes in Figure 4 compared to SETR appear to be quite marginal (mask boundaries are marginally sharper, but otherwise unchanged). If this is a representative example of the obtained improvement, then it is unclear how this method will improve prediction quality in downstream applications.

The statement that the proposed "encoder can easily adapt to arbitrary test resolutions without impacting the performance" is not adequately verified. In fact, I would expect this to be false: if I increase the resolution 100x of the input image, I would very much expect the model to struggle (not just computationally, but also in terms of propagating position information throughout the model). The experiment in Table 1c suggests that increasing the resolution and aspect ratio by a small factor leads to only little loss in performance, but this does not justify the above statement. 

Since the novel form of (implicit) positional encoding is a core contribution, it should be investigated more. For example, one could try to measure the model's ability to predict positional encodings by either: 1) giving the model PE as inputs (i.e. trivial auto-encoding) vs. 2) using the proposed position-free approach. This would allow for a good comparison of how 1) and 2) compare in terms of generalization to new image sizes.

Regarding the second novelty, i.e. skip-connections to the decoder: SETR introduces a related strategy termed "Multi-Level feature Aggregation (MLA)". A direct comparison against this technique (+ upsampling where necessary) would strengthen this contribution.

Clarity of the paper could be improved. There are many typos and grammatical mistakes, which should be easy to fix by doing another pass over the paper. The "Overlapped patch merging" strategy is described in a confusing way and parameters such as the padding size P are not explained. Looking at the code, this operation is simply implemented by a convolutional layer, and hence it would greatly simplify the description if the authors replaced this paragraph by simply saying that they use a convolutional layer after Transformer blocks, if I understand the method correctly.

Lastly, the impact of this method could be greatly increased by considering (and comparing on) other pixel-level prediction tasks such as depth prediction, which the architecture should support out-of-the box.


Other comments / questions, suggestions for improvement:
* The references [2,3] as cited in the sentence "In the deep learning era [...]" (Section 2, first sentence) look quite out of place compared to the other (seminal) works cited in this context. Were these cited by accident? I recommend explaining their relevance in the text (or dropping them).
* SETR uses SGD with momentum as optimizer whereas the present method uses AdamW. How much of the performance difference can be explained by the different choice of optimizers? Doing an ablation study over this choice would help clarify where the performance benefits come from (i.e. train with SGD and momentum in a setting similar to SETR).
* What is the unit for Params in Table 2?
* Why did the authors not compare on PASCAL Context (as in the SETR paper)?
* No error bars are reported and the authors state that the variance between seeds was low ("quite table mIoU results"). Would the authors be able to quantify this variance on at least one experiments, to get an idea whether results are significant?


[1] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021
[2] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In CVPR, 2019. 
[3] Wenhai Wang, Xiang Li, Tong Lu, and Jian Yang. Mixed link networks. In IJCAI, 2018.


----- UPDATE AFTER REBUTTAL ----

I would like to thank the authors for their very extensive and detailed response to my review. My concerns about positioning of the paper still remain, i.e. it is difficult to clearly pinpoint what is the main contribution of this work, as the method clearly benefits from a careful combination of several architectural contributions in conjunction with careful optimization. In this point I largely agree with the concerns raised by reviewer n5uX. 

At the same time, the authors have done an impeccable job at addressing my other concerns and at carefully investigating various aspects and components of the proposed model. Given the detailed experimental evaluation and the clear usefulness of this model as a strong baseline for the semantic segmentation community (as pointed out by reviewer LMud), I believe that this paper lies marginally above the acceptance threshold despite concerns around positioning and significance of the individual technical contributions. 
limitations_and_societal_impact: Limitations are not clearly discussed, apart from the following limitation mentioned by the authors:
"One limitation is that although our smallest 3.7M parameters model is smaller than the known CNN’s model, it is unclear whether it can work well in a chip of edge device with only 100k memory. We leave it for future work.". -- It is not entirely clear how number of parameters relates to memory consumption, as the latter is often dominated by intermediate activations and gradients. It would be good to discuss limitations specific to this particular model class. What are interesting next steps for future work?

In terms of (negative) societal impact the authors write "We do not foresee obvious undesirable ethical/social impacts at this moment." -- I would encourage the authors to consider highlighting potential negative societal impact stemming from future applications by techniques such as theirs, e.g. in the area of surveillance.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces SegFormer for semantic segmentation. The SegFormer consists:
(1) A Transformer encoder that extracts multi-scale features. The authors claim their Transformer encoder to be novel in the sense that it is "positional-encoding-free" and "hierarchical".
(2) A "MLP" based decoder that aggregates information from different layers. Although the authors claim it to be "MLP", the decoder is essentially a stack of several 1x1 convolutions with bilinear interpolation to upsample low-resolution features.

The authors further explore scaling the Transformer encoder which ends up with a series of models with different number of parameters and FLOPs. The authors show their models are competitive with SOTA on multiple semantic segmentation datasets and claim that their model is more robust on corrupted images than DeepLabV3+ (on Cityscapes-C).
main_review: ## Limitations of the paper

First of all, I don't agree with the position of this paper. The authors claim that both the Transformer encoder and the new "MLP" decoder are equally important to make SegFormer work. However, I don't see any reason why other decoders cannot be used with the SegFormer Transformer encoder, and why other Transformer encoder cannot be used with Segformer decoder. For me, this paper is more like a study of Transformer design that is not making a proper comparison with other Transformers: the authors did not show their number on ImageNet classification, which is fine if the model only works well for semantic segmentation. However, the comparison with other Transformers on semantic segmentation is completely unfair which I will explain later.

Although this paper shows good performance on multiple semantic segmentation datasets, it has two major problems: (A) the authors overclaimed their contribution (or did not clearly summarize their contribution) and (B) the experiments in this paper cannot support claims made by the authors. Next, I will explain my reasons.

### A. The authors overclaimed their contribution (or did not clearly summarize their contribution)

1. The authors claim they propose a "novel positional-encoding-free and hierarchical Transformer encoder" (L61).
    * In L165, it looks like the authors follow CPVT [54] by using 3x3 convolution to replace positional encoding. Furthermore, in L166-167, the authors argue "positional encoding is actually not necessary for semantic segmentation" without any explanation. I'm not sure how the authors come up with this conclusion, since it is something not obvious. In the Table 1 (c), the authors only compare "PE" with "Mix-FFN", but it is not enough to say positional encoding is not necessary. Given these facts, I don't think "positional-encoding-free" or "Mix-FFN" is something contributed by the authors, as there are many works already uncover this fact like CPVT [54] and CvT [58].

    * As for the hierarchical Transformer, I find it is very hard to understand what is the key difference with other hierarchical Transformers like PvT [8] from current description (the current text only describes "Hierarchical Feature Representation", "Overlapped Patch Merging", "Efficient Self-Attention" which seems to be already used by multiple hierarchical Transformer papers).

2. The authors claim their "All-MLP" decoder is lightweight and less complex than other decoder. First of all, "complex" is very subjective. Why one decoder is more complex than another decoder without a proper metric? Second, the decoder used in the paper is **not** "All-MLP": the decoder is essentially several 1x1 convolutions. And here is the question again: why 1x1 convolutions are less "complex" than 3x3 convolutions?

### B. The experiments in this paper cannot support claims made by the authors

1. The experiments do not show both proposed components (new Transformer encoder and "MLP" decoder) are necessary.
    * In Table 1, the authors only compares their Transformers with CNN backbones which is not a very fair comparison. In order to really show the benefit of this new Transformer, the authors should compare their Transformer with other Transformer designs (e.g., PvT, CvT, Swin Transformer etc.) to really show this is a better Transformer.
    * As for the decoder design, the authors only compare their decoder with different channel dimensions. However, the claim made by the authors is the "MLP" decoder is better than other "more complex" decoder. In order to backup this claim, the authors should compare the "MLP" decoder with the "more complex" decoder they refer to.

2. Design of the experiments is not consistent with the position of the paper. The authors position their paper as a Transformer design for semantic segmentation. Thus, it is okay to not comparing with other Transformer models on ImageNet. However, the authors show at least have a proper comparison with existing Transformers to show why their design is better than other Transformers. For example, the authors can compare their Transformer with Swin Transformer using **the same decoder and the same training recipe**. However, from the description of the paper, I do not see why this special design is more suitable than other Transformer specifically for semantic segmentation.

## Summary

In summary: (1) I believe the claims made by the authors in this paper is not strong enough, (2) experiments cannot support the claims made by the authors as well; and (3) the position of this paper is weird. I'm especially concerned about (2) and (3). I do not see why the design of this Transformer is specifically suitable for semantic segmentation, if the model does not perform better in ImageNet classification. Since the paper never offers an apple-to-apple comparison with other Transformer models for semantic segmentation task, I'm afraid the authors might have tuned training parameters very hard to make it work on semantic segmentation (while other Transformers mainly focus on reporting numbers for classification). If this is the case, I don't feel this paper is very valuable and I think it is a clear rejection. But I hope to see authors explanation if I make any mistake.



--------

\#### post-rebuttal update ####

I have read carefully all reviews and author response. The authors have addressed most concerns well and the major disagreement between reviewers (e.g., me and Reviewer GD12) and the authors lies in the position of this paper. I have to say that I'm still not fully convinced and I believe it is better to position the paper as a new ViT design instead of a specific model design for semantic segmentation. However, the authors argue they mainly show improvements on semantic segmentation which is the reason that they position it as a model for semantic segmentation specifically, which is a fair point and I respect their decision.

Besides the problem of "novelty" raised by all other reviewers, another major problem in the original submission is that there is no fair comparison which leads to my original decision of rejection. The authors have addressed this problem by running the suggested experiments and thus I will raise my score to borderline. However, I believe the paper requires a major revision to take into account all these discussions and I'm not sure if the authors will make the revision as they promised since I cannot see the revised version. So I will change my score to 5 and let the AC to decide whether this should be taken into account.

limitations_and_societal_impact: yes
review_title: null

Meta-review: The authors propose an efficient semantic segmentation model composed of a hierarchical Transformer encoder and a lightweight MLP decoder. The transformer encoder employs overlapped patch merging, efficient self-attention module, and a depthwise convolution inserted between two MLP layers in FFN to replace positional embeddings. The empirical evaluation demonstrates better accuracy/speed tradeoffs compared to competing models.
The paper was reviewed by 4 experts who appreciated the clarity of exposition, simplicity of the design, the effectiveness of the proposed approach in dealing with test-train resolution mismatch, as well as competitive performance. During the discussion the reviewers pointed out several concerns related to positioning and the significance of technical contributions which were mostly addressed. Finally, the reviewers agreed that the approach is a simple and competitive baseline for the semantic segmentation community. I will recommend acceptance.
Decision: Accept (Poster)