Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces SegViT, a semantic segmentation framework with plain ViTs as backbones. One of the core technical contributions is the proposed Attention-to-Mask (ATM) block, which generating masks from the intermediate attention maps between class embeddings and key maps. In addition, a shrunk structure is then proposed to save computational cost while maintaining the performance. Based on plain ViT networks only, SegViT obtains state-of-the-art results on three semantic segmentation datasets (ADE20K, PASCAL-Context and COCO-Stuff-10K). 
strengths_and_weaknesses: Pros

1. The paper is well motivated. Recently many works (e.g. [*1]) have realized even plain ViTs could have rich representation capacity, which however requires special optimization (e.g. masked image modeling) or other architectural modifications for downstream tasks. I am pleased that the paper demonstrates that plain ViTs can obtain as good results as the hierarchical counterparts (e.g. [15, 47]) on segmentation tasks, which may encourage simpler and unified network design principles. 

2. Strong results are reported in the paper. For example, on ADE20K val a model with ViT-L backbone achieves 55.2 mIoU, which is very competitive even among more sophisticated networks, such as Swin-L and MViT. 

3. The motivation of ATM module sounds reasonable to some extent: intuitively a good attention mask should cover the foreground of the given object (or class). Therefore, it is possible to generate mask directly from the attention matrix. 

Cons

1. My major concern is that the technical novelty is relatively limited. The overall framework is very similar to MaskFormer [15] and Mask2Former [47]. Compared with [15], the major difference on the technical details is, [15] generate masks from the product of the mask embedding and the per-pixel embedding, while in the paper the mask is directly derived from the attention weights. However, I do not think it differs much. Although [15, 47] mainly evaluate on hierarchical backbones, theoretically they can also be equipped with plain networks. In addition, the proposed QU/QD layers are not novel (also sounds irrelevant to the main topic of the paper), since many previous works, e.g. PVT [17], also adopt similar blocks to reduce computational cost. In conclusion, I think the contributions claimed in the introduction seems not significant.

2. According to Table 4 and Line 234-239, in the proposed ATM block, the separated supervision of classification and mask prediction is the most important design principle. However, it is not originally proposed in the paper as [15, 16] already introduces the paradigm. It further weakens the significance of the proposed method. 

[*1] Li et al. Exploring Plain Vision Transformer Backbones for Object Detection. Tech report. 

questions: SegViT still utilizes FPN-like architecture to fuse low-level and high-level features from the backbone. However, some previous works [*1, *2, *3] challenge the essential of such feature fusion. It will be interesting if the authors show experiments where ATM module only relates to the last layer of ViT encoder. For example, like the experiments in Table 5, the authors may try a cascade design while utilizing the last layer multiple times, or just follow ViTDet fashion [*3].

[*1] Zhang et al. ExFuse: Enhancing Feature Fusion for Semantic Segmentation. ECCV 2018. 

[*2] Chen et al. You Only Look One-level Feature. CVPR 2021.

[*3] Li et al. Exploring Plain Vision Transformer Backbones for Object Detection. Tech report. 

limitations: Limitations are mentioned in the conclusion. Although I think more discussion and comparisons with [15] are required in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors deal with ViT-based semantic segmentation. In particular, they use a set of learnable tokens (each corresponding to a semantic class) which decode the outputs of the ViT-based backbone into per-class semantic masks. This is accomplished by multiple layers of cross attention between class token and ViT tokens. Rather than use a dot product like mechanism to produce similarity between a class token and spatial features, they directly supervise the cross attention maps using a sigmoidal output. Furthermore, they introduce a down/upsampling technique to mimic the general idea of an efficient multi-scale prediction head. Their results are quite good even when compared to some of the best recent models and their QD module provides some computation/performance tradeoffs.
strengths_and_weaknesses: Strengths:

1. This is a well written paper and the approach is quite clean
2. The results presented are quite good as well, achieving at/near SOTA against competitive models

Weaknesses

1. The idea is still related to the idea of dot product based segmentation (from some class embedding). I think a good deal of experiments might need to be performed to actually understand the technical contribution.
2. While the results are good, related work like Segmenter is not far off from the performance presented here and shares some significant similarities with this method.
questions: 1. I'm slightly confused about why one needs class predictions on the output tokens? Is this because the attention mechanism is bad at producing low confidence values when the class does not exist? Do you just multiply this probability by the sigmoid output?
2. Table 4 could be more convincing. The loss formulation is not novel, so why not also apply it to SETR?
3. It seems reasonable to provide a "dot product"-based baseline (similar to the decoder in Segmenter) to tease where attention (which itself is a dot product to some degree in this scenario) actually makes a difference.
4. Does this work when just supervising the aggregated output (a single per class segmentation loss)?
limitations: I believe so.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposed a plain-ViT based semantic segmentation framework, which uses an Attention-to-Mask decoder to aggregate image features and a Shrunk structure to save computational cost.
strengths_and_weaknesses: Strengths:
1. The paper proposed SegViT framework and achieved a SOTA performance based on a plain ViT backbone. 
2. The paper is well written and clear to understand.

Weaknesses:
1. I doubt the novelty of the design of ATM module. Since the MaskFormer framework has been proposed for over half a year, the ATM module is similar to the MaskFormer transformer decoder module. The only difference is that the mask output of MaskFormer is generated by the multiplication of the final output query tokens and image features, while the mask output of ATM is from the multiplication of an intermediate variable K inside the transformer layer and the image features. The difference is not obvious. The SegViT framework is just like MaskFormer + ViT + multi-layer-feature.
2. Table 4 shows that ATM has a relatively low performance gain of about 0.5% to SETR. It shows that the performance of ATM is even worse than Segmenter (Since the result of Segmenter is 0.8% better than SETR in Talbe 1)?
3. Also, in table 4, it shows that by using L_mask loss the mIoU result increases about 2.6% than using CE loss only. However, Table 1 shows that the result of SegViT is about 2.3% better than Segmenter baseline (which only uses CE loss). If it shows that the performance gain is all from the new loss design but not from the framework architecture. 
questions: 1. Considering the similarity of MaskFormer and SegViT, I am curious about the result of MaskFormer + ViT + multi-layer-feature (use the same three ViT layer inputs to MaskFormer transformer decoder). I think the result may be similar with SegViT. 
2. In the paper, it claims that the SegViT framework has a quite good performance with Swin Transformer backbone and lists the results with Swin-Tiny. I am curious about SegViT + Swin-Large result and its comparison with MaskFormer and Mask2Former.

limitations: The authors have addressed the limitations and potential negative societal impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper present a semantic segmentation method based on the plain vision transformer (ViT). Specifically, it proposes the attention-to-mask (ATM) module to generate the pixel-level mask. In addition, to reduce the computational cost, it designs a query-based downsampling (QD) and upsampling module (QU) in the shrunk version. Experiments are conducted on three datasets and better results are obtained compared with previous methods.
strengths_and_weaknesses: **Strength**

1. Exploring plain architecture for semantic segmentation is an interesting and promising direction. This paper make a forward step towards this direction.
2. The performance of SegVIT seems to be better than previous state-of-the-art methods.

**Weakness**

1. About the Attention-to-Mask module (ATM), it is implemented in cross-attention manner. But in fact, there is little difference with the standard classifier (a fc layer to map features to probability) for per-pixel classification in the normal semantic segmentation framework. Each learned token could be viewed as a classifier layer to map the pixel-level features into a probability with a Sigmoid function. In this sense, the ATM is similar to a standard classification layer.

2. About the Shrunk structure, I am confused about the query-based downsampling operation (QD). In line 172, it says to use the nearest sampling to reduce the token numbers. In this sense, it has nothing with the query based downsampling and is simply a standard downsampling operation. I am also confused about the implementation details on query-based upsampling operation (QU). It says to use a standard transformer decoder structure to upsample features. Is there any special design on the transformer decoder by incorporating the spatial information? More details are required on the decoder design.

3. About the two QU operations in the version (c) of Figure 3,  the downsampling ratio of lower QU is 1/16, and it is natural to think its output have smaller downsampling ratio like 1/8. However, from line 181-183, its output size seems to be 1/16, which is confused for me.

4. I think this paper should compare with the previous works PerceiverIO, which employs a similar downsampling-upsampling architecture for dense prediction with transformers. More discussion on the difference is required to better motivate the proposed method.

questions: Please address the above concerns in the weakness.
limitations: The authors have addressed the limitations of the proposed method in large memory consumption.
review_title: null

Meta-review: metareview: This submission has received comments from 4 official reviewers. The authors have made very detailed replies to the reviewer's comments. The authors and reviewers had quite rich discussions. After these discussions, 3 reviewers recommended weak acceptance, and 1 recommended rejection. 

For the novelty concerns, the authors clarify them during the rebuttal. The reviewers have also recommended comparing with recent semantic segmentation methods using ViTs. Missing comparisons should be included in the final version, including comparisons with 

[1] Ma, Xuezhe, et al. "Luna: Linear unified nested attention." NeurIPS 2021.
[2] Ryoo, Michael, et al. "Tokenlearner: Adaptive space-time tokenization for videos." NeurIPS 2021.
[3] Wu, Yu-Huan, et al. "P2T: Pyramid Pooling Transformer for Scene Understanding", IEEE TPAMI, 2022.


Only reviewer Eyo8 recommends borderline rejection. The authors have made quite a detailed rebuttal but we have not heard from the reviewer after the rebuttal.

Thus, the AC would like to recommend acceptance. 

confidence: Certain
Decision: Accept