Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper accelerates vision transformers for dense prediction without finetuning. This is done by (i) using the token clustering layer to cluster the neighboring tokens; (2) using the token reconstruction layer to re-create the high-resolution representations.   The result of proposed method achieves state-of-the-art performance on five dense prediction tasks 
strengths_and_weaknesses: This paper proposes two layers (token clustering layer and token reconstruction layer) and apply them into Swin Transformer. The two ideas are simple and easy to follow. The setting of experiments is clear and the number of experiments is adequate.


This paper fulfills with texts and numbers. I think that authors should provide visual comparison for  detection, segmentation and depth estimation

questions: According to my understanding, author speed up Transformer by reducing token number and restore feature representation from these clustered tokens.  However, I do not understand why this approach could avoid finetune.
limitations: Could authors provide detailed discussion for reason why their acceleration method can avoid finetune.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: When using the Transformer for dense image prediction, detection, and segmentation, the large number of tokens can impose a heavy computational burden. Therefore, the core of this paper is to propose clustering tokens to obtain a low-resolution representation to reduce the computational burden, and to perform attention on the low-resolution token cluster, followed by a token reconstruction layer to reconstruct the high-resolution token representation. The authors' experiments verify that the proposed approach significantly improves the FPS and reduces the GFLOPS index for segmentation, detection, and depth estimation tasks when using Transformer with only a small loss of AP.
strengths_and_weaknesses: Strengths:
This paper introduces the superpixel scheme into the Transformer structure, which really reduces the computational burden on the Transformer. The author's solution ensures that the entire computational process is differentiable and is a practical solution.

Weaknesses
The Token Clustering Layer in the paper is similar to the algorithm cited in [30], where the center of the superpixel is obtained and then iterated until the result of the latest clustering is obtained. Although better results are obtained, the method is only introduced in and the innovation point is not sufficient. And I think the author's description of the two modules is not detailed enough and should be further illustrated with pictures. The experimental part should be more experimental on multiple datasets for each task. 
questions: The use of the superpixel method, while reducing the size of the token, must lead to a loss of semantic information. And after the reconstruction of the layers, there must be noise in the regained token. My question is how the loss of semantic information in each layer compares to that of the original ViT after using the two modules proposed by the authors
limitations: I personally believe that the work in this article has contributed to reducing the computational burden of the Transformer. However, the two modules proposed in the article should be further verified and reasoned in detail to show the reader more details.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces two non-parametric operators to efficiently down-sampling some intermedia tokens to accelerate the large-scale vision transformers for dense prediction. One is a token clustering layer  to decrease the number of tokens and the other one is a token reconstruction layer to recover the number of tokens. These two operators need no training or finetuning, while the related experimental results prove the effectiveness of them in typical benchmarks. 
strengths_and_weaknesses: Strengths:
The motivation of the paper is clear and solid. While the current existing works for efficient transfomer need training or finetuning, this work introduces two non-parametric operators and brings some meaningful insight for this topic.
The experimetal results are sufficient to support the proposed work. 
The paper is well written and easy to follow.

Weaknesses:
From Table 2, I see that the proposed operators actually have limited performance when increasing the downsampling stride. I think this limitation is normal for non-paramettic modules, but I also think the author should also compare with the other parametric methods such as [47,52,55] here to give a comprehensive discussion. 

Overall, I think this paper is technically sound and achieve good performance. Thus I vote for borderline accept.
questions: None.
limitations: See the above discussion about weaknesses.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors observed that the presence of local spatial representation redundancy and high definition representation in the vision transformer contribute to dense prediction. Therefore, this paper proposes the token clustering layer and the token reconstruction layer to reduce computation and memory cost. The proposed modules are non-parametric operations and are applicable to different models and multiple dense prediction tasks.
strengths_and_weaknesses: Strengths:
1. The paper is clearly motivated, well written and with sufficient experimentation.
2. The proposed token clustering and reconstruction are non-parametric operations and therefore can be implemented directly on the well-trained model without additional fine-tuning. Taking existing trained models, the proposed method significantly improves the efficiency of the large vision transformer on the dense prediction task at a low cost in terms of precision.
3. The authors' observations are insightful, and the methods proposed are clever and easy to implement. In addition to the five dense prediction tasks in the paper, I think the proposed approach also has the potential to be extended to other vision transformer-based tasks.
4. The authors also provides the extension strategy and experiments for Swin-Transformer in addition to ViT.

Weaknesses:
1. Since the authors claimed that there is redundancy in the tokens of the vision transformer and that the proposed method is based on clustering, some visualizations could better represent this observation.
2. Clustering attention has been explored by previous approaches such as clustering attention[1] and adaptive clustering attention[2]. Specifically, ACT can slim the high-resolution transformer without retraining. The author is highly encouraged to discuss relationship between previous works. 

Overall, I think the proposed approach is straightforward and effective. However, the paper lack comparison with previous similar approaches. The author is encouraged to discuss related works especially on clustering attention topic. 
[1] Fast transformers with clustered attention
[2] End-to-end object detection with adaptive clustering transformer
[3] SMYRF: Efficient Attention using Asymmetric Clustering
questions: I suggest that authors provide visualizations of the original features and clustering features in the paper or in the appendix, if possible.
limitations: I do not see any potential negative societal impact for this work.
review_title: null

Meta-review: metareview: This paper presents a method to reduce the computational cost of a trained vision transformer for dense prediction. According to the authors' presented experiments, the method can accelerate the transformers effectively without retraining. Although some experiments are not throughout (as discussed below), I see potential in this method and would give the research community a try to see whether the method can be further generalized to other architectures. 

The AC does see some strange experimental setups. For instance, it is strange that the authors chose Mask2Former to conduct experiments but uses Segmentor to conduct experiments on ADE20K. Mask2Former already provides quite a strong model on ADE20K. Why use Segmentor for ADE20K experiments? The AC also observe that the authors compare with ACT on Segmentor as well.

The code is strongly encouraged to release for letting the general public test the method on other architectures.
confidence: Less certain
Decision: Accept