Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new token downsampling method for vision transformer, called Token Pooling, to prune redundant tokens efficiently, so as to achieve a better flop-accuracy trade-off. Specifically, token pooling is a nonuniform data-aware downsampling method, which uses cluster algorithms to aggregate information from tokens automatically. To keep important information, the authors also proposed minimizing the reconstruction loss caused by downsampling. The authors performed experiments on the ImageNet-1k dataset, showing that the proposed token pooling can significantly improve the flop-accuracy trade-off over the existing downsampling methods.
main_review: Strengths
1. The authors analyzed the computational cost of vision-transformer components in detail. These pie charts (in Table 1) visually illustrate the computational bottlenecks of vision transformers are fully-connected layers.
2. The authors demonstrated that softmax-attention is a low-pass filter under mild assumptions, thus attention layers will generate redundant representations.
3. This paper is well written and easy to understand. The authors provide a clear explanation and motivation behind their token pooling method. As far as I'm concerned, it has an impressive feature in that it adopts clustering algorithms (such as K-Means and K-Medoids) to reduce the number of tokens.

Weaknesses
1. Despite claims that the proposed token pooling is an effective operator that can benefit many architectures, but in this paper, the authors only examined based on the DeiT architecture. If token pooling is a general method for vision transformers, the paper should apply it to at least the current SOTA vision transformers.
2. As mentioned in the introduction, max pooling and average pooling are widely-used downsampling methods. From my experience, average pooling is a competitive method to reduce the number of tokens. Why not do an ablation study with max/average pooling?
3. Personally, I would appreciate experiments with more vision transformers. If the authors proved that the proposed token pooling could be applicable to other vision transformers besides DeiT, it would strengthen its value. 
4. It would be better to provide more results and analysis of time consumption and inference speed, including the clustering process.


summary_of_the_review: In summary, this work presents a new token downsampling method for efficient vision transformers. The paper is well written and properly structured, but the experiment is only performed on DeiT and ImageNet-1k benchmark, and no time analysis is provided, which is not up to par.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose a novel token-pooling method to reduce redundancies in tokens for recent vision transformers. They analyze the computation cost distribution of vision transformers and the limitations of grid-based & score-based token downsampling methods. They further formulate a reconstruction loss and optimize it with the token pooling layer. The experimental results based on DeiT show that token pooling improves accuracy while reducing computation cost by a large margin. This idea can be applied to other vision transformers as well.
main_review: Strengths:

+ Solid experimental results

Compared with a well-optimized DeiT baseline, token pooling significantly improves accuracy and reduces computation. Ablations in the appendix on different clustering, convolution downsampling, models are well written. 

+ Well-structured analysis

The analysis in Table 1 clearly show the portion of different operators in vision transformer and figure 2 shows the limitation of score-based token downsampling, which motivates the authors to reduce redundancies with a novel token pooling method.

+ Impact on community

Vision transformers have been deployed on multiple vision tasks and are being a mainstream solution. Efficient vision transformers are important to be developed and the token pooling proposed in this paper can be used for other vision transformers without major modifications. It can replace convolution/patch-based downsampling and contribute to the community.

Weaknesses:

- Attention as a low-pass filter

I appreciate the analysis in section 3.3 on attention vs low-pass filter, but I don't get the relation between this observation and the token pooling methods. In the experiment at Appendix F, it seems token pooling is not affected by normalized or unnormalized Q/K vectors.
summary_of_the_review: Based on my main review above, I think this paper proposes a solid token pooling method for vision transformers that can be potentially applied to other vision transformers based on tokens. I recommend an acceptance to this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a token pooling approach based on the K-Means algorithm to improve the efficiency of vision transformers. The method is evaluated on image classification, where it reduces the computational complexity by half while maintaining similar performance.
main_review: Strength:
+ The submission is technically sound. In Section 3.3, the analysis on attention as a low-pass filter is interesting and makes sense. The K-Means based token down-sampling method is straightforward.
+ The submission is clearly written and well-organized.

Weakness:
- The K-Means clustering method for efficient transformers has been introduced in [1], which narrows the contribution of the paper. 
- According to Table 2, 3 and 4, the proposed framework is sensitive to the selection of hyperparameters. It seems that the method is tricky and heavily relies on manual tuning.
- The method is only evaluated on the DeiT. Can it apply to other vision transformers, e.g., PVT and Swin?
- Reducing the number of queries is unfriendly to apply to downstream tasks.
- It would be better to report the actual throughput and compare the proposed method with the previous dynamic vision transformers, such as [2] and [3].

[1] Roy A, Saffar M, Vaswani A, et al. Efficient content-based sparse attention with routing transformers[J]. Transactions of the Association for Computational Linguistics, 2021, 9: 53-68.

[2] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. DynamicViT: Efﬁcient vision transformers with dynamic token sparsiﬁcation. ArXiv, abs/2106.02034, 2021.

[3] Wang Y, Huang R, Song S, et al. Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length[J]. arXiv preprint arXiv:2105.15075, 2021.

summary_of_the_review: The paper proposes a new token pooling method to achieve efficient vision transformers, which is interesting and well-motivated. But the method heavily relies on manual tuning and the generalization is limited. I will increase the rate if the concerns are well addressed.
review_title: null

Meta-review: This submission receives mixed ratings initially. Two reviewers lean negatively while one reviewer is positive. The raised issues include 
whether the proposed method can be adapted to other vision transformers, the design choice of pooling strategy, the computational time cost, the similarity to an existing work, and the influence of the proposed method on downstream tasks. In the rebuttal, the authors have addressed several issues such as pooling strategy analysis and time consumption.

There are still some issues not completely solved. The proposed method introduces K-mean clustering on tokens between different layers. The K-mean clustering is prevalent and the weighted clustering does not make the technical contribution sufficient. Also as a general token pooling operation, the proposed method shall be integrated into various types of vision transformers (e.g., vanilla ViT [a], ConViT [b]), rather than one single DeiT. Besides, the downstream tasks in DeiT are not conducted in the proposed method. 

Overall, the AC feels the proposed method, although interesting, requires a major revision that addresses existing issues.  The authors are suggested to further improve the current submission and welcome to submit for the next venue. 

[a]. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Dosovitskiy et al. ICLR 2021.

[b]. ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. Ascoli et al. ICML 2021.
Decision: Reject