Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a simple method to merge tokens within ViTs with or without training, increasing the throughput by up to 2x across various image, video, and audio tasks. ToMe inherits the spirit of simplicity and can server as a drop-in replacement to increase the training speed or inference speed.
strength_and_weaknesses: Strength:

* The idea is simple yet effective and can benefit both ViT training and inference a lot.
* The authors clearly have deep insights about the (larger or smaller) ViT designs. The introduction of the literature is clear and informative.
* Using matching instead of clustering algorithm makes sense to me, results also demonstrate the expected throughput achievements.

Weakness:
* More direct comparisons are desired, e.g., in Tab. 3, you compare ViT-L with other ViTs. How about comparing them apple-to-apple. E.g., ViT-L w/ ToMe vs. ViT-L (attached in appendix; better to remove ahead), MViTv2 w/ ToMe vs. MViTv2, Swin w/ ToMe vs. Swin, etc.
* It may be easier to improve the throughput for large ViT models without hurting the model accuracy a lot. How about smaller ones, e.g., LeViT?
clarity,_quality,_novelty_and_reproducibility: Very clear and the quality is up to the bar of ICLR.
Token pruning methods are considered a lot in the literature, this paper drafts the story centering on token merging instead, which is novel to me.
summary_of_the_review: In short, the idea is interesting and work for ViTs. A lot of visualizations also intuitively explain and validate the idea. Thus I recommend to accept this work.

As for its generalizability, more experiments on both small models and large models can be considered as the cost is not so high.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper:  - This work proposes to reduce the computational redundancy in vision transformers (ViT) with Token Merging, without significantly changing the original architecture. 

 - With the introduced ToMe module, similar tokens are gradually combined with a simplified matching algorithm. The author also perform studies on the comparison with pruning.

 - Extensive experiments on image and video recognition are conducted, across a variety of transformer families. 
strength_and_weaknesses: #### **Strength**

 - The problem, reducing the computational burden in ViTs, has been an important issue in developing and deploying transformers. 

 - While redundancy in self-attention has already been investigated before, there are few solutions except for pruning-based methods, which require additional attention. The simple strategy in this work shows promising results, even without being trained. 

 - The authors perform concrete studies and experiments across different tasks and architectures, showing the reliability ToMe.

#### **Weakness**

 - This method could be viewed as a parallel implementation of the matching algorithm. We expect more discussion with previous slower variants, such as the clustering algorithm in [1].

 - The authors mentioned that the method could be used in training to reduce training complexity. One more common use case here is to adopt such idea to recent MIM-based methods such as MAE, where longer pre-training epochs are required.  However, the authors have only used it in fine-tuning MAE pre-trained checkpoints.

[1] Token Pooling in Vision Transformers. arxiv 2110.03860.
clarity,_quality,_novelty_and_reproducibility:  - This paper has a nice writing quality and presentations, except for some slightly-incorrect statement:

e.g. At Page 1 Introduction: "despite being overtaken in cost to performance, vanilla ViTs still have many desirable qualities". from my perspective, the statement here "being overtaken in cost to performance" might not be correct. As ViTs have many nice properties (also mentioned by the authors) and work well at many tasks, I would like the authors to rephrase this sentence. 
summary_of_the_review:  - Reducing the computational burdens in vision transformers is a non-trivial task. Given the simplicity and effectiveness of ToMe, I believe it's beneficial in further research.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper works on improving the efficiency of a ViT model by merging tokens each layer. Specifically, the authors first uniformly split ViT tokens into two sets and merge (i.e., replace two tokens with their mean feature) the top k similar pairs across sets in each layer. The merge runs in negligible time and can work on test time without retraining. Experiments on image, video, and audio data show the proposed method can reduce ~50% tokens within <0.5% accuracy drop.
strength_and_weaknesses: Strengths

+ The proposed method of greedily-merging tokens is very elegant and makes a lot of sense to me, especially that the method can work without retraining. This makes the proposed method easy to use for many off-the-shelf transformer-based models.

+ The paper shows results on three different modalities: images, videos, and audio, and all displayed competitive results. This shows the proposed method is general and robust.

+ The ablation studies in Table 1 are comprehensive, and show the proposed method is robust under different hyer-parameters.

+ The qualitative examples in Figure 4 are interesting. Please make sure they are not cherry-picked.

+ The paper is well motivated, well-written, and easy to understand.

Weaknesses

- The comparison to existing efficient transformers in Table 4 are not as exciting as expected, especially given that DynamicViT is also very simple. Most of the advances are in training speed instead of on speed-accuracy trade-off.

- It will be good to include other efficient transformer methods for videos and audio to provide more context (if there is any) about the numbers.

- [Minor] last line of the code in Appendix D. Should it be `return merge(k)`.?
clarity,_quality,_novelty_and_reproducibility: Clarity: very clear and easy to understand.

Quality: the paper is in high quality and the experimental results are good.

Novelty: the proposed method is novel as far as I can assess.

Reproducibility: the code sample provided in the appendix makes it easy to reproduce.
summary_of_the_review: This paper has a simple and effective idea on an important problem, and has great application values. All the technical contributions are well motivated and well ablated. The state-of-the-art comparisons to existing efficient transformer methods (Table 4) are currently a little bit below expectation, but this does not undermine the significance of this work, especially given that the model can work without retraining. My current recommendation is accept, and am happy to raise to strong accept if the authors address my concerns during the discussion session.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a token merging strategy for efficient inference of ViTs. A Bipartite Soft Matching method is proposed. It divides all the tokens into two parts, and draws an edge from each token in the first part to its most similar token in the second part according to the cosine similarity between pairs of tokens. The model efficiency can then be decided by how many similar tokens should be merged. 

Different from most previous works that aim at compressing ViTs with training, the approach presented in this paper can be conducted in the inference stage directly. Thorough experiments are performed on images, video, and audio, which reflect the effectiveness of the proposed approach.
strength_and_weaknesses: Strength
- The motivation of this paper is clear. For a large ViT model, there is no need to keep all the tokens involved in the decision making as some of them may contain useless information. Discarding some of the tokens that are similar to others can efficiently reduce the inference time.

- The proposed approach is simple and easy to follow.  It can be added in either the training or inference stage. For users who want to use large-scale models but are with limited computational resources, this paper provides an appropriate tool to conquer this issue.

- Experiments show that the performance does not drop much when r is small but the speedup is clear. Moreover, compared to other ViT based models, this paper also shows its advantages.

- Experiments on all images, video, and audio show that the proposed approach indeed behaves well.

- I really like the visualizations in Figure 4. It seems that for regions with similar textures or patterns, they can be merged for efficient inference.

Weaknesses
- Like the author said, the vanilla ViT shows great potential in visual recognition especially after the emerging of MIM-based methods, like MAE and the proposed approach works well with the vanilla ViT. However, in some cases, a pyramid structure of ViT is required especially for downstream tasks, like semantic segmentation. It would be better if the approach could be utilized in ViTs, like Swin Transformer.

- From the experimental results, we can see that for large models, increasing the value of r does not lead to too much performance drop. However, for small-sized or even tiny-sized models, the performance drops much.  Though the reason is clear but I still think it should be mentioned explicitly.

- From Figure 3(a), it seems that after compressing the ViT-L model, the trade-off between the performance and latency is not as good as ViT-B without compression.
clarity,_quality,_novelty_and_reproducibility: The novelty of this paper is clear. The paper is well-written as easy to follow.
summary_of_the_review: The novelty of this paper is clear but not that significant to match the score of 8. The experiments are thorough. I think this paper deserves a score of 6.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a novel method for increasing the throughput of transformers (without training), by progressively merging similar tokens in the model, instead of pruning them. The proposed method is very simple yet very effective, yielding up to 2x speed up across image, video, and audio classification. The motivation is clear, the experimental analysis is comprehensive, and the obtained results are compelling. A few weakness have been pointed out by the reviewers, including the lack of application of the method in modern/specialized transformer architectures such as Swin, MViT, etc. and during MAE pretraining. Overall, this is a good paper. All reviewers recommend acceptance and the AC agrees with this decision.
justification_for_why_not_lower_score: The proposed method is very simple, yet very effective. The AC believes this work will be of significant interest to a wide ICLR audience and practitioners in the field.
Decision: Accept: notable-top-5%