Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Research on Vision Transformer s ( ViTs is heavy Different from the prior investigations that focus on proposing ViTs structures, this work focuses on image tokens and study how to effectively wipe out useless ones to model acceleration This brings a potential advantage that
when maintaining the same time cost, models are able to take more tokens i.e, from a higher resolution image ) for prediction accuracy improvement. These two aspects are evaluated in the experiments.
main_review: PROS:
+	Image token identification via the class token guidance is interesting. The class token adaptively removes image tokens based on the corresponding category influence. The discarded tokens are reorganized/fused into one token to facilitate network training at the initial training stages.
+	The experiments on the efficiency and accuracy evaluations show the proposed method is effective for current Vit models.
CONS: 
-	The proposed method gradually removed tokens from different layers of ViT models. Is there any relationship between these removed tokens from different layers? Since the removed tokens are hardly maintained, will the proposed fusion strategy be able to restore them at the remaining layers?
-	Given a ViT model, is the design of the proposed method arbitrary for all the layers or some fixed layers? Where to drop and fuse tokens within a ViT model deserves further discussion.
-	What leads to the difference between Table 1 and Table 2(a)? A straightforward inattentive token fusion removal decreases the DeiT more, while without inattentive token fusion decrease the same model less? If the rate of Table 2(a) is set as 0, will the results be the same?
-	Reporting the actual time cost will benefit a thorough understanding of ViTs acceleration.
summary_of_the_review: The token reorganization is new. The results are fine. Addressing these raised issues above will make this submission more convincing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, an EVIT method is proposed for vision transformer speedup. It reduces image tokens based on the token attentiveness, which is measured by the class token. The inattentive tokens are reorganized as one to support attentive tokens. Experiments have shown on the
benchmarks for visual recognition.
main_review: Pros:
The proposed method is simple while effective. Using the class token to guide image token selection seems reasonable and is well motivated by the observations (Fig. 1 and Table 1). The whole computation does not bring much time cost while reducing inattentive tokens
during the ViT feed forward process. This design benefits both offline and online stages. 

In the experiments, the proposed method accelerates recent ViT models (DeiT and LV ViT) by increasing throughput and decreasing MAC. Meanwhile, the recognition accuracy does not drop much. Another perspective is that under a similar computational cost, using more tokens from higher resolution input images improves the prediction accuracies. This will benefit a series of ViTs without bringing additional time costs.

Cons:
The proposed method illustrates inattentive token removal at the 4th, 7th, and 10th layers. Whatis the rationale behind this choice? How to decide which layer to process when handling other ViT models? A clear illustration upon how to integrate this method into ViT models will be more convincing.

The token reorganization seems to fuse discarded tokens into one to supplement attentive tokens. As have explained in Sec. 3.3, this might be because the attentive token selection is not stable at the beginning of the training. If that is the case, a visualization upon token variations (initially random token selection, then the discarded token converges to the unrelated content) would be useful to support the claim.

summary_of_the_review: Token processing is promising as it potentially benefits all the ViTs. The proposed method improves ViTs from the efficiency and accuracy perspectives. This will bring a wide range of impacts on the ViTs development. This reviewer recommends acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a mechanism to dynamically drop transformer tokens, according to class token attention score. It is able to reduce computation with similar accuracy or improve accuracy with controlled computation.
main_review: ## Strengths
1. The paper is well written and easy to follow.
2. The method is simple and does not require a pre-trained model.
3. The visualization looks good and intuitive.
4. ViT models are accelerated by a large margin.

## Neutral
1. Marginal improvements over the baselines with the standard resolution and epochs. The gain is more significant with more input tokens and a longer training schedule. I think this is acceptable as the method reduces the computation cost for each iteration.

## Main Weaknesses
1. It is arguably difficult to combine the proposed method with multi-scale transformers, e.g. Swin transformer. It might be doable but the real throughput gain could be very limited. The current performance improvement on the vanilla ViT cannot cover this limitation.
2. The ablation difference of inattentive token fusion is very limited and close to pure training noise. Thus, the discussion of the vanilla model’s training instability and the improved training efficiency lacks enough justification.
3. It would be interesting to see if one can directly train DynamicViT from scratch. There seems no reason prevents us from doing so. As this is one of the main advantages of the proposed method over DynamicViT, more discussion is expected.
4. Apart from the class token, one could also use the full tokens-to-tokens attention to compute the attentive score. This is an obvious alternative but is not sufficiently studied.

## Suggestions
1. It is interesting to see if we can make use of the good DINO mask as an attentive region, although the DINO mask requires much longer training.
2. The interpretability of the learned mask could be evaluated just like what is done in DINO and then we may be able to see how the mask quality correlates with performance.
3. To better support the intuition of dropping tokens, one could visualize (in a naively pre-trained ViT) whether a token is never attended by the class token once it is less preferred (i.e. roughly monotonically decreasing attention from the class token). If there are cases where a token is only attended in the last layer, then the intuition of dropping-a-token-forever may not hold.

summary_of_the_review: I recommend accepting this paper for its technical novelty, the model acceleration, the performance gain (when it is trained with a long schedule), and the potential of improving more with a better attentive score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to expediting vision transformers by reducing the number of tokens. The main contribution is  the attentive token identification, which is based on calculating the attentiveness of the class token with respect to each image token. Experiments on DeiT and LV-ViT show that the proposed approach is able to successfully reduce the computational cost of the baseline models with nearly no performance drop.
main_review: Pros:

- The idea of this  paper is interesting. Different from the previous DynamicViT that reduces tokens with an extra sub-network,
this work starts from the working principle of the self-attention module and leverages the relationships between the class token and the normal image tokens. I believe this paper provides a promising way to reduce the computational cost of vision transformers for future research.

- The motivation and presentation of this paper are clear. Pseudo code is provided. It is easy to follow for readers. I am looking forward to the  release of the whole code.

- Experiments are thorough. Results on both DeiT and LV-ViT show that the proposed approach can efficiently compress the computational cost while maintain the baseline results.

Cons:

- One of the issue that I concern is the performance of the inattentive token fusion. As shown in Table 2, it seems that the classification performance gap between the two sub-tables is rather small. The question is why the authors introduce the inattentive token fusion?
In addition, as described in the late part of the experiment section, training on images with higher resolutions can improve the model performance. So, the question is how the inattentive token fusion would affect the model performance when taking images with larger image sizes as input?

- In Page 7, the authors report that training with a longer schedule benefits the model performance when utilizing the proposed approach. Is there any evidence/observation showing that what leads to such a phenomenon? 

- In the experiments, three reorganization layers are used. Have you attempted to use more? 

- The proposed approach can be regarded as a type of downsampling operations. So, would the proposed approach perform better than previous classic downsampling operations, such as pooling and convolution. A typical counterpart should be the PVT model.

- In Fig. 4, the authors show the comparison of the proposed method with other vision transformers, but there is no corresponding content describing the differences between the proposed approach and other methods. It would be great to add a new paragraph to give more description.

Typos:   In Page 7, 'wee' should be 'we.' In Page 9, 'reach' should be 'reaches.' Please carefully revised the manuscript and correct the typos.  


summary_of_the_review: In spite of some experiments missing, regarding the significance of the novolty of this paper, I think it deserves a positive score. If the concerns can be well addressed, I would like to lift the rating.
review_title: null

Meta-review: The paper presents an approach to select visual tokens in images and reorganize them for the object classification, within Transformers. All four reviewers find the paper interesting and novel, and they are also very positive about the experimental results. The authors also addressed minor concerns of the reviewers successfully through the discussion phase, clarifying details and adding experiments.

We recommend accepting the paper.
Decision: Accept (Spotlight)