Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a dynamic transformer network that allocate different computational cost to process different images with different complexity. The key idea is to change the number of tokens to represent an images, and based on the output confidence, the model decides if it goes to the next stage of computation to get more confident results. This paper further proposes two "reuse" strategies to reuse the results from the previous stage. Experiments show this improves the performance. 
main_review: Strength:
+ This paper successfully applied the mutl-stage (also called early exit, dynamic models, etc.) strategies to improve the efficiency of Vision Transformers. 
+ One unique contribution of this paper is that it tries to reuse computation from previous stages and use them to improve the performance. Unique to ViT, this paper proposes "feature reuse" and "relationship reuse". 

Weakness:
- The early-exit idea has been widely adopted in CNN models. The high-level idea of this paper is not that different from early-exit, though it is now applied on transformers. The novelty of this work is therefore limited. 

- Since the computation of such multi-stage network is inherently serial, the latency of such networks may be slow even though the theoretical FLOPs are low. 

- Different samples can have different latencies, this can be problematic for applications with hard latency constraint. Also, the batch-processing of such method can be much more complicated.

Question:
How is the actual throughput/latency measured?  I assume in this paper, it reported the "average" latency/throughput. Since the speed is measured on GPUs with batch processing, if one sample is more "difficult" and requires multiple stages of processing, how is the latency of this batch calculated? Assuming that a sample's latency is t_{b,i}, where b is the batch index and "I" is the index within the batch. The time it takes to finish inference of this batch should be max_i t_{b,i}, which means that even there are many simple (fast) samples, the inference time for this batch is determined by the slowest sample's processing time. I hope the author can clarify how their speed is measured, and how to avoid the problem I mentioned above. 
limitations_and_societal_impact: No societal impact discussion needed, in my opinion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces an efficient inference strategy for vision transformers (ViTs). The motivation is that there are some “easy” images which can be classified using a few tokens. Based on the motivation, the paper proposes a cascading inference framework which do not proceed to next stage if the prediction confidence is enough high (i.e., early exit  strategy). For further efficiency of the cascading scheme, the previous stage’s features are fused with the next stage features. Experimental results show that the proposed method outperforms the baseline methods both in inference speed and accuracy. 

main_review: ### Strengths
- This paper is well-written and the motivation is clear. 
- Experiments are extensive with various datasets and plenty of ablation studies. 


### Concerns and questions
- The proposed method provides technical contributions to accelerate vision transformers with an interesting motivation. However, the early-exiting is a  well-known strategy in CNN fields as the authors mentioned in the related work section ([21, 46, 24]). 
  - It would be nice to see in-detail comparisons against the existing multi-stage early-exiting strategies [21, 46, 24] in both empirical and conceptual ways. 
  - Could the previous methods [21, 46, 24] be naively adopted to vision transformer architectures? If not, discussing the reasons would be very interesting. 
- The proposed framework reuses the previous stage’s features to enhance the accuracy. The “reuse” mechanism is quite interesting, so it would be great if the paper provides a discussion about “reusing” the previous features to improve the inference speed of the next stages. 

### Post-rebuttal comments
- The author's response (reply) has resolved most concerns I raised so that I will keep my initial rating (weak acceptance).


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes how to dynamically apply a cascade of increasingly expensive ViT models in order to save compute resources at inference time. The architecture reuses features and attention logits between tokens of cheaper models as context for the more expensive ones. The resulting models achieve very strong performance on imagenet against competitive baselines when normalized by inference time and compute (flops). Detailed ablations verify individual design choices.
main_review: ## Strengths

* strong inference efficiency gains by coarse to fine processing of an image
* clearly written
* detailed ablations
* dynamic model that is able to adapt based on compute budget

## Weaknesses

* No training efficiency gains (yet), actually the proposed model is a bit more expensive because it involves training the entire cascade of models on every image (is this really necessary?). This fact should be made more explicit (!), as training efficiency is quite important, especially because ViTs have shown larger advantages in high-data regimes [1,2]. The paper typically talks about efficiency gains (in general), but should be more precise at times (e.g., table/figure captions, but also occasionally in text). 
* No comparison/discussion around other methods that speed up inference (e.g., distillation).
* Only ImageNet.


## Conclusion

Overall I like the idea of coarse to fine processing a lot, and combining it with dynamic processing even more. It goes against the main stream application of pyramids that start processing on the most fine grained level and building up ever more complex features. This is a very wasteful process. I would have liked the paper even more if the authors would have made an effort into translating these ideas also into the training process, which I think would be way more interesting than inference speed-ups, as there are other solutions to this.

I dislike the sole focus on ImageNet (which is unfortunately still common practice) because it is totally unclear to me how these speed ups would translate to other datasets or other tasks. It also makes me question how important some of the design choices are in general as ImageNet1k is fairly small by today's standards. I would urge the authors to at least include results for models pretrained on the freely available ImageNet 21k dataset.

Another important shortcoming of the paper is its lack of comparison to or at least discussion around related work in the space of inference speed-up, e.g. distillation methods.

## Comments & Questions

* capitalize everywhere -> Vision Transformers
* DeiT-small (w/o distillation) --> ViT-small (It is a ViT model with specs introduced in the DEIT paper, but it is still just ViT).
* Why concatenating the low res features before the MLP and why not simply concatenating the low res token reps with the higher res patch embeddings  passing them jointly as input to the transformer? This seems somewhat more elegant to me.

[1] ViT. https://arxiv.org/pdf/2010.11929.pdf
[2] CLIP. https://arxiv.org/pdf/2103.00020.pdf

## UPDATE

I stick with my initial judgement of the paper. I think it deserves acceptance.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: All three reviewers are positive about the paper (also after considering the authors' response and discussion). Here are the main points:

Pros:
1) Well motivated and well presented idea
2) Good inference speed improvements on ImageNet
3) Good ablation studies

Cons:
1) Somewhat related to existing early-exiting strategies
2) Doubts about practical gains in inference/training speed

The authors were successful at addressing the reviewers' concerns. Overall, the idea is clean, is presented well, and works well. Thus I recommend the acceptance of the paper.
Decision: Accept (Poster)