Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a method to improve the efficiency of vision transformers. Traditional transformers process maintain and processing embeddings equal to the number of patches sampled from the original visual input. The key idea of this work is to modify the existing architectures to allow dropping of redundant embeddings at intermediate stages. The experiments demonstrate that the approach improves efficiency, retains accuracy and effectively retains relevant patches from the input signal while discarding redundant areas. 
main_review: The paper is well written and is very easy to follow for the most part (see comment below). The introduction concisely motivates the problem and presents an overview of the paper. The related work section does a thorough job of contrasting with existing work in this domain. 

The idea of learning an additional model to drop redundant embeddings is novel and makes an interesting contribution towards improving the efficiency of transformers in computer vision (to the best of my knowledge). 

The proposed "Multi-head interpreter" architecture is intuitive and provides a simple way to implement the idea of redundancy reduction. The simplicity of this architecture makes this idea generally applicable to a wide-range of domains with minimal changes. I believe that this has the potential to be widely adopted. 

The experimental evaluation is thorough and exhaustive. The qualitative visualizations show conclusive evidence that the proposed "Multi-head Interpreter" focuses on highly relevant regions of the input signal while discarding redundant regions. The quantitative evaluation further demonstrates the benefits of this approach by showing improved efficiency without any loss in performance. 

The paper also presents some additional interesting insights - lower redundancy in cluttered images and orthogonality of model redundancy vs data redundancy. 

## Questions, Concerns and Suggestions

- Line 158-161, "the VIT is divided into D groups, ...". Here it is not clear that these D groups are sequential components of the whole VIT. This is especially confusing since the Figure 2 shows them as parallel pipelines. This could be explained better. 
- Line 162: "The network groups are optimized separately from the group right after the patch embedding layer to the group right before the prediction head." This line is not clear immediately and only makes sense after reading the next section. 

- Question: Why is REINFORCE used to train the interpreter? Why is it not trained in a supervised fashion using a gradient estimator (like the straight-through gumbel)? 

- Line 183-197 is very hard to follow. It would really help if a pseudo code for this training process was provided. 

- For the raw attention baseline, why is the attention chosen from block 1 specifically? Does it do better or worse (for results in Table 2) if the attention is chosen from a different block? 




limitations_and_societal_impact: The authors have adequately addressed the limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents a novel framework for hierarchical redundancy reduction in input tokens to vision transformers, resulting in fewer computation in multi head self-attention layers and improved inference time. The method achieves 1.4x speed up compared to DeiT or TimeSformer models when applied to image recognition and video action recognition tasks. The proposed method is also more interpretable compared to other baselines and raw attention learned by original vision transformer model.
main_review: STRENGTHS
- The approach reduces the number of tokens in a learnable manner, reducing the speed of inference considerably compared to models like DeiT or TimeSformer.
- The authors added good visualizations how hierarchical redundancy reduction process works.
- Interpretability results shown with the method appears to focus more on the informative regions compared to baseline methods. 

WEAKNESSES AND REMARKS

- Increased training time considerably? Given that the models are initialized from pretrained weights, and then again trained for ~90 epochs, this approach is increasing the training time considerably. The gains from this approach seems to be small over an attention only baseline(which doesn't need any retraining?). For image classification, this approach is better than an attention baseline by 0.4 and for video classification, this approach is actually worse than an attention only or random baseline by a large margin. 

- Given the different baselines and the IA-RED$^2$ approach give close performance results, it will be good to understand how different seeds affect the experiment results.

- The networks seem to be initialized from pretrained weights. How does this method work when training a model from scratch? If the method needs the model to be initialized from pretrained weights, is this approach better than training a student network containing fewer number of parameters OR having similar FLOPs as IA-RED$^2$ model distilled from a teacher Deit-B etc?

- Missing several important ablations for various hyper-parameters used in the methods. To understand how the hierarchical approach works, it would have been interesting to see the effect of changing the number of groups $D$. In addition, whenever $I_{i,j} < 0.5$ , the token is discarded for all the subsequent groups. How does different values for this threshold effect the final performance?

- The interpretable results in Figure 3 seems to be mixed (with raw attention attending to better regions in a few compared to IA-RED$^2$). Hence in some cases the message is not very clear. Are the raw attention results shown use all the tokens? Are the visualized results randomly chosen?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes the IA-RED2 framework, which tries to reduce the redundant tokens in vision transformer in a sequential manner. The redundant tokens are determined by the proposed multi-head interpreter (as a policy network). The multi-head interpreter is trained with REINFORCE, utilizing a reward function that considers the sparsity/accuracy tradeoff. The training of the full network follows a complicated curriculum learning training pipeline, which contains multiple stages and first training multi-head interpreter and then finetuning the rest of the networks in each stage. The authors conduct experiments on ImageNet1k classification with DeiT models and Video action recognition with the TimeSformer model.

This paper is among the first wave of sparse vision transformers with data-dependent sparsity, together with several other concurrent works like "Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length". It attains promising interpretable results, as the authors claimed. 
main_review: As a CV practioner, my main concern of this work is how I can use it? If I use it as a new sparse vision transformer model (after finetuning), its accuracy-speed trade-off is not comparable with data-independent sparse transformers, like Swin Transformer. For example, Swin_Tiny can reach ImageNet top1 accuracy 81.3 with fps >= 700, which in Table 2, the authors use IA-RED finetuned DeiT-B to achieve 80.9 top1 accuracy, with fps about 440. If I use it as a model interpretability method, then IA-RED2 fine-tuned the original model for the visualizations in  Figure 3 (the scores are from the second group). Moreover, for visualization/interpretability performance, IA-RED2 is only compared with GradCAM, not other popular methods like LIME and gradient integration methods. 

Therefore, if the authors want to focus on a new sparse vision transformer model, the authors may want to compare with other data-independent sparse vision transformers (HanoNet are published, Swin/ViL are are published). In fact, there are several data-dependent sparse transformers from the NLP literature, such as "Reformer: The efficient transformer", "routing transformers", "Sparse sinkhorn attention", "Informer: Beyond efficient transformer for long sequence time-series forecasting". These data-dependent sparse transformers are also model-agnostic and task-agnostic as the authors claimed, and may be good baselines for IA-RED2.

If the authors want to focus on interpretability, then the authors may want to compare with more interpretability methods (other than GradCAM) and other data-dependent sparse transformers (they are model-agnostic and task-agnostic although from NLP community), to validate the superiority of IA-RED2.

On both sides, other existing model-agnostic and task-agnostic methods from NLP community are valuable baselines to compare with. 

Minor comments:

1. In Equation (2), have the authors try the reward without the square? Just 1 - |u|_0/N. The complexity of vanilla self-attention is O(12ND^2 + 2N^2D), has both linear and quadratic part. In the DeiT case (N=196), the linear part in fact even larger than the quadratic part. 

2. In Equation (2), how did the authors determine the parameter \tau? Any insights here?

3. Line 241-242: "We generate the raw attention map by extracting the attention weight at the multi-head self attention module in the block 1." Since the attention is multi-head, the authors may want to elaborate on how they get the final visualization (how to aggregate attention scores from different heads).

 
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper proposes a framework for redundancy reduction in vision transformers. The reduction is obtained in a learnable manner, using REINFORCE. The results are encouraging, yielding an improvement over recent vision transformer architectures and providing an interpretable result. The tackled topic is timely and the execution of this submission is good. Therefore, I recommend the acceptance of this paper as a poster.
Decision: Accept (Poster)