Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper observes the problem of feature collapse in transformer, where the tokens in deep layers collapse to the same representation. It then proposes to add additional augmented shortcuts in parallel to the original shortcut connection in the multihead self-attention modules and MLP modules, based on the observation that simple identity shortcut alleviates the problem of feature collapse. In the experiments, the proposed method brings 1.1% performance gain in the ImageNet classification task compared to the baseline model. Performance gains can also be seen from COCO detection task and other downstream finegrained classification tasks, as compared to the baseline models.
main_review: In this paper, a simple add-on approach is proposed to encourage the feature diversity in deep transformer layers. It introduces additional shortcut connections, comprising of a linear layer and an activation non-linearity, on top of the existing shortcuts. The feature diversity is measured by the matrix norm of the difference with the best rank-1 approximation matrix to the feature matrix. This diversity measurement is improved compared to both the setting without any shortcut or with identity shortcut(Figure 3 and Figure 4).

Circulant martices are used as the weights of the linear layers introduced in the additional shortcuts to save computation and storage. As compared to using the full randomly initialized weight matrices, this approach uses roughly 1/200 the storage and 1/20 the G-FLOPs, with minimum performance drop.

The main modification centers around the shortcut connections in transformer architecture. Therefore it would be more convincing if more ablations could be done around the theme of shortcuts. Specifically, how does the choices of stochastic depth/dropout parameters influence the performance? How does the architecture of the shortcuts(e.g. multiple linear layers, choices of non-linearity, etc.) affects the diversity of the token feature outputs?

Moreover, the paper observes the decrease in feature diversity as the depth of the model increases, and that adding more shortcut connections can effectively solve this issue. An experiment with deeper transformer model, such as the CaiT family, is therefore much sought for. It would be more convincing if the paper can show improved diversity in token features of deeper transformer model.

------------------------------------------------Post Rebuttal--------------------------------------------------------------------------

The responses have addressed the issues listed in the previous comment, so I will  I will increase the final score to 6.
limitations_and_societal_impact: See above. The societal impact is shown one the last page of the manuscript.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes to boost the performance of vision transformers from shortcuts. As analyzed, the feature diversity of tokens declines as the layers go deeper while the shortcut connection provide a fix towards to potential ``feature collapse'' issue. To this end, additional branches of augmented shortcut is proposed to further improve the feature diversity as well as the final recognition power.
main_review: - The authors propose to augment the shortcuts in vision transformers. This technique is independent from the specific architecture designs, being able to be plugged into different vision transformers. 

- The demonstration of feature diversity is clear and interesting. The motivation of augmenting the shortcut is well explained through the figures and tables in Section 2. 

- This paper is well-written. All detailed techniques are clearly clarified. Extensive experiments are conducted to validate the significance of the proposed simple module.

- Weakness and Limitation: see next section.

---------------Post Rebuttal Feedback-------------
After reading the authors' rebuttal, some of my concerns are solved. Thus I increase my rating to 6.
limitations_and_societal_impact: Limitations:
- The motivation of this work is straightforward, from the observation that the rank, i.e. diversity, is lost when layers go deeper. However, this phenomenon does not appear merely in vision transformers. Instead, this issue has also be explored in modern CNNs structures. Based on this observation, lots of progress has been made, such as Channel Pruning [1], compression [2] and semantic segmentation [3]. Meanwhile, the lost-rank phenomenon of self-attention layer is originally investigated in [4], which this work is based on. Thus the contribution of this paper might not be adequate. 

- The technical novelty of the proposed module is also limited. There have been some prior works that transform/enhance the shortcut pathway. It's quite obvious that adding a re-parameterized shortcut branch is able to improve the performance. Furthermore, another concern is about the memory cost brought by additional shortcuts.

[1] Channel Pruning for Accelerating Very Deep Neural Networks. ICCV 2017

[2] On compressing deep models by low rank and sparse decomposition. CVPR 2017

[3] Tensor Low-Rank Reconstruction for Semantic Segmentation. ECCV 2020

[4] Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arxiv preprint
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a simple, effective and efficient way to address the feature collapse issue in current Vision Transformer models, which is add multiple parametrized shortcut paths in parallel to the original identity shortcut path, with an efficient approximation implementation. Experiments show the proposed method consistently improves over the baseline and the authors show transfer to detection task as well. 
main_review: 
The idea to use multiple parameterized shortcut to reduce the feature collapse is straightforward and effective, shown both theoretically and empirically. Actually, it is a bit supervising to see it works so well, e.g. 1.0% improvement on ImageNet by simply adding more shortcuts. 

The paper is extremely well structured, and easy to follow. The experiments are extensive and ablation studies are comprehensive to understand the proposed method.

The paper proposes to add additional shortcut paths, I’m wondering what’s the performance like if simply replace the original identify shortcut to parameterized one (1x1 shortcut)? 

There is a value projection inside the attention calculation, what if we simply use this projection output as augmented shortcut, what would the performance be like? in this way, no extra params would be introduced, and we get a parametrized shortcut as well. 

---------------Post Rebuttal Feedback------------- 

The authors' response make sense to me and solved my questions. I'll keep my rating. 


limitations_and_societal_impact: The authors have properly addressed the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper alleviates feature collapse in vision transformers by introducing an efficient module, named augmented shortcut. As shown in the theoretical and empirical results, the feature diversity is enhanced and obvious performance improvement is achieved. The authors conduct experiments with various SOTA vision transformer models and validate the effectiveness of the proposed augmented shortcuts. 
main_review: Positive:

The proposed method is interesting and has a clear motivation. The authors observe that massive features in vision transformers are very similar, which may restrict their representation capacity. It is interesting to solve the issue by improving the shortcuts. Multiple paths with parameters have stronger power to preserve the feature diversity. 

The block-wise circulant projection makes the augmented shortcut can be calculated very efficiently. The extra computation cost brought by the augmented shortcuts can be neglected. Thus the augmented shortcut can be flexibly implemented various models, just as the identity shortcut. 

Extensive experiments are conducted with various models (including ViT, PVT, T2T-ViT) on different mainstream tasks. For the classification task on ImageNet, the augmented shortcuts improve the performance by 1% top-1 accuracy, which is significant. For the detection task, all the metrics are improved. These experimental results validate the superiority of the proposed method.  

The components and hyper-parameters of the proposed method have been well analyzed with the ablation studies. It is glad to see that the proposed method is robust to these hyper-parameters. 

Negative:

Making more discussions about the experimental results in Table 2 will make the paper stronger. For example, it is interesting to see that the augmented shortcut can improve the performance of the deeper model more significantly. Could the authors make more analyses about the reason behind it?

The captions in Figure 3 are too small to read. The authors are required to refine them.

limitations_and_societal_impact: These aspects have been well discussed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduce a universal module to improve vision transformers. They show that vision transformers suffer feature collapse, which mean that features in deep layers will become similar. To address, this paper presents an augmented shortcut module to  be flexibly embedded into various transformer models. Extensive experiments are conducted on multiple tasks (image classification, object detection, and transfer learning), which show that augmented shortcuts can bring obvious performance improvements. That is a good paper to me because of the new and effective module which is a new investigation to the field.
main_review: 
+ The proposed method is novel and reasonable and extend the vision transformer from a new viewpoint and new investigation into the it. The augmented shortcuts can effectively alleviate feature collapse and enhance the representational ability, extensively validated both theoretically and empirically. The introduced block-wise circulant matrix can effectively improve computational efficiency.

+ The proposed augmented shortcut module has favorable flexibility and can be implemented on different transformer models for various tasks. Their module is generic and can be compatible with different transformer architectures on various models (e.g., ViT and SOTA variants PVT, T2T), which show obvious accuracy improvements. The commonality of the augmented shortcut make it have a lot of application scenarios.

limitations_and_societal_impact: - Figure 1 illustrates how to equip the MSA module with the proposed augmented shortcuts. What do patterns (rectangle, triangle, etc.) denote? Do the patterns with different colors denote different features from various patches? It will be more intuitive to add more detailed captions.

- The parameter b in block-wise circulant matrix can affect its computational cost. How do you determine its value? Is the parameter b sensitive to different architectures of vision transformers?  

[1] Deep Residual Learning for Image Recognition, CVPR 2016
[2] Densely Connected Convolutional Networks, CVPR 2017
review_title: null

Meta-review: This paper got mixed ratings initially. All the reviewers agree with the motivation and effectiveness of the proposed method, but they also raise the concerns on the novelty and technique contributions of this work. In particular, the feature collapse issue has been identified by a prior work [8]. The theorem listed in the paper was not developed by this work either. The reviewers also suggested to add more ablation experiments on this work. After the response, the authors address these concerns and all the reviewers give positive ratings. AC has read the submission, reviews and authors' response. Although the finding of feature collapse is not firstly identified by this work, the proposed shortcut augmentation is simple and the performance improvement is significant. Thus, AC agrees with reviewers that this work would be valuable for the community and recommend acceptance.  
Decision: Accept (Poster)