Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first analysis the difficulty of going deeper using transformer models. The challenge is that the mechanism of self-attention is applying a low-pass filter.  Keep stacking self-attention layers will lose the feature expressive power and only preserve the DC bias.  To alleviate this issue, this paper propose two modules to make the MHA to be an all pass filter. First module AttenScale combines the high-pass component and FeatScale reweights the featuremap. The experimental results demonstrate the improvement on accuracy for DeiT and CaiT when adopting the proposed method.
main_review: Overall, this paper proposes a novel idea of thinking self-attention weight matrix as a low pass filter. This explains the difficulty of convergence for deeper tranformer models. The proposed methods alleviates the low-passing representation of the transformers. The experiments also prove the superior performance using proposed modules. However, the experiments are not sufficient, the improvement is relatively minor, and it doesn't show whether the proposed approach can help deeper transformers converging. It would be very interesting to see it is helpful for deep models.
summary_of_the_review: Interesting and novel idea and good mathematical analysis. May need more experiments to demonstrate the claims in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores the reasons why ViTs cannot go deeper. The article provides clear and solid proofs, clarifying the rank collapse in attention matrix via Fourier analysis. Meanwhile, the authors suggest two types of possible solutions, AttnScale and FeatScale, which are useful when scaling up the ViTs architecture. The experiments on DeiT and CaiT show that the adjustment on attention helps avoiding rank collapse when going deeper, holding high-frequency information. 

main_review: However, some problems remain:
1.The article proves that the self-attention is a low-pass filter, thus neglecting the high-frequency information. The conclusion comes to the stage that pure attention leads to rank collapse in the network. FFN, skip connections and mutli-head self-attention mechanism which are used in ViTs help to hold high-frequency components. The current clarification in the paper are not very convincing to verify why the ViTs cannot go deeper, even though with FFN, skip connections and mutli-head.
2. The visualization of the modified attention is about the learnable parameters. However, the spectrum of the modified attention seems more convincing to show the change on high-frequency components. And it is more persuasive to show the situation when the depth is 24.

summary_of_the_review: The overall presentation is good. The main concerns are in above. 
The models (CaiT-S and DeiT-S) are fairly small. Can the authors provide more experiments on big models to further verify the effectiveness of the proposed method?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a viewpoint of spectrum domain to show self-attention amounts to a low-pass filter and will cause feature maps to only preserve direct-current components as depth increases. Based on this viewpoint, it proposes two technique named AttnScale, FeatScale to reweight low-pass, high-pass components with minimal parameters. Experiments are conducted on ImageNet. 
main_review: Strengths: 	
1.	The paper provides theoretical analysis of ViT from spectral domain, which explains the empirically findings of patch feature collapse of ViTs. 
2.	Reweight techniques of low-pass and high-pass components are proposed, which is sound according to the theory part.  

Weaknesses:
Experiments part does not strongly support the proposed method.
1.	The gain is limited, 0.2 ~ 0.3 improvements for 24-layer models. 
2.	With FeatScale, 12-layer DeiT has 1.0 improvement v.s. 24-layer DeiT has 0.3 improvement, this is kind of opposite to the paper’s claim, as layers increases, the collapse should be more severe, where one would expect the proposed techniques should be more helpful,  however the experiments results indicates the opposite. It questions the proposed methods’ ability to scaling depths further beyond 24 layers. 




------post-rebuttal-----
With the new provided experiments results, I raised my rating towards accept. 
summary_of_the_review: 
The paper address the feature collapse problem of ViT from spectral prospective, decomposing attention to low-pass and high-pass components. The therotical part is nice, while the experiments does not show adequate evidence that the proposed method would actually help scale depths of ViTs (e.g. beyond 24 layers) as title indicates.
review_title: null

Meta-review: The paper analyses the frequency filtering properties of self-attention in vision architectures, shows that it mainly acts as a low-pass filter, and proposes fixes that allow to better preserve the higher frequencies. These fixes yield moderate classification accuracy gains (~0.5-1%) for several existing attention-based architectures.

The reviewers are quite borderline about the paper, but after considering the authors' responses lean towards acceptance. Pros include interesting and novel analysis and sound model improvements leading to non-trivial empirical gains. The main con is that the experimental results are fine, but not outstanding.

Overall, I recommend acceptance. Empirical results are indeed good but not outstanding, but the theoretical analysis is interesting and it is good to see that it leads to actionable insights on the model design side that actually help in practice - even is not by a huge amount. One part that in my opinion is confusing (and might have been confusing to the reviewers too) is that the title seems to suggest the paper will present very deep vision transformers while it does not. Adding deeper models or adjusting the title would help here.
Decision: Accept (Poster)