Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an Aliasing Reduction Module (ARM) consisting of an anti-aliasing filter and an external smoothing module. The paper further iterates that the module is lightweight and compatible with most existing transformer architectures. Furthermore, the method shows comparisons between a naive Gaussian filter, a Learnable Convolution Filter, and the proposed Pre-defined Filter Bank. Experiments are performed to show that placing the anti-aliasing module after the Self-Attention block yields the best results. Additionally, the method illustrates that placing the anti-aliasing module in earlier layers translates to the performance boost while inserting the module at deeper stages leads to reduced performance.
main_review: - The paper is easy to understand. 
- Results are shown on various Vision Transformers, i.e., CoaT, DeIT, and SwinT.
- Results are shown on multiple high-level vision tasks of Image Classification, Object Detection, and Semantic Segmentation.

I have the following questions:

- It is a very common practice to use overlapping patch embedding [a, b]. Would these architectures suffer from aliasing? Did you try employing your module in these algorithms?
- The performance of the proposed ARM is demonstrated across various transformers with 224x224 image size. It would be interesting to study the impact of ARM on bigger image sizes (e.g. 384x384) where the jagged effect might occur less frequently.
- Most recent methods such as [c, d] use smaller patch sizes at the patch embedding stage. However, no comparisons are made between these algorithms with the proposed method on alleviating aliasing in attention maps and their performance.
- For Object Detection and Semantic Segmentation, it would be interesting to see visual examples where baseline methods show poor performance and when the proposed ARM is employed, the results are improved.
- SwinT shifted window version provides the best results for Semantic Segmentation on ADE20K dataset. In this paper, why is the sliding window variant of SwinT used in the experiments?
- The method should elaborate on how the proposed module is different from the previously used module in [42].
- The method hypothesizes that the jagged artifacts occur due to the extraction of non-overlapping patches at the patch embedding stage. The visualization is shown for the same in Figure 1. However, the method does not show any visualization for algorithms such as [7] which use overlapping patches. As stated in Lines 32-34, does the occurrence of aliasing decreased?
- In Section 3.2, the method gives a description of the Gaussian filter, a Learnable Convolution Filter, and the proposed Pre-defined Filter Bank. The effects of each of them on a baseline are shown in Table 1. However, no visual comparison is shown on attention maps for the three methods to describe how well the three algorithms perform.
- In Figure 1, the visualization of attention maps are shown for the DeiT-B architecture. Would this visualization differ for different architectures like SwinT and CoaT etc?

[a] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv 2021.

[b] Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv 2021.

[c] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv 2021.

[d] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv 2021.

* * * * *
### **Post Rebuttal Comments**

The authors have responded to my concerns with thorough experiments. The proposed method provides favorable results. Therefore, I am also leaning towards acceptance of this paper. Since the authors showed in rebuttal that the overlapping patch embedding models also suffer from aliasing, I would suggest including these findings in the paper.
limitations_and_societal_impact: The authors have sufficiently discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper discussed the aliasing effects for vision transformers and proposed a simple yet effective fix to it — applying anti-aliasing filters on the attention maps. The authors demonstrate such an improvement brings 0.5% - 1% performance gain on ImageNet classification tasks over two different vision transformer architectures. 

main_review: Pros: 
* Nice finding. I had similar concerns on ViT families to split the image into non-overlapped patches. It's good to see a minor fix could boost performance. 
* Thorough experiments. All the experiments deliver useful messages and come with a good analysis of the experimental results. 
* The presentation is clear. It's not hard to follow the paper. 


Cons:
* There's a lack of quantifiable measures on the aliasing effects. Besides, there are also not many qualitative examples for attention maps before and after anti-aliasing. As a result, it's not 100% convincing to me whether the proposed improvement and the performance gain it brought is due to "anti-aliasing". 
* I assume this effect might be more severe on dense prediction tasks, potentially resulting in block effects on the feature map. Have you considered applying the approach to those models?
* There's no discussion on whether a non-patch-based transformer would have similar issues, e.g., non-local blocks. 


Table 2: How do you apply ARM on patch embeddings? This is not straightforward to me. Have you considered adding an anti-aliasing feature to the input feature map before any embedding computation? 
In Table 2, which filters setting do you chose? Will the placement strategy's relative performances vary for different filter settings? 


You should be able to generate an attention map at a much higher "sampling rate" during training through overlapped image blocks followed by anti-aliased downsampling. Would anti-aliased downsampling on that high-res attention map get the ideal attention map with downstream performance gain? If yes, how about learn a filter to mimic that attention instead?




Table 3: The proposed approach seems to work only for the first layer. It is not clear whether this finding is generic to all different transformer models. And the author's justification for the experiments is not very convincing. Do you think Table 3's finding (only applying early layers matters) is generic for all ViT families and why? Have you visualized the attention maps to justify your conjectures?


Do you maintain the attention maps magnitudes through the filtering/smoothing operations?


The choice of the filter bank is quite adhoc. What are the other fixed filterbank options you have tried? How about DCT, or wavelets? Compared to fully learnable filters options, is this choice of fixed filterbank matter or the fact that the filters are low-rank matter? How about do learnable low-rank 5x5 filters? 




Is there any way to quantify the aliasing effects? How to convince readers the performance improvement brought by the proposed approach is due to the anti-aliasing effect? The author should consider building a stronger connection between aliasing effects and different discussed options, e.g., through spectral analysis of the attention map or measuring some form of invariance or equivariance. For instance, the Zhang et al. ICLR19 paper [9] measures the amount of shift-invariance of CNNs w/o anti-aliasing convs. No quantifiable metrics for aliasing is a primary concern for me as there could be other explanations why adding a conv filtering on the attention map could help improve tasks performance.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a plug-and-play anti-aliasing reduction method to solve the aliasing problem in the vision transformer. It analyzes the effectiveness and generalization ability in multiple tasks and various vision transformer methods. Experimental results demonstrate the effectiveness of the proposed method. 
main_review: This paper proposes a simple plug-and-play anti-aliasing reduction method. It analyzes the effect of the different filters (e.g., Gaussian filters, learnable convolution filter, pre-defined filter bank) on the anti-aliasing reduction and proposes an anti-aliasing reduction based on the pre-defined filter bank. The corresponding ablation studies in Section 4.1 demonstrate the effect of the anti-aliasing filters.

There are some unclear points in the paper. 

The aliasing in the transformer may be caused by low-resolution of the image patches. The analysis in Section 3.1 is not clear and does not support the motivation of the paper. 

It is not clear why the pre-defined filter bank performs better than the learnable convolution filter. In addition, the pre-defined filter bank needs to determine the number of the pre-defined filter dictionaries. Does the number of the pre-defined filter dictionaries affect the final results?

The contribution of the paper is marginal. For the aliasing, it is well-known that using Gaussian blur or other blur kernels can reduce such xx to some extent. As shown in Table 1. Using Gaussian filter is able to improve the performance. This is like an implementation trick. In addition, the paper does not provide insightful analysis on the proposed method.


---
The authors solve my major concerns.  It would be better to include the corresponding materials of the rebuttal in the revised paper properly.

limitations_and_societal_impact: The limitations of the paper is not explained clearly. More experimental results should provide to analyze the limitations of the proposed methods.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper analyzes the uncharted problem of aliasing in vision transformer and explore to incorporate antialiasing properties. To handle that, they propose a simple yet effect module named Aliasing-Reduction-Module(ARM) to smooth the self-attention output. The ARM contains a set of filters to augment the attention layer output. Extensive experiment results show such approach achieves a clear gain but with a mirror GFlop increase along with more efficiency and robustness. 
main_review: 
1. Overall, the motivation is very good and clear. Rather to design another transformer architecture, this paper tries to solve the aliasing problems brought by the tokenization process of VIT. This is an interesting problem. Also, it is complementary to many transformer design studies. 

2. The writing is good and easy to follow. 

3. The design of ARM is well motivated and the design of filtering process is well explored in both method and extensive experiments. Unlike tokens in natural language that stand for separated words, the tokens in vision transformers commonly represent manually-defined windows. As image contents may vary, the pairwise relationships between these tokens may be ill-posed and contain considerably amount of high frequency noise. It’s intuitive and novel to “smooth”the “aliased interactions” between tokens. To this end, this work proposes a simple yet effective design. The experiment results across different transformer architectures also validate the effectiveness of the plug-in design. Meanwhile, the work chooses to directly smooth the attention maps, which is compatible with other advances such as incorporating convolution into the patchify stem.

4. The findings in Section 4.1 and 4.2 are useful and crucial, where the choices of filter types and placements are explored. It’s interesting to observe that anti-aliasing techniques conducted on early transformer blocks are useful while too much smoothing on deeper layers might be harmful. The authors conduct extensive comparisons and discussion to explain the reason. These conclusions also provide some insights and directions to boost the development of vision transformers.

5. The proposed methods can be easily deployed into different transformer frameworks and leads to performance gains and more robust. These are well studied in experiment part.

Rather than tricky improvements on Vision Transformer ( I am tired with several attention layer extensions with SOTA performance),  this paper points out an important and potential problem of current VIT design named  Aliasing  Effect brought by tokenization process. It also give a simple yet effective solution.  Based on all the points, I give my init rate as accept. 

 




limitations_and_societal_impact: No major weakness.
Several points which are needed:

1.  During implementation, can the ARM module directly put after the softmax computation? Since the attention maps have lots of high frequency noise, i.e. aliasing effects. It seems more straightforward to anti-alias the attention scores compared to the post-projection features.

2. The anti-aliasing operation introduces new hyperparameters before training, such as the parameters of the selected Gaussian kernels. It’s worth noting about how to choose the proper kernels for different architectures and the training stability.
Here are several detailed concerns:

Fruther suggestions : 

1,  What about put both Pre-defined Filter Bank and learnable convolution(Depth-Wised)  into ARM ? (I wonder whether it have more performance boost).

2,  is input of ARM attention maps or value output of self-attention? Fig-3 looks a little confusing. I believe this is value output of self attention.

3,  Better put the figure on the top of pages. 

4, Better to show out experiment results on more exsiting transformer frameworks.
review_title: null

Meta-review: This paper initially receives mixed reviews. The authors successfully convince the reviewers with the rebuttal. The reviewers discuss extensively the strengths/weakness of this work and in the end all are in favor of accepting this paper. The area chairs agree with the reviewers' recommendation to accept this paper. 
Decision: Accept (Poster)