Blending Anti-Aliasing into Vision Transformer
Authors: Shengju Qian
Hao Shao
Yi Zhu
Mu Li
Jiaya Jia
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision transformer
Anti-aliasing
Image recognition

Abstract: The transformer architectures, based on self-attention mechanism and convolution-free design, recently found superior performance and booming applications in computer vision. However, the discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps, arising the traditional problem of aliasing for vision transformers. Aliasing effect occurs when discrete patterns are used to produce high frequency or continuous information, resulting in the indistinguishable distortions. Recent researches have found that modern convolution networks still suffer from this phenomenon. In this work, we analyze the uncharted problem of aliasing in vision transformer and explore to incorporate anti-aliasing properties. Specifically, we propose a plug-and-play Aliasing-Reduction Module (ARM) to alleviate the aforementioned issue. We investigate the effectiveness and generalization of the proposed method across multiple tasks and various vision transformer families. This lightweight design consistently attains a clear boost over several famous structures. Furthermore, our module also improves data efficiency and robustness of vision transformers.

Blending Anti-Aliasing into Vision Transformer
Shengju Qian
Hao Shao
Yi Zhu
Mu Li
Jiaya Jia
'The Chinese University of Hong Kong
Tsinghua University AAmazon Inc.
Abstract
The transformer architectures, based on self-attention mechanism and convolution- free design, recently found superior performance and booming applications in computer vision. However, the discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps, arising the traditional problem of aliasing for vision transformers. Aliasing effect occurs when discrete patterns used to produce high frequency continuous information, resulting in the indistinguishable distortions. Recent researches have found that modern convo- lution networks still suffer from this phenomenon. In this work, we analyze the uncharted problem of aliasing in vision transformer and explore to incorporate anti- aliasing properties. Specifically, Module (ARM) to alleviate the aforementioned issue. We investigate the effective- ness and generalization of the proposed method across multiple tasks and various vision transformer families. This lightweight design consistently attains a clear boost several famous structures. Furthermore, module also improves data efficiency and robustness vision transformers!.
propose plug-and- and--pay Aliasin-RReduction Reduction
Introduction
Transformers have led to impressive breakthroughs in language understanding. dominating wide range of natural language processing (NLP) tasks. Meanwhile, continuous researches strive to leverage transformers for vision tasks, revolutionizing the conventional inductive bias in convolutional neural networks (CNNs). After big leap made by vision transformer (ViT) [1], promising results on vision tasks have recently emerged [2-4], demonstrating a possibility of using transformers as primary backbone for vision applications.
While appealing advantages such as long-range context modeling and parameter efficiency are introduced by self-attention, this mechanism also brings inevitable aliasing issue to vision transformers. Aliasing [5] traditionally refers to the phenomenon that high-frequency signals become indistinguishable when undersampled [6]. This effect occurs when discrete patterns utilized to capture more continuous signal, resulting in frequency ambiguity. In terms of ViT [1], images are split into non-overlapping patches during tokenization, which then fed into transformer blocks. Compare to the more "continuous" representation of image, tokenization and self-attention performed on its discontinuous patch embeddings be regarded subsampling operations. While alleviating computational costs, these operations leave side-effect which leads to possible aliasing.
A simple solution to mitigate aliasing phenomenon is increase the sampling rate. Similar properties emerge in vision transformers, where overlapped tokens [7] and smaller patch sizes [1, 8] lead to improved performance. As increased sampling rates lead to quadratic computation costs, hereby conjecture that proper anti-aliasing filters that integrated into the "attending" process could also provide fix. A potential concern is that the general purpose of anti-aliasing: to introduce
Work done during an internship at Amazon. Code is made available at https: //gpt/h/ com/
wtwpe/chhannnnaaaaaaaaaaa transformer.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

"smoothness" to signals, might be contradictory with the goal of self-attention: to capture more significant and high-frequency features in contrast. How would traditional anti-aliasing techniques influence vision transformers? And what if we equip these state-of-the-an transformer architectures with suitable anti-aliasing options?
Transforme
Sea slug X
Blocks
Anti-Allasing
Transformer
Input
Blocks
Goldfish v
First Attention (Better zoom-in)
Thresholding
Final Attention
Prediction
Masks
Figure I: Visualization of attention maps from pre-trained DeiT-B B 19). The attention maps are visualized by averaging heads in the selected block. The masks obtained thresholding the self-attention maps following DINO [8]. The second row shows results after applying our anti- anti-aliasing module to first transformer block. earlier aliasing effect on fish tail region makes subsequent layers focus on the incorrect high frequency and make the wrong prediction. In contrast, the employed anti-aliasing maintains the overall smoothness, leading follow-up layers capture more continuous features along the real semantic contour.
To this end, we design the Aliasing Reduction Module (ARM) for vision transformers, which consists of anti-aliasing filter and an external modulation module. Extensive investigations are conducted primarily, including the filtering choices such as the basic Gaussian blurring filters in [10] and learnable convolutional filters, as well as different placements of these filters. Notably, based on the observations that the "jagged" phenomenon varies across images and locations, propose an adaptive filtering method which draws inspirations from dictionary learning [11]. Specifically, lightweight aliasing estimation network is constructed to predict the combination coefficients of the pre-defined low pass filter bank adaptively. Furthermore, the position of integrating our Aliasing Reduction Module in mighly-modularized transformers is crucial. The model benefits from an early anti alliasinn operation right after the self-attention computation, which is distinctive from the observations in CNNs [10, 12). As claborated in Figure 1, the module brings perceptually smoother attention, which leads to better feature localization and eventually correct prediction. By integrating ARM to state-of-the-ae af-t architectures, consistently find considerable margin of improvement.
To summarize, our contributions are:
We investigate anti-aliasing techniques for vision transformers, and propose Aliasing Reduction Module (ARM), which is compatible with most existing architectures.
We tailor various anti-alisan anti-aliasing strategies and different placements of aliasing reduction, which are distinctive from the observations CNN structures.
Experiments show that our simple yet effective design boosts sophisticated state-of-the-art vision transformers. Furthermore, we observe stronger generalization, robustness, and data-efficiency brought by our anti-aliasing strategy.
2
Related Works
Vision Transformers. The transformer structure was introduced in [13] for machine translation, which further became a general purpose model for many NLP tasks. The Vision Transformer (ViT) [1] firstly closes the gaps image classification with previous CNN models using pure transformer. DeiT [9] further improves training data efficiency of ViT, utilizing an additional distillation token for teacher-student training as well as stronger data augmentations [14, 15). Given its superior performance on classification, recent works apply transformer architecture to various vision tasks, including object detection [16, 17], segmentation [3, 18, 19], point cloud processing [20], image

---

generation [21-24] and so on. In order to obtain better transformer architectures that generalize well to general vision tasks, some concurrent advances have introduced hand-crafted designs. For example, CPVT [25] and Coa' [26] improves the original positional encoding. T2T [7] improves the naive tokenization process. Multi-scale information [27, 28], convolutional designs [29-31], and pyramid structures [32, 33] are also utilized in vision transformer extensions. Due to the inefficiency and unnecessity of self-attention, some researchers also explore approximated attention [34], local attention [31, 32] even replacing it with MLP structure [35]. Different from these designs, our work dive into the "feature-resampling" property of the self-attention mechanism, which leads to aliasing and performance degradation. To this end, investigate lightweight anti anti-aliasing module to help mitigate the effect. This design is general and compatible with most vision transformer architectures.
Anti-aliasing in CNNs. The definition of aliasing was originally covered in signal processing [6], which causes the distortion on sampled signals reconstruct original signals. The aliasing issue occurs in CNNs with any strided layers, such max pooling and strided convolution. While early smoothing methods like average pooling [36] degrades performance on modern models and bench- marks, the elegant Blurpool [10] introduces both anti-aliasing properties and increased accuracies to strong baselines. DDAC [12] further assigns adaptive filters for each spatial location and channel group. These anti-aliased CNNs demonstrate both better recognition, generalization to downstream tasks and robustness towards corruptions 38].
Differs from CNN structures, vision transformers do not have explicit "strided" subsampling except for the tokenization step, i.e. patch embedding. Furthermore, their potential downsampling [32. 33] choices are mostly censely-connected layers. Thus the observations and techniques for CNNs may not generalize well to transformers. In contrast, we argue that in transformer, self-attention on split patch tokens can be viewed as "resampling" step which tries to reconstruct the informative part on the original continuous pixel representation images. To this end, we investigate on anti-aliasing self-attention operation, in order to learn more continuous and smoothed attention representation.
3 Methodology
We propose to blend the anti-aliasing property into vision transformers by processing the inner representation with a lightweight Aliasing Reduction Module (ARM). ARM consists of an anti- aliasing filter as well as an external smoothing module. The anti- anti-aliasing operator can be chosen flexibly among a traditional low- wwwppass filter, e.g. Gaussian filter, a learnable convolutional filter, or pre-defined filter bank. In this section, we will first explain what raises the issue of aliasing in vision transformers, as well as difference compared with the twin problem in CNNs. Secondly, we provide the details about how we design the proposed ARM and the crucial choice of where to integrate it.
3.1
What makes for aliasing in vision transformer?
Frequency aliasing is the phenomenon occurs when performing subsampling any type of signal, for instance an audio, an image or a video. When the sampling rate is lower than bandwidth of the original signal and does not meet the Nyquist rate [39], the resampled signal will be aliased where its high frequency patterns interlace with the low frequency components additively. As demonstrated in Figure 2, this effect usually results in visible corruptions and artifacts.
Surptee Origial
Iaput
MLAA
Isput
ID signal
(b) 2D image
Rendering
(d) Sclf-attention
Figure 2: Different types of aliasing: (a) aliasing on ID signal; (b) aliasing on images [12]; (c) aliasing in graphical rendering [40]; (d) aliasing from self-attention. For (d), we visualize the attention map from pre-trained DeiT-B [9] by averaging the heads in the first block, where inconsistent semantic importance is observed.
As discussed in [10], convolutional networks also suffer from aliasing which caused by strided operations, i.c., strided convolution or max pooling, that commonly exist in widely adopted CNN
3

---

structures. The subsequent layers following the strided operations are likely to extract features from the distorted inputs. Such artifacts might affect the final prediction, as well as model's robustness towards natural corruptions.
Self-attention mechanism allows transformers to integrate information across all spatial locations, by resampling the importance of feature representations based on self-affinities. However, with quadratic cost in terms of large number of pixels, naive application of "per-word" tokenization in NLP does not scale to realistic image sizes. A widespread practice is to process each input image into a sequence of patches as in VIT [1]. The patch tokenization and embedding indeed alleviate computational costs while inevitably bring in the problem of aliasing due to the discontinuous processing. When there exists an aliasing phenomenon in very early layers, the subsequent transformer blocks will be overwhelmed by the misled signals. Similar to CNNs, vision transformers have another potential aliasing sources from the downsampling layers. Note that there exists no progressive downsampling in ViT-based architectures, we further analyze whether the feature map downsampling in recent transformer architectures [32, 30, 33] influences the model in Section 3.2 and Section 4.2.
In conclusion, the discontinuous tokenization process and its succeeding self-attention make for the aliasing phenomenon in vision transformer, further limiting their modeling power. Meanwhile, it's computationally infeasible to simply extend the finite token sets through smaller resolutions or even pixel-wise processing. To this end, we aim to develop generic anti-aliasing module that mitigates this omitted phenomenon in vision transformers.
3.2 Incorporating anti-aliasing module into vision transformer
In this section, we answer two questions raised above: how we design the structure of ARM and where to place it inside the highly-modularized transformer architecture.
We first risvisit the implementation of transformer block for vision transformer. Given an input & E Cx xW the block first splits x into a sequence = E NR with a length of N where the dimension of patch embedding is d. The sequence representation is then passed into fully connected layer. The FC layer projects the sequences into query, key and value with the same hidden dimension (g, k u] E IR Dx3D,, where D and Dr, represent the input and hidden dimension. Then self-attention S.A can be formulated as:
A = softmax(gk") /V Dr.)
(1)
SA(=)= Av
(2)
Then with the projection and normalization, the attention map is obtained. The attentive signal is accumulated back to the original input xr
As we've analyzed in Section 3.1, the self-attention operation is conducted split patch tokens from the original feature maps. This process "attends" to finite sets of discrete tokens and identifies relevant features from them, collecting stronger significance while bringing in aliasing. In other words, the attention map & is a resampled signal from the original representation x, which is more discrete and sparse. To this end, we choose to directly act on the aliased attention map and apply anti-aliasing operation on it. The general purpose is to make the attention signal more continuous and robust. A recent work FNet [41], which replaces self self-attention in BERT [42] with Fourier Transforms, may also share the same spirit. Besides, we' we've also tried to put the smoothing filter to the different positions, such as tokenization process or downsampling step in Swin Transformer [32]. Detailed discussion about the placement can be found in Section 4.2.
Figure 3 provides an overview of the proposed Aliasing Reduction Module (ARM). The module adaptively smooths aliased attention maps computed from the discontinuous token sets. We tailor three valid choices of the anti-aliasing filter as visualized in Figure 3:
Gaussian Filter. In order to investigate whether the model suffers from aliasing, the simplest approach is to smooth the attention map with traditional low pass filter. Hereby choose the Gaussian blur, which is effective at reducing image noise and leaving fewer sharp edges. The filter is kept frozen during training, introducing roughly zero computation cost training and testing.
Learnable Convolution Filter. While Gaussian blurring servers as an efficient operator at removing noises, applying such fixed smoothing to the entire feature map inevitably results in some valuable

---

ARM in Vision Transformer
Detailed design ARM
Bank
Coefiiiiena
BatchNorm
Sharical
Estimatoo
uttention maps
Anti-Allasing
External
Oierry
Filtering
Modulation
Newts
Flaton Patch Enshedding
Aliazed
Ganssinn
Convolation
Split Patch Tokens
Pooliins
atsention mage
Method overview. The left portion illustrates the placement of Aliasing Reduction Module (ARM). On the right, demonstrate the detailed design of ARM. Note that the filter could be chosen among Guassian kernel, a learnable filter, as well as the drawn filter bank.
Figure 3:
high frequency information loss. As has been discussed in [12], it's sub-optimal to apply the same filter across the varying content. To this end, we also propose a learnable convolution-based filter. The content-awes aware filter adaptively learns to produce varying anti-aliasing effect. On the contrary, this design incurs additional trainable parameters and extra GPU memory.
Pre-defined Filter Bank. Due the unconstrained nature attention representations, the training of shallow Anti-Aliasing convolution filter is unstable as the gradients oscillate between varying attention maps. While other strategies such as residual learning and encoder-decoder structures may alleviate these issues, this line of method demands even more computation costs. To balance the trade-off, we predefine a filter bank that contains disparate degrees of smoothness. A lightweight convolutional layer is optimized to estimate the combination coefficient of the atomic filters. Specifically, The filter F; k which denotes filter in the i-th channel the input attention maps, is constructed as a linear combination of n pre-defined filter dictionary D from the coefficient 4:
N
F.
2 D2DY j=1
Based on the predicted coefficient from the weight head, complex filter be derived from the pre-defined filter bank. Since the estimated filter evolves within the linear space constructed from the atomic filters in the dictionary, is feasible and fast to optimize. The linear assembling strategy has been proven effective in computational photography [43], parametric human body representation [44], network compression [45] and acceleration [46].
Specifically, we compose our pre-defined filter bank with Gaussian filters as well as a small portion of difference of Gaussians (DoG), which endow the feasibility when figg-preeeeving properties are required. As visualized in Figure 4, the filter bank is constructed by altering the covariance matrix of multivariate Gaussian kernels, reflecting in varying scale, rotation, and ellipticalness. Each filter is normalized to the summation of 1. Thus the weights of a k x k sized kernel is defined based on the probability density function (PDF) of the Gaussian distribution with covariance matrix E;
k(x Fo: Σ)
1
[( 20) "x- x (x-20)"
(4)
2T
Σ
Beyond the "internal" smoothing operation within each attention map, we also explore the effect brought by "external" modulation approach. Using buffered frames, temporal Anti- Aliasing (TAA) [47] in computer graphics tackles the aliased patterns that the dage-smmothing filters handle poorly. Inspired by this, we propose to ensemble our anti-aliasing filter with an ex- ternal regularization which reduces the internal shift in attention maps using batched samples. We operationalize the module using a batch norm [48] after the filtered attention maps. As the normal- ization operation has been analyzed to bring smoothness into the optimization process of neural networks [49], non-trivial be ensembled into our Aliasing Reduction Module for being able to mitigate separate patterns and oscillations, of which the bister-based methods incapable.
5

---

Gaussian kernel (0-5)
(6-7)
Rotation: =330"
Gaussian, - Gaussstan
Elipticul: x 0.5
scale: n x== 0.3
Figure 4:
Visualization a 5 x 5 pre-defined filter bank, which consists of 8 filters. Note that the parameters of J. rotation, scaling, and ellipticalness, are sampled stochastically. The experiment results are consistent using different random seeds. We provide more details about the generation of filter hank in the appendix.
In summary, our anti-aliasing component is composed of an anti-aliasing filter, which can either consist of Gaussian filter, the learnable convolution, and the proposed filter bank. Subsequently, an external modulation module connects to the filters, rescaling the filtered features. The module directly applies smoothness to the attention maps, thus being pluggable to most vision transformation architectures.
Experiments
In this section, first conduct extensive experiments analyze the suitable design choices, including which anti-aliasing filter to choose and where to integrate our anti-aliasing component. With these observations, apply our proposed aliasing reduction module to various state-of-the-art vision transformer architectures. To validate its effectiveness, we comprehensively perform downstream tasks, data-effieient training, well as robustness evaluation. In the end, present additional ablation studies to justify the contribution of each design.
4.1 Choices of Anti-Aliasing filter
As discussed in Section 3.2, three filters designed with different anti-aliasing effects. In order to compare their respective influence, we conduct apple-to-apple comparison and keep the placement consistent with the best conclusion made in Section 4.2. We choose Swin-T [32] main baseline for its dominating performance and efficiency.
All training and testing parameters remain consistent with its open-source implementation [50]. In Table we report the performance different anti-aliased variants on ImageNet [51] validation set with 50K images. All models are trained for 300 epochs.
Input size
Throughput ImageNet 2012 (image/s)
Method
# Parameters. FLOPs
Top-1 Acc
Original
224
28.3M
4.5G
746.0
81.2
W Gaussian
224
28.3M
4.5G
732.4
81.5
W Learnable
2242
28.7M
4.9G
668.9
81.6
Filter Bank (n=8)
2242
28.6M
4.6G
708.4
82.0
Table I: Comparison of different anti-aliasing filters on ImageNet classification.
Despite smoothing seems to be conflicting with the spirit of attention, we still see an increased accuracy in Table when fixed Gaussian filter is applied the strong baseline. The improvement indicates that vision transformer structures benefit from naive anti-aliasing. However, there is no significant boost when we switch the Gaussian filter to more powerful convolution operator. We thus reckon that as the attention maps vary across different heads and instances, it is extremely difficult capture the high-level relationship using such simple structure. The proposed filter bank strategy achieves the best results, which manifest the feasibility of anti-aliasing. The pre-defined filters provide several anti-aliasing templates which alleviate the training curse from the huge diversity between the attention of individuals.

---

Table 2: Comparisons of different placements
Table Comparison of different layers of transformer encoders being filtered. Top-2
of the proposed aliasing reduction module.
results are in bold size. Diff indicates the performance gap to the baseline where no
Model
Placement (after)
Top-1 1 Acc
DeïT-S
79.8
anti-aliasing performed.
DeîT-S
Patch Embedding
79.5
DeïT-S
Attention
80.7
Filtered Layers
Top-1
Acc
DeïT-S
Attention + shortcut
78.1
3
Val
Diff
Swin-T
81.2
81.2
Swin-T
Patch Embedding
81.0
82.0
+0.8
Swin-T
Attention
82.0
V
81.7
+0.5
Swin-T
Attention + shortcut
79.8
V
80.6
-0.6
Swin-T
Patch Merging
79.5
V
81.3
+0.1
4.2 Placements of Aliasing Reduction Module
Interestingly. we find that the correct placement of the anti-aliasing filter is crucial, as improper positions to insert such "smoothing" operation would potentially hurt valuable information.
As mentioned in Section 3.2, the aliasing issue potentially emerges during the discontinuous tok- enization and the self- ailf-attention's 's resampling process. It's straightforward to apply the anti- anti-aliasing operation to these positions including patch embeddings, attention maps, and the fused inputs with attention maps, i.e. after the shorteut connection in the transformer encoder. While aliasing in CNNs majorly happens after strided layer, there is no such downsampling in ViT [1] family. We hereby choose Swin-T the baseline it follows powerful and hierarchical structure, and experiment whether the fully connected downsampling (patch merging) brings in aliasing.
Table 2 illustrates the uneven effects from different placements of the anti-aliasing module. Training settings are consistent with the ones in Scction 4.1. Both DeiT [9] and Swin [32] transformers encounter minor performance drop when we choose to smooth their patch embeddings. In contrast, anti-aliasing on attention maps brings improvement to both baselines. Nevertheless, we move a step further and try to filter the post-shortcut features, significant degradation in accuracy is observed. As discussed, self-attention operations in vision are analogous to feature resampling on discontinuous tokens. It attends the overall feature a more significant representation and leads to potential aliasing. Applying the smoothing techniques to attention maps makes the signals more continuous and can reduce some impulse noise, thus improving the overall performance. From another perspective, low-pass filtering on the attention branch will not diminish the amount of total information as the values are still presented in the shortcut path. Conversely, filtering after shortcuts inevitably reduces the total information and leads to the accuracy drops in Table 2. Unlike previous findings in CNNs, smoothing the downsampling in transformer leads dramatic performance drop, suggesting that the densely-connede subsampling in vision transformers requires no anti-aliasing.
As we have chosen apply anti-aliasing to attention maps, another question arises: how many transformer blocks we choose to smooth? The cascaded transformer encoders in vision transformers gradually increase the significance of attention. Anti-aliasing on different depths might have different influences on the succeeding layers. Table 3 shows the respective influences on anti-aliasing different layers in transformers. Note that Swin-T has 2-2-6-2 layer number, which denotes how many transformer blocks exist in each layer. Then each row in Table 3 refers to filter all the blocks in the marked layers. The results show that anti-aliasing on earlier layers contributes to a substantial performance boost, while deeper integration leads to performance degradation. These observations indicate that the aliasing problem mainly occurs when the continuous images are split into separate tokens and become self-attended with the partial signals. As the layers go deeper, the "attended" mes- sage in the tokens has already been individual and separable, where such ver-aggressivv smoothing operations instead decrease the inherent salient information. Our observations accord with some recent findings on the robustness of transformers [52, 53]. In [52], people find that the CLSS token in VIT [1] changes slowly at earlier layers, but evolves rapidly in later layers where limited updates happen to representations of individual patches. Though no CLS token, the cascaded architectures in Swin identically build up the functionality that earlier layers refine the tokens embedding and the
7

---

latter layers consolidate the overall information for classification. These findings explain the uneven effects brought by different placements.
In conclusion, our anti-aliasing strategy for vision transformers to employ our module directly on the recomposed attention maps, and merely the earlier transformer layers.
4.3 Comparison of state-of-the-arts
To further validate that our anti-aliasing technique is versatile with different vision transformer architectures, we integrate the proposed module to several state-of-the-ard models, including DeiT [9], Coa'T [26], Token-2-Token ViT [7], and Swin Transformer [32]. These networks contain distinctive designs and pipelines for improving the performance of Vision Transformer. To allow a fair compari- son, utilize the original training scripts [54-56, 50J and keep the training parameters consistent with the provided ones. Except for T2T VIT [7]. which is originally trained for 310 epochs, all models trained for 300 epochs ImageNet [51] training set using Tesla V V100 GPUs. When applied with our component, the training configurations are maintained the same as the baselines, such as the optimizers and data augmentations, Table 4 shows how method influence these state-of-the fi-th---rt vision transformers. We here use the pre-defined filter bank which consists of randomly sampled 3 x kernels and integrate it into the first two transformer blocks analyzed in Section 4.2.
Throughput (image/s)
Transformer Family
Model
#Parameters.
Top-1 I Accuracy
CoaT-Lite-Tiny
5.7M
77.5
CoaT-Mini
10M
80.8
CoaT [26]
Coa Tiny w Ours
5.7M
78.4
C-aT-Mini Ours
10M
81.5
DeiT-S
22.1M
425.6
79.8
DeiT-B
86.6M
285.4
81.8
VIT [1]. DeiT [9] family
DeiT-S w Ours
22.3M
402.7
80.7
DeiT-B Ours
86.7M
259.1
82.4
T2T-ViT-14
21.5M
81.5
Token-to-Toker [7]
T2T-VTT-14 w Ours
21.7M
81.9
Swin-T
28.3M
746
81.2
Swin-S
49.6M
423.8
Swin Transformer [32]
Swin-B
88M
273.1
83.3
Swin-T w Ours
28.5M
708.4
82.0
Swin-S Ours
49.8M
401.5
83.5
Table 4: Top-1 accuracy on ImageNet validation set. All experiments use the input size 224 x 224.
Throughput is measured on Tesla V100 GPU using [57].
From Table 4, we can find that improvements are made across different transformer architectures. For the lightweight baseline CoaT-Lite [26], about percent improvement can be observed. By utilizing stronger backbones such as DeiT [9] and Swin [32], our anti-aliasing variants also outperform their "aliased" counterparts by clear margin. While T2T [7] utilizes the soft splits with overlapping and re- structurization strategies that could potentially reduce aliasing, our module still yields improvement, which manifests the consistent benefit from anti-aliasing strategy. Notably, the anti-alian anti-aliased Swin-S model surpasses Swin-B. B, which is roughly 2 times heavier. Due to space limits, we verify the effectiveness two downstream tasks, object detection and semantic segmentation, in the appendix.
Table 5: Ablation about varying sizes the pre- defined filter bank. Each row represents a model trained with n-sized bank. 10% and 100% denote used.
Table 6: Ablation about the external modulation. "Fil- tering" refers to the proposed filter and "External" de- notes our external modulation. The gray row shows the results when applied it to all four layers in Swin-T. T.
the percentage of ImageNet training
Top-1 Accuracy
Top-1
Accuracy Difference
Number of Kernels
Variants
Layer
10%
100%
Val
n =
43.8
81.6
Baseline
81.2
45.81
82.0
Filtering
81.7
+0.4
= 16
46.5
81.8
+ External
82.0
+0.3
24
46.1
81.5
External
1,2,3,4
80.6
-1.4
8

---

4.4
Ablation studies
While the choice of filter and placement ablations have been conducted in Section 4.1 and 4.2, there still exist potential factors that may influence the final performance. For efficiency and consistency, we perform all the following ablation studies using Swin-T with identical parameters to those in Section 4.3.
Number of filters. We increase the number of pre-defined filters to study the influence of a larger dictionary. As shown in Table 5, we can see trend of improvement from the growing bank when the number is smaller. However, larger dictionary doesn't 't help significantly when n > and further saturates. When trained with fewer data, the model benefits from relatively large filter bank which provides more details. Since more filters introduce redundancy and complexity, we choose a proper size of for all experiments unless otherwise stated.
External Anti-aliasing, In Table 6, we further analyze how much gain does the external modulation bring. It's casy to find that both filtering and modulation are beneficial. Besides, we can observe similar performance drop in Section 4.2, if we solely apply the external modular to all transformer blocks. The result in the gray row indicates that over-aggressive normalization on attention, especially on deeper layers, might be harmful.
Speed Analysis. During our implementations, the weights of anti- altisiaiasing filter are ensembled into convolution, i.e. 2dcon22 in PyTorch [58]. The process smoothing can be viewed as performing depth-wise convolution with defined kernels. Our technique only brings in negligible number of parameters since the filtering operation "depth-wise". Owing to the slower implementation of group convolution in cuDNN and PoToch,, about 4% to 5% lower throughput can be seen in Table 4. Given a better implementation, our module could further be accelerated.
4.5
Data efficiency and feature robustness
Another major curse of vision transformers is data efficiency. As VIT [1] is originally trained with hundreds of millions of images [59], previous methods have been making efforts to increase its data efficiency. We hereby investigate whether our module beneficial for data utilization. Thus we conduct comparisons by limiting the training data and training epochs. We train the baseline models as well as their anti-aliased ones using a smaller portion of ImageNet for only 100 epochs. Besides the consistent elevations in Table 7, we can find that relative improvements are even larger when the numbers of training samples become smaller. The results verify that our smoothing strategy is capable of enhancing data efficiency and improving generalization.
Top-1 Accuracy
Dataset
PCT (%)
Original
Ours
Improved
normalized
10
42.64
45.81
7.4%
# Params
FLOPs
mCE,
20
60.9
63.19
3.8%
[ID]
ImageNet-Ik
67.72
69.9
SWE
"3.30"
'60.7
71.41
72.89
2.1%
Swin- + ARM
28.4M
4.6G
59.8
50
73.6
74.92
1.8%
Table 8: Normalized mean corruption error mCE ImageNet-C,, which mea- sures the robustness towards corruptions.
Table 7:
Each row represents the accuracy validation when the model trained using percentage of the original ImageNet-Ik training set. The baseline architecture here follows Swin-T [32].
We further validate whether the proposed ARM upgrades the robustness of vision transformers towards natural corruptions on ImageNet-C [60], observed in anti-aliased CNNs [10]. Though transformers have demonstrated dominance against corruptions compared to CNNs [52, 61]. our module still enhances the robustness of the baseline structure Swin-T as shown in Table 8. We would like to emphasize that the baseline Swin-T is already very strong, with an absolute 12.7% reduction in terms of mCE comparing anti-aliasing ResNet50 [10].
4.6 Analysis of the Serial Characteristics of Transformer Blocks
We also present some analysis on understanding the serial information flow in vision transformers. It's natural conjecture that the attention signals grow more "peaked" in deeper blocks, leading to confidnnt final predictions. The gradually varying attention maps could potentially lead to different

---

degrees of aliasing in different layers. In order to study the serial characteristics in vision transformer, we perform a pioneering study on the implicit divergence between different layers.
37.5
Accuracy (%)
12.5
27.
25.0
22.5
14
16
Layer Depth
Figure 5: Accuracy of recognizing degrees of input blurring from attention maps at different layers. The classification model follows the ResNet50 architecture with 6-channel input. The attention maps are from a pre-trained DeiT- DeiT-Base on ImageNet.
Settings. We utilize a pre-trained DeiT-Base [9] model, which consists of 16 transformer blocks in total. To verify whether there exists unknown effects from different depths in vision transformer, we use 50000 images from ImageNet-val and process them with 5 different levels of Gaussian blurring.
The original 50K images are split into training split with 45K images and testing split with 5K images. By feeding the blurred images into the pre-trained vision transformer, we save their feature maps from the [0, 4, 8, 16]-th blocks and assign them one-hot labels based on the given levels of Gaussian blurring. Given these annotations, we train another estimation network examine whether the degrees of Gaussian blurring input images could be inferred from the inner representations. The network follows the default ResNet-50 [62] architecture and trained with cross-entropy loss based on the 5 classes from blurring levels. Then the model is evaluated on the testing split to recognize which level of blurring introduced. We respectively train four estimation networks using the features from [0, 4, 8, 16]-th blocks and report the results in Figure 5. Note that the accuracy is relatively low since we keep the blurring effects subtle which are hardly recognized by human.
Analysis. From Figure 5, we find that it's at least recognizable of those content-agnostic signals like Gaussian blurring on inputs earlier transformer blocks. However, these nuisance signals become hard to distinguish in deeper transformer blocks and result in a random guess. The results verify that, similar with the hierarchical design in CNNs [63. 37], vision transformers also have serial characteristic that gradually extracts high-level features from inputs. In consequence, different levels of aliasing occur with different instances and different layers. These observations indicate a dilemma between dealing with aliasing or truly-valuable high frequency information.
Furthermore, the accuracies (about 20%) on deeper transformer blocks show that the nuisance signals are filtered and the estimation process approximates a random guess. The comparative results indicate focusing on aggregating high-level semanties globally while the shallower blocks refines the extracted token embeddings from inputs, which better explain why aati-aliasing on deeper layers leads to a performance drop as in Table 3.
that deeper layers in vision transformers
Discussion and Conclusion
In this work, we explore feasible solutions for mitigating the aliasing issue in vision transformers. Unlike prior works introducing modified pipelines or increased sampling rates, we investigate a versatile aliasing reduction module which is compatible with these designs. Our method has shown promising results in terms of recognition, generalization, and robustness. The relatively small but noticeable improvements from the frozen Gaussian filter suggests that the power of self-attention on images may not be fully exploited.
We also note that anti-alin aati-aliasing for modern neural networks still remains open problem. Our findings initially draw novel insights towards understanding the source of aliasing in vision transformers. As such phenomenon is currently hard to quantify. we also expect a more interpretable and advanced operation. We also plan to generalize our module to other vision tasks beyond backbones.
10

---

Acknowledgements. This work is performed during Shengju Qian and Hao Shao's internship at Amazon Web Service. We thank the GPU resources provided by Amazon.
References
[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICIR,, 2021.
[2] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-basee object detection. arXiv preprint arXtv:2012.09958. 2020.
[3] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from sequence-to-sequenee perspective with transformers. In CVPR, 2021.
[4] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun. Mario Lučić, and Cordelia Schmid. Vivit: video vision transformer. arXiv preprint arXiv:2103.1569) 2021.
[5] Harry Nyquist. Certain topics in telegraph transmission theory. Trunsactions of the American Institute of Electrical Engineers, 1928.
[6] Rafael C Gonzalez, Richard E Woods, et al. Digital image processing, 2002.
[7] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint 10X.::2001///8000 2021.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint a00 14294, 2021.
[9] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training fata-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[10] Richard Zhang. Making convolutional networks shift-invariant again. In ICML. 2019.
[11] Weisheng Dong, Xin Li, Lei Zhang, and Guangming Shi. Sparsity-based image denoising via dictionary learning and structural clustering. In CVPR, 2011.
[12] Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets, In BMVC, 2020.
[13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[14] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: 180.00000000000 2018.
[15] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
[16] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[17] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[18] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, 2021.
[19] Fangrui Zhu, Yi Zhu, Li Zhang, Chongruo Wu, Yanwei Fu, and Mu Li. unified efficient pyramid transformer for semantic segmentation. In ICCV, 2021.
11

---

[20] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer. arXiv preprint arXiv:2012.09164, 091664,, 2020.
[21] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12091 2021.
[22] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[23] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML, 2020.
[24] Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. In ICCV, 2021.
[25] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
[26] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers, 2021.
[27] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103. 14899, 2021.
[28] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. arXiv preprint arXiv:2103 15358, 2021.
[29] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103. 11866, 2021.
[30] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2 15808, 2021.
[31] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXhv:2104.05707 2021.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint WxXi:2213.144030 2021.
[33] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.. 12122, 2021.
[34] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song. Andreea Gane, Tamas Sarlos. Peter Hawkins, Jared Quincy Davis. Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR, 2021.
[35] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mip-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
[36] Yann LeCun, Bernhard E Boser, John Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with back- propagation network. In NIPS, 1990.
[37] Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin. effective anti-aliasing approach for residual networks. arXiv preprint arXiv:2011.1067,, 2020.
12

---

[38] Samarth Sinha, Animesh Garg, and Hugo Larochelle. Curriculum by smoothing, NeurIPS, 2020.
[39] Alan Oppenheim, John R Buck, and Ronald W Schafer. Discrete-tine signal processing. Vol.
2. Upper Saddle River, NJ: Prentice Hall, 2001.
[40] Jorge Jimenez, Diego Gutierrez, Jason Yang. Alexander Reshetov, Pete Demorcuille, Tobias Berghoff, Cedrie Perthuis, Henry Yu, Morgan McGuire, Timothy Lottes, et al. Filtering approaches for real-time anti-aliasing. SIGGRAPH, 2011.
[41] James Lee- Lee-Thorp, Joshua Ainslie. Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824, 2021.
[42] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1800:088 2018.
[43] Pascal Getreuer, Ignacio Garcia-Dorado, John Isidoro, Sungjoon Choi, Frank Ong, and Peyman Milanfar. Blade: Filter learning for general purpose computational photography. In ICCP, 2018.
[44] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. TOG, 2015.
[45] Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In CVPR, 2019.
[46] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint UrXiv:1405.3860 2014.
[47] Jonathan Korein and Norman Badler. Temporal anti-aliasing in computer generated animation. In SIGGRAPH, 1983.
[48] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[49] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In NIPS, 2018.
[50] Microsoft. Swin transformer. htttps//giithen htttps///gttbb.cc///iicrooo//// Sww.sswiiiinraaaaarrr.rrr 2021.
[51] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[52] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. arXiv preprint arXiv:2103. 14586, 2021.
[53] Zhouxing Shi, Huan Zhang, Kai- Wei Chang, Minlie Huang, and Cho-Jui Hsieh. Robustness verification for transformers. In ICLR, 2020.
[54] Facebook Research. Deit. httpt//pr.ggggtttooooo/ccocoooee 2021.
[56] YITU. T2t vit. httppp///ggtaa...ooooo/oooaaaaaa
[55] Weijian Xu. Coat.
2021.
[57] Ross Wightman. Pytorch image models.
pytorch-imann imw-immmdaess.......---m-mmms 2019.
[58] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.0172 2019.
[59] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
13

---

[60] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
[61] Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Ye Yu Chen, and Cho-Jui Hsich. On the adversarial robustness of visual transformers. arXiv preprint 1rXiv:2103.15670 2021.
[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[63] Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-enn: hierarchical deep convolutional neural networks for large scale visual recognition. In ICCV, 2015.
14


Reviews:
[{"review_id": "ZLhhRIj8wTm", "reviewer": "Reviewer_HeAy", "summary": "This paper analyzes the uncharted problem of aliasing in vision transformer and explore to incorporate antialiasing properties. To handle that, they propose a simple yet effect module named Aliasing-Reduction-Module(ARM) to smooth the self-attention output. The ARM contains a set of filters to augment the attention layer output. Extensive experiment results show such approach achieves a clear gain but with a mirror GFlop increase along with more efficiency and robustness. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1. Overall, the motivation is very good and clear. Rather to design another transformer architecture, this paper tries to solve the aliasing problems brought by the tokenization process of VIT. This is an interesting problem. Also, it is complementary to many transformer design studies. \n\n2. The writing is good and easy to follow. \n\n3. The design of ARM is well motivated and the design of filtering process is well explored in both method and extensive experiments. Unlike tokens in natural language that stand for separated words, the tokens in vision transformers commonly represent manually-defined windows. As image contents may vary, the pairwise relationships between these tokens may be ill-posed and contain considerably amount of high frequency noise. It’s intuitive and novel to “smooth”the “aliased interactions” between tokens. To this end, this work proposes a simple yet effective design. The experiment results across different transformer architectures also validate the effectiveness of the plug-in design. Meanwhile, the work chooses to directly smooth the attention maps, which is compatible with other advances such as incorporating convolution into the patchify stem.\n\n4. The findings in Section 4.1 and 4.2 are useful and crucial, where the choices of filter types and placements are explored. It’s interesting to observe that anti-aliasing techniques conducted on early transformer blocks are useful while too much smoothing on deeper layers might be harmful. The authors conduct extensive comparisons and discussion to explain the reason. These conclusions also provide some insights and directions to boost the development of vision transformers.\n\n5. The proposed methods can be easily deployed into different transformer frameworks and leads to performance gains and more robust. These are well studied in experiment part.\n\nRather than tricky improvements on Vision Transformer ( I am tired with several attention layer extensions with SOTA performance),  this paper points out an important and potential problem of current VIT design named  Aliasing  Effect brought by tokenization process. It also give a simple yet effective solution.  Based on all the points, I give my init rate as accept. \n\n \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No major weakness.\nSeveral points which are needed:\n\n1.  During implementation, can the ARM module directly put after the softmax computation? Since the attention maps have lots of high frequency noise, i.e. aliasing effects. It seems more straightforward to anti-alias the attention scores compared to the post-projection features.\n\n2. The anti-aliasing operation introduces new hyperparameters before training, such as the parameters of the selected Gaussian kernels. It’s worth noting about how to choose the proper kernels for different architectures and the training stability.\nHere are several detailed concerns:\n\nFruther suggestions : \n\n1,  What about put both Pre-defined Filter Bank and learnable convolution(Depth-Wised)  into ARM ? (I wonder whether it have more performance boost).\n\n2,  is input of ARM attention maps or value output of self-attention? Fig-3 looks a little confusing. I believe this is value output of self attention.\n\n3,  Better put the figure on the top of pages. \n\n4, Better to show out experiment results on more exsiting transformer frameworks. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "TPGWEPjz-sj", "reviewer": "Reviewer_EQZA", "summary": "This paper proposes a plug-and-play anti-aliasing reduction method to solve the aliasing problem in the vision transformer. It analyzes the effectiveness and generalization ability in multiple tasks and various vision transformer methods. Experimental results demonstrate the effectiveness of the proposed method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a simple plug-and-play anti-aliasing reduction method. It analyzes the effect of the different filters (e.g., Gaussian filters, learnable convolution filter, pre-defined filter bank) on the anti-aliasing reduction and proposes an anti-aliasing reduction based on the pre-defined filter bank. The corresponding ablation studies in Section 4.1 demonstrate the effect of the anti-aliasing filters.\n\nThere are some unclear points in the paper. \n\nThe aliasing in the transformer may be caused by low-resolution of the image patches. The analysis in Section 3.1 is not clear and does not support the motivation of the paper. \n\nIt is not clear why the pre-defined filter bank performs better than the learnable convolution filter. In addition, the pre-defined filter bank needs to determine the number of the pre-defined filter dictionaries. Does the number of the pre-defined filter dictionaries affect the final results?\n\nThe contribution of the paper is marginal. For the aliasing, it is well-known that using Gaussian blur or other blur kernels can reduce such xx to some extent. As shown in Table 1. Using Gaussian filter is able to improve the performance. This is like an implementation trick. In addition, the paper does not provide insightful analysis on the proposed method.\n\n\n---\nThe authors solve my major concerns.  It would be better to include the corresponding materials of the rebuttal in the revised paper properly.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the paper is not explained clearly. More experimental results should provide to analyze the limitations of the proposed methods. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "D0q7OKEGd4", "reviewer": "Reviewer_fmAf", "summary": "The paper discussed the aliasing effects for vision transformers and proposed a simple yet effective fix to it — applying anti-aliasing filters on the attention maps. The authors demonstrate such an improvement brings 0.5% - 1% performance gain on ImageNet classification tasks over two different vision transformer architectures. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n* Nice finding. I had similar concerns on ViT families to split the image into non-overlapped patches. It's good to see a minor fix could boost performance. \n* Thorough experiments. All the experiments deliver useful messages and come with a good analysis of the experimental results. \n* The presentation is clear. It's not hard to follow the paper. \n\n\nCons:\n* There's a lack of quantifiable measures on the aliasing effects. Besides, there are also not many qualitative examples for attention maps before and after anti-aliasing. As a result, it's not 100% convincing to me whether the proposed improvement and the performance gain it brought is due to \"anti-aliasing\". \n* I assume this effect might be more severe on dense prediction tasks, potentially resulting in block effects on the feature map. Have you considered applying the approach to those models?\n* There's no discussion on whether a non-patch-based transformer would have similar issues, e.g., non-local blocks. \n\n\nTable 2: How do you apply ARM on patch embeddings? This is not straightforward to me. Have you considered adding an anti-aliasing feature to the input feature map before any embedding computation? \nIn Table 2, which filters setting do you chose? Will the placement strategy's relative performances vary for different filter settings? \n\n\nYou should be able to generate an attention map at a much higher \"sampling rate\" during training through overlapped image blocks followed by anti-aliased downsampling. Would anti-aliased downsampling on that high-res attention map get the ideal attention map with downstream performance gain? If yes, how about learn a filter to mimic that attention instead?\n\n\n\n\nTable 3: The proposed approach seems to work only for the first layer. It is not clear whether this finding is generic to all different transformer models. And the author's justification for the experiments is not very convincing. Do you think Table 3's finding (only applying early layers matters) is generic for all ViT families and why? Have you visualized the attention maps to justify your conjectures?\n\n\nDo you maintain the attention maps magnitudes through the filtering/smoothing operations?\n\n\nThe choice of the filter bank is quite adhoc. What are the other fixed filterbank options you have tried? How about DCT, or wavelets? Compared to fully learnable filters options, is this choice of fixed filterbank matter or the fact that the filters are low-rank matter? How about do learnable low-rank 5x5 filters? \n\n\n\n\nIs there any way to quantify the aliasing effects? How to convince readers the performance improvement brought by the proposed approach is due to the anti-aliasing effect? The author should consider building a stronger connection between aliasing effects and different discussed options, e.g., through spectral analysis of the attention map or measuring some form of invariance or equivariance. For instance, the Zhang et al. ICLR19 paper [9] measures the amount of shift-invariance of CNNs w/o anti-aliasing convs. No quantifiable metrics for aliasing is a primary concern for me as there could be other explanations why adding a conv filtering on the attention map could help improve tasks performance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Bp5WICUE7C6", "reviewer": "Reviewer_yhPz", "summary": "The paper proposes an Aliasing Reduction Module (ARM) consisting of an anti-aliasing filter and an external smoothing module. The paper further iterates that the module is lightweight and compatible with most existing transformer architectures. Furthermore, the method shows comparisons between a naive Gaussian filter, a Learnable Convolution Filter, and the proposed Pre-defined Filter Bank. Experiments are performed to show that placing the anti-aliasing module after the Self-Attention block yields the best results. Additionally, the method illustrates that placing the anti-aliasing module in earlier layers translates to the performance boost while inserting the module at deeper stages leads to reduced performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is easy to understand. \n- Results are shown on various Vision Transformers, i.e., CoaT, DeIT, and SwinT.\n- Results are shown on multiple high-level vision tasks of Image Classification, Object Detection, and Semantic Segmentation.\n\nI have the following questions:\n\n- It is a very common practice to use overlapping patch embedding [a, b]. Would these architectures suffer from aliasing? Did you try employing your module in these algorithms?\n- The performance of the proposed ARM is demonstrated across various transformers with 224x224 image size. It would be interesting to study the impact of ARM on bigger image sizes (e.g. 384x384) where the jagged effect might occur less frequently.\n- Most recent methods such as [c, d] use smaller patch sizes at the patch embedding stage. However, no comparisons are made between these algorithms with the proposed method on alleviating aliasing in attention maps and their performance.\n- For Object Detection and Semantic Segmentation, it would be interesting to see visual examples where baseline methods show poor performance and when the proposed ARM is employed, the results are improved.\n- SwinT shifted window version provides the best results for Semantic Segmentation on ADE20K dataset. In this paper, why is the sliding window variant of SwinT used in the experiments?\n- The method should elaborate on how the proposed module is different from the previously used module in [42].\n- The method hypothesizes that the jagged artifacts occur due to the extraction of non-overlapping patches at the patch embedding stage. The visualization is shown for the same in Figure 1. However, the method does not show any visualization for algorithms such as [7] which use overlapping patches. As stated in Lines 32-34, does the occurrence of aliasing decreased?\n- In Section 3.2, the method gives a description of the Gaussian filter, a Learnable Convolution Filter, and the proposed Pre-defined Filter Bank. The effects of each of them on a baseline are shown in Table 1. However, no visual comparison is shown on attention maps for the three methods to describe how well the three algorithms perform.\n- In Figure 1, the visualization of attention maps are shown for the DeiT-B architecture. Would this visualization differ for different architectures like SwinT and CoaT etc?\n\n[a] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv 2021.\n\n[b] Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv 2021.\n\n[c] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv 2021.\n\n[d] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv 2021.\n\n* * * * *\n### **Post Rebuttal Comments**\n\nThe authors have responded to my concerns with thorough experiments. The proposed method provides favorable results. Therefore, I am also leaning towards acceptance of this paper. Since the authors showed in rebuttal that the overlapping patch embedding models also suffer from aliasing, I would suggest including these findings in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have sufficiently discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper initially receives mixed reviews. The authors successfully convince the reviewers with the rebuttal. The reviewers discuss extensively the strengths/weakness of this work and in the end all are in favor of accepting this paper. The area chairs agree with the reviewers' recommendation to accept this paper. 
Decision: Accept (Poster)