Alias-Free ViT: Fractional Shift Invariance via Linear Attention
Authors: Hagay Michaeli
Daniel Soudry
Venue: NeurIPS 2025 poster ()
Keywords: Vision Transformers
alias‑free
aliasing
anti‑aliasing
shift invariance
fractional shifts
linear attention
cross‑covariance attention
translation robustness
ImageNet
XCiT
Primary area: deep_learning
TL;DR: We introduce an alias‑free ViT that combines anti‑aliasing with linear cross‑covariance attention to achieve fractional shift invariance, delivering ~99% consistency to sub‑pixel shifts and stronger translation robustness with competitive accuracy.

Abstract: Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance.  
Specifically, Vision Transformers (ViTs) are not translation‑invariant and are more sensitive to minor image translations than standard convnets.
Previous studies have shown, however, that convnets are also not perfectly shift‑invariant, due to aliasing in downsampling and nonlinear layers. 
Consequently, anti‑aliasing approaches have been proposed to certify convnets translation robustness. 
Building on this line of work, we propose an Alias‑Free ViT, which combines two main components. 
First, it uses alias-free downsampling and nonlinearities. 
Second, it uses linear cross‑covariance attention that is shift‑equivariant to both integer and fractional translations, enabling a shift-invariant global representation.
Our model maintains competitive performance in image classification and outperforms similar‑sized models in terms of robustness to adversarial translations.

Alias-Free ViT: Fractional Shift Invariance via Linear Attention
Hagay Michaeli
Daniel Soudry Technion
Technion Haifa, Israel hagaymi@campus wtw.aaappsssttcioon....mm technion. ac il
Haifa, Israel
daniel. wtw.ellssourr@gmmaa..oom com
Abstract
Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hin- der their potential performance. Specifically, Vision Transformers (ViTs) are not wranslationiinvariant and are more sensitive to minor image translations than stan- dard convnets. Previous studies have shown, however, that convnets are also not perfectly shift-invariant, due to aliasing in downsampling and nonlinear layers. Consequently, anti-aliasing approaches have been proposed to certify convnets translation robustness. Building this line of work, propose Alias-Free VIT, which combines two main components. First, it uses alias-free downsam- pling and nonlincarities. Second, it uses lincar cross-covariance attention that is shift-equivariant to both integer and fractional translations, enabling a shift- invariant global representation. Our model maintains competitive performance in image classification and outperforms similar-sin sized models terms of robustness adversarial translations
Introduction
Transformers, primarily designed for language modeling [88] have become dominant vision tasks [6 34. Since they were originally designed for sequential data, their underlying attention mechanism is not sensitive to the locality of information in visual data. As result, Vision Transformers (ViTs) exhibit a lack of shift-invariance, shortcoming that becomes evident in cases where small image translations lead to significant deviations in output 25 ST.
To mitigate this gap, many studies have been conducted the integration of convolutional priors, such as spatial hierarchy and nhift-invariance, into ViT architectures. For example, approaches include hierarchical patch merging (39], the incorporation of convolutional layers [61], and the design of relative positional encodings [62] Furthermore, some works have drawn parallels between self- attention and dynamic convolutions
27 motivating reinterpretations of the attention mechanism
through a convolutional lens.
Despite being more spatially aware than transformers, convnets are not perfectly shift-invariant due to aliasing introduced by strided convolutions and pooling layers 67]. This has to line of research that aims to restore shift-invariance, by methods including anti-aliasing filters
21 23 32 40 667 [ and adaptive sampling techniques (6 46). Building on these advances, recent works have adapted such methods for transformer architectures. For example, Adaptive Polyphase Sampling (APS) has been employed to achieve cyclic thift-equivariance ViTs [ 47 However, despite the latter approach efficiently guaranteeing shift-invariance for integer pixel cyclic shifts, it falls short in fractional (i.e., sub-pixel) shifts and "standard" shifts (i.e. imitating camera translation)
40 51).
'Our code is available at hgghhbb.commmmichhlllllaaaaaaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

x
nw
AF-Down 4x
xD
Hx HxWas x3
BN
AF-Up 2x
GeLU
AF-LN
XCA
AF-LN
AF-PI
AF-LN
AF-MLP
AF Class
Attention
Linear
AF Patch Embedding
AFT Block
AF-LN
XCA
Limear
AF
Figure 1: Overview of the Alias-Free Vision Transformer (AFT) architecture. The input image is first processed by alias-free patch embedding module composed of convolutional layers (Conv), batch normalization (BN), and alias-free activation, composed of upsampling (AF-Up). GELU and downsampling (AF-Down). The result is reshaped to token matrix form and fed through L Alias- Free Transformer blocks, each consisting alias-free layer normalization (AF-LN), cross-covariance attention (XCA), alias-free local patch interaction (AF-LPI), and alias-free MLP (AF-MLP) layers, interconnected by residual connections. The result concatenated with a learnable class token embedding and fed into two Alias-Free Class Attention blocks composed of an XCA layer and an MLP applied on the class token, The final representation is the updated class token, which is fed into final linear classifier. Detailed explanations of each component provided in Section 3
As aliasing is primarily related downsampling layers, which are not frequently used in ViTs, only few studies have been conducted on integrating ctusiing-rduuctinn techniques to achieve shift- invariance in ViTs. Qian al. 442 propose plugging a low-pass filter post self-attention reduce aliasing, however, this only provides partial solution, as it does not resolve the inherent lack of shift-equivariance self-attention and aliasing in other nonlinearities. Recent works study aliasing in latent diffusion models [2 64 70 that typically include attention layers, in order to improve their consistency i.e. in video generation. However, these works do not address the main issue in the self-attention operation. A similar problem may also hinder transformer neural operators which have as aliasing has been shown to related to discretization errors
recently become popular 28
5 44 69
Another emerging direction focuses on linear and softmax-free attention mechanisms, initially proposed to reduce the quadratic complexity of standard attention in large language models (LLMs)
59 In the vision domain, models such as XCIT [16] and SimA [35] demonstrate that alternative attention formulations, e.g., using cross-covariance
7
linear attention, can maintain competitive performance without directly computing a full attention map. Beyond improved efficiency, we now show that such mechanisms enable designing transformer-bat based architecture that is shift- equivariant, similar convnets.
Contributions. In this paper
We present in Section certain class of shift-equivariant attention layers (SEA), including linear attention and cross-covariance attention, which is useful for vision tasks.
We design in Section a shift-invariant, alias-free Vision Transformer (AFT) using cross- covariance attention and alias -ree nonlinearities and show it has competitive performance in image classification.
We show in Section 4 that the AFT robust (~ 99% consistency) fractional cyclic shifts, and significantly more robust to practical translations than other similar models, albeit with increased computational overhead due to the alias-free components.

---

2 Methods
2.1 Preliminaries: the Vision Transformer
The Vision Transformer (ViT) [15] transfers the Transformer architecture [58] from text to images by interpreting an image as sequence of visual tokens. Given an input RR Cx the image is partitioned into N = HW non-overlapping patches of resolution p x p. Each patch is flattened n²
and projected by learned linear layer into D-dimensional embedding, forming X NR For classification, learnable "class" token is prepended to X and later serves as global representation, propagated into the classification module. Learnable absolute positional encodings P E RNN+1)x are then added compensate for the
ietpuuttttionniviiiain of self-attention.
The resulting sequence X = X + P is processed by L identical Transformer encoder blocks. Each block composed a Multi-Head Self- Seff-Attention (MHSA) module with a two-layer Feed-Forward Network (FFN), interleaved with Layer Normalization (LN) and residual connections:
x x(-11) MHSA
LN X (X(-1))
8 8(0 FFN LN (8(3),
where <=1, L and 10) = X.
Multi-Head Self-Attention. Each of the h heads linearly projects the input into queries, keys and values, Q = XW,,, qr K
XW,,, V = XW,, with Wi WA. W. E R hxdr and dN = D/h.
Self-Attention then computed as
SA (X) softmax (Qk /V da V.
The outputs of all heads are concatenated and projected back to R by a final linear layer.
Feed-Forward Network. The FFN first expands the embedding dimension to 4D, applies GELU activation [30]. and projects back:
FEN (X) = W2 GELU (W,X + b1) +b2
with W1 E IDDD and W2 E DDAD.
2.2 Linear Attention
Notably, Equation 3 requires computing QKT E NNN explicitly, which has a quadratic cost in the number of tokens. This has motivated many linear complexity variants of self-attention 33 59 In vision, SimA removes softmax entirely by maintaining stability using & normalzlee Q and K 35 XCIT mixes channels instead of tokens with cross-covariance attention (XCA) (16),
XCA(X) V softmax (R (KT Q/T).
where Q, K are Ez-noraalieed along the token dimension and > 0 is a learnable temperature.
2.3 Alias-Free Vision Transformer
Next, we describe our proposed model, replacing every ViT component that is shift-equivariant with modification that restores equivariance. For brevity, the analysis is given for a one-dimensional signal. The same principles can be applied in the two-dimensional case by viewing the sequence of N tokens in two-dimensional representation, namely X E IR X
Similar to Karras al. 332 and Michaeli et al. [40], view the input discrete sampling of an underlying
continuous signal. The main difference from a standard convnet emerges
In practice two- noo-dmmensionl signals tokens are also stacked into one mie-dimensional sequence (formally the row-stack). Nevertheless, shift- equivariine in the two dimensions is still maintained under this representation.

---

after tokenization: the patch-embedding matrix XERNXD stores the sequence length N along its first axis, and the embedding dimension D the second axis. Thus, the spatial and channel roles are swapped compared to convnet feature maps, where the channel index comes first and spatial indices follow. Throughout our analysis, we therefore interpret the 1) columns of X as D "channels"" of a length-N N one-dimensional signal. Maintaining shift-equivariance then amounts to ensuring that each column transforms equivariantly under fractional translations in the continuous domain.
Shift invariance and equivariance. We reuse the definitions of Michaeli et al. [40]. Let x be discrete signal, I"' its sample spacing, and 2t)) the unique 27 -andd- limited signal with z|n] = z x(n]). For any (possibly non-integer) shift A E E.
Taa(N]
2(n'T A).
An operator is shift-equivariant f(TAz) = (a) and shift-invarinn if f(T2a) = f2) for all and A.
In some cases, we may claim that value is shift-equivariant. This is slight abuse of the definition to say the overall operator computing this value is shift-equivariant w.r.t. the input signal.
Patch embedding. Algebraically, the tokenization process described in Section 2.1 iis equivalent to a convolution with kernel size p. stride p, and D output channels. By separating this stride-p -p convolution into a stride-1 convolution followed by an dires-free downsampling [21]. the composite operator becomes shift-equivariant [32
40 67 However, this requires inserting single low-pass filter with cut-off 1/p, which would severely attenuate high-frequency content, especially for large patches. Instead, employ convolutional patch-embedding that replaces the single stride-p layer with short convnet of progressively smaller strides, often used in hybrid models [16] [4 61 66). Here, can avoid aliasing by plugging a low-pass filter with larger cut-off before each downsampling
layer, and by using an alias-free activation function (32 40). This gradual approach still enables the network to learn high-frequency features, despite the anti-aliasing components.
Positional encoding. Absolute positional encoding injects the global index of cach token and therefore breaks bhitt-qquivriianc. Several relative schemes have been proposed that depend only on pairwise offsets and thus preserve chift-equivariance at the token level R 39
50] These methods,
however, do not guarantee equivariance to pixel-level translations, as they cause the patch contents themselves to change. Notably, the convolutional patch embedding already breaks permutation invariance of the tokens, possibly reducing the need for additional positional signals. Moreover, recent studies demonstrate that hybrid transformers can learn effectively without any explicit positional encoding [3 68). Consistent with this observation, we find empirically (Section 4.4) that positional encoding may be unnecessary in architectures that include convolutional layers inside the transformer blocks, such as XCIT [16].
Shiit-Equivariant Attention. We next show a class of attention operations that, by removing the softmax in Equation (3). are shift- shift-equivariant. This primarily includes the linear attention, which, although still less common, is attractive for its lower complexity [33 54] and yields competitive vision results (35] Formally,
SEA (X) = QS (VTVV),
(6)
where we keep the existing notation Q XW K XW = XW, and let f DDD D be arbitrary function.
We now establish the desired property in three steps.
Proposition Q, K and V are shift-equivarianl.
Proof. Each column of Q. K or V is fixed linear combination of the columns of X. As the columns of X are merely the channels of the same signals, they translate together, and any linear combination of them is also shift-equivariant.
Proposition 2. f VV is shift-invariant.

---

Proof sketch. Consider the matrix entry (K Vi K2 Vs, where K2 and Vj denote columns. By Parseval's theorem,
1 - [ Ki(w) Vj (w) du. 2x
K,V,
fractional translation by r multiplies both Fourier transforms by the same phase factor e which cancels in the product. Hence, every entry of K' V remains unchanged due to translation (see formal proof in appendix (A). An important observation is that since K V is inttiiivarint any matrix operation can be applied it without compromising this property.
Proposition 3. V = Q (KTV) is shift-equivarant.
Proof. Column j of V satisfies
V =
i.e. sum of shifl-equivariant columns Qi (Proposition 1 scaled by shift-invarianl coefficients (K" (KTV),, (Proposition P). The resulting column therefore shift-equivarian.
As mentioned above, since V is shift-invariant, any matrix operation f can be applied on it without compromising the overall shift-equivariance, By the same argument, Q is also shift- invariant; thus a row-wise softmax on K Q as used in XCA (Equation (5) is permissible. Note that in XCA, the columns of Q and K are l2- no-oomallieed along the token dimension, which as well maintains thift-qquivariance 40
MLP. The MLP linear layers apply the same linear transformation on all tokens, maintaining shift-equivariance in Proposition]] [ However, pointwise nonlinearities induce aliasing that breaks fractional bhift--quivariance. This can be solved similarly to [40] by alias-free activation function, which includes upsampling before the nonlinearity and downsampling back after low-pass filtering.
Layer normalization. Per-token LayerNorm rescales each column differently and breaks equiv- ariance. The same problem has been addressed by Michaeli et al. [40], by using a global variant, namely
XiJ
Xi-P Vo² +c'
P: = +.=bExa Xij, o² = NDD Σ(x - -2)
4
where z is computed per token and 0² per layer.
Class token. Prepending the "class" token interferes with the signal representation of the columns of the embedding matrix, and breaks shift-qquivariance. A simple solution is to instead construct a global representation using global average pooling over the embedding dimension after the last transformer block, i.e.
X
X (D)
ERD
(8)
similar to the common approach in convnets and few other ViTs shift-equivariance is maintained, yields shift-invariant global representation
24
39
65
This, assuming
40
Class Attention. Some ViTs append the class token only after L patch-only blocks and update
through few class-attention (CA) layers [10 16 56]. These layers usually fix the patch embeddings and update only the class token embedding by attending it to itself and the frozen patch embeddings. We find that when using SEA after concatenating the class token, Propositions to hold w.r.t. the patch tokens, and additionally the class token remains shift-invariant (see proof in appendix (A). Retaining similarity to the original class attention blocks, we only propagate the class embedding through the MI.,, which also prevents aliasing in the nonlinearity. We use this approach instead of global average pooling since we find it performs slightly better.
5

---

3 Implementation
We implement the Alias-Free Vision Transformer (AFT) based on XCIT architecture 16 XCIT
replaces standard self-attention with cross-covariance attention, described in Equation (5) This, by Proposition 3 preserves shift-equivariance. The remaining modifications focus on eliminating aliasing in the patch-embedding, local patch interaction (LPI), and MLP blocks. The overall model is presented in Figure
Conv Patch Embedding. XCIT patch embedding (PE) module is sequence of blocks composed of strided convolution, batch normalization, and GELU. Similar to [40), we replace the strided convo- lution with stride of 1 and insert alias-free downsampling at the end of the block, implemented by truncation of freuunreqeescss the Fourier domain, using Fast Fourier Transform (FFT) [22 45
We replace zero-padding with circular padding. We use alias-free activation functions by wrapping the GELU activation with upsampling and alias-free downsampling layers [32 40). In contrast to 40 find that replacing the GELU with polynomial activation to get perfect shift-invariance degrades the model performance significantly. Conversely, we find that keeping GELU affects the manslation-robusteess marginally. We do not add positional encodings and do not prepend class token at this stage; the class token is appended only before the class attention blocks.
AFT Block. Following the patch embedding, XCiT is composed sequence of Blocks, each consisting of three components: cross-covariance attention (XCA), local patch interaction (LPI), and MIP. As argued in Section 2.3 the XCA already shift-equivariant. The I.PI consists of two convolutional layers, batch normalization, and activation layers, and we treat each of them in the PE, forming an alias-free version we denote AF-LPI. The MLP applies shared two-layer FFN on each token, and can be viewed as convolutional layer with kernel size 1. Therefore, the only required modification to form the alias-free variant (AF-MLP) is converting the activation function into an alias-free activation. We replace all LayerNorm instances, applied before XCA, LPI, and MLP, with the alias-fres layer norm described Section 2.3
Classification. After the last AFT block append learnable class token and apply two AF class attention blocks. Each block consists of AF-LN, an XCA layer, and an MLP applied only to the class token. Residual connections
used around XCA and the MLP, mirroring the AFT blocks. By the SEA properties, the patch tokens remain shift-equivariant and the class token is shift-invariant (see Proposition 4 The final prediction is obtained by linear classifier applied to the class token.
Experiments
We evaluate Alias-Free Transformer (AFT) the ImageNet dataset [2] and compare its accuracy and shift consistency with the baseline XCTT model. We additionally compare our method with the adaptive polyphase sampling (APS) approach [13 47. which implement by replacing the strided-convolutions in the PE with stride- convolutions followed by APSPool 6 and using the standard class attention blocks for classification (maintaining integer shift-invariance. We the nano and small XCIT versions with 12 layers and patch-size 16, processing inputs of size 224 x 224.
We train all models for 400 epochs, following the XCIT training recipe [66, using PyTorch [41], single machine with NVIDIA RTX A6000. We observe a slight improvement for the AF models when training with a smaller batch size; therefore, we reduce the batch size from 1024 to 512 for the AF versions (See additional details in Appendix D.I).
In Sections 4.1 and we evaluate the baseline, APS, and AFT models using eyclic translations and implement m/ in-Iractional translation by translating m pixels the n-upsampled image using iin-interpollaton,
our and the APS models were initially designed under those assumptions. In Section 4.3 we use more "realistic" types of translations, and add additional publicly available ViTs to the comparison.
4.1 Accuracy and shift consistency
The results in Table I (left) show the classification accuracy and consistency, defined as the percentage of validation samples whose prediction did not change after random translation. The alias free

---

Table 1: ImageNet accuracy and eyclic shift consistency. Left: the percentage of test samples that did not change their prediction following a random translation. The alias-free models have similar aceuracy to the baseline models, and much higher consistency in both integer and half-pixel translations. Right: Adversarial accuracy is defined as the percentage of correctly classified samples the worst case at the corresponding grid (Equation (9)) See
Shift consistency is defined as
Appendix B for additional results.
Test accuracy
Integer shift consist.
Half-pixel shift consist,
Adversarial integer grid
Adversarial half-pixel grid
Model
XCiT-Nano (Baseline)
70.4
83.7
82.0
50.9
52.9
XCTT-Nano-APS
68.4
100.0
87.5
68.4
62.9
XCiT-Nano-AI (ours)
70.5
99.2
98.7
69.9
69.5
XCiT-Small (Baseline)
82.0
91.4
89.8
70.9
71.3
XCiT-Small-APS
81.3
100.0
94.0
81.3
78.2
XCiT-Small-AF (ours)
81.8
99.5
99.4
81.3
81.1
models have similar accuracy to the baseline models, and much higher consistency in both integer and half-pixel translations. The APS models, on the other hand, achieve near- 100% consistency under integer translations, as expected. However, they have a more modest improvement in consistency to half-pixel shifts.
4.2 Adversarial robustness
To show practical implication of shift consistency, ask whether adversary, free to translate the image within prespecified grid, can find any shift that flips the label. We define adversarial accuracy the fraction of images that classified correctly in the worst-case this grid. In Table (righ[[(righ)),,, we show results for cyclic integer and half-pixel translations, reporting adversarial accuracy over the following grids
T integer-= = (6.))) i,j
Thair =
The AFT models maintain high accuracy under both integer and half-pixel attacks, having 2% relative accuracy reduction the nano version and less than 1% reduction the small model. This is in contrast to the baseline models, with relative accuracy reductions of 25% and 14% in the nano and small models respectively. As expected, the APS models maintain their accuracy under the integer grid adversarial attacks. Their accuracy under half-pixel grid attacks decreases slightly in comparison to the baseline models. The reason for this is that the APS is invariant to any two half-pixel translations, as these differ in exact integer translations. We expect the APS accuracy to decrease more under arbitrary fractional translations, as can be seen in Section 4.3 and 40) See Appendix B for additional results.
4.3 Robustness realistic shifts
The experiments above use cyclic translations, which may leave unnatural image artifacts in realistic cases (where the input is not sample of some periodic signal). We therefore test two more realistic perturbations and measure adversarial accuracy exactly as in Section 4.2 See Appendix C for visualizations of the used translations.
Crop-shifts. The image first center-cropped, then cropped in offsets (8z, 8y) with (6.],, 15y s. This mimics a camera translation that moves content out of view instead of wrapping it around.
Bilinear fractional shifts. To simulate sub- sub-pixel motion, we translate the image by (8./6, 6//6) with (8.. jyy I s, using bilinear interpolation. Here, leave an edge of one pixel of the original image in each direction avoid edge artifacts.
We compare the baseline, APS and AFT XCIT-Small models with other publicly available trained models, in similar scale as XCiT-small (26M) (indicates number of parameters): CvT-13 (20M), Swin-T (28M), and ViT-Base (86M). We repeat these experiments with (max-shift) in the range 0
7

---

XCIT (Baseline)
82-
XCIT-APS
Roversarial Accuracy (%6)
Adversarial Ac Accuracy (%)
XCIT-AFT (ours)
80
CVT
Swin
76-
72
74-
68
4/6
5/6
6/6
Max Shift (pixels)
Max Shift (pixels)
(a) Adversarial "Crop-shifts"
(b) Adversarial Bilinear fractional shifts
Figure 2: ImageNet adversarial accuracy under realistic translations. Adversarial accuracies under (a) "Crop-shifts," simulating camera translations, and (b) "Bilinear fractional shifts," simulating realistic sub-pixel image translations. The Alias-Free Transformer (AFT) consistently outperforms baseline XCTT, APS, and other vision transformer variants (CvT, Swin and ViT), demonstrating superior robustness against realistic translations.
6. The results in Figure 2 show show model has improved robustness both these types of realistic shifts, despite not being specifically designed for them. Among the additional baselines, the vanilla ViT degrades the most under crop-shifts, whereas under bilinear fractional shifts it is surprisingly competitive and in fact better than the hybrid backbones (XCTT, Swin, CvT).
4.4 Ablation study
We conduct an ablation study on XCiT-Nano evaluate the impact each of the alias-free mod- ifications on the model performance. We train the baseline model ImageNet with one specific change, and report the results Table 2 Surprisingly, replacing the layer norm with flias-free layer-norm and replacing class attention with average pooling cause much larger degradation in accuracy in comparison to the marginal degradation in the final alias-free model. On the other hand, removing the positional encoding leads small improvement in accuracy, emphasizing that it may be unnecessary in hybrid architectures.
Table Ablation study on alias-free compo- nents of XCiT-Nano (ImageNet). Evaluation of isolated alias-free modifications to the base- line model. Alias-free layer normalization (AF- LayerNorm) and replacing class-attention with average pooling (AvgPool) result in notable curacy degradation individually. Removing posi- tional encoding slightly improves performance. The final combined alias free model retains near- baseline accuracy.
Table 3: Training runtime. Train time was mea- sured on 8 NVIDIA RTX A6000 using batch size 1024 for the baseline model and batch size 512 in the APS and AF models, due to memory constraints.
Train time [hours]
Model
XCiT-Small (Baseline)
69
XCiT-Small-APS
98
XCTT-Small-AR (ours)
487
Model
Accuracy
Change (%)
Baseline
70.4
Cyelic convolution
70.4
+0.0%
AvgPool
69.1
-1.8%
AF-LaaerNorm
69.6
-1.1%
positional encoding
70.7
+0.4%
AF (AvgPool)
70.4
+0.0%
AF (AF Class Attention)
70.6
+0.3%
8

---

5
Related work
A few studies have shown broad effect of aliasing in deep neural networks, e.g., breaking shift-
equivariance in convnets 4 67], inconsistencies in image generation [32 64 70 and breaking
continuous-disrette equivalence in neural operators S 55 69
Shift invariant convnets. For a long time, convolutional neural networks have held dominance in vision thanks to their useful inductive biases, including translation invariance. However, previous studies have shown their output can in fact change in a large extent due to small translations 4 17 This has led to extensive research to find the root causes and resolve this problem. Azulay and Weiss 4 Zhang [67] have identified infttiinvariance breaks as a result of aliasing in downsampling and nonlinear layers. Consequently, other studies suggested solving this problem by plugging a low-pass filter before downsampling 23
22 67]. and preventing aliasing in nonlincarities by using smooth 57 and by applying activations after upsampling [32 40 60). Other
activation functions [31 40
works suggested maintaining
in convnets by downsampling
adaptive grids 6 46
Specifically, the adaptive sampling method (APS) 6 46 has been shown to retain perfect consistency to integer cyclic translations, while the anti- anti-aliasing approach maintains consistency in fractional translations as well. Worth mentioning here are recent works that propose transforming the input into "canonic" shift-invariant representation 11 [1] which theoretically makes equivariance of the following neural network unnecessary.
Shift invariant transformers. Vision transformers have gained dominance despite not having the convnet priors and being even more sensitive to image translations. Some studies have proposed hierarchical ViT architectures similar to convnets [14 18
49 62). and "hybrid" architectures including convolutional layers directly [6]] Furthermore, some works have drawn parallels between self-attention and dynamic convolutions
39
27 motivating reinterpretations of the attention mechanism through convolutional lens. Few studies have taken inspiration from these studies aiming to retain shift-invariance in convnets and implemented their ideas into ViTs. Qian et al. 42] proposes plugging& low-pass filter post self-attention to reduce aliasing, partially improving consistency similar to Zhang (67]. Ding et al. [13, Sojas- et al. [47] adapt the adaptive sampling method (APS) into ViT layers, certifying consistency to integer cyclic translations. Other studies proposed more general framework for group equivariant attention 48 63 Yet, to the best of our knowledge, no other work has dealt with the invariance of ViTs to fractional shifts.
Linear attention. The standard Transformer architecture relies on softmax-based attention (58), characterized by quadratic computational complexity in the number of tokens. To overcome scalability limitations, linear and dernel-basee attention mechanisms have been proposed 7 [3), substantially reducing complexity while maintaining performance. For example, linear attention leverages a linear approximation of the softmax kernel, achieving significant efficiency gains (59]. In vision, models
like SimA 35 and XCTT [16] utilize simplified normalization schemes to replace the expensive softmax operation, enabling to avoid a direct computation of full attention maps.
Discussion and limitations
In this paper, we propose a shift-invariant alias-free vision transformer by introducing a class of shift-equivariant attention operations. We show that the AFT maintains competitive accuracy and superior robustness to fractional shifts, compared to other ViTss. We next discuss a few of our model limitations.
Polynomial activation function Michaeli al. 40 propose replacing nonlinear activation func- tions, such as GELU, with polynomial approximations. This mathematically ensures the overall activation layer (including upsampling) shift--qqivariant w.r.t. continuous domain, namely perfectly consistent to fractional shifts. In our experiments, we observe that this leads to a significant reduction performance, which caused specifically due to the activation replacement in the pateh-embedding stage (see Appendix B). On the other hand, we observe that the filtered activation function using GELU leads to a rather small reduction in consistency. Notably, the certified consistency limited to cyclic shifts and fractional shifts performed by sine-interpolation, both induce artifacts that do not

---

appear in natural images. We find that similar to the AFC, our model also has significant improvement robustness realistic translations despite the imperfect consistency to cyclic shifts.
Runtime performance The alias-free modifications we perform in our model to attain shift invariance, despite not requiring any additional parameter, cause a substantial runtime increase, as shown in Table This is mainly due to the downsampling and upsampling, which are implemented in 70), seemingly underoptimized
the Fourier domain using FFT, similar to other works [2 23 40
43
for GPU of today 220 53).
Acknowledgments and Disclosure of o Funding
The research of DS was funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of the Schmidt Career Advancement Chair in Al.
References
[1] Jean-Mare Andreoli. Convolution, attention and structure embedding. May 2019, URL httpss//arxiv..
http::///arrv.oog//a////0.0... wttgg/////990.022200 arXiv: 1905.01289.
[2] Md Fahim Anjum. Advancing Diffusion Models: Alias-Free Resampling and Enhanced Rotational Equivariance. November 2024. httttps:/arxxv..rg///// www.o//gg///.....////o 2411.0911441
arXiv: 1.09149174.
[3] Bouzid Arezki, Fangchen Feng, and Anissa Mokraoui. Convolutional Transformer-Basse Image Compression. In 2023 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), pages 154-159, September 2023. doi: 22291///AA99660022... URL htttttttp//xiiiii htwp:aawwwi..raaaaaaaeoe httrg/abs/24499 04118 arXiv:2409.0k 00:.:2499.4118 [eess].
[4] Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly to small image ?ransformations? Journal of Machine Learning Research, 20: 20:1-25, 2019. URL
arXiv: 1805.12177v4 ISBN: 1805.12177v4.
[5] Francesca Bartolucci, Emmanuel de Bézenac, Bogdan Raonić, Roberto Molinaro, Siddhartha Mishra, and Rima Alaifari. Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning. Advances in Neural Information Processing Systems, 36, May 2023, ISSN 10495258. TRL/https:// Publisher: Neural information processing systems foundation.
org/ abs. 2305 19913v2
arXiv: 19913
[6] Anadi Chaman and Ivan Dokmanić. Truly shift-invariant convolutional neural networks. Pro- ceedings the
the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, November 2020. ISSN 10636919, doi: (0.485500/rxiv.200 1. 14214. URL hntpps:// http//axxvvv.. org/ abs/ 14214v4
pages 1777-37782
arXiv: 201 14214 Publisher: IEEE Computer
Society ISBN: 9781665445092.
[7] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking Attention with Performers, November 2022. URL
http://arxiv.oxt http://aaaii..orraa//////// abs///bbb///0./. 14794 arXiv:2009.144 14794 [cs].
[8] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional Positional Encodings for Vision Transformers, February 2023. URL httt:::/arxiv.ooom hbggg//bb/12222
10882 1XXi:2102211088 [cs].
[9] Jean Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the Relationship between Self-Attention and Convolutional Layers. 8th International Conference Learning Repre-
sentations, ICLR 2020, November 2019. URL U /ttttt:ss///reeeee warxiv.org/abbl. www.oogg////o...ooooooo..ooo. arXiv: 191 1.0 .03584 Publisher: International Conference Learning Representations, ICLR.
htt//bss11111 /991.. 03584v2
10

---

[10] Marco Cotogni, Fei Yang, Claudio Cusano, Andrew D. Bagdanov, and Joost van de Weijer. Exemplar-fre free Continual Learning of Vision Transformers Gated Class-Attention and Cascaded Feature Drift Compensation, July 2023.
htttp://arxiv.org/aaatt
12292
#rXiv:2211.12222 [es].
[11] Berken Utku Demirel and Christian Holz. Shifting the Paradigm: A Diffeomorphism Between Time Series Data Manifolds for Achieving Shift-Invariacy in Deep Learning, February 2025.
http:ttt:///att www.aaawivvv...aaaaae org/abs/2502 19921 arXiv:2502. 19921 [es].
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255. IEEE, June 2009. ISBN 978-1 42444-4992-444 doi: 10.1109/CVPR. 2009.5206848.
[13] Peijian Ding, Davit Soselia, Thomas Armstrong. Jiahao Su, and Furong Huang. Reviving Shift Equivariance in Vision Transformers. June 2023. BtLphttss://axxtit http//xxxvv.rrr//e owg.iii..//......... axg/abs/2306
07470v1 arXiv: 2306.07470.
[14] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows, January 2022. URL httt:://txiiiii www.oorg////aaaeeeeeeeee abs/ 00652 arXiv:2107.0063 [cs].
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021 9th International Conference on Learning Representa- tions, October 2020. URL https: //axxvv.vvvv..////v//aaa//aiav org/ abs/2010. 11929v2 arXiv: 2010.11929 Publisher: International Conference on Learning Representations, ICLR.
[16] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jegou. XCIT: Cross-Covariance Image Transformers, June 2021. URL http:://axxxx org/
abs/2106 096811 arXiv:2106.099 096681 [cs].
[17] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the Landscape of Spatial Robustness. 36th International Conference on Machine Learning, ICML 2019, 2019-Juee3218- -23388, December 2017. doi: 1445550/axxv.1112..222222 wwwaaaawwwaaaaaaaaaeaeeaaaaeeeee abs/ /11/ 02779v4 arXiv: 1712.02779 Publisher: International
Machine Learning Society (IMLS) ISBN: 9781510886988.
(18) Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale Vision Transformers. Proceedings of the IEEE Inter- national Conference on Computer Vision, pages 6804 6815, April 2021. ISSN 15505499. URL https: /tttps://arxiv httg/abs/21000 11227v1 Publisher: Institute of Electrical and Electronies Engineers Inc. ISBN: 078166542811 125.
doi: 10. (U9/CCCV48922222110000
arXiv: 2104. 1227
[19] V. Fanaskov and I. Oseledets. Spectral Neural Operators, May 2022. URL http://arxiv brg/abs/2200 10573 arXiv:2205.10571 [cs, math].
[20] Daniel Y. Fu, Hermann Kumbong, Eric Nguyen, and Christopher Ré. FlashFFTConv: Efficient Efficiinn Convolutions for Long Sequences with Tensor Cores, November 2023. URL http://arx/ org org/abs/233 ht//aa////3111.///// 05908 arXiv:2311.05908 [cs].
[21] Julia Grabinski. FrequencyLLooCCt Pooling Plug Play against Catastrophic Overfitting. Tech- nical report.
wowp:///oooooo.eee.aaaaaaaaaa http:/ffccpoooogggg///eee arXiv: 2204.00441v2.
[22] Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. Frequency Pooling Plug and Play Against Catastrophic Overfitting. Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, volume 13674, pages 36-57. Springer Nature Switzerland, Cham, 2022. ISBN 978-3-031- doi:
19780-2 978-3-031-1977
URL https: //link

---

springer Science.
Series Title: Lecture Notes in Computer
[23] Julia Grabinski, Janis Keuper, and Margret Keuper. Fix your downsampling ASAP! Be natively more robust via Aliasing and Spectral Artifact free Pooling, July 2023. URL http::/arxx arXiv:2307.099 [cs].
[24] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. ViViT: A Vision Transformer in ConvNet's Clothing for Faster Inference. pages 12299-1226,, 2021. (RRLltttss///ppaaccee
thecvf com/contet//
(CCV2021/acmm//raammmaooo LeVii V__iiinnnn... Transformer in Convlets__Coohhnng hhmm .tml
[25] Suriya Gunasekar. Generalizzation translation shifts: a study in architectures and augmenta- tions, July 2022. UtLtttts::/ http://///aaaii...aaaaaab//a/ htrg/abs/2207 2ttpp//227.0029999
arXiv: 2207.02349.
[26] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang. Yiman Zhang, and Dacheng Tao. A Survey on Visual Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45 (1):87-110, December 2020. doi: 10. 1090 31522477. URL BRRp:ttp://aatt org/ abs/2012. 22552022.12556 arviv: 2012. 12556v6 Publisher: IEEE Computer Society.
[27] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming Ming Cheng, Jiaying Liu, and Jingdong Wang. On the Connection between Local Attention and Dynamic Depth-wise Convolution. ICLR
2022 - 10th International Conference on Learning Representations. 3, June 2021. URLhttpp 7/arxiv. abs/ /206. 04263v5 arXiv: 2106.04263 Publisher: International Conference on Learning Representations, ICLR.
[28] Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu,
Ze Cheng, Jian Song, and Jun Zhu. GÑOT: A General Neural Operator Transformer for Operator Learning, June 2023. URL http://arxiv. org/ abs 2302. 14376. arXiv:2302.14376 [cs].
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. February 2015. URL http://arxivv org/ aa/a//bb///100 01852 arXiv: 1502.01852.
[30] Dan Hendrycks and Kevin Gimpel. GAUSSIAN ERROR LINEAR UNITS (GELUS). arXiv: 1606.08415v4,
[31] Md Tahmid Hossain, Shyh Wei Teng, Ferdous Sohel, and Guojun Lu. Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and Activation Function, October 2021. URL
https://arxiv org/ abs/2110. 00899v1
Xiviv: 21 2110.00899
[32] Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-Free Generative Adversarial Networks. In Advances in Neural Information Processing Systems, volume 34, pages 352-863. Curran Associates, Inc., 2021. URL https
//arxiv.org/abley
12423
[33] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. 2020. arXiv:
[34] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in Vision: A Survey. ACM Computing Surveys, 54(10), January 2021. ISSN 15577341. doi: 10. 10.15555550244. URL TRRLhttps:// https://aaxxiv org/abs/ 2101. 01169v5 arXiv: 2101.01169 Publisher: Association for Computing Machinery.
[35] Soroush Abbasi Koohpayegani and Hamed Pirsiavash. SimA: Simple Softmax-Free Attention for Vision Transformers. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2607-2617, January 2024.
[36] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine-Grained Categorization. In 2013 IEFE International Conference on Computer Vision Workshops, pages 554-561, December 2013. doi: 10. 10/11////.........77... URL. https //ieeexplore. ieee. org/ www.eee.wwo...ooe/////e1ee1 /tt///mmmmmt66555555555/
12

---

[37] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images.
[38] Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for Partial Differential Equations' Operator Learning. Technical report. JRLhtts::///gihhbb con//Brratiabb OFormer.
[39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the IEEE International Conference on Computer Vision. pages 9992-10002, March 2021. ISSN 15505499, doi: 10. 10.1199/CCV488922.2000 2021000986. URL https::p//// :
/attps:/aane //txxaax.vv//////////eaa/eaaaav. org /bs/ 2103. 14030v2 arXiv: 2103. 14030 Publisher: Institute of Electrical and Electronics Engineers Inc. ISBN: 978 978116566 665428125.
[40] Hagay Michaeli, Tomer Michaeli, and Daniel Soudry. Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 16333-16342, 2023, doi: 10.48550/ arXiv.2303.08085. ISSN: 23318422.
[41] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitooo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorc. 2017.
[42] Shengju Qian. Hao Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blending Anti-Aliasing into Vision Transformer. Advances in Neural Information Processing Systems, 14154166-4229 December 2021. URL Stt.hhtts::// /tstps://githubt teggtthuub... com/ mmmazonnreeeearchanniiiii-
transformer
[43] Md Ashiqur Rahman and Raymond A. Yeh. Truly Scale Equivaraui Deep Nets with Fourier Layers, November 2023. URL aattp://arxiz org/ httg//b//2/0 http://3....22292 arXiv:2311.02922 [cs].
[44] Bogdan Raonić, Roberto Molinaro, Tim De Ryck, Tobias Rohner, Francesca Bartolucci, Rima Alaifari, Siddhartha Mishra, and Emmanuel de Bézenac. Convolutional Neural Operators for robust and accurate learning of PDEs. February 2023. URL URL/httss///axxi org/ httggabb/2302 arXiv: 2302.01 178 ISBN: 2302.01 178v3.
01178v3
[45] Oren Rippel. Jasper Snoek, and Ryan P. Adams. Spectral Representations for Convolutional Neural Networks, June 2015.
03767 arXiv:1506.0333
[stat].
[46] Renan A. Yeh. Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Net- works. October 2022. doi: 00-48550/axxvv22100.000 URL htttss://arxx:: org/ abs/
Teck- Yian Lim, Alexander G. Schwing, Minh N. Do, and Raymond A.
2210. 011008800vv
arXiv: 2210.08001.
[47] Renan A. Rojas-Gomes Teck- Yian Lim, Minh N. Do, and Raymond A. Yeh. Making Vision Transformers Truly Equit--qqivariant. May 2023. URI. https://arxiv. ortpr////rrraa///3555/
16316v1 arXiv: 2305.16316.
[48] David W. Romero and Jean- Saan-Bapistte Cordonnier. Group Equivariant Stand-Alone Self-Attention For Vision, March 2021. URL [cs].
20000/0000/00000077777 arXiv:2010.00977
[49] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A Hierarchical Vision Transformer without the Bells- Ani-Whistles, June 2023. URL http:://ttahii o306 [cs].
arXiv:2306.00989
[50] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-Attention with Relative Position Representations, April 2018. (ttp:/tt:///axxi wwt//bs//80/.....o. 02155 arXiv:1803 803.02155 [cs].
13

---

[51] Ofir Shifman and Yair Weiss, Lost in Translation: Modern Neural Networks Still Struggle With Small Realistic Image Transformations. April 2024. URL https:://///tiii ab2//444 arXiv: 2404.07153.
07153v1
[52] Benjamin Shih, Ahmad Peyvan, Zhongqiang Zhang, and George Em Karniadakis. Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity, May 2024.
URL. http:tttt///iii.orrgggggo/ www.orgg//ba///eeeeee 19166 arXiv:2405.191 [es].
[53] Matteo Spanio and Antonio Rodà. TorchFX: A modemn approach Audio DSP with Py- Torch and GPU acceleration, April 2025. URL http://arxivi org/obgg//o.o hrrg/bb//2504... arXiv:2504.08624 [eess].
2504. .86244
[54] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive Network: A Successor Transformer for Large Language Models. July 2023, URL /tttps://arxxiv.or//aaa 200.bb/221000082214
arXiv: 2307.08621
[55] Karn Tiwari, N. M. Anoop Krishnan, and Prathosh A. P. CoNO: Complex Neural Operator for Continuous Dynamical Systems, October 2023. 2BLLattp.//arriivoorr/a////221.0 02094 arXiv:2310.0200 [nlin, physics:physies).
[56] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with Image Transformers, April 2021. URL http://arxiv..rrrr www.oogg////////o/eee/o.o
17239 arXiv:2103.17239 [es] version: 2.
[57] Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin. An Effective Anti-Aliasing Approach for Residual Networks. November 2020. doi: 10448550/axiv.22111.065 URL aottpa://arxinn org/ httg/bbs/20010 10675v1 2011.10675.
arXiv:
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. June 2017. arXiv: 1706.03762.
[59] Sinong Wang. Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- Attention with Linear Complexity, June 2020. WRLphttp://arii http:pp//aaaxii..g/aaa/////////e arXiv:2006.04768 [cs].
04768
[60] Emmy Wei. Aliasing-Free Convolutional Nonlinear Networks Using Implicitly Defined Functions. March 2022. URL BtRLhhtps:://// //htps://hal....
prrchiees-ouverter httpp:///hllaarccieee--ooooe
htehhal-0347560
[61] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang CvT: Introducing Convolutions to Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22-31, October 2021.
[62] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10033-10041, 2021.
[63] Renjun Xu, Kaifan Yang, Ke Liu, and Fengxiang He. -Equiivaai Vision Transformer, July 2023. URL http:://///// / htt///rrvi.ooorabrae www..org/abbb/33.0077222 arXiv:2306.06722 [es].
[64] Cuihong Yu, Cheng Han, and Chao Zhang. DMFFT: improving the generation quality of diffusion models using fast Fourier transform. Scientific Reports. 15(1):10200, March 2025. ISSN 2045-2322.
doi: 10.
598-025-94481-8 URL httphhttptst/tttt h /www http://www.ttuuut.ee nature. com/ Publisher: Nature Publishing Group.
articles/s41598-00 025
[65] Weihao Yu. Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. MetaFormer Is Actually What You Need for Vision, July 2022. URL. 11418, arXiv:2111.11418 [cs].
http:///axxiv.rr/aabbe
[66] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis E H Tay, Jiashi Feng. and Shuicheng Yan. Tokenn-to-Token ViT: Training Vision Transformers from Scratch on ImageNet. URL. 2101.11986v3.
arXiv:
14

---

[67] Richard Zhang. Making convolutional networks shift-invariant again. 36th International
Conference on Machine Learning, ICML 2019, 2019-Jume:127122122711 2019. URL URL/https:/ arXiv: 1904. 1 1486 ISBN: 978.510886988.
richzhang- github. antiallaseddcnnsn
[68] Tianxiao Zhang, Wenju Xu, Bo Luo, and Guanghui Wang- Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets. Neurocomputiing, 617:128998, February 2025. ISSN 09252312. doi: wtw.m//.eccmm.222...2....
URL hRttt:////arii wwwpzi/v.rrr/// abs/
2407 19394 arXiv:2407.19394 [cs].
[69] Jianwei Zheng, Wei Li, Ni Xu, Junwei Zhu, Xiaoxu Lin, and Xiaoqin Zhang. Alias-Free Mamba Neural Operator. Technical report. URL http:////giiaabb.ccc///egg//gg//
[70] Yifan Zhou, Zeqi Xiao, Shuai Yang, and Xingang Pan. Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space. CVPR, 2025.
[71] Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving Deeper into Anti-aliasing in ConvNets. International Journal of Computer Vision 2022, pages 1-15, August 2020. ISSN 15731405. doi: 10. 1007/S I 1263-022-01672- Y/FIUREES/11 URL http: http///xxivv.rrg/// arXiv: 2008.09608 Publisher: Springer.
abs/ abs/2008.0 09604
15

---

A Full proofs
Proposition 2. f (KTV) is shift-invariant.
Proof. The problem can be simplified by considering an arbitrary entry in K V, since
(K (KTV),s K, Vj,
where K₁, Vj are the i-th and j-th columns of K and V. representing 1-dimensional signals. By Parseval theorem, this inner product between signals equals to their inner product Fourier domain:
1 2T /
V
K, (w) Vj dw) dw.
(10)
Denote K" and V the queries and values of r-shifted input signal w.r.t. the input signal of and V. Following Proposition [ a translation by of the input x yields translation by of both and
V. The Fourier transform of the translated signals differs by a phase which cancels out in the inner product, thus we get the same product:
1
I KJ (w) Vi (w) du
KJ V
(11)
2T
1
K2 c
cJuT (V) (w) edur)) dw
(12)
2T
1
K₁ (w) Vi (w)
(13)
2T
= K, V3
(14)
Class Attention. Below we formalize the statement regarding ihift-invariance of the Class Attention layer in Section 2.3
Proposition 4. Let Q XW yr K X and XW be the query, key and value matrices of
patch sequence X NxD. Append the sequence with a class token,
i [ gels]
K [[KT, kcle]]
V == [VT, Ucls
(15)
with learnable vectors Gelas kick; Vels E R.P. Our CA layer applies the SEA update
V SEA SEA(Q. K, v) = ( E (((++)xDD
(16)
where f: RDDxD) + Dx is any matrix finction. Denote the output by V [V'T, ddda] TT
with
V' E NND, Then
1. Patch equivariance: V is shift-equivariant.
2. Class invariance: v/ax shift-invariant
Proof. Let Q KT and V be the keys and values obtained from the 7-translated input X, and define
Q = [(QT), gels
k = [KT) Kila
er = [(VT), Vels
(17)
Shift-invariance of the attention weights. Since the translation acts only patch tokens, we get
RT = (KT)T V + T = KTV + keta = K TTTV
(18)
where the middle equality uses Proposition The rank-one class term ki dent of the input translation), hence f(K V) is shift-invariant.
U. T is constant (indepen-
16

---

It holds that
F'' = Qs ( O ( (RTV) detal (RTP)]
(19)
Patch equivariance. From Equation 19 the patch tokens post CA are V QJ ( where
( (RTV) is shift-invariant, therefore V shift-equivarint similar Proposition 3
Class invariance. From Equation 19 the class token post CA golsd (K (RT)). which is shift- invariant.
B Additional results
B.1 Additional datasets
We evaluate all models from section 4.1 on three additional classification benchmarks - CIFAR-10
CIFAR-100 [37] and Stanford Cars [36 under two protocols: (i) fine-tuning ImageNet-pretrren pretrained checkpoints and (ii) training from scratch. In both protocols, we train each model using the ImageNet recipe of Table X with 1,000 epochs, where in the fine-tuning protocol,
initialize the model weights using the checkpoints from section We report top-1 accuracy together with cyclic shift consistency for integer and half-pixel translations, defined exactly as in Section 4.1
Across all datasets, AFT maintains near near-perfect shift-equivariance (above 99% consistency in integer and half-pixel shifts). Additionally, when fine-tuned, the XCIT-Small-AF model is slightly but consistently more accurate than the baseline on all three datasets, suggesting that the ihift-invariance prior benefit larger transformers when adapting small datasets.
Table 4: CIFAR and Stanford Cars (fine-tuning): accuracy and cyclic shift consistency. We fine-tune ImageNet-pretrained checkpoints on CIFAR-10/100 and Stanford Cars. Metrics are top-1 test accuracy and consistency to integer and half- palfl cyclic translations (as in Section 4.1p
CIFAR-10
CIFAR-100
Stanford Cars
Text
Integer consist.
Half-pixel consint.
Test accuracy
Integer
Half-pisel shift
Test accuracy
Integer consist.
Half-pixel
accuracy
XCTT-Nano (onooBBusite (Baseline)
98.0
98.0
98.1
84.3
89.4
89.9
92.3
95.2
95.1
XCTT-Nanc-APE
97.7
100.0
99.4
84,1
100.0
96.9
92.2
100.01
97.0
XCiT-Namo-AA (ours)
XCIT-Small (Baseline)
98.2
98.4
98.5
85.6
87.4
89.6
92.6
95.4
96.3
XCYT-Small-APS
98.3
100.0
99.5
85.4
100.0
96.2
92.2
100.0
97.2
XCIT-SmalI-AF (purs)
98.4
85.8
Table 5: CIFAR and Stanford- (from scratch): accuracy and cyclic shift consistency. Models are trained from scratch on each dataset using the ImageNet training setup with 1,000 epochs; metrics as in Section 4.1
CIFAR-10
CIFAR-100
Stanford Cars
Integer consist.
Half-pixel consist.
Integer consist.
Half-pisel shift
Integer consist.
Half-pisel consist.
accuracy
sccuracy
accuracy
XCTT-Nano ( (Baseline)
97.2
98.0
98.0
82.4
90.3
90.2
86.5
91.6
91.7
XCIT-Naao-APS
97.2
100.0
99.2
82.3
100.0
97.1
84.1
100,0
93.3
XCIT-Nano- (ours)
XCIT-Small (Baseline)
98.3
U8.7
98.7
85.3
91.3
91.3
89. 6
95.1
94.5
XCIE-SmalL-APS
98.0
100.0
99.4
85.1
100.0
95.7
90.5
100.0
96.5
XCIT-SmalI-AF (ours)
88.5
B.2 Global average pooling vs AF Class Attention
we propose two mechanisms get shift-invariant global representation out of the AFT global average pooling over the embedding dimension and an alias-fred free class attention that leverages SEA to maintain shift-invariant class token. We compare the final AF class attention (AFCA) head with a global average pooling (AvgPool) head within the AFT architecture. As shown Table 6 both AFCA and AvgPool maintain near-perfect shift consistency. AFCA demonstrates consistent improvement over AvgPool in top-1 accuracy, most visible in the Small variant.
In section 2
17

---

Table AF class attention vs. global average pooling (AFT). Top-1 aceuracy and cyclic shift con- sistency (integer and half-pixell on ImageNet. AvgPool and AFCA retain near-perfect equivariance, while AFCA provides a consistent accuracy gain, most notably for the Small variant.
Test accuracy
Integer shift consist.
Half Hall- shift consist.
Model
XCiT-Nano-AF (AvgPool)
70.35
99.0
98.6
XCiT-Nano-AF (AF-CA)
70.48
99.2
99.4
XCiT-Small-AF (AvgPool)
80.70
99.5
99.4
XCiT-Small-AF (AF-CA)
81.81
99.5
99.4
B.3 Polynomial vs GELU comparison
Similar the Alias-Free ConvNet (AFC) of Michaeli et al. [40] certified shift-invariance in the Alias-Free Transformer (AFT) can be achieved by replacing the filtered GELU activations with polynomial approximations. We therefore train an alias-free XCiT-Nano variant whose activations are polynomials with learnable coefficients per embedding channel, following Michaeli et al. [40]. the AFT blocks and degree 3 in the patch-embedding stage (PE), which remains alias-free thanks to the downsampling layers following the activations in the PE. The results in Table 7 show that the full polynomial model ("Poly") has near- 100% shift consistency, with small gap that can be attributed to numerical errors, similar to the case in the APS model (see Table D However, we observe that unlike the AFC, polynomial activations lead significant reduction in accuracy.
We use polynomials degree
Interestingly, when the four GELU activations in the PE are retained and only the block activations are replaced ("GELU (PE), Poly (Blocks)"), most of the lost accuracy is recovered. This may indicate that polynomial activations limit the representational capacity of the convolutional PE which much shallower than the convnet tested in AFC.
Table 7: Effect of polynomial activations on ImageNet performance and shift consistency, Top-1 accuracy and consistency (%) of XCiT-Nano with the standard filtered GELU, full polynomial replacement (Poly), and a hybrid that keeps GELU in the patch- beucc-eebeddinng (PE)
Test accuracy
Integer shift consist.
Half-pixel shift consist.
Model
GELU
70.4
98.8
98.4
Poly
65.8
99.7
99.6
GELU (PE), Poly (Blocks)
68.5
99.4
98.7
C Translation visualization
We provide visual examples of the translations described in the paper in Figures B Bland
D Experiments details
As mentioned in Section 4 we used the same training settings as in XCIT [16), except for lowering the batch size for the AF model. We report the used training hyperparameters in Table
D.1 Batch size choice
Aiming to avoid expensive hyperparameter tuning, we used the XCIT original recipe [16] However, our early experimentations, we observed fluctuations the training curves of the AF and APS models, which were alleviated by reducing the batch size. To decide fair batch sizes, we trained the Nano variants of Baseline, APS, and AF with batch sizes $12 and 1024 and picked, for each method, the configuration that performed best. The Baseline favored 1024 (top-1 70.4 vs. 70.1 at 512), while
18

---

(a) Original image
(b) Circular integer shift
(c) Circular half-pixel shift
validation-set image. (b) Circular shift of 16 pixels in horizontal and vertical axes. (c) Circular shift of 16.5 pixels in horizontal and vertical axes. The original image is upsampled by factor 2, circularly shifted by 33 pixels, and downsampled by factor 2.
Figure 3: Visualization of cyclic shifts. (a) Original ImageNet [12]
(a) Original image
(b) Crop-shift
(c) Bilinear fractional shift
Figure 4: Visualization of realistic shifts. (a) Original ImageNet 12 validation-set image
224 x 224 center crop of the original 256 x 256 image. (b) Crop-shift the original image of 16 pixels in the horizontal and vertical axes. The cropped area is shifted by 16 pixels with respect the cropped area in the original image. (c) Bilinear fractional shift 0.5 pixels in horizontal and vertical axes. We use 226 226 center crop of the original 256 x 256 image and simulate fractional shift using grid-sample with fractional offset.
Table 8: Hyperparameters. Unless stated otherwise, the same settings apply to all models.
Category
Parameter
Value
Optimizer
Adam W (0.9, 0.999)
Optimizer
(B1, 82) Weight decay
0.05
Base LR
1x x 10 (Baseline), 5 10 (AF, APS)
Warm-up epochs
Learning rate scheduling
LR decay
Cosine
Min LR
x 10-5
Resolution
224 224
Data
Batch size
1024 (Baseline), 512 (AF, APS)
Layer scale (e init)
1.0
Regularization
Stochastic depth
0.0 (Nano), 0.05 (Small)
APS and AF favored 512 (APS: 68.7 vs. 67.2 at 1024; AF: 70.4 vs. 70.1 1024). We therefore used batch size 1024 for the Baseline and 512 for APS and AF throughout the paper. Note that the figures
19

---

above for the AF and APS models are with an AvgPool head. We applied the same choice to the Nano and Small variants with CA and AFCA heads without further tuning.
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes],
Question:
Justification: We describe the proposed alias-free model in sections 2 and and evaluate its performance in section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the proposed model compared to other studies in the field section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
21

---

The main theoretical results are presented in propositions | to each followed by a proof or proof sketch (in this case we provide a full proof in the supplementary material).
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We explain in detail all parts of the proposed model in sections 2 and Some elements (including the baseline model, layers in our proposed model, models we evaluate for comparison, and training recipe) are based on other public manuscripts to which we refer.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We intend to release the code and trained models upon acceptance of the paper. The model implementation and training details explained in the paper, and the only dataset we used (ImageNet) is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nipscc. tttpcc//pbbb. muublc//guidee wttggdedeeeeeeeuebmmmmmeeee www.//ooieeessiiieooca wommissssoooiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training regime is explained in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We train all models once due to high computation costs. The rest of the results are deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the type of hardware we used in Section and report training runtime in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp////eeerppp..c/pppbbbbhccic
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The proposed method is image classification model trained standard datasets (ImageNet) consisting mostly of animals and objects, and has no higher risk of negative societal impact than other studies in the field.
Guidelines:

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release data. The proposed method an image- classification model trained standard datasets (ImageNet) and has high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All relevant assets, including models, datasets, and the implementation frame- work, are mentioned with proper citation and have standard open-source licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The model and training details are fully explained in the paper. We intend to release the code and trained models upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) hbtte:///eerrrppcccc//ooeeeeeee /ct/////ooeeeeeee//22////.. for what should or should not be described.
27


Reviews:
[{"review_id": "qvDYq0Gz0f", "reviewer": "Reviewer_he7X", "summary": "The authors adapt a visual transformer to be less sensitive to translations. Specifically, they adapt downsampling components, non-linearities and the attention mechanism. The authors build on existing work in this field, but combine various elements in a novel way. The resulting model is significantly more robust against certain standard and adversarial translations, while retaining a standard test accuracy that is mostly comparable with an adaptive polyphase sampling (APS) baseline. (See Table 1.) The most convincing results are in Fig 2, which compares the adversarial accuracy of different approaches on realistic shifts applied to Imagenet.", "questions": "1. Were you able to reproduce your findings on other datasets?\n2. How exactly was HP tuning performed?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper proposes an adapted ViT that seems practical and useful.\n- Work is clearly presented and well motivated.\n- Experimental results seem solid (but also see weaknesses).\n\nWeaknesses:\n- For a practical paper that combines existing ideas carefully and methodically, I expected additional empirical results. All results are on Imagenet. Confirming results for additional datasets would have strengthened the paper.\n- The sentence “We observe a slight improvement for the AF models 212 when training with a smaller batch-size; therefore, we reduce the batch-size from 1024 to 512 for the 213 AF-versions” seems to indicate that some form of hyperparameter tuning was performed. It is not clear what was done and which measures were taken to ensure a fair comparison across the different approaches.", "was_revised": true, "final_justification": "The paper proposes an adapted ViT that seems practical and useful, and addresses very small translations in images. Work is clearly presented and well motivated. Experimental results were extended during the review process and seem solid. \n\nMy only concern is the lack of hyperparameter (HP) tuning when comparing different approaches.  As the paper relies on its empirical results, I feel some additional analysis of the sensitivity of the approaches to HP tuning is required for the results to be fully trustworthy. Coupled with the error that slipped in (included in the main paper, found during the supplementary period), I am going to keep my score as a 4. I am still leaning towards accepting the paper, but with some reservations.", "rebuttal": "We thank the reviewer for their endorsement and constructive feedback. We are happy they found our work novel, practical and useful, well-motivated, clearly presented, and supported by solid experimental results.\nIn the following, we address the questions that have been raised. We will incorporate these clarifications and additional results into the paper (or its appendix).\n\n> Were you able to reproduce your findings on other datasets?\n\nWe evaluated our model solely on ImageNet-1K, as this is the only classification benchmark reported in our baseline model paper (XCiT [A]). This allowed us to avoid expensive hyperparameter tuning. In [A], the authors report results on additional detection and segmentation datasets, but these require training Mask R-CNN, which includes modules that are not trivial to convert to be fractional-shift equivariant under our framework (e.g., RoI Align, which uses bilinear interpolation). Therefore, we leave this as an interesting direction for future work and focus this paper on classification tasks.\n\nIn response to the reviewer’s request, we evaluate our model on additional classification datasets. We fine-tune the models reported in the paper (pre-trained on ImageNet-1K) following the regime of [B]. Note that the ImageNet pre-training regimes of [A] and [B] are identical. For each dataset, we report the accuracy, integer shift consistency, and half-pixel shift consistency, as in the left part of Table 1 in the paper. Although the hyperparameters may not be optimal, the AF models maintain competitive accuracy while exhibiting near-perfect shift consistency. Note that we do not report results for the XCiT APS models, as they diverged with the current hyperparameters.\n\n| Model \\ Dataset           | | CIFAR-10   |             |  | CIFAR-100  |         |  | Stanford Cars  |  |\n|---------------------------| :-:|:-------------:|:---------:|:--:|:-------------:|:-------:|:-:|:------------------:|:--:|\n| | Accuracy | Int. shift-consistency  | Half-pixel shift consistency | Accuracy | Int. shift-consistency  | Half-pixel shift consistency | Accuracy | Int. shift-consistency  | Half-pixel shift consistency |  \n|XCiT-Nano (Baseline)  | 98.2 | 98.0 | 98.0     | 85.5 | 88.3 | 88.1  | 92.4 | 94.9 | 95.2|\n|XCiT-Nano-AF (Ours)  | 97.5 | 99.9 | 99.9     | 85.1 | 99.3 | 99.3 | 92.1 | 99.8 | 99.7|\n||||||\n|XCiT-Small (Baseline)| 98.8 | 98.7 | 98.8      | 88.9 | 88.0 | 90.0  | 92.7 | 94.9 | 96.2 |\n|XCiT-Small-AF (Ours) | 98.6 | 100.0 | 100.0 | 87.1 | 99.7 | 99.6  | 92.4 | 99.8 | 99.7 |\t\n  \n\n> How exactly was HP tuning performed?\n\nWe refrained from performing full hyperparameter tuning due to limited resources. Thus, we retained the same hyperparameters as the XCiT paper (reported in Appendix C), with the following exception:\nWe observed instability in training the AF models, which was resolved by reducing the batch size (and consequently the learning rate, scaled with the same ratio, as in [A] and [B]). We trained all Nano models (Baseline, AF, and APS) using batch sizes of 1024 and 512, and chose the best-performing configuration for each. We then used the same configuration for the Small versions of each model (i.e., 1024 for the baseline and 512 for APS and AF). Below, we report the accuracies for each of the Nano models under these batch sizes.\n\n| Model \\ Batch Size  | 512    | 1024 |\n|---------------------|--------|------|\n|XCiT-Nano (Baseline) | 70.1   | 70.6 |\n|XCiT-Nano APS        | 68.7   | 67.2 | \n|XCiT-Nano-AF (Ours)  | 70.4   | 70.1 | \n  \n---\n\n[A] XCiT: Cross-Covariance Image Transformers, NeurIPS 2021.\n\n[B] Training data-efficient image transformers & distillation through attention, ICML 2021."}, {"review_id": "7F6efluyCQ", "reviewer": "Reviewer_jU3t", "summary": "The authors, through a series of architectural modifications, present an alias-free vision transformer (ViT) that is invariant to both integer and fractional translations. In particular, they remove positional embeddings, use alias-free convolutional embeddings, replace self-attention with cross-covariance attention, and implement alias-free activation and layer normalization. They then experimentally validate superior performance to existing models on fractional shifts.", "questions": "No questions.", "limitations": "Limitations are discussed in Section 6.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* The presented work, to the best of my knowledge, seems to be the first to focus on invariance of ViTs with respect to fractional shifts.\n* An ablation study was performed analyzing the impact of the individual changes on the overall model performance.\n\n**Weaknesses**\n* The proposed model can only outperform existing models when evaluated on fractional translations. However, based on the results reported in Tab. 1, the difference decreases substantially for the larger model, raising the question of relevance on even larger ViTs.\n* Fractional shifts are evaluated synthetically, e.g., using bilinear interpolation, which may not be representative of fractional shifts observed in real-world distributions. However, it is noted that this is a general problem within this domain and not particular to this work.", "was_revised": true, "final_justification": "The authors present a vision transformer that surpasses previous work in achieving invariance to fractional shifts, rather than just integer shifts. While their approach largely involves recombining many known methods, I am confident that the result is greater than the sum of its parts. I am also less concerned than other reviewers about any potential hyperparameter tuning that may have taken place. Thus, I stand by my original \"Accept\" rating.", "rebuttal": "We thank the reviewer for their endorsement and constructive feedback. We are happy that the reviewer found our work novel.\nIn the following, we address the concerns that have been raised. We will incorporate these clarifications and additional results into the paper (or its appendix).\n> The proposed model can only outperform existing models when evaluated on fractional translations. However, based on the results reported in Table 1, the difference decreases substantially for the larger model, raising the question of relevance on even larger ViTs.\n\nUnfortunately, we were unable to evaluate our method on larger models due to limited resources. However, in the following, we evaluate the shift consistency of larger variants of the baseline model (XCiT [A]), as in the third and fourth columns of Table 1 in the paper. Seemingly, the improvement in shift consistency by increasing model size is limited and never reaches the consistency levels of our AF model (>98).\n\n| Model \t          | Params | Integer shift consistency  | Half-pixel consistency |\n|---------------------|--------|----------------------------|------------------------|\n|XCiT-Nano 12 (Tab. 1) | 3M     | 83.2 \t\t\t\t\t\t| 81.4\t\t\t\t\t|\n|XCiT-Tiny 12         | 7M     | 87.5 \t\t\t\t\t\t| 86.5\t\t\t\t\t|\n|XCiT-Small 12 (Tab. 1)| 26M    | 90.3 \t\t\t\t\t\t| 88.8\t\t\t\t\t|\n|XCiT-Small 24        | 48M    | 92.2 \t\t\t\t\t\t| 90.9\t\t\t\t\t|\n|XCiT-Medium 24       | 84M    | 92.6 \t\t\t\t\t\t| 91.2\t\t\t\t\t|\n|XCiT-Large 24        | 189M   | 93.0 \t\t\t\t\t\t| 91.4\t\t\t\t\t|\n \n\n> Fractional shifts are evaluated synthetically, e.g., using bilinear interpolation, which may not be representative of fractional shifts observed in real-world distributions. However, it is noted that this is a general problem within this domain and not particular to this work.\n    \nIndeed, “natural translations” (e.g., due to physical camera movements) are hard to simulate precisely given a pixel representation of an image.\n\n---\n[A] XCiT: Cross-Covariance Image Transformers, NeurIPS 2021."}, {"review_id": "7mIB70qbfX", "reviewer": "Reviewer_N1LY", "summary": "The authors pointed out that ViTs are not translation-invariant and are more sensitive to minor image translations than convnets. To overcome that, the authors proposed the  Alias-Free ViT, which utilizes several shift-invariant models, including alias-free\ndown-sampling and non-linearities, linear cross-covariance attention. The results show that Alias-Free ViT achieves better robustness to adversarial translations.", "questions": "Please see the weakness part.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n- This paper is well-motivated. To be specific, Alias-Free ViT is proposed to overcome the sensitivity of ViT to minor image translations, which is a reasonable starting point.\n\n- The authors give module-by-module analysis in order to get an alias-free ViT and implement an Alias-Free ViT based on XCiT architecture.\n\nCons:\n\n- Based on [1], this work seems to be incremental. Even though this paper focuses on transformer architecture, there are many insights originating from [1], e.g., MLP, layer normalization.\n\n- The experimental results are not enough to illustrate the superiority of Alias-Free ViT. For example, the authors only evaluate XCiT, while there are many ViT variants. Also, the baselines (only APS) are not sufficient to make the result convincing.\n\n- As shown in Table 1, the Alias-Free version of XCiT cannot beat XCiT-AFS in 3 aspects out of 5. \n\n[1] Alias-free convnets: Fractional shift invariance via polynomial activations. CVPR, 2023.", "was_revised": true, "final_justification": "Thanks for the detailed response. It solved my concerns on the novelty and evaluation. Therefore, I decide to improve the score to Borderline accept.", "rebuttal": "We thank the reviewer for their constructive feedback. We are happy the reviewer found our work well-motivated.\nIn the following, we address the concerns that have been raised. We will incorporate these clarifications into the paper (or its appendix).\n\n>   Based on [A], this work seems to be incremental. Even though this paper focuses on transformer architecture, there are many insights originating from [A], e.g., MLP, layer normalization.\n    \nIndeed, many shared components between ViTs and convnets were already “solved.” Therefore, this work is focused on the attention mechanism, as this is the main difference between the two architectures. We believe this work may also have a broader impact, as the attention mechanism is commonly used in many architectures. For example, [B] used existing alias-free components to improve shift equivariance, yet the attention mechanism was overlooked (i.e., it was not modified and, therefore, was not alias-free).\n\nMoreover, we report in the paper additional methodologies and findings that did not originate in [A]. For example, we found that we need to remove the positional encoding and class attention, and to use GeLU instead of polynomial activations in the patchifier. Also, we found that the AF-LayerNorm performs better in the AFT compared to the AFC (in contrast, in AFC it was the main cause of performance degradation). Combining all these small modifications and insights was critical to achieving good performance, and their discovery required careful and extensive experimentation.\n\n>  The experimental results are not enough to illustrate the superiority of Alias-Free ViT. For example, the authors only evaluate XCiT, while there are many ViT variants. Also, the baselines (only APS) are not sufficient to make the result convincing.\n\nWe assume this comment relates to Table 1, where we compare three variants (Baseline, APS, and AF (ours)) of XCiT, as this is our baseline model. The main idea of this table is to show that: (1) our model maintains classification accuracy, and (2) it is almost perfectly robust to circular shifts, as theoretically claimed. We compare our model to other ViT variants (Swin and CvT) in Figure 2, where we evaluate “standard shifts” that better represent practical cases. Note that common models usually perform worse on circular shifts than on standard shifts, as shown in [C].\n\nWe are unaware of any other relevant baselines (i.e., techniques to improve vision transformers’ translation robustness) with public implementation, other than [C] and [D], represented by APS. We would be happy to add additional baselines to the camera-ready version in case we have missed any.\n\n> As shown in Table 1, the Alias-Free version of XCiT cannot beat XCiT-AFS in 3 aspects out of 5.\n\n*Table 1 had an issue which gave APS an unfair advantage, but we already corrected this issue in Section B of the SM (Table 4)*. \nSpecifically, upon the main paper submission, we detected an error in our implementation of the APS model (in the results presented in Table 1 and Figure 2), leading to training with an effective patch size of 8 instead of 16, as was used in the Baseline and AF models. This led to better accuracy for the APS model and an unfair comparison vs. the other methods.\n\nThe corrected results in Table 4 (in the SM) show that the APS model has lower accuracy than the AF model (APS 68.7 vs. AF 70.6 in the Nano versions and APS 80.1 vs. AF 80.7 in the Small versions). Consequently, the APS and AF models have similar adversarial accuracy under the integer grid (APS 68.7 vs. AF 68.9 in the Nano versions and APS 80.1 vs. AF 80.0 in the Small versions). Notably, the AF model has better adversarial accuracy under the half-pixel grid, with a larger margin than reported in Table 1 (APS 62.9 vs. AF 69.2 in the Nano versions and APS 76.0 vs. AF 79.9 in the Small versions).\n\n  ---\n[A] Alias-free convnets: Fractional shift invariance via polynomial activations. CVPR, 2023.\n\n[B] Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space. CVPR, 2025\n\n[C] Making Vision Transformers Truly Shift-Equivariant. CVPR 2024\n\n[D] Reviving Shift Equivariance in Vision Transformers, arXiv 2023"}, {"review_id": "mYGcECXoW0", "reviewer": "Reviewer_GB1B", "summary": "The authors propose a modification of the vision transformer architecture based on XCiT (that uses linear attention) to ensure invariance to fractional shifts. The authors theoretically establish shift invariance by stepping through each component of the architecture and empirically validate the method on ImageNet classification. The authors compare against XCiT using the small and nano architectures as well as a shift invariant baseline method APS. The authors also compare the training time required for each method.", "questions": "- can you include additional figures for what the natural shifts used in Figure 2 are? It would be great to show for example an original image and transformed variants in contrast to the fractional / integer shifts that the authors present as more artificial. This would help illustrate to readers the benefit of the method as well.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "* It's nice to see the authors building on prior work and using more efficient linear attention as the foundation for their approach. \n* I also appreciate the clarity of the exposition overall, particularly in establishing theoretical shift invariance for the proposed architecture. \n* Importance of fractional shifts. Figure 2: shows the importance of the issue of fractional intelligence in translations that are likely to come up in real settings. I recommend the authors highlight this finding to illustrate the importance of fractional shifts as a research problem.\n\n* It would be good to include a vanilla ViT as an additional baseline or reference to illustrate the gains from linear attention and lack of invariance to translation for the dominant ViT architecture. \n* The proposed method comes with several tradeoffs. First, compared to APS the method achieves lower test accuracy for both the nano and small XCiT architectures. Second, the method achieves slightly lower integer shift consistency, particularly for the nano architecture (Table 1). Third, and perhaps most important is that the method comes with a much higher training cost (509 hours versus 303 for APS and only 77 for the baseline XCiT small model). This is a significant cost that must be clearly highlighted to make the trade off clear to readers, particularly in the abstract, which seems to ignore these important tradeoffs. \n    * Specifically the abstract read \"Our model maintains competitive performance in image classification and outperforms similar‑sized models in terms of robustness to adversarial translations\" while omitting discussion of these tradeoffs. I recommend including a more detailed discussion of these tradeoffs in the introduction as well. \n* One other suggestion to bolster the experiments is to consider also adding object detection and segmentation experiments to further validate the proposed architecture, as was done in prior work such as XCiT. \n* I acknowledge resource constraints may not make this possible and believe the current experiments on their own are certainly interesting, but I would be curious what happens when the both the model and training data is scaled (to say ImageNet-22k or larger).  The original ViT paper for example demonstrated the performance of the transformer architecture in vision can change significant with scale (and that ViT is generally data hungry). If the authors are resource constrained, I would recommend at least noting the scale of the experiments (model sizes and training data size) as limitations.", "was_revised": true, "final_justification": "This work proposes a modification to the popular vision architecture to tackle fractional shifts—while a nice contribution, the limited scope of the problem and limited empirical verification in terms of hyperparameters are the primary justifications for the rating.", "rebuttal": "We thank the reviewer for their endorsement and constructive feedback. We are happy the reviewer found our work clearly explained and well-motivated.\nIn the following, we address the questions that have been raised. We will incorporate these clarifications and additional results into the paper (or its appendix).\n\n>   It would be good to include a vanilla ViT as an additional baseline or reference to illustrate the gains from linear attention and lack of invariance to translation for the dominant ViT architecture.\n\nWe thank the reviewer for this suggestion. We find that vanilla ViT is indeed less robust to crop shifts than the hybrid models, and our AFT in particular. For example, ViT-Base adversarial accuracy for crop shifts with max-shift 3 (see Figure 2a) declines from 80.4 to 71.7 vs. 80.0 to 76.4 in our model. We will include complete results in the revision of the paper.\n\n>   The proposed method comes with several tradeoffs. First, compared to APS the method achieves lower test accuracy for both the nano and small XCiT architectures. Second, the method achieves slightly lower integer shift consistency, particularly for the nano architecture (Table 1).\n\n*Table 1 had an issue which gave APS an unfair advantage, but we already corrected this issue in Section B of the SM (Table 4)*. \nSpecifically, upon the main paper submission, we detected an error in our implementation of the APS model (in the results presented in Table 1 and Figure 2), leading to training with an effective patch size of 8 instead of 16, as was used in the Baseline and AF models. This led to better accuracy for the APS model and an unfair comparison vs. the other methods.\n\nThe corrected results in Table 4 (in the SM) show that the APS model has lower accuracy than the AF model (APS 68.7 vs. AF 70.6 in the Nano versions and APS 80.1 vs. AF 80.7 in the Small versions). Consequently, the APS and AF models have similar adversarial accuracy under the integer grid (APS 68.7 vs. AF 68.9 in the Nano versions and APS 80.1 vs. AF 80.0 in the Small versions). Notably, the AF model has better adversarial accuracy under the half-pixel grid, with a larger margin than reported in Table 1 (APS 62.9 vs. AF 69.2 in the Nano versions and APS 76.0 vs. AF 79.9 in the Small versions).\n\n> Third, and perhaps most important, is that the method comes with a much higher training cost (509 hours versus 303 for APS and only 77 for the baseline XCiT small model). This is a significant cost that must be clearly highlighted to make the trade-off clear to readers, particularly in the abstract, which seems to ignore these important trade-offs.\n \nIndeed, the proposed model currently comes with a high additional computational cost. As mentioned in the discussion, we believe this issue may be alleviated by carefully optimizing components that are currently not commonly used within neural architectures. For example, [A] reports a custom CUDA kernel for their alias-free activation function, which uses an approximation of sinc-interpolation upsampling, yielding a 10× training speedup. Nevertheless, we will make sure this trade-off is clearly noted.\n\n>   One other suggestion to bolster the experiments is to consider also adding object detection and segmentation experiments to further validate the proposed architecture, as was done in prior work such as XCiT.\n\nWe thank the reviewer for the suggestion. We also find it intriguing to implement translation-robust detection and segmentation models using the anti-aliasing approach. XCiT is indeed used for detection and segmentation as a backbone to Mask R-CNN. However, this includes additional modules that are not trivial to convert to be fractional-shift equivariant under our framework (for example, RoI Align, which uses bilinear interpolation). Therefore, we leave this as an interesting direction for future work and focus this paper on classification tasks.\n\n>   I acknowledge resource constraints may not make this possible and believe the current experiments on their own are certainly interesting, but I would be curious what happens when the both the model and training data is scaled (to say ImageNet-22k or larger). The original ViT paper for example demonstrated the performance of the transformer architecture in vision can change significant with scale (and that ViT is generally data hungry). If the authors are resource constrained, I would recommend at least noting the scale of the experiments (model sizes and training data size) as limitations.\n\nUnfortunately, we were unable to scale the size of the experiments due to limited resources. We will mention this in the limitations section of the camera-ready version.\n\n> Can you include additional figures for what the natural shifts used in Figure 2 are? It would be great to show for example an original image and transformed variants in contrast to the fractional / integer shifts that the authors present as more artificial. This would help illustrate to readers the benefit of the method as well.\n\nWe thank the reviewer for this suggestion. Unfortunately, we are unable to attach figures in the rebuttal, but we will include a visual example in the revision. We kindly refer the reviewer to visual examples in [B], where Figure 6 demonstrates circular shift and crop shift, and Figure 5 demonstrates bilinear fractional shift. Note that, unlike in Figure 5, we interpolated the translated images while leaving an edge of one pixel instead of padding with zeros, to avoid any edge artifacts\n\n---\n\n[A] Alias-Free Generative Adversarial Networks, NeurIPS, 2021.\n\n[B] Alias-free convnets: Fractional shift invariance via polynomial activations. arXiv, 2023."}]

Author rebuttal:

Meta-review: This paper presents an alias-free Vision Transformer (AF-ViT) designed for invariance to both integer and fractional image translations. Through targeted architectural modifications, the authors empirically demonstrate that their model achieves superior robustness to fractional shifts while maintaining competitive accuracy.

The paper's primary strength is its novel and well-motivated focus on achieving fractional shift invariance in ViTs, a significant and previously underexplored problem. The proposed method is technically sound and systematically constructed.

Initial weaknesses included limited empirical evaluation and concerns regarding fair hyperparameter tuning, which cast some doubt on the robustness of the results compared to baselines.

The paper is accepted because it makes a solid contribution to a novel problem. The authors comprehensively addressed all major concerns during the rebuttal by providing additional experiments and clarifications, ultimately presenting a convincing and well-validated method.
Decision: Accept (poster)