Title: Alias-Free ViT: Fractional Shift Invariance via Linear Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Transformers

alias‑free

aliasing

anti‑aliasing

shift invariance

fractional shifts

linear attention

cross‑covariance attention

translation robustness

ImageNet

XCiT
Abstract: Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance.  
Specifically, Vision Transformers (ViTs) are not translation‑invariant and are more sensitive to minor image translations than standard convnets.
Previous studies have shown, however, that convnets are also not perfectly shift‑invariant, due to aliasing in downsampling and nonlinear layers. 
Consequently, anti‑aliasing approaches have been proposed to certify convnets translation robustness. 
Building on this line of work, we propose an Alias‑Free ViT, which combines two main components. 
First, it uses alias-free downsampling and nonlinearities. 
Second, it uses linear cross‑covariance attention that is shift‑equivariant to both integer and fractional translations, enabling a shift-invariant global representation.
Our model maintains competitive performance in image classification and outperforms similar‑sized models in terms of robustness to adversarial translations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a modification of the vision transformer architecture based on XCiT (that uses linear attention) to ensure invariance to fractional shifts. The authors theoretically establish shift invariance by stepping through each component of the architecture and empirically validate the method on ImageNet classification. The authors compare against XCiT using the small and nano architectures as well as a shift invariant baseline method APS. The authors also compare the training time required for each method.\n\nSTRENGTHS AND WEAKNESSES:\n* It's nice to see the authors building on prior work and using more efficient linear attention as the foundation for their approach. \n* I also appreciate the clarity of the exposition overall, particularly in establishing theoretical shift invariance for the proposed architecture. \n* Importance of fractional shifts. Figure 2: shows the importance of the issue of fractional intelligence in translations that are likely to come up in real settings. I recommend the authors highlight this finding to illustrate the importance of fractional shifts as a research problem.\n\n* It would be good to include a vanilla ViT as an additional baseline or reference to illustrate the gains from linear attention and lack of invariance to translation for the dominant ViT architecture. \n* The proposed method comes with several tradeoffs. First, compared to APS the method achieves lower test accuracy for both the nano and small XCiT architectures. Second, the method achieves slightly lower integer shift consistency, particularly for the nano architecture (Table 1). Third, and perhaps most important is that the method comes with a much higher training cost (509 hours versus 303 for APS and only 77 for the baseline XCiT small model). This is a significant cost that must be clearly highlighted to make the trade off clear to readers, particularly in the abstract, which seems to ignore these important tradeoffs. \n    * Specifically the abstract read \"Our model maintains competitive performance in image classification and outperforms similar‑sized models in terms of robustness to adversarial translations\" while omitting discussion of these tradeoffs. I recommend including a more detailed discussion of these tradeoffs in the introduction as well. \n* One other suggestion to bolster the experiments is to consider also adding object detection and segmentation experiments to further validate the proposed architecture, as was done in prior work such as XCiT. \n* I acknowledge resource constraints may not make this possible and believe the current experiments on their own are certainly interesting, but I would be curious what happens when the both the model and training data is scaled (to say ImageNet-22k or larger).  The original ViT paper for example demonstrated the performance of the transformer architecture in vision can change significant with scale (and that ViT is generally data hungry). If the authors are resource constrained, I would recommend at least noting the scale of the experiments (model sizes and training data size) as limitations.\n\nQUESTIONS:\n- can you include additional figures for what the natural shifts used in Figure 2 are? It would be great to show for example an original image and transformed variants in contrast to the fractional / integer shifts that the authors present as more artificial. This would help illustrate to readers the benefit of the method as well.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
The authors pointed out that ViTs are not translation-invariant and are more sensitive to minor image translations than convnets. To overcome that, the authors proposed the  Alias-Free ViT, which utilizes several shift-invariant models, including alias-free
down-sampling and non-linearities, linear cross-covariance attention. The results show that Alias-Free ViT achieves better robustness to adversarial translations.

STRENGTHS AND WEAKNESSES:
Pros:

- This paper is well-motivated. To be specific, Alias-Free ViT is proposed to overcome the sensitivity of ViT to minor image translations, which is a reasonable starting point.

- The authors give module-by-module analysis in order to get an alias-free ViT and implement an Alias-Free ViT based on XCiT architecture.

Cons:

- Based on [1], this work seems to be incremental. Even though this paper focuses on transformer architecture, there are many insights originating from [1], e.g., MLP, layer normalization.

- The experimental results are not enough to illustrate the superiority of Alias-Free ViT. For example, the authors only evaluate XCiT, while there are many ViT variants. Also, the baselines (only APS) are not sufficient to make the result convincing.

- As shown in Table 1, the Alias-Free version of XCiT cannot beat XCiT-AFS in 3 aspects out of 5. 

[1] Alias-free convnets: Fractional shift invariance via polynomial activations. CVPR, 2023.

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
The authors, through a series of architectural modifications, present an alias-free vision transformer (ViT) that is invariant to both integer and fractional translations. In particular, they remove positional embeddings, use alias-free convolutional embeddings, replace self-attention with cross-covariance attention, and implement alias-free activation and layer normalization. They then experimentally validate superior performance to existing models on fractional shifts.

STRENGTHS AND WEAKNESSES:
**Strengths**
* The presented work, to the best of my knowledge, seems to be the first to focus on invariance of ViTs with respect to fractional shifts.
* An ablation study was performed analyzing the impact of the individual changes on the overall model performance.

**Weaknesses**
* The proposed model can only outperform existing models when evaluated on fractional translations. However, based on the results reported in Tab. 1, the difference decreases substantially for the larger model, raising the question of relevance on even larger ViTs.
* Fractional shifts are evaluated synthetically, e.g., using bilinear interpolation, which may not be representative of fractional shifts observed in real-world distributions. However, it is noted that this is a general problem within this domain and not particular to this work.

QUESTIONS:
No questions.

LIMITATIONS:
Limitations are discussed in Section 6.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
The authors adapt a visual transformer to be less sensitive to translations. Specifically, they adapt downsampling components, non-linearities and the attention mechanism. The authors build on existing work in this field, but combine various elements in a novel way. The resulting model is significantly more robust against certain standard and adversarial translations, while retaining a standard test accuracy that is mostly comparable with an adaptive polyphase sampling (APS) baseline. (See Table 1.) The most convincing results are in Fig 2, which compares the adversarial accuracy of different approaches on realistic shifts applied to Imagenet.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper proposes an adapted ViT that seems practical and useful.
- Work is clearly presented and well motivated.
- Experimental results seem solid (but also see weaknesses).

Weaknesses:
- For a practical paper that combines existing ideas carefully and methodically, I expected additional empirical results. All results are on Imagenet. Confirming results for additional datasets would have strengthened the paper.
- The sentence “We observe a slight improvement for the AF models 212 when training with a smaller batch-size; therefore, we reduce the batch-size from 1024 to 512 for the 213 AF-versions” seems to indicate that some form of hyperparameter tuning was performed. It is not clear what was done and which measures were taken to ensure a fair comparison across the different approaches.

QUESTIONS:
1. Were you able to reproduce your findings on other datasets?
2. How exactly was HP tuning performed?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03