Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials
Authors: Yifan Pu
Jixuan Ying
Qixiu Li
Tianzhu Ye
Dongchen Han
Xiaochen Wang
Ziyi Wang
Xinyu Shao
Gao Huang
Xiu Li
Venue: NeurIPS 2025 poster ()
Keywords: vision transformer
architecture design
Primary area: deep_learning

Abstract: Vision Transformers (ViTs) have become a universal backbone for both image recognition and image generation.  Yet their Multi–Head Self–Attention (MHSA) layer still performs a quadratic query–key interaction for \emph{every} token pair, spending the bulk of computation on visually weak or redundant correlations.  We introduce \emph{Visual–Contrast Attention} (VCA), a drop-in replacement for MHSA that injects an explicit notion of discrimination while reducing the theoretical complexity from $\mathcal{O}(N^{2}C)$ to $\mathcal{O}(N n C)$ with $n\!\ll\!N$.  VCA first distils each head’s dense query field into a handful of spatially pooled \emph{visual–contrast tokens}, then splits them into a learnable \emph{positive} and \emph{negative} stream whose differential interaction highlights what truly separates one region from another.  The module adds fewer than $0.3$\,M parameters to a DeiT-Tiny backbone, requires no extra FLOPs, and is wholly architecture-agnostic.  Empirically, VCA lifts DeiT-Tiny top-1 accuracy on ImageNet-1K from $72.2\%$ to \textbf{$75.6\%$} (+$3.4$) and improves three strong hierarchical ViTs by up to $3.1$\%, while in class-conditional ImageNet generation it lowers FID-50K by $2.1$ to $5.2$ points across both diffusion (DiT) and flow (SiT) models.  Extensive ablations confirm that (i) spatial pooling supplies low-variance global cues, (ii) dual positional embeddings are indispensable for contrastive reasoning, and (iii) combining the two in both stages yields the strongest synergy.  VCA therefore offers a simple path towards faster and sharper Vision Transformers. The source code is available at \href{https://github.com/LeapLabTHU/LinearDiff}{https://github.com/LeapLabTHU/LinearDiff}.

Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials
Yifan Pul*
Jixuan Ying
Qixiu Li
Tianzhu Ye
Dongchen Han'
Xiaochen Wang
Ziyi Wang'
Xinyu Shao'
Gao Huang
Xiu Li
Tsinghua University
Peking University
Abstract
Vision Transformers (ViTs) have become a universal backbone for both image recognition and image generation. Yet their Multi-Head Self-Attention (MHSA) layer still performs quadratic query-key interaction for every token pair, spending the bulk of computation on visually weak or redundant correlations. We introduce Eixal-CContrie Attention (VCA), drop-in replacement for MHSA that injects explicit notion of discrimination while reducing the theoretical complexity from (NN2C) to (NNCC with n < N. VCA first distils each head's dense query field into a handful of spatially pooled visual-contrast tokens, then splits them into a learnable positive and negative stream whose differential interaction highlights what truly separates region from another. The module adds fewer than 0.3 parameters to a DeiT-Tiny backbone, requires no extra FLOPs, and is wholly architecture-agnue ImageNet- K from 72.2% to 75.6% (+3.4) and improves three strong hierarchical VIT's by up 3. 1%, while in class-conditional ImageNet generation it lowers FID-50K by 1 to 5.2 points across both diffusion (DiT) and flow (SiT) models. Extensive ablations confirm that (i) spatial pooling supplies low-variance global cues, (ii) dual positional embeddings are indispensable for contrastive reasoning, and (iii) combining the two both stages yields the strongest synergy. VCA therefore offers a simple path towards faster and sharper Vision Transformers. The source code is available
Empirically, VCA lifts DeiT-Tiny top-1 accuracy
Introduction
Since the Vision Transformer (ViT) demonstrated that the same machinery that revolutionised natural language processing can match carefully designed CNNs on ImageNet [12), self attention has become a central ingredient of modern computer vision architectures. It now underpins recognition models (e.g., DeiT
[3)), dense predictors, and even high-fidelity generators such as Yet the way self attention is executed vision has changed little from its language origin: for an image unfolded into N tokens every layer computes N x N similarity matrix, leading to (NNCC) multiplications and activations (C' the hidden width). With dozens of layers, quadratic self attention dominates both training and inference budgets, often forcing practitioners to shrink the patch size or the backbone depth and thus give up accuracy.
(55 Swin
DiT 56 61 62 97 96
A first family of methods reduces the matrix size by limiting the receptive field: sliding windows 4 dilated blocks (30), stripes or criss-cros cross patterns [10] rely on the observation that many visual interactions are local. While they cut cost, they also prune long-range cues priori, so they must juggle between speed and the ability to model global structures such as symmetry or repeated textures.
Equual contribution.
8 Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A second family keeps the global field of view but approximates the attention map with low-rank projections 772 4 or fourier kernels 833. These schemes are orthogonal to locality, yet they treat all correlations as equally useful. The network still has to wade through a sea of weak, often redundant similarities, which can drown signals and slow down convergence during training.
Inspired by recent progress in language modelling, differential attention (88) argues that the difference between two attention maps carries more discriminative signal than either map alone. Duplicating queries and keys and subtracting their softmaxes helps large language models focus on tokens that set one sentence apart from another, but the technique remains quadratic and ignores the particular redundancy structure of images. We start from simple premise: it is better to compress the dense query field first and postpone any expensive comparison. Natural images exhibit spatial smoothness, which means neighbouring patches usually carry almost identical information. By leveraging this property, we can shrink the query set to just a handful of prototypes before matching. This idea materialises as Visual-Contrast Attention, drop-in substitute for Huad Self-Attention that injects an explicit notion of contrast and lowers the computational burden to ((Nnd) where n < N.
In the first stage, the model pools the scene for every attention head. Specifically, it average-pools hew
the H x W query feature map to a coarse h w grid (e.g., 8 x 8), flattens this grid into visual-contrast tokens, and adds two distinct positional embeddings so that the tokens form a positive stream and negative stream. Each stream attends independently to all keys and values; the two resulting outputs are subtracted and normalised, which produces a global contrast map that highlights the differences between two pooled views of identical content.
In the second stage, the module refines information at the patch level. Every one of the original N patch queries re-attends the contrast map through lightweight differential operation. Because the contrast map contains only n tokens, the three matrix multiplications that follow (query +> contrast and contrast ++ value) scale with n.N rather than N2. Consequently, the module preserves the global receptive field characteristic of Vision Transformers, yet each attention weight now measures how much a patch stands out instead of merely capturing raw similarity.
This redesign yields three practical pay-offs in a single stroke. (i) Replacing quadratic MHSA with Visual Contrast Attention turns the per-layer complexity into strictly linear form, shrinking both runtime and memory by roughly the ratio N/n, e.g., a 256 image patchified at 16x16 enjoys a 256x cut, without touching residual paths, layer norms, or any training cypeerparameters, (ii) The added machinery is tiny: each head only stores two n-dimensional positional embeddings, amounting to less than 0.3 M parameters contrast stage reuses existing key-value tensors. (iii) Because VCA dispenses with window masks, dilations, or kernel tricks, any ViT-style backbone that processes a 2-D patch grid can adopt by swapping one block, leaving downstream decoders and pretrained heads entirely intact.
DeiT-Tiny and introducing essentially no new FLOPs because the
We validate these claims two demanding tasks. For image classification, inserting VCA into vanilla DeiT backbone raises ImageNet- top-1 accuracy from 72.2 % to 75.6 % without adding
FLOPs, and integrating VCA into three hierarchical backbones (PVT 773. Swin 45 CSwin (T))
yields consistent gains of up to 3.1 percentage points. For image generation 18
17
54
53
53 38
45 46), replacing the attention blocks in class-conditional generators lowers FID-50K 256 x 256
ImageNet by 2.1 to 5.2 points across diffusion models (e.g.. DiT (566) and flow-based models (e.g.,
SiT (52)), across both Small and Base scales, and across patch sizes of 8, 4, and 2-again without extra compute.
Comprehensive ablation studies reinforce three crucial design choices. First, spatial pooling reli- ably supplies low-variance global cues. Second, the dual embeddings are absolutely essential for disentangling positive from negative evidence. Third, applying the symmetrically to both streams in both stages consistently unlocks the full benefit of the method.
recipe
Our contributions are fourfold. First, we introduce Visual- Attention, which is the first linear- time attention module that embeds explicit contrast into Vision Transformers. Second, we provide a detailed complexity analysis verifying its linear computation complexity. Third, we demonstrate consistent accuracy and quality improvements in both image classification and image generation while keeping training budgets unchanged. Fourth, we show that VCA is architectur--gnootiic, so it can serve as drop-in upgrade for a wide range of Vision Transformer models.
In summary, Visual- Contrastt Attention reconciles global reasoning with practical efficiency and offers principled route toward faster and more descriptive Vision Transformers.
2

---

2 Related Work
Attention with Linear Complexity. A first group of studies attains linear time complexity by limiting the receptive field, such as Shifted-window attention [44] and Neighborhood Attention [31] By re-introducing locality into the ViT architecture, these methods lower cost but partially sacrifice global context. A second research line tackles the problem directly with linear attention. The seminal work of [39] eliminates the Softmax and applies a feature map 6 to Q and K, reducing complexity to O(N) the cost of noticeable accuracy loss. Follow-ups proposed better approximations: Efficient Attention (4 applies Softmax separately to Q and K"; SOFT (50) and Nyströmformer [33 rely on
matrix decompositions: Castling- ViT [89] uses full Softmax only as an auxiliary during training; FLatten Transformer
[99] introduces focus function and depth- wise convolutions to enrich features.
MLLA [2]] incorporate the key design in Mamba into linear attention, while InLine [20] introduces
an injective linear attention mechanism. More recently, Agent Attention 222.. Anchored Stripe Attention [1]. and Efficient DiT [6]] insert an extra token set that mediates between queries and keys, an equivalent linearization that attains strong results for recognition and low-level vision. Our work is built on this architecture, interpreting the additional visual contrast tokens as a semantic compression.
Vision Transformer. Since the arrival of the Vision Transformer (ViT) self-attention has flourished in computer vision, yet the quadratic cost of conventional Softmax attention [66] remains hurdle. Numerous remedies have been proposed. PVT 773 sparsifies global attention by down- sampling K and V; Swin [43] confines attention to local windows and shifts them to cover the whole image; NAT (32 mimics convolution by attending within each feature's 's neighborhood: DAT [80] introduces deformable, data-dependent patterns; Biforrerr [101] routes queries to salient regions via bi-level scheme; GRL
42 mixes stripe, window, and channel attentions for restoration. Nevertheless, these strategies either cap the global receptive field or are tailored to specific patterns, which limits their plug-and-pg play versatility.
Diffusion Transformer. State-of- art diffusion models 9
33
47
16
86 are traditionally tokenizes time,
built on U-Net [63] yet recent works explore ViT backbones 87 56
2.
U-
shows ViT scales favorably, outperforming U-Net on ImageNet; SiT [52] extends DiT to continuous time with more general coefficients. MaskDiT [99] adopts masked training cut cost, while MDT [14] and MDTv2 refine masked latent modeling for better FID and faster learning. HDiT (5) trains high resolutions with cost linear in pixel count. FiT (99] treats images as variable-sized token sequences, enabling flexible resolution and aspect ratios. These results verify that transformer backbones are both effective and scalable for generative diffusion, yet their internal architectural choices remain under-explored.
conditions, and noisy patches, adding U-Net-style skip connections. DiT
(56)
Dynamic Neural Network. Unlike static networks with fixed graphs and weights, dynamic adapt structure or parameters per input, gaining advantages in ac-
neural networks [24
75
curacy. adaptability
85
95] 98), efficiency 84 68
77
91]. and representation capac- 58 69 79), spatial- 51) Following DETR's query
ity [60]. They are commonly classified as sample-wise
13
34
76 27
23
wise 770 35 28 26
25
81
55 78), and temporal-wise 29
74
paradigm [3], query- dynamic branch has also emerged [59] Our method could be classified
as a type of sample-wie wise dynamic network, since different sample generate different visual contrasts
token in the first stage.
3 Approach
3.1 Preliminaries
Standard Attention in Vision. We first revisit the attention mechanism [67 in Vision Transform-
ers 12 56 (2). The Vision Transformer takes visual token sequence ž1-1 E N from the previous layer 1 as input (N is the token number and is the hidden dimension), then projects into the query, key, and value token sequences with three different linear projection layers, denoted as
Throouhouu the paper we use Vision Transformer (VTT) collectively denote the original ViT 12] and its diffusion variant Diffusion Transformer (DiT) (56). A DiT block is identical to a VIT block except that each LayerNorm is augmented with timestep-condition conditioned scale and shift parameters (AdaLN) generated from the contribution.
diffusion timestep embedding. This distinction is orthogonal
3

---

W W ERCXO (we omit the bias term for simplicity):
21-1 W k = -1 Wk v 21- W.
(1)
Then q. k, v R are divided into M heads ((m), , v((m) E R in terms of channel C. with head dimension of d= C/M (C is always divisible by M). Within each head, the similarity of each query q q(m) and key k(m) computed as:
A (m) Softmax (g ( /va).
(2)
where the attention map A (m) an NxN matrix containing elements in the range [0, 1], and the sum of each row is normalized to 1. The attention mechanism reweights the value sequence according to the attention map, h (m) = A (m²) ( E IR Nxd to dynamically adjust the outputs based on the dependency of each token in the inputs. In the end, each head of the reweighted representation is
concatenated together to produce the final output of this layer I, written as:
h Concat (h..,..., (M)),
where h E RRNCC Wo ERCCC (the bias term is also omitted for simplicity) is linear projection layer to promote interaction between different heads in the multi-head attention layer.
Differential Attention in Language. We further revisit the recent proposed differential attention
mechanism which is primarily used in language modeling [88]. Given the token squences from the previous layer #1-1 E ERNxC (N tokens, each with hidden dimension C), differential attention first produces two sets of queries and keys (1q), ki } and {42, k2)) with two sets of linear projections ((W9, WhJ and {W7. w }) and one set of values v with a linear projection WV:
[q1: 92 = 21-1 [W W
k1i = 21-1 (W; Wh),
V 0= D
where WT. W W E (C/2)xC, 91 92, k1, E
WVERCXG WERN ETERRXCDERNXG Then , 92 k ((()) ky (m)) Nxdd/222 ((() Nxd xd with
q1 92, k1, k2, v are divided into M heads (m)
(m)
(double) head dimension of d= C/M. Within each head, compute two attention maps
A (m)) Softmax
(g(T))
k (m)T / /Vd22).
A (m)
Softmax (
(g2 k
(m)
(m)) /Vd12) (5)
and take their difference as the final attention weight of each head:
A (m) A {m)) A (m)
== exp(( -&a) exp(2 Agp Aio)) + Ainit
(6)
Here À is learnable scalar parameterized by scalar Aini and vectors Ag Aga ARe Axg ERd The attention map A (m) is an N N matrix. The attention mechanism reweights the value sequence according to the attention map followed by a RMSNorm Layer, and scaled by (1 - Xiii) to match
Transformer's gradient flow:
h (m) A (m) (m)
h (m) (1 Ainit) RMSNorm(h(()) (m())
where h (m), h (m) Nxd, In the end, each head of the reweighted representation is concatenated together to produce the final output of this layer I, written as:
h Concat (1 ...,,,! (A0),
= hwo
(8)
where h NxC. W CxC (the bias term is also omitted for simplicity) is linear projection layer to promote interaction between different heads in the multi-head attention layer.
3.2 Our Approach
Visual Contrast Attention. To extend differential attention to vision and trim the quadratic complexity, besides the query "i)) key k(m"), and value ((m) tokens, we introduce a compact pair of
visual contrast tokens for each head: of positive visual contrast tokens (()) and set of negative visual contrast tokens & (m) These pair of visual contrast tokens are both with x d shape, where

---

is the visual contrast token length and < N. Intuitively, the two sets act as the same mediator token viewed through two coloured lenses. Stage lets the two skim the whole image and return a contrast map; Stage II lets the original patch queries exploit that map.
Stage -- global contrast. This pair of visual contrast tokens first each attends to all key tokens and all value tokens individually to get the intermediate results a(m)) (!m))
(m)
Softmax
(+ (m) (())T /Vd) v(m) i (m)) Softmax ((T)(mm) /va] (m))
(9)
where 64m)
(m) Raxd Then the visual contrast results is obtained by performing differential
operation in the intermediate results, followed by RMSNorm and a (1 - > scalar factor:
RMMSNrM (m)) = (1- init
(6(m) (amm~~<0 (()(m - exp(2 xx)) exp(A
(1) (1) -
42
init
(10)
Stage patch-wise differential attention. The pair visual contrast tokens interact with the query tokens and extracts the results from the intermediate result ((m) in differential way. To be specific, the query tokens ((m) derive its attention scores with both the positive visual contrast tokens t (m) and the negative ones ((m) within each head:
A (im) Softmax (mm)T (Vvd))
A (m) Softmax (q(m)((m) /Vd).
(11)
and take their difference the final attention weight of each head:
A (m)
(mm) 22) (m)
2 = exp(A (2) (2) - exp (2)
(2) lo2
+
init:
(12)
where (2) follows the same parameterisation as (1) The attention map (m) is an x matrix. The attention mechanism reweights the value sequence according to the attention map followed by RMSNorm Layer, and scaled by (1 X to match Transformer's gradient flow:
h (m)) (m)) (())
(m) =0 (2)) RMSNorm(h('))
(13)
where (m), (m) E Nxd, In the end, each head of the reweighted representation is concatenated together to produce the final output of this layer 1, written as:
h Concat (1
(M))
21 hwo
(14)
where h E NXC. W CxC (the bias term is also omitted for simplicity) a linear projection layer to promote interaction between different heads in the multi-head attention layer.
The visual contrast tokens are distilled directly from the query tokens through spatial average pooling. Since our attention module operates on visual latent features, the query matrix (m)) E R.xxd can be rearranged back to its 2-D spatial layout, i..., ((m) - qi(m) E T xxx xd with H x W = N. We then apply average pooling along the spatial dimensions, with kernel size and stride chosen to reduce the resolution from x W to h w:
Visual Contrast Token Generation
((m (vg
kxxwxd
(15)
To further disentangle helpful and distracting correlations, we split the visual contrast branch into positive and a negative stream, inspired by the core idea of Differential Transformer. Specifically, we add two distinct learnable positional embeddings, et, Exxxxxd, create two groups of visual contrast tokens:
f(m) + +
(m))
(() +e"
(16)
Finally, we flatten the positive and negative tensors over the spatial axes to obtain the visual contrast token matrices:
5

---

(m)
(())) Flatten (fLm)). Flatten (t_m) E Rodd h. < N.
(17)
Each visual contrast token thus represents the average feature non-overlapping image patch, providing compact summary. The target spatial size (h, w) and thus the number of visual contrast tokens n n-iis a tunable nyper-parameter that balances computational cost and representational fidelity.
3.3 Complexity Analysis
We retain the notations of Section 3.2 N visual tokens, n visual contrast tokens (n< d features per head, M heads and C Md channels in total.
Stage 1- global contrast. For each head, the positive and negative contrast tokens execute the two attention steps in Equation 9
Softmax (m) (mm) /Vd)
(mm)
(m)
(18)
ROXN
RNKd
Each of the two matrix products (n d by d x N, then x N by N d) costs O(Nnd); performing them for both "+" and "_" streams therefore costs at most
Stage I:
Nnd O(Nnd).
(19)
The subsequent subtraction, normalisation, and scalar modulation are all O(nd) and thus negligible in the big-O sense.
Stage II paseh-wise differential attention. In each differential stream, we calculate the two attention map in Equation (1I
Softmax (m) (m)JT /va)
E NRn
(20)
Each map requires one N x d by d multiply, i.. O(Nnd). Both maps together give cost of 2 (ONNdd) The differential combination A (m)
(m)) X A (m) is only O(Nn).
Finally, value aggregation in Equation another ((Nd).. Hence
1
multiplies an N x n matrix by an n x d matrix, adding
Stage
3 Nnd = O(Nnd).
(21)
Per-head and per-layer complexity. Both stages are linear N, n, and For one head
Chead (Nnd),
(22)
and for the whole layer
Clayer M Chead ONNnC).
(23)
Comparison with vanilla self-attention. Standard self-attention forms an N attention map, incurring ((N2C) time. Replacing the quadratic query-key interaction by the two linear contrast and contrast -ee interactions reduces the cost by factor of N/n:
O(N2C) O(NCC)
(24)
Because & N, the proposed visual-contrast attention substantially lowers computation while preserving global context. Overheads from RMS normalisation and the learnable scalars are at most O(Nd) ((nd) and therefore non-dominant.

---

Table 1: Image classification results on ImageNet-
Method
#Params
FLOPs
Top-1 1 Acc.
Method
#Params
FLOPs
Top-1 1 Acc.
DeïT-T
5.7M
1.2G
72.2
Swin-T
28.9M
4.5G
81.3
+Ours
6.0M
1.2G
(5.6()3.4)
+Ours
28.5M
4.6G
82.3(
DeïT-S
22.1M
4.6G
79.8
Swin-S
49.7M
8.7G
83.0
+Ours
22.6M
4.6G
80.7(10.9)
+Ours
49.6M
8.7G
83.7
Swin-B
88.1M
15.4G
83.5
PVT-T
13.2M
1.9G
75.1
+Ours
87.9M
15.5G
(1.4)
+Ours
11.6M
2.0G
78.2
PVT-S
24.5M
3.8G
79.8
CSwin-T
20.5M
4.3G
82.7
+Ours
20.6M
4.1G
(1.3)))
+Ours
20.4M
4.3G
83.3
PVT-M
35.9M
7.0G
81.2
CSwin-S
32.8M
6.8G
83.6
+Ours
35.8M
7.2G
83.2()2.0)
+Ours
32.7M
6.8G
84.0(10.4)
Experiments
In section, we empirically evaluate our visual contrasts attention method on both image recognition and generation tasks. We first introduce the detailed experiment setup in Section 4.1 including datasets and training configurations. Then the main results of our method with various backbone
architectures on different tasks are presented Section 4.2 and Section 4.3 Finally, the ablation study in Section 4.4 further validate the effectiveness of the proposed method.
4.1 Experiment settings
Datasets The ImageNet-IK 7 recognition dataset contains 1.28M training images and 50K validation images with total of 1,000 classes. For image recognition experiments, images are trained and evaluated 224 224 size. The top-1 accuracy on the validation set is adopted as the evaluation metric. For image generation tasks, we train and evaluate the images in 256 x 256 size, following the commonly used practice in class-condition generation. We use FID-50K as the evaluation metric, which measures the Fréchet distance between the Inception-V3: features of 50 000 generated images and 000 real validation images.
Training Configuration For image recognition experiments, we use the same training setup as the baseline models ensure fair comparison. All models are trained from seratch using the AdamW [48] optimizer for 300 epochs. We apply cosine learning rate decay, starting with 20 epochs of linear warm-up. and set the initial learning rate x 10-3 with weight decay of 0.05. The data augmentation and regularization methods include RandAugment (6], Mixup [3 CutMix [92 and
random erasing [100]. We also follow CSwin I0 and use EMA during training. For image generation tasks, we follow DiT (56) and SiT
522 to train class-conditional diffusion transformer
models on the ImageNet- IK 8 dataset. All models are trained with the Adam [40 49 optimizer and weight decay. For 256 X 256 resolution, 256 for 400,000 iterations. The learning rate is kept constant at x 10-4, We use only random horizontal flip for data augmentation during training. Additionally, apply exponential moving average (EMA) to the model weights with a decay rate of 0.9999.
train from scratch with a global batch size of
4.2 Image recognition
The image recognition experiments are conducted on ImageNet-IK 7 dataset. We conduct Ima- geNet classification on both plain vision transformer architectures (c.g.. DeiT (65]) and hicrarchical counterparts (e.g., PVT [63]. Swin [43] CSwin (TO). On the plain Vision Transformer line, as is illustrated in Table D our method consistently enlarges the accuracy-efficiency Pareto front: DeiT- Tiny takes a +3.3 accuracy gain (from 72.2 75.6 %) with only 0.3 additional parameters and no extra computational cost, while DeiT-Small still enjoys a +0.9 performance improvement under the same computational budget. For hierarchical vision transformer architectures, which include the multi-stage PVT, the shifted-window Swin, and the cross-shaped CSwin architectures, the proposed block remains universally beneficial. PVT experiences the largest margins, up to +3.1 percentage point PVT-Tiny and +2.5/+2.0 on PVT-Small/PVT-Medh respectively. On more

---

Table 2: Class-conditional image generation results on ImageNet-Ile IK with 256 x 256 resolution.
Method
#Params
FLOPs
FID-50K(L)
Method
#Params
FLOPs
FID-50K(L)
DiT-S/8
33.0M
0.4G
151.9
SiT-S/8
33.0M
0.4G
149.5
+Ours
33.8M
0.4G
148.3 143.3
+Ours
33.0M
0.4G
147. 142.4(12.1)
DiT-S/4
32.9M
1.4G
97.9
SiT-S/4
32.9M
1.4G
84.0
+Ours
33.6M
1.5G
(2.7()
+Ours
33.6M
1.5G
(0.9
DiT-S/2
33.0M
6.1G
67.2
SiT-S/2
33.0M
6.1G
57.3
+Ours
33.6M
6.0G
62.3
+Ours
33.6M
6.0G
(3.0(14.3)
DiT-B/8
130.7M
1.4G
118.4
SiT-B/8
130.7M
1.4G
106.0
+Ours
132.1M
1.5G
114.4
+Ours
132.1M
1.5G
102.1
DiT-B/4
130.4M
5.6G
68.3
SiT-B/4
130.4M
5.6G
55.9
+Ours
131.8M
5.8G
(6.0
+Ours
131.8M
5.8G
53.6(12.3)
DiT-B/2
130.5M
23.0G
42.9
SiT-B/2
130.5M
23.0G
35.3
+Ours
131.8M
22.9G
(8.9()4.0)
+Ours
131.8M
22.9G
(2.7(12.6)
strong baselines like Swin and CSwin, our method still receive steady gains between +0.4 and +1.0 with negligible (< 5%) overhead. These results demonstrate that the proposed visual contracts attention is architecture-agnostii:: complements both global self-attention in plain ViTs and the localized or cross-shaped attention patterns employed by state-of-the-art hierarchical designs.
4.3 Image generation
We evaluate our approach on class conditional ImageNet-IK image generation at 256 x 256 resolution, taking the diffusion based DiT (56) family and the flow-based SiT [52) family as basclines. For cach backbone consider two model sizes, Small (~333 M parameters) and Base (~ 131 M), and three different patch size (8, and 2), which together sweep a wide range of computation budgets from 0.4 G to 23.0 GFLOPs. All networks trained under the original recipes released by the authors: DiT models follow the DDPM (3 schedule with 1 100 denoising steps, while SiT counterparts are optimized with the latent flow objective; the only change is that replace the original attention with the proposed visual contrast attention, adding fewer than 1.3 M parameters and at most 0.1 GFLOPs. Following standard protocol we report Fréchet Inception Distance 50000 validation samples (FID-50K), computed with the same code base as DiT [56] paper.
As summarized in Table 2 our method consistently lowers FID across various configuration. Along the model-size axis, the absolute gains on Small backbones reach 3.6 to 5.2 points for DiT and 2.1 to 4.3 points for SiT, whereas Base models still benefit by 2.3 to 4.0 and 2.3 to 3.9 points respectively, indicating diminishing but non-negligible returns as capacity grows. Along the patch-resolution axis, the most fine-gra fine-grained patches (e.g.. /2) exhibit the largest relative improvement (up 4.9 gain in FID in DiT-S/2), yet even the largest variants (e.g.. /8) obtain solid reductions in FID-50K of 2.1 to 4.0. Finally, comparing the two training paradigms (e.g., diffusion- that the proposed module is agnostic to the underlying generative mechanism: it offers similar FID reductions for the DDPM pipeline of DiT and the rectified flow pipeline of SiT, thereby confirming its general applicability to both diffusion and flow-based training configurations.
flow-base based,
observe
4.4 Ablation Studies
Ablation on Detailed Model Architectures. We first investigates where the standard Multi-Head (olf-Attention(MHAA)) are replaced by our modified counterparts, incluing the first attention operation in Stage I (global contrast) and that in Stage II (patch-wise differential attention) in all the attention blocks. We also ablate the result by using the original differential attention [88] in both stages. We conduct the ablation studies both on image classification with DeiT-Tiny and image generation task with DIT-S/2 model. The quantitative results in Table 3 reveal three clear tendencies. First, the two components of our Visual-Cone -ontrast Attention contribute additively. Activating only the Stage-I global-contrast branch raises DeiT-Tiny aceuracy from the implicit vanilla baseline to 75.4 and reduces the DIT-S/2 FID to 64.6, while switching
only the Stage-Il patch-wise differential branch is slightly more effective (75.5 % 64.3). When the two branches are combined, their effects accumulate almost linearly, pushing the score to 75,6 % and 62.3 FID without introducing any extra

---

Table 3: Ablation on detailed model architectures across image classification and generation tasks.
Attention Type
Image
Classification on DinT-Tiny
Image
Generation on DiT-S/2
Stage
Stage II
Params
FLOPs
Top-1 I Acc.(t)
Params
FLOPs
FID-50K(L)
Ours
Vani.
6.0M
1.2G
75.5
33.6M
5.9G
64.3
Vani.
Ours
6.0M
1.2G
75.4
33.6M
5.9G
64.6
Diff.
Diff.
5.7M
1.2G
75.1
33.0M
5.8G
63.9
Ours
Ours
6.0M
1.2G
75.6
33.6M
6.0G
62.3
Table 4: Ablation on visual contrast token generation across image classification and generation tasks.
Token Type
Image
Classification on DeiT-Tiny
Image Generation on DiT-S/2 Params
Pos. Str.
Neg. Str.
Params
FLOPs
Top-1 Acc.
FLOPs
FID-50K
Emb.
Emb.
6.0M
1.2G
75.1
33.6M
6.0G
63.7
Pool
Emb
5.9M
1.2G
75.5
33.3M
6.0G
64.1
Pool+Emb.
Pool
5.9M
1.2G
75.3
33.3M
6.0G
63.5
Pool+Emb.
Emb
6.0M
1.2G
75.6
33.6M
6.0G
62.3
FLOPs and with only ~0.3 M additional parameters. Second, we compare VCA with the language- oriented differential attention (Dif)) (88] applied to both stages. Although Diff' already improves over the single-branch variants (75.1 % 63.9), replacing with our
VCA brings further relative gain of +0.5 percentage points in classification accuracy and a -1.8 improvement in FID, This superiority indicates that (i) summarising the scene with small set of learnable visual-contrast tokens in Stage and (ii) letting Stage Il queries interact with those tokens in a differential manner both crucial for vision, and that the proposed formulation exploits their synergy more effectively than simply duplicating the original differential attention design.
Ablation on Visual-Contrast Token Generation. Table investigates how the two visual-contrast streams that feed the subsequent differential operation should be formed. Each stream can be pure learnable embedding (EMB.), a query representation obtained by spatial average pooling (POOL), or the pooled query features augmented with an independent positional embedding (POOL+EMB.) that is adopted in our final model. Using embeddings for both the positive and negative streams already gives noticeable improvement over the vanilla backbone (75.1 % Top-1 1 Accuracy 63.7 FID-50K), confirming that explicit differencing is helpful even with the same randomly initialised tokens for different input images. Replacing the positive stream by pooled queries while leaving the negative one unchanged (Poot. POOL+EMB.) yields a marginal additional gain (75.5 % 64.1), whereas performing the opposite substitution (POOL+EMB. POOL) produces a larger jump to 75.3 % and 62.3, suggesting that injecting real image statistics into the positive branch is more influential. When both streams adopt the full POOL+EMB. recipe, performance peaks at 75.6 % and 62.3, outperforming the embedding-oni variant by +0.5 percentage points and -1.4 FID with no additional parameters and identical FLOPs. These results demonstrate that (i) spatial pooling supplies informative, low-variance global cues, (ii) separate positional embeddings remain essential for disentangling complementary correlations, and (iii) combining the two ingredients for both streams yields the strongest synergy across classification and generation tasks.
5 Conclusion
We have presented Visual-Contrast Attention, plug-and-play replacement for MHSA that couples linear complexity with an explicit notion of discrimination. By first summarising the image into handful of pooled tokens and then splitting these tokens into antagonistic positive/negative streams, VCA highlights genuinely informative relationships while discarding redundant ones. The module is parameter-light, nudget-neutral in FLOPs, and universally beneficial: boosts classification accuracy across plain and hierarchical ViTs, and further sharpens generative quality in both diffusion- and flow-based models. We hope our findings encourage the community to rethink attention not only as similarity measure but also stage for explicit contrast.

---

Limitations
VCA reduces the quadratic burden of self-attention but is not cure-all: (i) task-agnostic average pooling may miss edge-rich details; (ii) the added micro- micro-attention may shrinks speed gains on small
images; (iii) extensions video [38. 3-D 36 37,, or more efficient language [90 94 71 tasks are still unexplored.
Acknowledgement
This work is supported in part by the National Key R&D Program of China under Grant 2024YFB4708200, the National Natural Science Foundation of China under Grants U24B20173 and 42327901, and the Scientific Research Innovation Capability Support Project for Young Faculty under Grant ZYGXQNSXYCXXLZCXM..P
References
[1] Ramesh Aditya, Dhariwal Prafulla, Nichol Alex. Chu Casey, and Chen Mark. Hierarchical text-conditional image generation with clip latents. arXiv:2204.0612), 2022.
[2] Fan Bao, Shen Nie. Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All worth words: A vit backbone for diffusion models. In IEEE CVPR, 2023.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020.
[4] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In ICLR, 2021.
[5] Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Kaplan, and Enrico Shippole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. In ICML, 2024.
[6] Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPRW, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE CVPR, 2009.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
[10] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
[13] Jingwen Fu, Ming Xiao, Chao Ren, and Mikael Skoglund. Computatiio--esouuc---fffffiii task-oriented communications, IEEE TCOM, 2025.
10

---

[14] Shanghua Gao, Pan Zhou, Ming-Miin Cheng, and Shuicheng Yan. Masked diffusion trans- former strong image synthesizer. In IEEE ICCV, 2023.
[15] Jiayi Guo, Chaofei Wang, You Wu, Eric Zhang, Kai Wang, Xingqian Xu, Humphrey Shi, Gao Huang, and Shiji Song. Zero-shot generative model adaptation via image-specific prompt learning. In IEEE CVPR, 2023.
[16] Jiayi Guo, Xingqian Xu, Yifan Pu, Zanlin Ni, Chaofei Wang, Manushree Vasu, Shiji Song, Gao Huang, and Humphrey Shi. Smooth diffusion: Crafting smooth latent spaces in diffusion models. In IEEE CVPR, 2024.
[17] Jiayi Guo, Chuanhao Yan, Xingqian Xu, Yulin Wang. Kai Wang, Gao Huang, and Humphrey Shi. Img: Calibrating diffusion models via implicit multimodal guidance. In ICCV, 2025.
[18] Jiayi Guo, Junhao Zhao, Chaoqun Du, Yulin Wang, Chunjiang Ge, Zanlin Ni, Shiji Song, Humphrey Shi, and Gao Huang. Everything to the synthetic: Diffusion-driven test-time adaptation via synthetie-domain alignment. In CVPR, 2025.
[19] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. FLatten transformer: Vision transformer using focused linear attention. In JEEE ICCV, 2023.
[20] Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang. Bridging the divide: Reconsidering softmax and linear attention. In NeurIPS, 2024.
[21] Dongchen Han, Ziyi Wang. Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. In NeurIPS, 2024.
[22] Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. In ECCV, 2024.
[23] Yizeng Han, Dongchen Han, Zeyu Liu, Yulin Wang, Xuran Pan, Yifan Pu, Chao Deng. Junlan Feng, Shiji Song, and Gao Huang. Dynamic perceiver for efficient visual recognition. In ICCV, 2023.
[24] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang, Dynamic neural networks: survey. IEEE TPAMI, 2021.
[25] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Yitian Zhang, and Haojun Jiang. Spatially adaptive feature refinement for efficient inference. IEEE TIP, 2021.
[26] Yizeng Han, Zeyu Liu, Zhihang Yuan, Yifan Pu, Chaofei Wang, Shiji Song, and Gao Huang. Latency-aware unified dynamic networks for efficient image recognition. IEEE TPAMI, 2024.
[27] Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song, Junfen Cao, Wenhui Huang, Chao Deng, and Gao Huang. Learning to weight samples for dynamic carly-exiting networks. In ECCV, 2022.
[28] Yizeng Han, Zhihang Yuan, Yifan Pu, Chenhao Xue, Shiji Song, Guangyu Sun, and Gao Huang. Latency-aware spatial-wise dynamic networks. NeurIPS, 2022.
[29] Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Lioma. Neural speed reading with structuralljump-lstts In ICLR, 2019.
[30] Ali Hassani and Humphrey Shi. arXiv:2209.15000 2022.
Dilated neighborhood attention transformer.
[31] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In IEEE CVPR, 2023.
[32] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In CVPR, 2023.

---

[33] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[34] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018.
[35] Gao Huang, Yulin Wang, Kangchen Lv, Haojun Jiang, Wenhui Huang, Pengfei Qi, and Shiji Song. Glance and focus networks for dynamic visual recognition. IEEE TPAMI, 2022.
[36] Rui Huang, Songyou Peng, Ayea Takmaz, Federico Tombari, Mare Pollefeys, Shiji Song. Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. In ECCV, 2024.
[37] Rui Huang, Henry Zheng, Yan Wang, Zhuofan Xia, Marco Pavone, and Gao Huang. Training open-vocabulary monocular 3d detection model without 3d data. In NewrIPS, 2024.
[38] Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. In ICMI, 2025.
[39] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In ICML, 2020.
[40] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[41] Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Luc Van Gool. Efficient and explicit modelling image hierarchies for image restoration. In IEEE CVPR, 2023.
[42] Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Lue Van Gool. Efficient and explicit modelling image hierarchies for image restoration. In CVPR, 2023.
[43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE ICCV, 2021.
[45] Zeyu Liu, Weicong Liang, Zhanhao Liang, Chong Luo, Ji Li, Gao Huang, and Yuhui Yuan. Glyph-byt5: A customized text encoder for accurate visual text rendering. In ECCV, 2024.
[46] Zeyu Liu, Zanlin Ni, Yeguo Hua, Xin Deng, Xiao Ma, Cheng Zhong, and Gao Huang. Coda: Repurposing continuous vacs for discrete tokenization. In ICCV, 2025.
[47] Zhixuan Liu, Peter Schaldenbrand, Beverley-Claire Okogwu, Wenxuan Peng, Youngsik Yun, Andrew Hundt, Jihie Kim, and Jean Oh. Scoft: Self-contrastive fine-tuning for equitable image generation. In CVPR, 2024.
[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2018.
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[50] Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. In NeurIPS. 2021.
[51] Kangchen Lv, Mingrui Yu, Yifan Pu, Xin Jiang, Gao Huang, and Xiang Li. Learning to estimate 3-d states of deformable linear objects from fingle-frame occluded point clouds, In ICRA, 2022.
12

---

[52] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijan and Saining Xie. SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV, 2024.
[53] Zanlin Ni, Yulin Wang, Renping Zhou, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Shiji Song, Yuan Yao, and Gao Huang. Revisiting -on-atorrerressive transformers for efficient image synthesis. In CVPR, 2024.
[54] Zanlin Ni, Yulin Wang, Renping Zhou, Rui Lu, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Yuan Yao, and Gao Huang. Adanat: Exploring adaptive policy for token-based image generation. In ECCV, 2024.
[55] Xuran Pan, Tianzhu Ye, Zhuofan Xia, Shiji Song, and Gao Huang. Slide trad-f-rrasfommrr Hierar- chical vision transformer with local self-attention. In IEEE CVPR, 2023.
[56] William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE ICCV, 2023.
[57] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization. 1992.
[58] Yifan Pu, Yizeng Han, Yulin Wang, Junlan Feng, Chao Deng, and Gao Huang. Fine-grained recognition with learnable semantic data augmentation. IEEE TIP, 2023.
[59] Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, and Gao Huang. Rank-detr for high quality object detection. In NewrlPS, 2024.
[60] Yifan Pu, Yiru Wang, Zhuofan Xia, Yizeng Han, Yulin Wang, Weihao Gan, Zidong Wang, Shiji Song, and Gao Huang. Adaptive rotated convolution for rotated object detection. In IEEE ICCV, 2023.
[61] Yifan Pu, Zhuofan Xia, Jiayi Guo, Dongchen Han, Qixiu Li, Duo Li, Yuhui Yuan, Ji Li, Yizeng Han, Shiji Song, et al. Efficient diffusion transformer with step-wise dynamic attention mediators. In ECCV, 2024.
[62] Yifan Pu, Yiming Zhao, Zhicong Tang, Ruihong Yin, Haoxing Ye, Yuhui Yuan, Dong Chen, Jianmin Bao, Sirui Zhang, Yanbin Wang, et al. ART: Anonymous region transformer for variable multi-layer transparent image generation. In CVPR, 2025.
[63] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[64] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In WACV, 2021.
[65] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[68] Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang. Efficient knowledge distillation from model checkpoints. In NeuriPS, 2022.
[69] Jiangshan Wang, Yifan Pu, Yizeng Han, Jiayi Guo, Yiru Wang, Xiu Li, and Gao Huang. Gra: Detecting oriented objects through group-wiss wise rotating and attention. In ECCV, 2024.
[70] Shenzhi Wang. Liwei Wu, Lei Cui, and Yujun Shen. Glancing the patch: Anomaly localization with global and local feature comparison. In IEEE CVPR, 2021.
13

---

[71] Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Iim reasoning. In NewrIPS, 2025.
[72] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. 0rXiv:2006.047600 2020.
[73] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[74] Yulin Wang. Zhaoxi Chen, Haojun Jiang. Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In JEEE ICCV, 2021.
[75] Yulin Wang, Yizeng Han, Chaofei Wang, Shiji Song. Qi Tian, and Gao Huang. Computation- efficient deep learning for computer vision: A survey. Cybernetics and Intelligence, 2023.
[76] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. In NeurIPS, 2021.
[77] Yulin Wang, Zanlin Ni, Yifan Pu, Cai Zhou, Jixuan Ying, Shiji Song, and Gao Huang. Infopro: Locally supervised deep learning by maximizing information propagation. IJCV, 2025.
[78] Yulin Wang, Yang Yue, Yang Yue, Huanqian Wang, Haojun Jiang, Yizeng Han, Zanlin Ni, Yifan Pu, Minglei Shi, Rui Lu, Qisen Yang, et al. Emulating human-like adaptive vision for efficient and flexible machine visual perception. Nature Machine Intelligence, 2025.
[79] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. GSVA: Generalized segmentation via multimodal large language models. In IEEE CVPR, 2024.
[80] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR, 2022.
[81] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In JEEE CVPR, 2022.
[82] Zhuofan Xia, Xuran Pan, Shiji Song. Li Erran Li, and Gao Huang. Dat++: Spatially dynamic vision transformer with deformable attention. SXXi:2309900433/ 2023.
[83] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention.
In AAAI, 2021.
[84] Qisen Yang. Shenzhi Wang, Matthicu Gactan Lin, Shiji Song, and Gao Huang. Boosting offline reinforcement learning with action preference query. In ICML, 2023.
[85] Qisen Yang. Shenzhi Wang, Qihang Zhang, Gao Huang, and Shiji Song. Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance. IEEE TNNLS, 2023.
[86] Qisen Yang. Zekun Wang, Honghui Chen, Shenzhi Wang, Yifan Pu, Xin Gao, Wenhao Huang, Shiji Song, and Gao Huang. Psychogat: A novel psychological measurement paradigm through interactive fiction games with Ilm agents. In ACL, 2024.
[87] Xiulong Yang, Sheng-Min Shih, Yinlin Fu, Xiaoting Zhao, and Shihao Ji. Your ViT is secretly a hybrid dissriminnaive-geneeaatt diffusion model. arXiv:2208.07791, 2022.
[88] Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. In ICLR, 2025.
[89] Haoran You, Yunyang Xiong, Xiaoliang Dai, Bichen Wu, Peizhao Zhang, Haoqi Fan, Peter Vajda, and Yingyan Lin. Castling-vit: Compressing self-attention via switching towards linear- attention during vision transformer inference. In IEEE CVPR, 2023.
14

---

[90] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? In NeurlPS, 2025.
[91] Yang Yue, Yulin Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang, Shiji Song, Jiashi Feng, and Gao Huang. Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution. In NeurIPS, 2024.
[92] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features, In ICCV, 2019.
[93] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[94] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self self-play reasoning with zero data. In NeurlPS, 2025.
[95] Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li. Yibing Song, Kai Wang, Zhangyang Wang, and Yang You. A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms. In CVPR, 2025.
[96] Wangbo Zhao, Yizeng Han, Jiasheng Tang. Kai Wang, Hao Luo, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dydit++: Dynamic diffusion transformers for efficient visual generation. arXiv:2504.. SoKIv:2504.06803, 2025.
[97] Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang. and Yang You. Dynamic diffusion transformer. In ICLR, 2025.
[98] Wangbo Zhao, Jiasheng Tang, Yizeng Han, Yibing Song, Kai Wang, Gao Huang, Fan Wang, and Yang You. Dynamic tuning towards parameter and inference efficiency for vit adaptation. In NeurlPS, 2024.
[99] Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. TMLR, 2024.
[100] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, 2020.
[101] Lei Zhu, Xinjiang Wang. Zhanghan Ke, Wayne Zhang, and Rynson WH Lau. Biformer: Vision transformer with bi-level routing attention. In CVPR, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification: We use public data to conduct experiments. However, in the submission and the reviewing period, do not provide open source data to prevent disclosure of author information.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //ipps.ccppblic//uu.dde sss ssiiisosisyyyyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and testing details in the submission.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We provide information on the computer resources in the first part of the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics, in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses attention mechanism and has direct societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks it is foundational research focusing attention paradigm.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use all assets properly according to their licenses, and give credits the creators in Section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is only used during polishing the writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy httpp///aaaarppcoccooooooerrreee for what should or should not be described.
22


Reviews:
[{"review_id": "FP7Ne89nHs", "reviewer": "Reviewer_ptg9", "summary": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "questions": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "limitations": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "As I notified the AC in June 2nd, I deeply apologize but due to an unexpected change in my professional schedule, I will not be able to perform my reviews this year.", "was_revised": true, "final_justification": "Please ignore my rating. I still don't understand why I have to go through this even though i notified early on that i would not able to perform my reviews this year.", "rebuttal": "We sincerely thank the reviewer for their time and completely understand that unexpected professional obligations can prevent a review.\n\n- We noticed that while your comment explains the situation, the submission still includes numerical scores and a final rating of \"Borderline Reject.\" We would like to highlight that this stands in contrast to the detailed and very positive assessments from the other reviewers. They described our work as \"an excellent work\" and \"well-structured,\" praising its \"impressive experimental results,\" \"clear logic,\" and the \"significant performance improvement\" achieved without additional computational cost.\n\n- Given this, we would be very grateful if your schedule now permits you to take a look at our paper; the strengths identified by others might be of interest to you as well. If a full review is not feasible, we would kindly and respectfully ask if you might consider adjusting the numerical scores to reflect that a detailed assessment was not performed. Our concern is that the current \"Borderline Reject\" rating, particularly with a confidence score of 1, could have an unintended influence on the final decision, which would not align with the thorough and positive evaluations provided by the other reviewers.\n\nThank you again for your time and understanding."}, {"review_id": "YFxVPjxRWA", "reviewer": "Reviewer_f3ww", "summary": "The paper introduces Visual–Contrast Attention (VCA), a drop-in replacement for the quadratic Multi-Head Self-Attention (MHSA) used in Vision Transformers. Specifically, VCA pools the image into a few “positive” and “negative” contrast tokens, attends globally through them, and then lets every patch re-attend to this compact map. This cuts the attention cost from quadratic to linear in the number of patches and yields better ImageNet accuracy and lower FID in image generation, all with minimal extra parameters.", "questions": "**Questions**\n\n- Please report actual training and inference latency, peak GPU memory, and throughput on ImageNet for DeiT-T/S and DiT-S/8, comparing MHSA, VCA, FlashAttention-2, and Performer. A ≥10 % wall-clock gain would significantly raise confidence; a negligible or negative gain would lower it.\n\n- How does accuracy/FID vary with the pooled grid size (e.g., 4 × 4, 8 × 8, 16 × 16)? A plot of n vs accuracy/runtime would clarify the trade-off and help practitioners pick n.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Paper Strengths**\n- The paper is overall well-structured and easy to follow. The authors provide clear motivation, detailed methodology, and thorough experimental validation.\n- The method has broad applicability, which can work plug-and-play across many Transformer-based mdoels. VCA adds < 0.3 M parameters to a DeiT-Tiny backbone and introduces “essentially no new FLOPs,” because the contrast stage reuses existing K/V tensors, and each head only stores two n-dimensional positional embeddings. Any ViT-style block can adopt it by a single swap—no window masks, dilations, or kernel tricks needed\n- The authors provide well-documented experiments to prove that it potentially benefits both discriminative and generative vision transformers with minimal code changes. For example, replacing MHSA with VCA lifts DeiT-Tiny top-1 from 72.2 % to 75.5 % (+3.3) with identical FLOPs, and yields performance gains on DeiT-Small, PVT-T/S/M, Swin-T/S/B, and CSwin-T/S—all under the backbones’ original training recipes\n\n**Weaknesses**\n- Lacks wall-clock latency / memory benchmarks and statistical significance (no error bars).\n- Speed-ups are only theoretical; pooling overhead & two extra mat-muls may erode gains for small images—no profiling provide.\n- One suggestion is that to visualise the positive vs negative contrast maps on representative ImageNet images and discuss failure cases (e.g., texture-rich scenes). This could clarify where VCA helps (or hurts).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough evaluation and constructive suggestions. We are pleased that the reviewer finds our paper well-structured, broadly applicable, and experimentally solid. Below, we address each concern point-by-point.\n\n---\n\n1. **Wall-clock Latency, Peak GPU Memory, and Throughput**\n\n    - We have benchmarked our models to provide concrete performance metrics. We measured training latency (for one ImageNet epoch), peak GPU memory per GPU during training, single-image inference latency (batch size = 1), and inference throughput (batch size = 128).\n\n    - Crucially, our **Visual Contrast Attention (VCA) is fully compatible with optimizations like FlashAttention-2**. The core operations in our method (Eq. 9 and Eq. 11 in our submission) can be implemented with `F.scaled_dot_product_attention`, allowing VCA to inherit the speed and memory benefits of such fused kernels. Therefore, we report metrics for both a vanilla implementation and an implementation using FlashAttention-2 (`+FA2`).\n    \n    - Due to its focus on language modeling and implementation complexity, we were unable to benchmark Performer within the limited rebuttal period. However, we did test a similar linear attention architecture; please see our response to Reviewer Z3tv for these results.\n\n    - **Setup**: Training benchmarks were run on 8xA100 GPUs with a global batch size of 512. Inference benchmarks were run on a single A100 GPU with `torch.compile()` enabled. For inference, we averaged the results over 1000 runs, discarding the first 100 for warmup. For the Diffusion Transformer (DiT), we report the time for a single denoising step (1-NFE) and exclude VAE decoding time.\n\n    | Model | Setting | FLOPs (G) | Acc ↑ / FID ↓ | Peak Mem (GB) | Training Latency (min/epoch) | Inference Latency (ms/image) | Throughput (img/s) |\n    |:---|:---|:---:|:---:|:---:|:---:|:---:|:---:|\n    | DeiT-T | MHSA | 1.2 | 72.2 | 18.9 | 4.1 | 4.2 | 3734 |\n    | DeiT-T | **VCA** | 1.2 | **75.5** | 19.2 | 4.2 | 4.3 | 3701 |\n    | DeiT-S | MHSA | 4.6 | 79.8 | 36.9 | 8.8 | 4.3 | 1321 |\n    | DeiT-S | **VCA** | 4.5 | **80.6** | 37.3 | 8.9 | 4.3 | 1324 |\n    | DiT-S/8 | MHSA | 0.4 | 153.8 | 9.8 | 12.1 | 7.8 | 15637 |\n    | DiT-S/8 | **VCA** | 0.4 | **146.3** | 9.9 | 12.1 | 7.8 | 15602 |\n    |--- |--- |--- |--- |--- |--- |--- |--- |\n    | DeiT-T | MHSA+FA2| 1.2 | 72.2 | 16.5 | 3.1 | 3.3 | 4499 |\n    | DeiT-T | **VCA+FA2**| 1.2 | **75.5** | 16.7 | 3.1 | 3.3 | 4494 |\n    | DeiT-S | MHSA+FA2| 4.6 | 79.8 | 31.1 | 8.6 | 3.4 | 1588 |\n    | DeiT-S | **VCA+FA2**| 4.5 | **80.6** | 31.4 | 8.7 | 3.4 | 1602 |\n    | DiT-S/8 | MHSA+FA2| 0.4 | 153.8 | 9.8 | 12.1 | 7.6 | 16075 |\n    | DiT-S/8 | **VCA+FA2**| 0.4 | **146.3** | 9.8 | 12.1 | 7.6 | 16078 |\n\n    - As the results show, **VCA delivers significantly better accuracy/FID with virtually no additional computational overhead**. For example, VCA boosts DeiT-T's accuracy by 3.3 points while maintaining the same FLOPs and nearly identical latency and throughput. The minor overhead from pooling and normalization is effectively optimized away by `torch.compile`, and compatibility with FlashAttention-2 ensures VCA matches the speed of a highly optimized baseline.\n\n---\n\n2. **Speed-up Compared to Models with Similar Performance**\n\n    - A key advantage of VCA is its ability to achieve the performance of larger, more computationally expensive models with a more efficient architecture. When comparing VCA-equipped models to stronger baselines with similar accuracy, the speed-up is substantial.\n\n        | Model | Setting | FLOPs(G) | Acc | Throughput (img/s) |\n        |:---|:---|:---:|:---:|:---:|\n        | PVT-T | MHSA | 1.9 | 75.1 | 2770 |\n        | DeiT-T | **VCA** | **1.2** | **75.5** | **3701 (↑33.6%)** |\n        |--- |--- |--- |--- |--- |\n        | Swin-T | MHSA | 4.5 | 81.3 | 1428 |\n        | PVT-S | **VCA** | **4.0** | **82.3** | **1764 (↑23.5%)** |\n        |--- |--- |--- |--- |--- |\n        | Swin-S | MHSA | 8.7 | 83.0 | 952 |\n        | PVT-M | **VCA** | **7.2** | **83.3** | **1076 (↑13.0%)** |\n\n    - This comparison highlights VCA's efficiency. For instance, **VCA-DeiT-T surpasses PVT-T in accuracy while offering a 33.6% increase in throughput** on lower FLOPs. This trend holds for larger models, where our VCA-PVT variants outperform stronger Swin Transformer counterparts with **13.0%-23.5% higher throughput**.\n\n---\n\n3. **Statistical Significance**\n\n    - To address the concern about statistical significance, we trained the VCA-DeiT-Tiny model three times with different random seeds.\n\n        | Model | FLOPs(G)| Run | Top-1 Acc. |\n        |:---|:---:|:---:|:---:|\n        | VCA-DeiT-T| 1.2 | 1 | 75.5 |\n        | | 1.2 | 2 | 75.5 |\n        | | 1.2 | 3 | 75.4 |\n\n    - The results show minimal variance, with a standard deviation of just 0.05. This demonstrates that the performance gains from VCA are consistent and the training process is stable.\n\n---\n\n4. **Visualization of Contrast Maps**\n\n    - Due to updated NeurIPS policies, we cannot include images in this rebuttal. Instead, we provide a textual description of our visualization analysis, as requested.\n\n    -   Visualization Method: We visualize the contrast map in the VCA-DeiT-Tiny model, which has 12 attention blocks. For each block, we pick the head with the largest L2-norm of the Stage-I contrast map. We export the positive $\\text{A}^+=\\text{Softmax}(\\boldsymbol{t}_{+}^{(m)} \\boldsymbol{k}^{(m) \\top} / \\sqrt{d})$ and negative $\\text{A}^-$ matrices, compute $\\Delta = \\text{A}^+ - \\lambda \\text{A}^-$ , average the $n$ rows, reshape to 2-D grid, upsample to the original image size and overlay it as a heatmap on the original image—highlighting patches that VCA block promotes or suppresses.  \n\n    -  **Observations**: In **early blocks** (e.g., block 3), the positive map $A^{+}$ tends to activate on coarse object edges, while the negative map $A^{-}$ activates on background textures. The resulting differential map $\\Delta$ often exhibits a classic center-surround pattern, separating the foreground subject from its context. In **late blocks** (e.g., block 10), $A^{+}$ sharpens its focus on semantic parts (e.g., the head and tail of a bird), while $A^{-}$ becomes more diffuse, indicating that distracting correlations have been effectively filtered out.\n\n    -  **Failure Cases**: We observed that the contrastive separation in $\\Delta$ is less effective in two main scenarios: (i) **texture-dominated scenes**, such as an image of \"leopard skin,\" where both positive and negative maps activate strongly on the same texture; and (ii) images with **very small objects** that are smaller than the resolution of our pooled grid (e.g., 7×7), making it difficult to generate distinct contrastive views. In these cases, VCA's benefit is diminished.\n\n---\n\n5. **Ablation on Pooled Grid Size**\n\n    -  As requested, we analyzed how performance varies with the number of visual contrast tokens ($n$), which is determined by the pooled grid size. This clarifies the trade-off between computational cost and model fidelity.\n\n    -  **Image Classification (VCA-DeiT-Tiny)**: We ablated the number of tokens by pooling the original 14×14 feature map to 5×5, 7×7, and 9×9 grids.\n\n        | Grid $h$×$w$ | $n$ | Params | FLOPs | Top-1 Acc. | Throughput (img/s) |\n        |:---|:---:|:---:|:---:|:---:|:---:|\n        | 5×5 | 25 | 5.9 M | 1.2 G | 75.4 | 4517 |\n        | **7×7 (ours)** | 49 | 6.0 M | 1.2 G | 75.5 | 4494 |\n        | 9×9 | 81 | 6.1 M | 1.3 G | 75.7 | 4263 |\n    \n    - The results show a clear trade-off: increasing $n$ yields modest accuracy gains at the cost of slightly more parameters and reduced throughput. We chose **7×7** as our default because it offers a strong balance and corresponds to half the spatial resolution of the input feature map.\n\n    -  **Image Generation (VCA-DiT-S/2)**: We pooled the 16×16 latent map to 4×4, 8×8, and 16×16.\n\n        | Grid $h$×$w$ | $n$ | Params | FLOPs | FID-50k ↓ |\n        |:---|:---:|:---:|:---:|:---:|\n        | 4×4 | 16 | 33.2 M | 5.6 G | 65.5 |\n        | **8×8 (ours)** | 64 | 33.6 M | 5.9 G | 62.1 |\n        | 16×16 | 256 | 33.6 M | 7.3 G | 58.4 |\n\n    - A similar trend appears, where a larger grid improves the FID score. We chose **8×8** for our main experiments, as it again represents half the latent's spatial resolution and provides a compelling trade-off between generation quality and computational cost.\n    \n---\n\nWe hope this clarifies our contributions and performance benefits, and we await your feedback."}, {"review_id": "JKA8yiWVxj", "reviewer": "Reviewer_sZxA", "summary": "This paper introduces Visual-Contrast Attention (VCA), a novel attention mechanism designed to address the computational inefficiency of standard Multi-Head Self-Attention (MHSA) in Vision Transformers (ViTs). Traditional MHSA computes quadratic interactions between all token pairs, which is computationally expensive and often processes redundant or weakly correlated visual features. VCA reduces this complexity to linear scale (O(NnC), where (n≪N) while enhancing discriminative power by explicitly modeling visual contrasts.", "questions": "1、The paper uses average pooling to generate visual-contrast tokens. Why was average pooling chosen over max pooling or other sparse sampling methods? Were alternative pooling strategies experimentally compared?\n2、While the paper claims a theoretical complexity reduction from $O(N^2C)$ to O(NnC), it lacks hardware benchmarks (e.g., wall-clock time on GPUs/TPUs). Does VCA actually accelerate training/inference in practice?\n3、The paper does not compare with recent efficient attention methods (e.g., FlashAttention, RetNet). How does VCA outperform or differ from these approaches in both accuracy and speed?", "limitations": "Experiments focus on static images. Can VCA handle video or 3D data, where token redundancy differs? If not, what modifications would be needed?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper validates VCA across both discriminative (ImageNet classification) and generative (DiT/SiT) tasks, demonstrating consistent improvements in accuracy (e.g., +3.3% for DeiT-Tiny) and sample quality (FID reductions of 2–7.5).\nWhile FLOPs are reduced, the paper does not benchmark actual wall-clock speedups on hardware (e.g., GPUs/TPUs), leaving practical deployment gains unclear.", "was_revised": true, "final_justification": "I have read through all the reviews and author responses. I have reached a final conclusion based on the rebuttal and discussion. The authors provide comprehensive explanations that have addressed all my main concerns. The supplement reinforces the quality and contribution of the paper. Thus, I raise the final rating from borderline reject to borderline accept, which stands that the paper is ready for publication in the conference.", "rebuttal": "Thank you for your constructive comments and important questions. We have addressed your main concerns below by adding practical hardware speed benchmarks, including a comparison against recent methods like FlashAttention, and clarifying our design choices.\n\n---\n\n1. Ablation on Visual Contrast Token Generation\n\n- We conduct ablation studies on VCA-DeiT-Tiny: for MaxPool we directly replace the AvgPool layer with a MaxPool operator; for ConvPool, we replace it with a 3×3 stride-2 Conv2d so that the 14 × 14 input still shrinks to 7 × 7. All training hyperparameters, data augmentations, and evaluation protocols were kept identical. The results are listed below:\n\n    | Token generator  | Params | FLOPs | Top-1 Acc. (%) |\n    |------------------|--------|-------|----------------|\n    | **AvgPool(ours)**| 6.0M   | 1.2G  | 75.5 |\n    | MaxPool          | 6.0M   | 1.2G  | 75.2 |\n    | ConvPool         | 10.0M  | 1.4G  | 75.6 |\n\n- Average pooling outperforms max pooling because it preserves information from every patch instead of discarding everything except the single strongest activation. Moreover, its gradients flow through the entire window rather than a lone peak, which makes optimization smoother and more stable. In practice, it raised Top-1 accuracy by 0.3% in our run while costing exactly the same compute and parameters.\n\n- Compared with the 3 × 3 stride-2 ConvPool alternative, AvgPool is far more efficient. ConvPool inflates the backbone by 4 M extra params and 0.2 G additional FLOPs with only 0.1% accuracy gain. Moreover, AvgPool is parameter-free, deterministic and already highly optimised in hardware libraries, whereas ConvPool introduces extra weight initialisation, regularisation choices and kernel-specific latency.\n\n- Hence, AvgPool offers the best trade-off among information preservation, training stability, and computational efficiency, and is adopted as the default visual-contrast token generator.\n\n---\n\n2. Wall-clock training/inference benchmarking & FlashAttention\n\n- We have benchmarked our models to provide concrete performance metrics. We measured training latency (for one ImageNet epoch), peak GPU memory per GPU during training, single-image inference latency (batch size = 1), and inference throughput (batch size = 128).\n\n- Crucially, our **Visual Contrast Attention (VCA) is fully compatible with optimizations like FlashAttention-2**. The core operations in our method (Eq. 9 and Eq. 11 in our submission) can be implemented with `F.scaled_dot_product_attention`, allowing VCA to inherit the speed and memory benefits of such fused kernels. Therefore, we report metrics for both a vanilla implementation and an implementation using FlashAttention-2 (`+FA2`).\n    \n- Due to its focus on language modeling, implementation complexity, and limited rebuttal time, we have not provide RetNet (Retentive Network: A Successor to Transformer for Large Language Models) results within this period.\n\n- **Setup**: Training benchmarks were run on 8xA100 GPUs with a global batch size of 512. Inference benchmarks were run on a single A100 GPU with `torch.compile()` enabled. For inference, we averaged the results over 1000 runs, discarding the first 100 for warmup. For the Diffusion Transformer (DiT), we report the time for a single denoising step (1-NFE) and exclude VAE decoding time.\n\n    | Model | Setting | FLOPs (G) | Acc ↑ / FID ↓ | Peak Mem (GB) | Training Latency (min/epoch) | Inference Latency (ms/image) | Throughput (img/s) |\n    |:---|:---|:---:|:---:|:---:|:---:|:---:|:---:|\n    | DeiT-T | MHSA | 1.2 | 72.2 | 18.9 | 4.1 | 4.2 | 3734 |\n    | DeiT-T | **VCA** | 1.2 | **75.5** | 19.2 | 4.2 | 4.3 | 3701 |\n    | DeiT-S | MHSA | 4.6 | 79.8 | 36.9 | 8.8 | 4.3 | 1321 |\n    | DeiT-S | **VCA** | 4.5 | **80.6** | 37.3 | 8.9 | 4.3 | 1324 |\n    | DiT-S/8 | MHSA | 0.4 | 153.8 | 9.8 | 12.1 | 7.8 | 15637 |\n    | DiT-S/8 | **VCA** | 0.4 | **146.3** | 9.9 | 12.1 | 7.8 | 15602 |\n    |--- |--- |--- |--- |--- |--- |--- |--- |\n    | DeiT-T | MHSA+FA2| 1.2 | 72.2 | 16.5 | 3.1 | 3.3 | 4499 |\n    | DeiT-T | **VCA+FA2**| 1.2 | **75.5** | 16.7 | 3.1 | 3.3 | 4494 |\n    | DeiT-S | MHSA+FA2| 4.6 | 79.8 | 31.1 | 8.6 | 3.4 | 1588 |\n    | DeiT-S | **VCA+FA2**| 4.5 | **80.6** | 31.4 | 8.7 | 3.4 | 1602 |\n    | DiT-S/8 | MHSA+FA2| 0.4 | 153.8 | 9.8 | 12.1 | 7.6 | 16075 |\n    | DiT-S/8 | **VCA+FA2**| 0.4 | **146.3** | 9.8 | 12.1 | 7.6 | 16078 |\n\n- As the results show, **VCA delivers significantly better accuracy/FID with virtually no additional computational overhead**. For example, VCA boosts DeiT-T's accuracy by 3.3 points while maintaining the same FLOPs and nearly identical latency and throughput. The minor overhead from pooling and normalization is effectively optimized away by `torch.compile`, and compatibility with FlashAttention-2 ensures VCA matches the speed of a highly optimized baseline.\n\n---\n\n3. **Speed-up Compared to Models with Similar Performance**\n\n- A key advantage of VCA is its ability to achieve the performance of larger, more computationally expensive models with a more efficient architecture. When comparing VCA-equipped models to stronger baselines with similar accuracy, the speed-up is substantial.\n\n    | Model | Setting | FLOPs(G) | Acc | Throughput (img/s) |\n    |:---|:---|:---:|:---:|:---:|\n    | PVT-T | MHSA | 1.9 | 75.1 | 2770 |\n    | DeiT-T | **VCA** | **1.2** | **75.5** | **3701 (↑33.6%)** |\n    |--- |--- |--- |--- |--- |\n    | Swin-T | MHSA | 4.5 | 81.3 | 1428 |\n    | PVT-S | **VCA** | **4.0** | **82.3** | **1764 (↑23.5%)** |\n    |--- |--- |--- |--- |--- |\n    | Swin-S | MHSA | 8.7 | 83.0 | 952 |\n    | PVT-M | **VCA** | **7.2** | **83.3** | **1076 (↑13.0%)** |\n\n- This comparison highlights VCA's efficiency. For instance, **VCA-DeiT-T surpasses PVT-T in accuracy while offering a 33.6% increase in throughput** on lower FLOPs. This trend holds for larger models, where our VCA-PVT variants outperform stronger Swin Transformer counterparts with **13.0%-23.5% higher throughput**.\n\n---\n\n4. Extension to video and 3D data\n\n- While our experiments in the paper concentrate on static images, the Visual Contrast Attention paradigm is inherently agnostic to the underlying token structure. By replacing the 2D patch pooling with other sparse sampling techniques, one can extend VCA to videos and 3D data with only a few key changes:\n\n- For structured 3D tokens such as videos and dense voxels, 2D pooling could be replaced with adaptive 3D pooling: for video, divide $T$ frames into t segments and average each segment’s $H \\times W$ features to yield $t \\times h \\times w$ tokens per stream, then add fixed spatiotemporal embeddings $E^{+},E^{-} \\in \\mathbb{R} ^{t \\times h \\times w \\times d}$. For volumetric data, pool along depth, height, and width to compress $D \\times H \\times W$ into $d \\times h \\times w$ positions, attach separate volumetric embeddings, and flatten into n×d positive/negative token sets. To handle variable‐length inputs, one can initialize each embedding table to the maximum sequence length in the dataset and then slice or truncate it to match the current sample’s temporal (or spatial) extent.\n\n- For unstructured 3D data like point clouds and meshes, one could sample or coarsen to $n \\ll N$ tokens: for point clouds, one could use FPS plus local feature averaging, coarse voxelization, or lightweight clustering to extract n representative features and then duplicate them into positive/negative streams with distinct embeddings; for meshes or graph-structured data, one could apply edge-collapse mesh-coarsening or graph-pooling (e.g. top-k selection, SAGPool) to reduce the vertex/node count to $n$ and replicate their features similarly.\n\n- In summary, VCA’s two‐stage mechanism only requires swapping the 2D spatial pooling for a suitable spatiotemporal or volumetric pooling/sampling module and adding the corresponding positional embeddings. Everything else—positive/negative streams, differential subtraction, RMSNorm scaling, and two‐stage re-projection—remains unchanged. This design preserves global context at linear cost in the video and 3D domains, just as it does for static images.\n\n---\n\nWe hope this clarifies our contributions and performance benefits, and we await your feedback."}, {"review_id": "3Ih0NZFPGY", "reviewer": "Reviewer_aXpT", "summary": "This paper introduces a novel attention module named Visual-Contrast Attention (VCA), designed as a drop-in replacement for the MHSA layer in standard ViTs. The core idea of VCA is to explicitly inject the notion of \"contrast\" through a two-stage process to enhance the model's discriminative power, while simultaneously reducing the computational complexity of the attention mechanism from quadratic O(N²C) to a near-linear O(NnC), where n ≪ N. The module is plug-and-play, with a minimal increase in parameters and computational overhead (FLOPs). The authors validate VCA's effectiveness on two mainstream tasks: image classification and image generation. The experimental results demonstrate that VCA significantly improves the performance of various ViT architectures without requiring an additional computational budget.", "questions": "My questions are listed in the weaknesses section. If the authors address my concerns, I would prefer to greatly improve the final score.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This paper is an excellent work in general, but with some concerns to be address.\n\n**Strengths**\n1. Explicitly embedding the core idea of discriminative learning—\"contrast\"—into the attention mechanism is a valuable exploration of attention's fundamental nature (from \"similarity\" to \"difference\"). The transition of differential attention design from language models to vision models is reasonable. The linear complexity and its \"plug-and-play\" nature make it easy to integrate into ViT architectures.\n\n2. The experimental results are impressive. The substantial improvement in different variants of models indicates that the method can effectively mine and enhance the model's representation capabilities. Its universal applicability across different architectures and tasks proves that VCA is a highly practical base module.\n\n3. The paper is well-written with clear logic. The flow from motivation, method design, complexity analysis, to experimental validation is coherent. \n\n**Weaknesses**\n1. The ablation studies are not explicit. For example, the choice of $n=h\\times w$ for each model is not specified in the experiment settings. Is it a constant ratio to the number of patch tokens across different models, or a cherry-picked choice for different models and different tasks?\n\n2. It lacks an ablation study on the designs of Visual Contrast Token Generation. For example, MaxPool or parametric ConvPool could be potential designs and compared with the simple AvgPool.\n\n3. The setting for Table 3 is not clear. How to apply Stage I (global contrast) and Stage II (patch-wise differential attention)? A more detailed explanation is required.\n\n4. There are typos in this paper. In L147, there exist two identical symbols $q_1, k_1$. The \"+\" and \"-\" are not the same in Eq. 9 and the equation under L195. And there are many equations without number.\n\n5. The shape of ${\\mathbf{t}^{(m)}_{+}}$ and the negative counterpart is not specified in the neighbour of Eq. 9.\n\n6.  In Eq. 11, the scale factor is still $\\sqrt{d}$ when the dimension of $\\mathbf{v}^{(m)}_{+}$ has become $d/2$. It should be $\\sqrt{d/2}$ or you can provide other explanation.\n\n7. In L302, the authors mentioned \"crucial for vision\". Why is it crucial for vision? The authors could provide more explanation for why the two stages are compatible with the attributes of vision modality and task.\n\n8. If the authors only substitute half of the MHSA block with the VCA block, resulting in an interleaved structure of the two blocks, what performance could it achieve? Is it better to keep both \"similarity\" and \"difference\" pattern learning?", "was_revised": true, "final_justification": "I have read through all the reviews and author responses. I have discussed with the author during the phase and reached a final conclusion based on the rebuttal and discussion. The authors provide comprehensive explanations that have addressed all my main concerns. The supplement reinforces the quality and contribution of the paper. Thus, I raise the final rating from borderline accept to accept, which stands that the paper is ready for publication in the conference.", "rebuttal": "Thank you for your constructive comments and insightful questions. We have addressed your main concerns below by clarifying key experimental settings and notation, adding ablation studies regarding our core design choices, and providing a deeper elaboration on the conceptual basis for our method and potential hybrid architectures.\n\n---\n1.  Ablation on the choice of $n=h \\times w$\n\n- For image classification task, we ablated the number *n* of visual-contrast tokens in VCA-DeiT-Tiny by pooling the original 14 × 14 feature map to 5 × 5, 7 × 7 and 9 × 9 grids (*n* = 25/49/81). Accuracy rises gradually while the parameter and FLOP budgets increase proportionally. We adopt 7 × 7 in the main paper because it is exactly half the original feautre spatial resolution for a 224 × 224 input.\n\n    | grid *h* × *w* | *n* | Params | FLOPs | Top-1 Acc. |\n    |----------------|-----|--------|-------|-----------|\n    | 5 × 5          | 25  | 5.9 M  | 1.2 G | 75.4      |\n    | **7 × 7 (ours)** | 49  | 6.0 M  | 1.2 G | 75.5      |\n    | 9 × 9          | 81  | 6.1 M  | 1.3 G | 75.7      |\n\n- For image generation task,  On VCA-DiT-S/2 we start from a 16 × 16 latent map (for a 256 × 256 noised image) and pool it to 4 × 4, 8 × 8 and 16 × 16. The same trend appears: FID improves as *n* grows. We keep 8 × 8 in the submission, again corresponding to one-half of the original latent spatial size.\n\n    | grid *h* × *w* | *n* | Params | FLOPs | FID-50k ↓ |\n    |----------------|-----|--------|-------|-----------|\n    | 4 × 4          | 16  | 33.2 M | 5.6 G | 65.5      |\n    | **8 × 8 (ours)** | 64  | 33.6 M | 5.9 G | 62.1      |\n    | 16 × 16        | 256 | 33.6 M | 7.3 G | 58.4      |\n    \n - In general, we keep the `h=H/2, w=W/2` choice for all the models, even some model have different feature spatial size in different depth.\n\n---\n\n\n2. Ablation on Visual Contrast Token Generation\n\n- We conduct ablation studies on VCA-DeiT-Tiny: for MaxPool we directly replace the AvgPool layer with a MaxPool operator; for ConvPool, we replace it with a 3×3 stride-2 Conv2d so that the 14 × 14 input still shrinks to 7 × 7. All training hyperparameters, data augmentations, and evaluation protocols were kept identical. The results are listed below:\n\n    | Token generator  | Params | FLOPs | Top-1 Acc. (%) |\n    |------------------|--------|-------|----------------|\n    | **AvgPool(ours)**| 6.0M   | 1.2G  | 75.5 |\n    | MaxPool          | 6.0M   | 1.2G  | 75.2 |\n    | ConvPool         | 10.0M  | 1.4G  | 75.6 |\n\n- Average pooling outperforms max pooling because it preserves information from every patch instead of discarding everything except the single strongest activation. Moreover, its gradients flow through the entire window rather than a lone peak, which makes optimization smoother and more stable. In practice, it raised Top-1 accuracy by 0.3% in our run while costing exactly the same compute and parameters.\n\n- Compared with the 3 × 3 stride-2 ConvPool alternative, AvgPool is far more efficient. ConvPool inflates the backbone by 4 M extra params and 0.2 G additional FLOPs with only 0.1% accuracy gain. Moreover, AvgPool is parameter-free, deterministic and already highly optimised in hardware libraries, whereas ConvPool introduces extra weight initialisation, regularisation choices and kernel-specific latency.\n\n- Hence, AvgPool offers the best trade-off among information preservation, training stability, and computational efficiency, and is adopted as the default visual-contrast token generator.\n\n---\n\n3. A more detailed explanation on Tabel 3\n\n- To isolate the effect of Stage I (Line 1 in Table 3), we can combine it with a standard attention mechanism in the second stage, replacing the patch-wise differential attention. In this setup, Stage I proceeds exactly as described. \n\n- To isolate the effect of Stage II (Line 2 in Table 3), we can precede it with a simplified, non-contrastive summarization step. In this variant, we still perform pooling on the query tokens $\\tilde{\\mathbf{q}}^{(m)}$ to generate a single set of mediator tokens $\\tilde{\\mathbf{t}}^{(m)}$, but we do not create positive and negative versions. This single set of mediator tokens $\\mathbf{t}^{(m)}$ attends to the full key and value sequences to produce one intermediate summary value set, $\\hat{\\mathbf{v}}_{\\text{summary}}^{(m)}$, without any subtraction as in Eq.(10). Then, Stage II is applied as described in the paper, but using this non-contrastive summary. \n\n- For comparsion with differential attention, we also use both differential attention in Stage I and Stage II, as is shown in Line 3, Table 3. This is because differential attention could be plug-and-played and replace with vanilla attention.\n\n---\n\n4. Typos\n\n - We are very grateful to the reviewers for pointing out the typo in our submission.\n - In L147, it should be $\\{\\mathbf{q}_1, \\mathbf{k}_1\\}$ and $\\{\\mathbf{q}_2, \\mathbf{k}_2\\}$\n - The \"+\" and \"-\" symbol should use the same symbol, thank for your carefulness.\n - We have make sure each equation have its number in the revision.\n\n---\n\n5. Tensor shape\n\n- Good suggestion! We have clarity its shape upon Eq.(9) appear in the revision, appraciate it!\n\n---\n\n6. Scale factor\n\n- Thank you for your careful review and for pointing this out. You are correct; the scaling factor in Eq.(11) is a typographical error in the manuscript. Our implementation for all experiments correctly used the standard scaling factor of $\\sqrt{d/2}$. We apologize for this oversight and will correct the equation in the revision.  \n\n- For completeness, we also ran an ablation study on the VCA-DeiT-Tiny model using the incorrect $\\sqrt{d}$ factor from the paper. As shown in the table below, this yielded identical performance to using the correct $\\sqrt{d/2}$ factor, which suggests our model is robust to this specific hyperparameter in this setting.  \n\n\n    | Model | Attention Scaling Factor | Top-1 Acc. (%) |\n    |---------|------------|--------|\n    | VCA-DeiT-Tiny | $\\sqrt{d/2}$ (Correct, as used in our experiments) | **75.5** |\n    | VCA-DeiT-Tiny | $\\sqrt{d}$   (Typo in paper, tested for ablation) | **75.5** |\n    \n---\n\n7. Reason for both stage are crucial for vision\n\n- This two-stage architecture is crucial because it mirrors the principles of efficient visual processing, first addressing the immense spatial redundancy in images by distilling a global \"figure-versus-ground\" context map, akin to identifying the forest before the trees. Subsequently, the patch-wise differential stage allows each local region to query this global map and disambiguate its own role with high precision, deciding whether it belongs to the salient subject or the contextual background. This hierarchical strategy of global abstraction followed by local refinement is not only computationally efficient for high-resolution vision tasks but also aligns with how biological vision systems prioritize and process complex scenes.\n\n- Our ablation results also verify the point our made.\n\n---\n\n8. Results for interleaved structure\n\n- We thank the reviewer for this insightful question. To investigate the performance of a hybrid approach, we conducted two new experiments on the DeiT-Tiny architecture, creating interleaved structures that substitute half of the standard MHSA blocks with our proposed VCA blocks. The results directly address the question of whether it is beneficial to combine both \"similarity\" and \"difference\" pattern learning.\n\n- The first experiment, placing VCA in odd-numbered layers and MHSA in even layers, achieved 75.0% accuracy. The second experiment, with the reverse configuration, yielded 74.3% accuracy. As detailed in the table below, both hybrid models underperform our baseline which uses VCA exclusively, even though they have slightly fewer parameters. The all-VCA model remains superior with an accuracy of 75.5%.\n\n    | Configuration | Parameters | FLOPs | Top-1 Acc. (%) | \n    | :--- | :---: | :---: | :---: |\n    | **All VCA (Baseline)** |**6.0M** | **1.2G** | **75.5%** | \n    | Odd Layers VCA, Even Layers MHSA | 5.8M | 1.2G |75.0% | \n    | Even Layers VCA, Odd Layers MHSA | 5.8M | 1.2G |74.3% | \n\n- These findings suggest that it is more effective to consistently apply a single, coherent learning strategy. Standard MHSA learns through \"similarity,\" while our VCA is fundamentally designed to learn through \"difference.\" The superior performance of the all-VCA model indicates that committing fully to the \"difference\" learning paradigm is more advantageous for this architecture than alternating between the two patterns. This reinforces our central claim that the proposed contrastive attention mechanism is a powerful and sufficient replacement for, rather than a supplement to, standard attention.\n\n---\n\nWe hope this clarifies our contributions and performance benefits, and we await your feedback."}, {"review_id": "NSkP8HaP7H", "reviewer": "Reviewer_Z3tv", "summary": "This paper introduces Visual–Contrast Attention, a novel and lightweight module designed to replace conventional multi-head self-attention in Vision Transformers. VCA enhances inter-token discrimination by introducing visual contrast tokens, which amplify the variance between different visual regions. Extensive experiments and ablation studies demonstrate that VCA significantly improves performance on both image classification and image generation tasks, without introducing substantial computational overhead.", "questions": "Please refer to the weaknesses and provide some explanation for them.  I would be happy to improve the rating.", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The introduction of contrast tokens to promote sparse attention is both intuitive and conceptually appealing.\n\n- VCA is lightweight and can be seamlessly integrated into existing Transformer architectures.\n\n- The proposed method demonstrates consistent improvements across both classification and generative tasks.\n\nWeaknesses:\n\n- The paper lacks comparisons with other attention mechanisms designed for linear complexity, making it difficult to assess the efficiency benefits in a broader context.\n\n- The novelty of the method remains somewhat unclear, especially as it appears to be an adaptation of techniques from the language domain.\n\n- Most experiments are conducted on relatively small-scale models. Given the paper’s emphasis on efficiency, evaluating VCA on larger backbones such as ViT-B and ViT-L would strengthen the claims.\n\n- The performance gain on ImageNet classification is limited, and an ablation study on vanilla ViT is missing, which raises concerns about generalizability.\n\n- The paper would benefit from visualizations and analysis of the VCA mechanism to improve interpretability and deepen the understanding of its behavior.", "was_revised": true, "final_justification": "The rebuttal has satisfactorily addressed most of my concerns. The proposed VCA backbone offers a practical, plug-and-play solution with minimal overhead, which I believe can benefit the community. Overall, I consider this paper to be above the acceptance threshold.", "rebuttal": "Thank you for your valuable feedback. We've addressed your key concerns in our response by expanding our experiments to include larger models and comparisons to other methods, clarifying our method's novelty, and providing new visualizations to improve interpretability.\n\n---\n\n1. Comparion with other linear attention mechanisms\n\n- We conducted additional experiments on both image classification and image generation tasks, comparing our Visual Contrast Attention with several strong linear attention baselines.\n\n- We first benchmarked our method against other efficient attention mechanisms on the ImageNet-1k classification task. All methods were integrated into a DeiT-Tiny backbone with a similar configuration to ensure a fair comparison. As the results summarized in the table below show, our Visual Contrast Attention achieves the highest Top-1 accuracy of 75.5%, surpassing the next best method, Focused Linear Attention, by 1.4%. This performance gain is achieved with a comparable number of parameters and only a marginal increase in FLOPs, highlighting the effectiveness of our two-stage approach in capturing crucial visual information.\n\n    |  Linear Attention   | Params| FLOPs | Top-1 Acc. (%) |\n    |---------------------|-------|-------|-----------|\n    | Hydra Attn          | 5.7 M | 1.1 G | 68.3      |\n    | Efficient Attn      | 5.7 M | 1.1 G | 70.2      |\n    | Linear Angular Attn | 5.7 M | 1.1 G | 70.8      |\n    | Focused Linear Attn | 6.1 M | 1.1 G | 74.1      |\n    | **Visual Contrast Attn (ours)** | 6.0 M | 1.2 G | **75.5** |\n\n- Furthermore, to evaluate generative capabilities, we replaced the standard attention layers in a DiT-S/2 with our method and the baselines. We measured the Fréchet Inception Distance (FID) on 50,000 generated samples, where a lower score indicates better image quality. The results in the following table again demonstrate the superiority of our method. Visual Contrast Attention achieves the lowest FID score of **62.1**, indicating that it generates higher-fidelity images than the other linear attention mechanisms. \n\n    |  Linear Attention   | Params| FLOPs | FID-50k (lower is better)   |\n    |---------------------|-------|-------|-----------|\n    | Hydra Attn          | 33.0 M | 5.4 G | 121.3     |\n    | Efficient Attn      | 33.0 M | 5.1 G | 86.5      |\n    | Linear Angular Attn | 33.0 M | 5.6 G | 78.2      |\n    | Focused Linear Attn | 33.8 M | 5.6 G | 64.7      |\n    | **Visual Contrast Attn (ours)** | 33.6 M | 5.9 G | **62.1** |\n\n- These new experiments provide a clear and compelling demonstration of our method's advantages in a broader context. We have incorporated these results and a detailed discussion into the revised version of our manuscript.\n\n---\n\n2. Novelty in vision\n\n- We thank the reviewer for this insightful question, as it allows us to clarify the core novelty of our work. While the concept of differential attention originates in the language domain, our contribution is a fundamental architectural redesign of this idea to solve problems unique to vision. The original differential attention is \"flat,\" operating across all token interactions, which is computationally prohibitive for high-resolution images. In contrast, our Visual-Contrast Attention (VCA) introduces a novel two-stage, hierarchical bottleneck architecture: it first uses a small set of learnable \"contrast tokens\" to distill a compact global summary of the image's most salient information, and then allows the individual patches to query this summary differentially. This \"global distillation, local differential query\" mechanism is entirely absent in the original work and is specifically engineered to handle the spatial redundancy and quadratic complexity of visual data.\n\n- This architectural innovation allows us to uniquely solve two problems with a single, elegant mechanism. It simultaneously reduces computational complexity from $O(N^2)$ to near-linear and, by forcing information through a contrastive bottleneck, significantly enhances the model's discriminative power, leading to superior performance. Unlike other efficient ViTs that rely on hand-crafted heuristics like local windows (Swin) or striding (PVT), VCA achieves its dual benefit through a more principled, perception-inspired approach. Therefore, the novelty is not in the subtraction operation itself, but in the creation of a sophisticated architecture that intrinsically unifies computational efficiency and representation enhancement, providing a more fundamental and versatile building block for vision transformers.\n\n---\n\n3. Results on larger model\n\n- We thank the reviewer for the constructive suggestion regarding model scalability. To address this, we have conducted new experiments on DeiT-Base, which shares an identical architecture with ViT-B. As shown below, our method boosts the Top-1 accuracy by a notable +0.6% with negligible parameter overhead and no change in FLOPs, validating our efficiency claims on a larger scale. Due to the limited rebuttal period, we were unable to finish the ViT-L experiments, but we commit to including these results in the final camera-ready revision.\n\n    |    Model    | Params |  FLOPs   | Top-1 Acc. (%) |\n    |-------------|--------|----------|----------------|\n    | DeiT-Base   | 86.6 M | 17.6 G   |      81.8      |\n    | +ours (VCA) | 87.1 M | 17.6 G   |      82.4      |\n    \n- Since our claim in efficiency focus on its low complexity w.r.t token length, we add an addition ablation with large image resolution (224 -> 448) on VCA-DeiT-Small, to match the FLOPs of DeiT-Base. The results below verified our method can use a smaller model to handle high resultion and longer token sequences, which achieving better final results. \n\n| Model      | Resolution | Attn Type | Params | FLOPs  | Top-1 Acc. (%) |\n|------------|------------|-----------|--------|--------|----------------|\n| DeiT-Base  | 224x224    | MSHA      | 86.6 M | 17.6 G | 81.8           |\n| DeiT-Small | 448x448    | **VCA**       | 22.6 M | 17.7 G | 83.3           |\n\n---\n\n4. Final performance and Concerns on vanilla ViT\n\n- Regarding the performance on ImageNet, we respectfully argue that characterizing the performance gain as \"limited\" may not fully account for the highly competitive nature of the ImageNet benchmark and the strict constraints of our experiments. Specifically, our VCA module achieves a +3.3% Top-1 accuracy gain on DeiT-Tiny, which is a highly significant result. On the heavily optimized ImageNet benchmark, any improvement over 1%, especially when realized with zero additional computational cost (FLOPs), is typically considered a substantial breakthrough. Crucially, this gain is achieved under \"plug-and-play\" conditions, without any re-tuning of the original models' hyperparameters. This demonstrates that VCA offers a fundamental improvement in representation quality, not just a marginal one, and its consistent effectiveness across diverse architectures (including DeiT, PVT, Swin, and CSwin) strongly supports its generalizability.\n\n- Regarding the missing ablation on vanilla ViT, we chose DeiT as our primary baseline precisely because its network architecture is identical to that of the original Vision Transformer (ViT). The key contribution of DeiT was its significantly more powerful and data-efficient training strategy, which has since established it as the standard and most widely-used baseline in the Vision Transformer research community. Therefore, performing our analysis on DeiT is not only equivalent to testing on the ViT architecture but also ensures our results are directly and fairly comparable to contemporary state-of-the-art methods. This provides a more meaningful and relevant measure of our method's generalizability and practical impact. \n\n---\n\n5. Visualization of Contrast Maps\n\n- Due to updated NeurIPS policies, we cannot include images in this rebuttal. Instead, we provide a textual description of our visualization analysis, as requested.\n\n-   Visualization Method: We visualize the contrast map in the VCA-DeiT-Tiny model, which has 12 attention blocks. For each block, we pick the head with the largest L2-norm of the Stage-I contrast map. We export the positive $\\text{A}^+=\\text{Softmax}(\\boldsymbol{t}_{+}^{(m)} \\boldsymbol{k}^{(m) \\top} / \\sqrt{d})$ and negative $\\text{A}^-$ matrices, compute $\\Delta = \\text{A}^+ - \\lambda \\text{A}^-$ , average the $n$ rows, reshape to 2-D grid, upsample to the original image size and overlay it as a heatmap on the original image—highlighting patches that VCA block promotes or suppresses.  \n\n-  **Observations**: In **early blocks** (e.g., block 3), the positive map $A^{+}$ tends to activate on coarse object edges, while the negative map $A^{-}$ activates on background textures. The resulting differential map $\\Delta$ often exhibits a classic center-surround pattern, separating the foreground subject from its context. In **late blocks** (e.g., block 10), $A^{+}$ sharpens its focus on semantic parts (e.g., the head and tail of a bird), while $A^{-}$ becomes more diffuse, indicating that distracting correlations have been effectively filtered out.\n\n-  **Failure Cases**: We observed that the contrastive separation in $\\Delta$ is less effective in two main scenarios: (i) **texture-dominated scenes**, such as an image of \"leopard skin,\" where both positive and negative maps activate strongly on the same texture; and (ii) images with **very small objects** that are smaller than the resolution of our pooled grid (e.g., 7×7), making it difficult to generate distinct contrastive views. In these cases, VCA's benefit is diminished.\n\n---\n\nWe hope this clarifies our contributions and performance benefits, and we await your feedback."}]

Author rebuttal:

Meta-review: The submitted paper addresses the attention computation in transformer and replaces the traditional MHA by a new learnable variant of differential transformers, where dense queries are grouped into positive and negative parts. The paper received 4 reviews with ratings which were all above borderline after the author/reviewer discussion phase. One additional 5th reviewer was in the system but couldn't provide a review and the AC was unable to remove them. The corresponding review and rating were completely ignored during the peer review process.

The reviewers appreciated the motivation, clear methodology, the idea of embedding the notion of contrast into an attention mechanism, and experimental validation, with multiple tasks having been tested.

Weaknesses were 
- lukewarm gains in concrete speed-ups / exact numbers on this
- Some missing ablations, and analysis of token generation.
- the lack of baselines with linear attention 

The authors have provided a rebuttal with new experiments, including detailed wallclock timings, and comparisons with the requested baselines, but also improved the justification of the methodology.

The reviewers agreed that this paper is a valuable contribution to the community and the AC concurs.
Decision: Accept (poster)