DiffCLIP: Differential Attention Meets CLIP
Authors: Hasan Abed Al Kader Hammoud
Bernard Ghanem
Venue: Accepted by TMLR ()

Abstract: We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures. Differential attention was originally developed for large language models to amplify relevant context while canceling out noisy information. In this work, we integrate this mechanism into CLIP's dual encoder (image and text) framework. With minimal additional parameters, DiffCLIP achieves superior performance on image-text understanding tasks. Across zero-shot classification, retrieval, and robustness benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably, these gains come with negligible computational overhead, demonstrating that differential attention can significantly enhance multi-modal representations without sacrificing efficiency.

Published in Transactions on Machine Learning Research (08/2025)
DiffCLIP: Differential Attention Meets CLIP
Hasan Abed Al Kader Hammoud
King Abdullah University of Science and Technology (KAUST)
Bernard Ghanem
bermand.ghannem@@auut..ckk
King Abdullah University of Science and Technology (KAUST)
Reviewed on OpenReview: https: // openreview. net/ ftt/fform? id= 2d22fTerr
Abstract
We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures. Differential attention originally developed for large language models to amplify relevant context while canceling out noisy information. In this work, we integrate this mechanism into CLIP's dual encoder (image and text) framework. With minimal additional parameters, DifCCLIP achieves superior performance on image- text understanding tasks. Across zero-shot classification, retrioval, and robustness bench- marks, Dif[CLIP consistently outperforms baseline CLIP models. Notably, these gains come with negligible computational overhead, demonstrating that differential attention can signif- icantly enhance multi-modal representations without sacrificing efficiency. Code and models can be found at setps://gitonn com comp///anooorhsssss///if...
1 Introduction
Vision-language models (VLMs) have made remarkable progress in bridging the gap between textual and visual modalities, enabling powerful capabilities such as zero-shot image classification, image-text retrieval, and descriptive captioning (Radford et al., 2021; Jia et al., 2021). By aligning images and text in joint embedding space, these models capture broad semantic relationships across modalities and often excel at out- of- of-distribution generalization. Among VLMs, Contrastive Language Language-Image Pre-training (CLIP) (Radford et al., 2021) stands out as foundational approach, demonstrating strong zero-shot performance on numerous benchmarks with minimal fine fine-tuning.
While CLIP's contrastive training regime has been widely adopted, its attention mechanism can sometimes focus on irrelevant spurious features in both the image and text encoders. This attention noise hamper fine-grained understanding, particularly when precise localization or explicit contextual knowledge is required. Interestingly, recent. language modeling research has proposed a differential atlention mechanism (Ye et al., 2024), which subtracts complementary attention distributions to suppress noise and highlight salient tokens. However, whether a similar strategy would be effective for multimodal tasks has remained an open question.
"Can differential attention be adapted to vision-language models in a way that meaningfully improves their ability to focus on relevant features across modalities?"
Motivated by this question, introduce DiffCLIP, extension of CLIP that integrates differential attention into both the vision and text encoders. By learning two attention maps and subtracting one from the other, DiIECLIP effectively cancels out misaligned or noisy signals, enabling a more precise alignment of images and text. Crucially, this enhancement introduces only a negligible overhead in model parameters and computational cost. Our results show that DiffCLIP consistently outperforms standard CLIP wide range of tasks, including linear probing, few-shot classification, image text retrieval, outside-domain robustness, and fine-grained visual understanding, highlighting the efficacy of differential attention in multimodal

---

Published in Transactions on Machine Learning Research (08/202))
CC3M Pretraining CLIP vs. DiffCLIP
Linear Probing
Few Shot
Image Retrieval
80
30
Accuracy (%6
60
40
20
40
20
20-
0
CLIP DiffCLIP
CLIP DiffCLIP
CLIP DIffCLIP
Text Retrieval
ImageNet Zero-Shot
Zero Shot OOD
1.55
40
15
10.0
Accuracy
10-
20
5.0
5
10
2.5
CLIP DifíCLIP
CLIP DiffCLIP
CLIP DiffCLIP
Figure CC3M Pretraining: CLIP vs. DiffCLIP Across Six Tasks. We compare standard CLIP (bluc) and our DiffCLIP variant (pink) on linear probing, few-shot classification, image /tee/ttttttttee retricval, zero- shot ImageNet, and zero-shot OOD. In each case, DifICLIP consistently oulperforms CLIP, highlighting the effectiveness of differential attention with only 0.003% extra parameters.
setting. As shown in Figure 1, DifICLIP is capable of improving performance across various benchmarks with only 0.003% extra parameters. Figure 2 also shows how DiffCLIP is capable of suppressing attention noise compared to CLIP models with vanilla non-differential allention.
Our contributions are threefold:
We propose DifICLIP, the first integration of differential attention into CLIP-based VLMs, yielding simple yet effective approach reducing attention noise in both vision and text streams.
Through extensive experiments on Conceptual Captions 3M/12M pretraining, we demonstrate con- sistent gains over baseline CLIP across diverse suite of tasks, with a minimal parameter overhead of roughly 0.003%.
We perform detailed ablations, showing that (i) dynamic initialization can boost zero-shot perfor- mance, and (t)) applying differential attention solely in the vision encoder already captures most of the benefits, suggesting flexible and cost-elfective path to improved multimodal learning.
The remainder of this paper is organized follows. Section 2 surveys previous work training-centrit model-centric, and data-centric strategies for enhancing CLIP. Section 3 provides overview of the standard Transformer attention mechanism, the differential attention concept, and the CLIP framework. Section 4 details our experimental setup, empirical results, and ablation studies, while Sections 5 and 7 concludes with a discussion of future research directions and wrapping up of the paper.
2 Related Work
Vision-language pre-training (VLP) has advanced our ability to learn joint representations images and text, lcading to improvements in tasks such as image retrieval, visual question answering, and zero-shot classification (Gan al., 2022: Zhang et al., 2024). CLIP (Radford et al., 2021) has been central to this progress by using contrastive loss to align image and text embeddings from large-scale image-caption data.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Original
CLIP
DiffCLIP
CLIP
DiffCLIP
Original
CLIP
DiffCLIP
CLIP
DiffCLIP
Figure 2: Comparing CLIP vs. DiffCLIP Attention Maps. For two images (rows), we visualize where CLIP and DiffCLIP attend when matching each image against different textual queries. While CLIP allocates attention to irrelevant background regions, DiffCLIP more effectively centers query-relevant objects, highlighting how differential attention can reduce noise and improve focus. Queries: First Row: "Mug", Lamp", ; Second Row: Flower", Dog".
Despite CLIP's strong zero-shot performance, researchers continue to explore improvements in its training, architecture, and data collection strategies. These efforts generally fall into three categories: ceaning-centric, model-centric, and data-centric approaches.
Mhaiiig-CCettii Approaches A common strategy is to enrich CLIP's contrastive framework with ad- ditional objectives. For example, SLIP (Mu et al., 2022) adds masked image modeling to boost downstream results, while DeCLIP (Li al., 2021) uses nearest-neighbor supervision enhance data efficiency. SigLIP (Zhai et al., 2023) replaces the standard softmax temperature with sigmoid loss, allowing larger batch training and improving generalization and robustness to noisy labels. Retrieval-Fnhancee CLIP (Iscen et al., 2023) leverages external memory of image-text pairs at inference, achieving significant gains on finc-grained zero-shot tasks. Further, novel training objectives, such as those proposed by Yang et al. (Yang et al., 2022) and PyramidCLIP (Gao et al., 2022), aggregate information across multiple semantic levels, highlighting the benefit of diversified training signals for improved CL.IP performance.
Model-Centric Approaches Another line of work modifies CLIP's architecture for greater efficiency accuracy. The original CLIP (Radford et al., 2021) employs Transformer (Vaswani et al., 2017) for text and either ResNet (He et al., 2016) or Vision Transformer (VIT) (Dosovitskiy et al., 2020) for images. Subsequent studies incorporate ideas from object detection and segmentation to capture finer visual details, such as region-level representations (Xu et al., 2022; Zhong et al., 2022). Recently, ViTamin (Chen et al., 2024) proposed a specialized vision transformer architecture tailored specifically for multimodal models, demonsttatingg improved zero-shot results compared to standard ViTs under similar training setups. Other researchers attempt unify image and text encoders into single Transformer (Tschannen et al., 2022), although this approach is less common. Notably, few methods have altered the attention mechanism within CLIP. Our work addresses this gap by adapting Differential Atlention (Ye al., 2024), originally proposed for language models, to CLIP's multimodal setting. This adaptation aims to reduce attention noise and enhance representation quality.
Approaches Data-centric methods emphasize improving the size, diversity, and quality of pre-training datascts. Initial efforts focused on scaling datasets (Jia. al., 2021; Radford al., 2021), while more recent approaches prioritize richer and cleaner supervision. VeCLIP (Lai al., 2024) uses large language models (LLMs) to generate detailed and enriched captions, enhancing textual supervision.
3

---

Published in Transactions on Machine Learning Research (08/2025)
Similarly, CLIPS (Liu et al., 2024b) utilizes truncated synthetic captions to improve visual grounding and retrieval performance, showing that carefully controlled synthetic textual inputs can surpass standard image-caption pairs. SynthCLIP (Hammoud et al., 2024) explores training entirely on synthetic image-text pairs. Further methods employ filtering techniques to climinate noisy or irrelevant samples (Gadre et al., 2023; Abbas et al., 2023), while Cluster Masking (Wei al., 2024) proposes masking clusters of similar image patches, leading to faster training and improved representation quality. These efforts underline the potential of data curation and augmentation strategies in bolstering the efficacy of CLIP-based models.
Beyond performance, fairness and compositionality have also received incrcased attention. FairCLIP (Luo et al., 2024) addresses demographic biases found in models like CLIP by using opaimall-raussoot-bbsedd fea- ture alignment across demographic groups. Meanwhile, iterated learning approaches (Zheng et al., 2024) tackle the compositional limitations of large vision-language models, promoting representations that gener- alize more reliably to complex and compositional linual-linguisti scenarios.
In this paper, we contribute to the model-centric direction by adapting Diffcrential Attention (Ye et al., 2024) to CLIP's dual-encoder archilecture. Through this adaptation, we aim reduce atlention noise and enhance performance across various image-text understanding tasks.
3 Preliminaries
In this section, we outline the fundamental concepts that essential for approach. We begin by reviewing the Transformer self-attention mechanism (Vaswani et al., 2017), which is widely used in modern sequence modeling. Next, we introduce differential attention (Ye et al., 2024), technique designed to reduce attention noise by leveraging complementary attention distributions. Finally, we summarize the Contrastive Language-Imago Pre-training (CLIP) framework (Radford et al., 2021), which learns to align images and text in a shared representation space. These components form the basis for our model and experiments.
3.1 Transformer Attention
Transformer networks (Vaswani et al., 2017) capture relationships among elements in a sequence through a self- aelf-attention operation. Let
X E
be input sequence of N tokens (or image patches), each embedded in d-dimensional space. The Transformer maps X to queries (Q), keys (K), and values (V) using learned weight matrices:
Q= x K X XW
where Wº, WK WV Rddd Self-attention scores are then computed via scaled dot-products:
A softmax QK"
and these scores are used to weight V:
Attn AV.
To capture different types of relationships, Transformers use multi-head attention (MHA). An MHA module with h heads splits each projection into lower-dimensional parts of size d, = d/h. In each head i,
Attn:(X) = softmax
Q.K Vil
V
where Qi K, V, xxdd The head outputs are concatenated and projected back:
MHA(X) [Attn,(X) ... (ttn, w
with Wo E ddd Despite remarkable success in many areas, standard attention can assign non- negligible weights to irrelevant tokens (often called attention noise) (Kamradt, 2023; Liu et al., 2024a), which degrade performance in settings requiring precise focus.
4

---

Published in Transactions on Machine Learning Research (08/2025)
3.2 Differential Attention
Differential allention (Ye et al., 2024) addresses allention noise by learning two separate attention distribu- tions and subtracting one from the other, effectively canceling out spurious alignments.
Single-Head Differential Attention. Let X E R be the input to single attention head. We split Q and K into two halves, denoted by subscripts and 2:
Qi Q2] = x WQ; Ki; K2) = XWK V=
where Q1, Q2, K1 K2
N Each half computes its attention distribution:
A = softmax
9.K Va/2
= softmax
Q2k. Vaf2
The output is formed by subtracting the second distribution (scaled by learnable parameter A) from the first:
DiffAttn(X) (Ai - A V.
The parameter X is trained to control how strongly the second distribution subtracted:
A = exp(>g: Ak,) - exp(>xx )ka) + Amit.
where Agu Aks Aga Ab are learnable weights and Ajni is a hyperparameter. This subtraction often yields sparser, more focused attention map, which can improve results in scenarios sensitive to background or redundant signals (Ye et al., 2024).
Multi-Head Extension. Like standard attention, differential attention can be extended to multiple heads. In Differential Multi-Head Attention (Diff MHA), each head i applics the differential step independently:
QI.:K. /dn/2
Q2 2.1
) V
DiffAttn,(X)
softmaxx
Asoftmax
V /dn/2
where Q1,i, Q2 K K2, E (Nddd/2)) The final output is then
DiffMH((X- [DiffAttn;XX) ... [iffAttn,(X)) wo.
By learning complementary attention maps in each head and subtracting them, Diff MHA aims to amplify relevant patterns while reducing noise.
3.3 CLIP Training
Contrastive Language-Image Pre-training (CLIP) (Radford et al., 2021) learns image and text embeddings in shared space using large collection of paired image-text examples {(T,, T J=z It consists of two encoders: one for images (fo) and one for text (go). Their outputs are normalized to unit length:
fo(T.)
g (T,).
llg
For a batch pairs, CLIP forms a similarity matrix
uf Us
S
where F is a (learned or fixed) temperature parameter. The text-to-image contrastive loss
N
log exp(S.s j= exp(Si;))
Lu = N

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Classification Performance (Linear Probing and Few-Shot). We compare CLIP and Dif- ICLIP on nine classification tasks with two pretraining sets (CC3M and CC12M). The top block reports linear probing accuracy, while the bottom block shows few-shot results. Numbers in parentheses indicate
absolute gains or drops for DiffCLIP relative to CLIP.
Pretraining
Model
Caltech-101
DTD
Pets
Flowers
SUN397
Aireraft
CIFARIO
CIFARI00
Food-101
Avg.
Probing
CC3M
CLIP
72.5
58.7
61.0
85.8
54.1
35.7
83.5
63.4
59.1
61.8
CC3M
DIFCLIP
(6.2(-3.7)
(0.2(-155)
(-12
66.0 (+0.9)
(422()
(-1.))
(-0.4)
63.7 (-0.3)
(0.4
(418
CC12M
CLIP
88.3
71.2
79.5
92.6
68.3
48.8
92.0
74.7
77.5
77.0
CC12M
DIUCLIP
(9.5
(-.8
(-8
93.5 (+0.0
69 (++1))
46.4
(0.7
73.3
77.7 (-0.))
(
Few-Shot
OCSM
90.4
44.6
67.0
73.9
CC3M
DIICLIP
91.6(-12)
73.2 (-.2
(-.6
92.9 (.0.4
92. (.100)
(5.4
(2.4
(--5)
08.3 (-13)
74.6
CC12M
CLIP
97.1
81.9
86.3
96.9
96.5
56.1
81.3
85.1
86.0
85.3
CO12M
DIFCLIP
97.6(-0.2)
(-.2(-0.))
(-
(+0.4)
96 (+0.3)
56.2 (0.0)
80.3
(...
(7.5(
(-04
Table 2: Zero-Shot Retrieval and ImageNet Zero-shot Accuracy, We report image and text retrieval (Recal1@5, %) and zero-shot ImageNet accuracy (%) for CLIP vs. DiffCLIP, using CC3M or CC12M pretraining data. Values in parentheses reflect absolute gains drops for DiffCLIP relative to CLIP.
Pretraining Model
Image Retrieval (R@5)
Text Retrieval (R@5)
Zero-Shot
Flickr30k
Fllckr8k
MSCOCO
Avg.
Flickr30k
Fliekrök
MSCOCO
Avg.
ImageNet
CCBM
CLIP
31.8
35.4
19.4
28.9
43.4
46.2
25.4
38.3
13.6
CC3M
DIFCLIP
32.9
36.5
20.9
(0:1))
41.7 (+18)
47.8 (718()))))
(7.6(+2.2)
(0.1
144(40.8)
CC12M
CLIP
62.5
41.3
55.3
77.7
69.4
CC12M
DIICLIP
62.2 (-0.3)
61.5 (-0.0)
(4-3
55.3 (49.0)
77.4 (+0.8)
(7.4
(-1.)))
70.1
33.8
and the imago-to-text counterpart is
1 N N
La
log
exp( S:.)
== exp(S,))
The overall objective
LCLIP = {Lux + Cx)
By encouraging matching image-text pairs have high similarity (and mon-mac pairs have low sim- ilarity), CLIP learns robust features that often transfer well downstream tasks like zero-shot classification and retrieval.
4 Experiments
We present an extensive empirical study to investigate whether differential attention can benefit CLIP-style vision-language models. We first describe our dataset sources and training configurations, then evaluate both standard CLIP and our DiffCLIP variant under linear probing, few-shot classification, and image-text retrieval. We also test robustness to distribution shifts (via OOD ImageNet) and fine-grained features (via MMVP), and conclude with ablation studies the initialization of the differential attention parameter Annt and on applying differential attention to only the vision encoder.
4.1 Experimental Setup
Datasets. We pretrain on Conceptual Captions 3M (CC3M) (Sharma al., 2018) and Conceptual Captions 12M (CC12M) (Changpinyo et al., 2021). After downloading using img2dataset. (Beaumont,

---

Published in Transactions on Machine Learning Research (08/2025)
CC3M Pretraining
CC12M Pretraining
0.8
3.3
Accuracy (%)
15
0.2
0.3
40
A 2.0
10
30
1..0
2.8
2.1
0.1
20
0.4
10
ImageNet
IN-v2 IN-A
IN-R
IN-Sketch
Avg.
ImageNet
IN-V2
IN-A
IN-R
IN-Sketch
Avg.
CLIP
DiffCLIP
Figure 3: OOD Zero-Shot ImageNet Performance. Comparison of zero-shot accuracy (%) on Image Net, ImageNet-V2,, ImageNet-A, ImageNet-R, and ImageNet-Sketch, plus the average. Bars show performance of CLIP (blue) versus DifiCLIP (pink), trained on CC3M (left) or CC12M (right). Numerical deltas above the bars indicate the absolute improvement or drop for DiffCLIP relative to CLIP. DiffCLIP improves on average the zero-shot performance on OOD ImageNet datascts as compared CLIP.
2021) (with shorter edge resized to 224), we end up with about 2.3M image-text pairs for CC3M and 7.9M for CC12M. For CC3M, we train on four A100 GPUs, while CC12M uses eight A100 GPUs to reduce training time. Text data minimally processed, limited basic tokenization.
Training Parameters. All models train for 40 epochs, using epoch of lincar warmup, global batch size of 4096, and AdamW optimizer (Loshchilov & Hutter, 2017). We set the base learning rate to 5 10-1 with weight decay of 0.5. For DiffCLIP, every attention layer in both the vision and text encoders is replaced with differential attention. We initialize each layer's X at 0.8 unless stated otherwise. This setup introduces only a minor parameter overhead: roughly 0.003% additional parameters relative to a standard CLIP-B/16 Training parameters are chosen similar to SynthCLIP (Hammoud al., 2024) and training code adopted from SLIP (Mu et al., 2022).
Evaluation Protocol. We follow established practices for linear probing and few-shot evaluation (El Ba- nani et al., 2023) on nine image-classffiationn datasets: DTD (Cimpoi al., 2014), Flowers (Nilsback & Zis- serman, 2008), Pets, Caltech-101 (Li et al., 2022), Aircraft (Maji al., 2013), CIFAR-10 (Krizhevsky et al., 2009), SUN397 (Xiao et al., 2010), CIFAR-100 (Krizhevsky el al., 2009), and Food-101 (Bossard et al., 2014). For retrieval (image-to-text and text-to-image) on Flickr8k (Rashtchian et al., 2010), Flickr30k (Young et al., 2014), and MSCOCO (Lin et al., 2014), we use the LAION CLIP Benchmark framework (Schuhmann et al., 2022). We measure zero-shot robustness on ImageNet (Russakovsky et al., 2015) and variants (ImageNet- V2 (Recht et al., 2019), ImageNet-A (Hendrycks et al., 2021b), ImageNet-RR (Hendrycks al., 2021a), and ImageNet-Sketch (Wang al., 2019)). Finally, use the MMVP-VLM benchmark (Tong al., 2024) to check how well each model focuses on fine-grained visual details.
4.2 Do CLIP Models Benefit from Differential Attention?
To evaluate the effectiveness of our proposed DifICLIP, test its performance across lasks involving image classification, image-text retrieval, and zero-shot generalization, following common bench- marks ostablished in prior literature (Hammoud et al., 2024).
Motivation.
Results. We compare a baseline CLIP-B/16 to our DiICLIP-B/166 (with differential atlention in both vision and text encoders). Table shows linear probing and few-shot classification results for models pre- trained on both CC3M and CC12M. DiffCLIP outperforms standard CLIP on almost every dataset. For example, with CC3M pretraining, DiffCLIP achieves about +1% gain in linear probing and +0.7% in few- shot accuracy.

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2 presents retrieval metries and zero-shot ImageNet. DiffCLIP again surpasses CLIP image and text retrieval: for CC3M, we see average improvement. of about 1.2% (image retrieval) and 1.8% (text retrieval). On zero-shot ImageNet, DifCCLI--CCM increases accuracy by 0.8%, with even larger gains of +2.0% when using CC12M.
Conclusion. Even though DifCCLIP only adds a tiny fraction extra parameters, consistently outperforms standard CLIP on classification and retrieval benchmarks. This suggests that differential attention is lightweight yet effective way to enhance vision-language representation.
4.3 Does Differential Attention Improve cut-of-Domain Robustness?
Motivation. Having observed improvements on in-distribution ImageNet, we ask if these gains carry over to more challenging out-of-domain variants. Real-world applications often involve domain shifts, and CLIP's zero-shot adaptability has been tested on ImageNet- ImageNct-A, ImageNet-R, and ImageNet- Sketch benchmarks known stress model robustness beyond standard ImageNet. Understanding how differential attention influences robustness in such scenarios is crucial for assessing its practical utility in deployment settings. We aim to see if differential attention helps maintain or improve performance under such shifts.
Results. Figure 3 summarizes zero-shot performance across ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-Sketch. Models with differential attention outperform standard CLIP by an average of 2.1%, suggesting that subtracting noisy attention patterns yields features that generalize more robustly, even under significant distribution shifts.
Conclusion. DifICLIP not only enhances in-distribution performance but also strengthens zero- shot robustness against substantial domain shifts, further demonstrating the benefits of differential attention.
4.4 Does DiffCLIP Improve Fine- Grained Visual Understanding?
MMVP-VLM Benchmark. To test fine-grained visual understanding, we employ the MMVP-VLM benchmark (Tong et al., 2024). This benchmark mcasures how well vision-language models capture mu- anced visual properties, such object orientation, presence, and relational context, beyond straightforward recognition. Both CLIP and DiffCLIP are pretrained CC12M under identical settings.
Results. On average, DIfICLIP improves MMVP-VLM accuracy by 5.7% relative to baseline CLIP. A radar plot (Figure 4) shows DiffCLIP surpassing matching CLIP on nearly all categories except one (state and condition). This suggests that subtracting noisy attention patterns (via differential attention) helps the model attend to more subtle details images.
Interpretation. The MMVP benchmark is specifically designed to require precise visual focus on relevant object details in order to answer fine-grained questions correctly. Therefore, the observed +5.7% absolute improvement over baseline CLIP provides strong quantitative evidence that DiffCLIP's reduced attentional noise translates directly into more accurate and discriminative visual focus. This aligns with our qualitative visualizations, where DiffCLIP consistently attends to the most semantically relevant regions, supporting our central claim that differential attention enhances the quality of attention maps in CLIP-style modols.
Conclusion. By mitigating extrancous context through differential attention, DiffCLIP achieves stronger fine-grained visual understanding. These gains highlight the effectiveness of explicitly can- celing irrelevant attention weights in multimodal settings.

---

Published in Transactions on Machine Learning Research (08/2025)
State and Condition
Presence of Specific Features
Positional and Relational Context
80%
50%
20%
Orientation Direction
Color and Appearance
Model Performance CLIP (avg: 21.9%)
(avg: 27.6%)
Viewpoint and
Perspective
structural Characteristics
Figure MMVP-VLM Benchmarking. Radar plot illustrating performance on different fine-grained visual categories. Both models (CLIP blue, DiffCLIP in pink) are evaluated on properties like orientation, positional context, and color appearance. DifICLIP (average 27.6%) consistently oulperforms CLIP (average 21.9%), demonstrating more focused attention subtle visual details.
4.5 Dynamic or Static Amnit? ?
Motivation. All previous experiments used fixed initialization Aini 0.8 for differential attention.
However, (Ye et al., 2024) proposes dynamic schedule:
Amit(() 0.8 0.6 exp(-0.31),
where / is the layer index. We denote the model using this schedule DiffCLIP".
Results. Figure 5 summarizes six tasks: linear probing, few-shot classification, image retrieval, text re- trieval, zero-shot ImageNet, and zero-shot OOD. Compared to the baseline CC12M CLIP, DiffCLIP" im- proves zcro-shot ImageNet by +2.8% and text retrioval by +1.5%. It also raises zero-shot OOD accuracy by +1.3%. However, relative to standard DiffCLIP (with fixed Amit = 0.8), DiffCLIP" is +0.8% better on zero-shot ImageNet and +0.8% on text retrieval, but underperforms or only slightly improves on other tasks. For instance, in zero-shot OOD, DiffCLIP* is -0.8% behind standard DifICLIP.
Conclusion. A dynamic À schedule yields notable gains on zero-shot ImageNet and text retrieval, though lags behind the simpler constant initialization on several other benchmarks. Future work might explore how best to tune or combine these schedules to achieve consistent improvements.
4.6 Does Applying Differential Attention Vision Only Suffice?
Motivation, Because the vision encoder often plays a dominant role in CLIP models, one might ask whether differential attention is necessary in both encoders. We define DiffCLIP" as a variant that integrates differential attention only in the vision encoder, leaving the text encoder with regular attention.
Results. Figure compares CLIP, DiffCLIP, and DiffCLIPt across six tasks: lincar probing, few-shot classification, image retrieval, retrieval, zero-shot ImageNet, and zero-shot OOD. DiffCLIP' improves upon baseline CLIP by +0.1% in linear probing, +0.3% in few-shot, +0.4% in image retrieval, +1.2% in text

---

Published in Transactions on Machine Learning Research (08/2025)
CC12M Pretraining
100
CLIP DiffCLIP
DiffCLIP* DiffCLIP
Accuracy (%)
75
25
Linear Probing
Few Shot
Image Retrieval
Text Retrieval
ImageNet Zero-Shot
Zero Shot OOD
Figure 5: Comparing Different DiffCLIP Variants. We evaluate four models on six tasks (linear probing, few-shot, image retrieval, text retrieval, ImageNet zero-shot, and zero-shot OOD), all pretrained on CC12M. CLIP (blue) the baseline, DifCCIIP (pink) uses a fixed differential attention parameter, DiffCLIP* (purple) employs & dynamic schedule for differential attention, and DiffCLIP! (yellow) applies differential allention only to the vision encoder.
retrieval, +1.9% on zero-shot ImageNet, and +2.3% on zero-shot OOD. Compared to DiffCLIP, DiffCLIP surpasses or matches performance on few-shot, image retrieval, text retrieval, and zero-shot OOD, but is slightly behind on linear probing and standard zero-shot ImageNet.
Conclusion. Applying differential attention solely to the vision encoder already brings sizable gains. Interestingly, DifECLIP can even match or exceed full DifCCLIP on certain tasks, suggesting that most of the performance boost may come from more robust visual feature extraction.
5
Future Directions & Limitations
5.1
Beyond CLIP
An intriguing question for future research is how vision encoder trained with differential attention within the CLIP framework would perform when integrated into larger, more sophisticated vision-language models such LLaVA (Liu al., 2023) or TinyLLaVA (Zhou et al., 2024). To provide initial insights into this possibility, conducted preliminary experiments by combining our SIFCLIP-CC121 vision encoder with the Doven-2---nnstrutt-0.55 (Yang et al., 2024) language encoder. We followed a typical two-stage training procedure: first, a linear projector was trained to align visual tokens with the language embedding space, freezing all other components; second, both the projector and the language encoder underwent instruction fine-tuning.
For the projection training, we utilized the LAION-CC-SBU dataset (558K image-text pairs) used in the LLaVA training setup. For instruction fine-tuning, adopted the COCO (Lin et al., 2014) subset (approx- imately 350K pairs) also used by VA. All experiments were conducted using the TinyLLa VA repository
Table 3: CLIP DiffCLIP POPE Hallucination Benchmark. We compare models across three POPE categories, showing accuracy, precision, and recall. Absolute improvements of DiffCLIP over CLIP
are highlighted in pesentheses.
CLIP
DiffCLIP
POPE
Accuracy
Precision
Recall
Accuracy
Precision
Recall
Random
50.14
50.07
98.21
50.41 (4-.27)
50.21 (+0.14)
98.56 (+0.35)
Popular
50.27
50.13
99.33
50.27 (+0.00)
50.13 (+0.00)
99.47 (+0.14
Adversarial
50.17
50.08
99.33
50.20 (+-003))
(+0.02)
99.47 (40.44)
10

---

Published in Transactions on Machine Learning Research (08/2025)
4 A100-80GB GPUs. The hyperparameters for fine-tuning included batch size of 48 samples per GPU, learning rate of 2 x 10-5, xero weight decay, a warm-up ratio of 0.03, and cosine decay scheduling. The projection pretraining similarly employed 48 samples per GPU, a learning rate of 1 x 10-3, no weight decay, a warm-up ratio of 0.03, and cosine decay scheduling.
We evaluated the resulting models on the POPE (Li et al., 2023) hallucination dataset, which assesses models' susceptibility visual hallucinations. Despite the modest size of observed improvements, DifICLP--CI2M consistently outperformed the CLIP-CC12M baseline across all metries (Table 3). These initial findings suggest that differential invention-trainee vision encoders could enhance performance when integrated into broader vision-language architectures, making it promising direction for further exploration.
5.2 Scaling Data and Architecture
Training CLIP models with a ViT-B/16 backbone on the CC12M dataset (7.9M samples) currently requires approximately 10 GPU-days on A100 GPUs, translating to roughly $600 using Google Cloud Platform (GCP). natural future direction would involve exploring how differential attention performs when scaling to larger architectures (e.g., VIT-L ViT-H) and substantially bigger datasets (e.g., LAION-400M). In- vestigating such scaling could reveal whether the performance gains observed with DiffCLIP persist even amplify as model size and dataset scale increase, offering insights into the broader applicability and benefits of differential attention in vision-language pretraining.
6 Limitations and Future Work
Although DiffCLIP demonstrates consistent gains in a wide range of tasks, several limitations remain. First, due to computational constraints inherent to academic settings, we are unable to perform large-scale multi- seed pre-training runs (e.g., on LAION-400M). Instead, we focus on smaller scale datasets and we provide statistical measures only for nondeterministic evaluations (linear probing, few-shot), and rely the breadth and consistency of results supporting evidence. Second, while preliminary results integrating DifCCLIP-trained encoders into larger vision-language models (e.g,, L.LVV--slyle are promising, exploring this direction at scale remains future work. We believe that these extensions, particularly scaling to larger datasots and architectures, will further reveal the potential of differential attention in multimodal learning.
7 Conclusion
We introduced DiffCLIP, which integrates differential attention into CI.IP-based vision-language models better filter out noisy alignments. Through extensive experiments on classification, retrieval, robustness, and fine-grained benchmarks, DiffCLIP consistently improves over standard CLIP with minimal overhead. Further ablations highlight the flexibility of dynamic attention schedules and vision-only setups. We hope these findings inspire future rescarch on more efficient, robust attention mechanisms in large multimodal learning.
8 Acknowledgements
The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940.
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023. 4
Romain Beaumont. img2dataset: Easily turn large sets of image urls to an image dataset. https://github htmn//oomm504//mm2daaaaeee 2021.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 3ood-101-mining discriminative components with random forests. In ECCV, pp. 446-461. Springer, 2014. 7
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pro-training to recognize long-tail visual concepts. In CVPR, 2021.
Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang-Chied Chen. Vitamin: Designing scalable vision models in the vision- laigunla era. In CVPR, pp. 12954-12966, 2024.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014. 7
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXic:2010. 11929, 2020.
Mohamed El Banani, Karan Desai, and Justin Johnson. Learning Visual Representations via Language- Guided Sampling. In CVPR, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next. generation of multimodal datascts, 2023. 4
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision-language pre- training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 14(3-4):163 (1213322,,2..2 352, 2022. 2
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hicrarchical feature alignment for vision-language model protraining. NewrIPS, 35:35959-3599 2022. 3
Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthclip: Are we ready for a fully synthetic clip training? arXiv preprint arXiv:2402.0 01832, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn
CVPR, pp. 770 778, 2016. 3
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of ofttof-diitribbtiin generalization. In ICCV, pp. 8340-8345 2021a. 7
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- plcs. In CVPR, pp. 15262-15271, 2021b.
Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Retrieval-enhanced contrastive vision- text models. arXiv preprint orXtx:2306.071 07196, 2023. 3
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pp. 4904-4916. PMLR, 2021. 1,
Greg Kamradt. Needle in a Haystack pressure testing LLMs. http:////ggtuuu..oogmaaaaaatttte
NeedleInee hww.eeaaaaatttcckrerrraaaaaaaaaa 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Alcksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, et al. Veclip: Improving clip training via visual-enriched captions. In ECCV, pp. 111-111111111 127. Springer, 2024. 3
Fei-Fei Li, Marco Andreeto, Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022. 7
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm.
arXiv preprint trXic:2110005200 2021.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallueina- tion in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), ACL, pp. 192-305, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.200 URL https: / http////caaaatlog/o/ooo wrceloogyoorg///ee 2223/ emnlp-main wwa.mmaan......................m 11
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
7, 10
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. NewrIPS, 36:
34892 34916, 2023. 10
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173 2024a.
Yanqing Liu, Xianhang Li, Zeyu Wang, Bingchen Zhao, and Cihang Xie. Clips: An enhanced clip framework for learning with synthetic captions. arXiv preprint arXiv:2411.16885 2024b. 4
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ar preprint arXiu:1711.05100 2017. 7
Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Munceb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, et al. Fairelip: Harnessing fairness in vision-language learning. In CVPR, pp. 22289-12300 2024. 4
S. Maji, Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In ECCV, pp. 529 544. Springer, 2022. 3, 7
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sizth Indian conference on computer vision, graphics & image processing, pp. IEEE, 2008
7
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishknnn Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp.
PmLR, 2021. 1, 2, 3, 4,
Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACL HLT 2010 workshop creating speech and language data with Amazon's Mechanical Turk, pp. 139-147, 2010. 7
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, pp. 5389-5400. PMLR, 2019.
Olga Russakovsky, Jia Dong, Hao Su, Jonathan Krause, Sanjeev Sathoesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. 1JCV, 115:211-250 12222222222222 2015. 7
13

---

Published in Transactions on Machine Learning Research (08/2025)
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NewrIPS, 35:25278 25294, 2022. 7
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In ACL, 2018. 6
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, pp. 9568-9578 2024. 7,
Michael Téchannen, Basil Mustafa, and Neil Houlsby. Image-and-language understanding from pixels only. arXiv preprinl Xic:2221 08045, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NewrIPS, 30, 2017. 3, 4
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric Xing. Learning robust global representations by penalizing local predictive power. NeurIPS, 32, 2019.
Zihao Wei, Zixuan Pan, and Andrew Owens. Efficient vision-language pre-training by cluster masking.
CVPR, pp. 26815 26825, 2024. 4
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, pp. 3485-3492. IEEE, 2010. 7
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. arXiv preprint arXiv:2202. 11094, 2022. 3
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511: 2024. 10
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, pp. 19163-19173, 2022. 3
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer.
arXiv preprint ar Xiv:22 10. 05258, 2024. 1, 3, 4, 5,
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual de- notations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. 7
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pp. 11975-11986, 2023. 3
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-langual models for vision tasks: A survey. IEEE TPAMI, 2024. 2
Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, and Ranjay Krishna. Tterated learning improves com- positionality in large vision-language models. In CVPR, pp. 13785 13795, 2024. 4
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionelip: Region-based language-image pretraining. In CVPR, pp. 16793 16803, 2022. 3
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint ar 2ix:22242. 14289, 2024. 10
14

---

Published in Transactions on Machine Learning Research (08/2025)
A Extra Visualizations
In this appendix, provide more visualizations of the attention maps similar to Figure 2.
Original
CLIP skull
DiFICLIP:
197722
#197723
#197723
Original
dog
dog
Original
CLIP: seapot
DITCCLPP teagot
Original
CLIP: sypewriter
DxfrCLIP typewriter
Figure 6: Additional Visualizations of DiffCLIP Attention vs CLIP Attention
On the Difficulty of Quantitatively Evaluating Attention Maps. While qualitative comparisons (as in Figure 6) clearly show that DifCCLIP produces more focused and object-centric attention patterns than CLIP, turning these visual differences into a robust quantitative metric is challenging. A common approach would be to compare attention maps to ground-truth object masks (e.g., from segmentation datasets) using
15

---

Published in Transactions on Machine Learning Research (08/2025)
IoU scores. However, this requires converting soft attention maps into binary masks, which introduces several issues: (i) selecting a single threshold is problematic because optimal values vary widely across images, especially between small and large objects; (ii) attention scores may be diffuse by design for some heads, making them poorly suited to binary evaluation; and (iii) per-image threshold tuning risks introducing bias and undermining reproducibility. Given these constraints, we rely instead on downstream task performance (e.g-; the +5.7% gain on the MMVP benchmark) as an indirect but more meaningful quantitative indicator that the attention mechanism improves focus on semantically relevant regions.
Statistical Significance
In this appendix, we run the non-deterministic linear probing and few-shot evaluation from Table 1 on three seeds and report the mean and standard deviation performance.
The following tables (4 and 5) present. the aggregated results, enabling a clearer comparison that accounts for non-determinism.
Table 4: Three-seed Linear Probing (mean sd, %).. Each cell reports the average accuracy and
standard deviation over three seeds for non-dcterministic evaluation
Pretraining Model Caltech-101
DTD
Pets
Flowers
SUN397
Aircraß
CIFAR10
CIFAR100
Food-101
Avg.
CC3M
CLIP
22.5
68.7 + 0.0
61.0 # 0.0
85.8 + 0.0
54.1 + 0.0
35.6 + 0.0
83. + 0.0
63.4 + 0.0
39.1 0.0
63.7 0.0
OC3M
DIECLIP
7..2::0.0
622
866
839 0.0
63.7
38.4
CC12M
CLIP
8.4
71.2
79.9 : 0.8
926 0.1
68.3 + 0.0
48.8 UD
92.0 0.0
74.7 0.0
77.5
77.1 0.
CC12M
DIECLIP
718
69.1
73.3
77.1000
Table 5: Three-seed Few-Shot (mean + sd, %). Average accuracy and standard deviation over three
sceds.
Pretraining
Model
Caltech-101
DTD
SUNS97
Aircraft
CIFAR10
CIFAR100
Food-101
Avg.
CC3M
CLIP
90.3
72.5 0.5
70.6 # 0.2
92.6 # 0.4
91.7 # 0.1
44.6 # 03
63.8 + 0.2
72.8 + 0.3
66.9 +.5 0.5
+..9
DIRCLIP
0.3
72.9 + 0.5
72.6 0.2
93.0 + 0.3
92.7 +
62.6 +
73.2 0.4
08.3 03
74.7
CC12M
CLIP
97.3
81.9 0.4.
8688 0.2
96.5 1 0.3
98.5 0.2
56.5 1 03
80.7 1 0.2
818 0.2.
86.1 86.1..0.11
65.2 15.2.00.00
CC12M
DiECLIP
07.6
82.2 = 0.4
88.6 = 0.3
97.1 # 0.2
96.6 # 0.1
55.1 #
79.5 + 0.1
83.2 a 0.2
87.4 0.1
85.3 0.0
C Intuitive Explanation of Differential Attention
While Scction 3.2 presents the formal definition of differential attention, here we offer more intuitive explanation to complement the mathematical formulation.
Differential attention learns two separate attention maps, A and Ag, for the same input:
Q
A softmax
(1)
dd/2
Q2Kg
AQ softmax
(2)
vd/2
The final attention output is computed as:
DiflAttn(X) = (A1 - X A2) V.
(3)
Interprotation.
A1 encouraged by the contrastive objective to highlight the most salient, task-relevant features (e.g., the "dog" in image).
16

---

Published in Transactions on Machine Learning Research (08/2025)
Ag is learned through separate projections and, via the subtraction, is implicitly encouraged to cap- ture more diffuse, high-entropy patterns-often often corresponding to moise such as background textures ("grass," "sky") or features spuriously correlated with the main object.
Subtracting scaled version of A2 from A reduces these irrelevant activations, producing sparser and more discriminative attention distribution.
This mechanism helps the model cancel out noisy or distracting visual patterns while amplifying the signals that are most relevant the text query. In practice, we find that this results in attention maps that more tightly focus semantically important regions, which contributes to the improved performance and robustness observed in Section 4.
17


Reviews:
[{"review_id": "HLFnL8cd5T", "reviewer": "Reviewer_wZrT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The motivation for the use of differential attention in CLIP models is straight-forward given the Differential Transformer paper\n* Performance gains seem remarkably consistent across a variety of benchmarks, which is a strong point in favor of the proposed method\n\n### Weaknesses\n* The downside of such a straight-forward application of differential attention is that it makes for an arguably incremental contribution which might have limited interest to the community\n* A major methodological issue of this paper is that it does not provide error bars around the reported results to establish statistical significance of the performance gains. This is a common problem in applied machine learning that fortunately the community is striving to overcome (the latest NeurIPS guidelines for example explicitly require reporting of error bars). This is particularly problematic specifically for this paper given that the performance gains are generally quite modest, making it very unclear whether they are statistically significant or not. And in general, as the community is moving towards more rigorous standards for reporting results, not reporting uncertainty estimates should start to be flagged as a major methodological flaw.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DiffCLIP, an extension of the Contrastive Language-Image Pre-training (CLIP) model where self-attention blocks are replaced by differential attention modules.\nDifferential attention is a previously published attention layer that was developed for LLMs and which produces attention maps consisting of the difference between two regular attention maps, a construction which was motivated by the observation that regular attention maps often contain noise that can be mitigated by the subtraction operation.\nDiffCLIP is constracted in such a way that the additional number of parameters added to the corresponding CLIP model is negligible, and the computational overhead is also minimal.\nDespite this minimal increase in parameters and computation cost, the paper demonstrates that the proposed method delivers consistent performance gains across classification, retrieval, robustness, and fine-grained vision benchmarks, although these gains are generally quite modest.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Provide uncertainty estimates (e.g. error bars) around the reported results to establish statistical significance of the performance gains, for instance by running multiple trials with different random seeds and reporting the standard deviation of the results, or by establishing some other null hypothesis testing procedure.", "broader_impact_concerns": "nothing", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e94SLuLuUO", "reviewer": "Reviewer_gGUL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe core idea of adapting differential attention to a dual-encoder VLM like CLIP is novel and well-motivated. The paper clearly articulates its goal: to reduce attention noise and improve multimodal representations.\n\nA major strength is the minimal overhead. Achieving consistent performance gains with only a 0.003% increase in parameters and negligible computational cost makes DiffCLIP a very practical and appealing modification.\n\nThe authors evaluate DiffCLIP on a wide array of tasks: linear probing, few-shot classification, image-text retrieval (Flickr30k, Flickr8k, MSCOCO), zero-shot ImageNet, and out-of-distribution (OOD) robustness (ImageNet-V2, -A, -R, -Sketch). They also include fine-grained visual understanding using the MMVP-VLM benchmark. Experiments are conducted on two different pretraining dataset scales (CC3M and CC12M), demonstrating consistency. Across the board, DiffCLIP generally outperforms the standard CLIP baseline. While some individual gains are modest, the consistency across diverse tasks and datasets is a strong indicator of the method's effectiveness. The OOD and MMVP results are particularly noteworthy, suggesting better generalization and focus.\n\nThe paper is generally well-written, clearly explaining the motivation, the differential attention mechanism in the context of CLIP, the experimental setup, and the results. Figures and tables are informative.\n\n\n\n\nWeaknesses:\n\nThe preliminary experiments integrating DiffCLIP's vision encoder into an LLaVA-style model (on the POPE benchmark) hint at the potential for this technique to benefit larger, more complex multimodal systems. More explorations can be done in the future.\n\nThe experiments use CC3M (2.3M pairs) and CC12M (7.9M pairs). While standard, this is smaller than the original CLIP's private 400M dataset. It's a limitation (common to many academic works) that the full potential at massive scale remains an open question, though the positive trend is encouraging. This is more of a scope limitation than a flaw.\n\nWhile the paper cites Ye et al. (2024) and provides the core equations, a slightly more intuitive explanation of why subtracting one learned attention map from another helps cancel noise (e.g., what characteristics make the second map \"complementary\" or \"noisy\") could be beneficial for readers not deeply familiar with the original work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DiffCLIP, a novel modification of the CLIP architecture that integrates a differential attention mechanism into both its vision and text encoders. The motivation stems from the observation that standard CLIP attention can focus on noisy or irrelevant features. Differential attention, adapted from recent work in large language models, aims to mitigate this by learning two complementary attention distributions and subtracting one from the other, thereby amplifying salient features and suppressing noise. The primary contribution is the successful adaptation and evaluation of differential attention for multimodal CLIP-style architectures, offering a lightweight method to enhance feature representation and task performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the paper cites Ye et al. (2024) and provides the core equations, a slightly more intuitive explanation of why subtracting one learned attention map from another helps cancel noise (e.g., what characteristics make the second map \"complementary\" or \"noisy\") could be beneficial for readers not deeply familiar with the original work.\n\nThe pretrained models and training code can be released, or add a statement, perhaps in the conclusion or as a footnote, indicating the authors' intent to release the code for DiffCLIP. They would be helpful for the community.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0jecsNCLWF", "reviewer": "Reviewer_mLBQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nAttention noise and spurious features are critical problems in vision-language tasks.\nShowing that the differential attention mitigates attention noise in CLIP is also beneficial for practitioners.\n\n## Weaknesses\n- W1: The Introduction claims that attention noise in CLIP leads to poor performance on localization and fine-grained tasks. However, this statement lacks supporting evidence, such as citations or preliminary experiments, to substantiate the claim.\n- W2: The experimental evaluation is rather superficial; it merely mentions an improvement in average accuracy without providing a deeper analysis or insights.\n- W3: Fig. 2 and Tab. 3 are not referenced or discussed in the experimental section, leaving the reader unclear about their relevance or how they support the paper's findings.\n- W4: An analysis of CLIP-specific perspectives, such as modality gap[a,b,c], which is known to be critical for CLIP's generalization, would be intriguing.\n\n\n[a] Liang et al., Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning, NeurIPS 2022.     \n[b] Yamaguchi et al., Post-pre-training for Modality Alignment in Vision-Language Foundation Models, CVPR 2025.    \n[c] Eslami, et al., Mitigate the Gap: Improving Cross-Modal Alignment in CLIP, ICLR 2025.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DiffCLIP, a CLIP model using the differential attention mechanism, which was recently proposed in the LLM literature.\nThe experimental results show that DiffCLIP improves performance on various tasks compared to the counterpart CLIP.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major\n- For W1 and W3: Demonstrating that the vanilla CLIP has poor localization, such as in Fig. 2, would support the claim. It would also be interesting to add more examples of failure cases of attention and quantitative evaluation of localization and spurious features and how DiffCLIP alleviates them.\n- For W2: A possible explanation or analysis for why DiffCLIP underperformed the vanilla CLIP on Aircraft and CIFAR10/100 in Tab. 1 would provide deeper insights and help clarify the limitations.\n- For W3: Providing experimental details and discussions for Fig. 2 and Tab. 3 would be beneficial.\n- For W4: Evaluating and visualizing the modality gap in the embedding space would provide more insights for CLIP.\n\n\n## Minor\n- Bar plots such as Figs. 1, 3, and 5 should be replaced with tables. The score differences are generally small (~1 percentage point), making the bar lengths visually indistinguishable. Presenting the exact numerical values would be more informative.\n- Numbering all equations would facilitate easier referencing and improve the readability of the technical discussion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)