DiffCLIP: Differential Attention Meets CLIP
Authors: name: Hasan Abed Al Kader Hammoud; name: Bernard Ghanem
Published: 2025-03-09 14:04:09

Abstract: We propose DiffCLIP, a novel vision-language model that extends the
differential attention mechanism to CLIP architectures. Differential attention
was originally developed for large language models to amplify relevant context
while canceling out noisy information. In this work, we integrate this
mechanism into CLIP's dual encoder (image and text) framework. With minimal
additional parameters, DiffCLIP achieves superior performance on image-text
understanding tasks. Across zero-shot classification, retrieval, and robustness
benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably,
these gains come with negligible computational overhead, demonstrating that
differential attention can significantly enhance multi-modal representations
without sacrificing efficiency. Code can be found at
https://github.com/hammoudhasan/DiffCLIP.