Seeing What’s Not There: Negation Understanding Needs More Than Training
Authors: Bhuvan Aggarwal
Amit More
Mudit Soni
S Divakar Bhat
Venue: ICLR 2026 Poster (2026)
Keywords: Negation
Zeroshot
VisionlanguageModels
MachineLearning
ComputerVision
DeepLearning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Understanding the negation in a sentence is an important part of compositional
understanding and logic in natural language. Many practical AI applications, such
as autonomous driving, include precise instruction with negations. For example,
following instruction to an AI assistant ”locate a parking spot without a vehicle”
requires the assistant to not confuse between presence and absence of vehicles. Al-
though joint embedding-based Vision Language Models (VLMs) like CLIP have
revolutionized multi-modal tasks, they struggle to interpret negation. To address
this limitation, recently many works proposed to solve the problem through a data-
centric approach by introducing additional datasets with hard-negative samples for
both image and text data. Contrary to these approaches, we present a zero-shot
approach to tackle the negation understanding problem. We probe the properties
of CLIP text embeddings and show that they follow compositional arithmetic op-
erations, which allow the addition or removal of semantic information directly in
the embedding space. We then present a rule-based approach to extract negated
text from given caption and then use it to explicitly remove corresponding se-
mantic information from original embedding, improving negation understanding
in VLMs. Our approach does not require expensive training process to induce
negation understanding into the model, and achieves the state-of-the-art perfor-
mance on popular benchmark for negation understanding. We improve baseline
CLIP model performance on NegBench from 25.5% to 67.0% for MCQ and from
50.9% to 56.1% for retrieval tasks. Even NegCLIP model which is fine-tuned on
negtion datasets, our approach boosts its MCQ accuracy from 54.03% to 66.22%
and retrieval accuracy from 59.25% to 60.1% showing strong performance.

Published as a conference paper at ICLR 2026
SEEING WHAT'S NoT THERE: NEGATION UNDER- STANDING NEEDS MORE THAN TRAINING
Bhuvan Aggarwal
Amit More
Mudit Soni
S Divakar Bhat
Honda R&D, Tokyo
A person with no handbag
christmas tree devoid of any decoration
CLIP
NegCLIP*
Ours
CLIP
NegCLiP
Ours
Figure 1: Vision-Language Models struggle with negation understanding. Standard CLiP-based Text-to-Image generation models fail to understand the negation in input text, generating images containing the negated objects. Even when models are fine-tuned on exclusive datasets with hard- negatives for negation understanding, they struggle to understand negations (NegCI.IP*). Our pro- posed embedding correction approach on CLIP text embeddings successfully addresses the negation understanding failures, generating images which exclude the negated objects.
ABSTRACT
Understanding the negation sentence is an important part of compositional understanding and logic in natural language. Many practical Al applications, such as autonomous driving, include precise instruction with negations. For example, following instruction to an Al assistant "locate parking spot without vehicle" requires the assistant to confuse between presence and absence of vehicles. Al- though joint embedding- revolutionized multi-modal tasks, they struggle to interpret negation. To address this limitation, recently many works proposed to solve the problem through data- centric approach by introducing additional datasets with hard-negative samples for both image and text data. Contrary these approaches, we present a zero-shot approach to tackle the negation understanding problem. We probe the properties of CLIP text embeddings and show that they follow compositional arithmetic op- erations, which allow the addition or removal of semantic information directly in the embedding space. We then present rule-based approach to extract negated text from given caption and then use it to explicitly remove corresponding se- mantic information from original embedding. improving negation understanding in VLMs. Our approach does not require expensive training process to induce negation understanding into the model, and achieves the state-of-the-art perfor- mance popular benchmark for negation understanding. We improve baseline CLIP model performance on NegBench from 25.5% to 67.0% for MCQ and from 50.9% 56. % for retrieval tasks. Even NegCLIP model which is fine-tuned negation datasets, our approach boosts its MCQ accuracy from 54.03% to 66.22% and retrieval accuracy from 59.25% to 60.1 % showing strong performance.
Vision Language Models (VLMs) like CLIP have
INTRODUCTION
Vision-Languee models (VLMs) have demonstrated remarkable capabilities in multi-modal tasks. This is generally achieved using a contrastive learning framework by aligning image and text infor- mation in joint-embedding space, such in CLIP models Radford al. (2021): al. (20226).

---

Published as a conference paper at ICLR 2026
These models widely used for tasks such as text-to-image generation Choi al. (2022)
Rom-
bach et al. 2022); : Ramesh et al. (2021). cross-modal retrieval et al. (2023); Luo et al.
2022)
image captioning Vinyals et al. 2015 Xu et al. (2015) and referring image segmentation Wang
et al. 2022); Lee et al. (2023) y Liddecke & Ecker (222)).
However, it has been shown that these models generally struggle with compositional understanding of text Fan et al.] (2024); Yuksekgonul et al. (2022): |Hsieh et al. (2023); Ma et al.] (2023): y: Lewis
al. (2022) and, particular, negation Singh et al. (002)); Li et al. (2025), core linguistic aspect
of understanding and reasoning. In general, the CLIP models often ignore the negation words such
as "no", "not" and "without", referred to as the "Affirmation Bias" Li et al. (2025). Negation is
essential for precise semantic interpretation in natural language with many practical applications. For example, following text prompt to a diffusion model, "a photo of a car with tires" and "a photo of a car without tires" will result in similar generated images, both with tires, which is very detrimental for their performance and generalization.
The failure to induce negation exposes the shortcoming in how these models structure and inter- pret semantic concepts in the embedding space. This limitation of joint-embedding models can be attributed to both the training data and process. The CLIP model is trained on web-scale image- text pairs using contrastive learning. The negation data is highly underrepresented in these pairs as shown in Yoon al. (2025), with Laion-400M dataset (Schhhman et al. (202I) containing less than 1% negated captions. Even when present, the negated captions don't 't align with corresponding im- age in terms of the negation context. This problem is further aggravated by the contrastive learning approach which encourages the model to interpret the text as a bag-of-words rather than understand- ing its compositionality Yuksekgonul et al. (2022). In response to this, most of the previous research has tried to tackle this problem using a data-centric approach by introducing new datasets with ex- plicit hard- nard-iegatives to improve model learning. Though effective, these approaches still struggle to understand negations completely and solely relying on large-scale datasets for performance im- provement is not an optimal solution.
In the present work, we focus on approach orthogonal to data-centric methods and provide an alternative solution to improve negation understanding. We ask an important question "Do we re- ally need fine-tuning
special datasets with hard negatives improve negation understanding in VLMs?". We show that a simple post-hoc correction to the text embeddings of CLIP (not fine- tuned on such datasets) can outperform even the fine-tuned variants by large margin. In particular, we propose to solve negation understanding problem in a novel zero-shot, training-fret free approach through explicitly enforcing compositionality constraints on the text embeddings to reflect the effect of negation in the embedding space. We note that negation manifests itself as a directional offset in embedding space Li et al.) (025). We first estimate this directional offset correction signal for negation understanding and then construct negation-aware embedding vector.
Our approach is inspired by the linearity shown by the word vector models and vision-language embeddings Mikolov et al. (2013); Couairon et al. (2022) where semantic information can be ma- nipulated using basic arithmetic operations such as "King" "Man" + "Woman" = "Queen". The idea that CLIP embeddings can be manipulated directly via vector arithmetic is not new, and is dis- cussed in detail Section 2.3. Our approach is primarily motivated by an empirical and geometric observation. We test our approach on various benchmarks and show that our zero-shot approach significantly boosts the negation- underrtaaddinngggg of CLIP models.
The key contributions of our work are summarized as follows:
We identify and characterize the limitation of existing models in terms of negation under- standing. We show that CLIP text embeddings can be corrected using compositional arith- metic operations. Our approach computes the correction signal using directional offset and generates final negation- awwre embedding for better negation understanding.
We propose a training-free zero-shot embedding correction approach to improve CLIP's ability to interpret negation. Our approach achieves best results using a baseline CLIP model-not not trained on explicitly curated negation datasets-outperffmee outperforming best models in the literature which are trained on such hard-negative datasets.
We show strong limitations of existing data driven approaches which simply curate mil- lions of image-text pairs and hope to solve negation understanding by existing contrastive learning framework. Simplicity of our method-trainee free approach and strong perfor-
2

---

Published as a conference paper at ICLR 2026
NegCl
OL Lake 20
Mepour)
Accuracy
Dataset
Mariad Da
Figure 2: (1) We compare zero-shot image classification accuracies across multiple datasets for different CLIP models. * indicates models fine-tuned on the CCI2M-NegFull dataset from Li al. (2025) improving score on NegBench-MCC benchmark (shown on left) but also leading to reduced foundational knowledge and decrease in zero-shot image classification accuracies. (2) Confusion matrices on Oxford Pets dataset shows that NegCLIP* have increased confusion between categories, demonstrating how negation-focuse focused fine-tuning can degrade general capabilities of model.
mance-Suggest that real improvement to negation understanding may need a new training framework to effectively utilize given data.
RELATED WORK
2.1 LIMITATIONS OF VISION-LANGUAGE MODELS
Early investigations into CLIP model's capabilities include the ARO Yuksekgonul et al. (2022) and CREPE Ma et al. (023) benchmarks. ARO aims at evaluating the VLMs ability to understand relationships, attributes and order and proposes NegCLIP while CREPE presents benchmark to evaluate VLMs on two important aspects of compositionality: systematicity and productivity, Hsich al. (2023) show that these benchmarks are biased and proposes an unbiased dataset SugarCRI:PI. using ----eenrrated hard negatives for fluent and meaningful captions. Further, Hsiih et al. (2024) proposes Graph-based captioning to enhance the training data using explicit compositional structure.
2.2 ADDRESSING NEGATION
ConCLIP Singh al. (2024) introduced CC-Neg dataset with hard-negatives, built on the CC-3M dataset Sharma et al. 2018). However, CC-Neg contains hard negatives only as a distraction and not
target ground truth. Following this, many works [i et al. (2025), Yoon et al. (2025) have intro- duced better benchmarks, datasets, and fine-tuned CLIP variants. Li et al. 2025) creates large-scale synthetic (itusett-CI2M-NegFFll. based on CC-12M Changpinyo al. (2021) and the NegBench benchmark for evaluation. Similarly, TripletCLIP Patel et al. (2024a) introduces hard negative sam- ples in visual domain. General trend here, in research community, training datasets with minor modifications to contrastive loss by adding distractor samples or hard negatives. However, relying on retraining exposes models to catastrophic forgetting of the founda- tional knowledge, affecting the generalization performance unseen domains or negation contexts.
to improve the quality of
2.3 TEXT EMBEDDING EDITING AND ZERO-SHOT COMPOSITION
Orthogonal to the data-centric approach, our method explores manipulating the text embeddings directly. The embedding arithmetic method has been shown to be effective in the traditional NLPP literature, also known as the "Vector-offset method" Mikolov et al. (2013). often used for zero- shot predictions, disentangling object-attribute pairs, image editing, retrieval and debiasing Chen et al. (2018); 8); Vo et al. (2019); Bolukbasi et al. (2016). The embedding arithmetic has become quite popular in many VLM applications. Douze et al. (2022) shows latent text representations to ex- hibit geometric regularities and that text-defined transformations can be applied image modality, demonstrating the application in image retrieval. (Tewel et al. (2022) uses visual embedding arith- metic for LM-side steering to enable zero-shot image-to-text generation. A recent work Zhang al.
3

---

Published as a conference paper at ICLR 2026
CLIP
ConCLIP
CLIP*
NegCLIP
Confined aftemeetens and negatiors
& wi
offimnatiors
Chertoed offimationn
negations difirent objects
Collapsed regations
negatione
Caption
Objects
Affirmation
cat
bicycle
airplane
train
bird
sheep
elephant
zebra
Negation
dog
boat
truck
cow
bear
giraffe
Figure 3: Embedding space visualization with negated captions. PCA projection of embeddings
show: (1) CLIP fails distinguish between affirmative and negative samples, (2) ConCLIP col- lapses all negative embeddings, (3,4) CC12M-NegFull trained models do separate out the negated embeddings from affirmative ones, however, the negated embeddings are still tightly clustered.
2023) has used text embedding editing method improve CLIP model's object counting ability by transferring counting knowledge between different objects. Recently, KKoihigariia et al. (2025) showed that CLIP text embeddings follow compositional properties. In general, the CLIP feature space is well studied, however, it has not been explored for improving the negation understanding capabilities. Building on these insights, we propose a training-fre free text embedding editing method, which improves the semantic fidelity of text prompts containing negations.
PROPOSED METHOD
In this section, we present our method for embedding space correction to improve negation un- derstanding. We first investigate some limitations of existing data centric approaches. Then we analyze. if CLIP embeddings allow arithmetic manipulations of embeddings to improve general compositional understanding? With the newfound insights, we finally introduce our approach.
3.1 HIDDEN LIMITATIONS OF TRAINING ON CUSTOM NEGATION DATASETS
We begin by investigating zero-shot image classification accuracies of CLIP Cherti et al. (2023) and NegCLiP Yuksekgonul al. (2022) models fine-tuned on CC12M-NegFull. As shown the Fig. 2(1) while these fine-tuning approaches improve the scores on NegBench-MCQ benchmarks. comes at the cost of reduced performance on the general zero-shot classification tasks. There is substantial drop across datasets like ImageNetlk, Caltech101, Flowers 102, Stanford Cars. and
Oxford Pets Parkhi et al. (2012); Krause et al. (2013): Nilsback & Zisserman (2008): Deng et al.
2009); Li et al. (2022a), suggesting that specializing for negation understanding compromises on general representations and increases inter-class confusion as shown in Fig. 2()
We generate affirmative captions for several noun words such as "cat", "bicycle", "train", etc. and their negated versions such as "not cat", "not bicycle" and "not train", following the caption template from Li et al. (2025) can be noted that the original CLIP model is not able to distinguish between the affirmative and negated examples. ConCLIP, trained on CC-Neg dataset can separate positive and negated examples but results in collapsed negated embeddings. Models fine-tuned on CC12M-NegFull et al. (2025) do show better discriminability between both affirmative and negated captions. However, negated embeddings from different categories appear to be tightly clustered in comparison to the positive ones, suggesting inherent limitation of these models.
Next, we visualize the text embeddings for different models in Fig. 3
3.2
Do CLIP EMBEDDINGS ALLOW ARITIMETTCC OPERATIONS TO INTRODUCE COMPOSITIONALITY?
We consider two object nouns such as "Cat" and "Flower" and compare the embeddings for com- posite word "Cat and Flower" using sentence template "A photo of a (}". We denote corresponding CLIP embeddings e ef and ef respectivell. We note that the embedding ef is quite similar to
4

---

Published as a conference paper at ICLR 2026
10.
Siifurty to
64-
62
Le
-16
Figure 4: Embedding space arithmetic visualization (Left): We plot embeddings of elementary captions as e and e' and plot all the other embeddings for composed captions ex e + e eet - ef and eet - ef + a with respect to axis ec and ef. Figure show that compositional em- bedding of can be synthesized using e + ef. Further the elementary embedding e can also be synthesized with the help of anchor embeddings said to be correct they have high cosine similarity with respect to desired embedding. For the example of cat and flower, c + e has cosine similarity of 0.86 with e Similarly, cx - and eat e + a have a similarity of -0.33 and 0.83, respectively with target embedding e". (Right): We also visualize embeddings for several examples.
ex - ef + a. We note that embeddings are
e + ee (in terms of cosine similarity), suggesting that the semantic information in text embeddings can be added together to generate semantically accurate embeddings for the composed sentence.
We further test whether we can reconstruct the embeddings e and ef using composed embedding eat. Intuitively, simple subtraction between ex and ef should provide candidate embedding for es, however, note that such embedding given by eaf eef is quite far away from the desired e We hypothesize that, in general text embeddings have some non-informative bias which offsets them into correct region in embedding space. The subtraction operation removes such common bias that results in incorrect embedding. Thus, we argue that we need to add such bias back into the resultant embedding
the form of an anchor embedding, a. One may estimate the embedding a. as a bias computed as the mean of all embeddings on a large text corpus. One can even use embeddings of some oon-informative words, such as "neutral". "balanced", etc. as an anchor, since these words generally convey the semantic meaning of umbiasedness and should be closer to all embeddings in general. Further, anchor embedding estimated in this way are dataset independent and computationally efficient. Thus we use the latter approach to estimate The resultant embedding given by e e + a is indeed similar to the target embedding e" We repeat above analysis for several random pairs of nouns and show all the original and recomputed embeddings in Fig. 4
3.3 CAN IMPROVING NEGATION UNDERSTANDING WITH ARITHMETIC OPERATIONS?
Based on the analysis presented in the previous section, we now present our approach where we explicitly manipulate embeddings for better negation understanding. For a caption such as "A photo of cat but not flower". negation understanding requires the final embedding not have any seman- tie information about the negated concept "flower". However, due to affirmation bias, models may ignore the negation word and as result the embedding is quite similar to the affirmative sentence such as A photo of a cat and flower".
In particular, let e neg and e" be embeddings for given full caption C with negation, the negated concept Cneg (such as "flower" in above example) and final corrected embedding, respectively. A naive formulation for correct embedding can be given as
e
neg +
Our experiments show that the above implementation results in an improved performance Neg- Bench. However, we argue that such formulation is suboptimal for following reasons. First, the baseline CLIP models may not completely ignore the negation but only struggle to eliminate it, lead- ing to poor negation understanding. Secondly, when fine-tuned on negation datasets, models such III learn to suppress negated concept more aggressively in embedding (hence improved
5

---

Published as a conference paper at ICLR 2026
Cit
Qui
Ort
but
enbeoding closer
Ecrociion
Cal
Dog
"dog'
CAP
DOG
Dog
Figure 5: Overall pipeline of our zero-shot negation correction method. Given an input caption containing negation, we first apply rule-based splitting to separate the affirmative part ("An image of a cat but") and negated part ("not dog"). The CLIP text encoder then generates embeddings for the original caption e and the negated part enes, Our approach then corrects the embedding for better alignment with images.
accuracy) but may not succeed in completely eliminating it. Further, the CLIP embeddings are semantically dense Bhalla al.) (2024) and represent multitude of related semantic concepts, and models may already have removed subvet of negated semantic information in e from e correctly. For these reasons, we must first estimate the information about ee which present e and then only remove such semantic information. Thus, propose the following formulation.
e" e -A(-
< <eee, nnn
neg
eueg eneg
The term < neg represents projection of e on " representing common information .enog that must be removed. This term effectively represents the directional offset required to generate corrected negation- embedding e* from e We further note that, we need hyperparameter allow control over the adjustment depending specifie model and datasets.
3.4 OVERALL APPROACH
We summarize overall approach in Fig. 5 We define comprehensive corpus of nega- tion words including explicit negators (never, no, nothing, nowhere, none, not), absence in- dicators (absence, empty. devoid, lacks, ab- sent), and contracted forms (n't, haven't, hasn't t, can't, etc.). If none of the negator words are present in the given caption C, the embedding not updated. If negator words present in- stead, then we extract the negated concept Cneg sequence of words affected by the negator. We then use Eq. 2 to get the updated embed- ding. We present the summary in Algorithm !
Algorithm Embedding Correction
Input: Text Caption C Parameters: Set N of negator words
Set A of anchor words
Output: Corrected embedding e*
for w/ E do
2: a; CCI...Encoe._Tere
3: end for
4: a Mean(a;)
5: C = Null
6: for n; E N do
7: if n: E C then
8: (neg Negated C*)
3.4.1 EXTRACTION OF NEGATED CONCEPT
9: break
Omeg
10: end if
11: end for
We develop rule-based negation scope de- tection algorithm, inspired from Loper & Bird (2002). The negation scope defines the se- quence of words which are influenced by nega- tion term in given caption. We begin to process the text sequentially and when a negation word is detected, the negation scope is defined de- pending on the type of negator and all the words inside the scope taken the (me
12: if Cneg Null then
13: e =CCL..EncooeeTex((( Text(C))
14: return e*
15: end if
16: ef, enog
CLIP. Eii.PEncode_Txxe Cneg)
e- eiee e neg -
17: e* = e
18: return e*

---

Published as a conference paper at ICLR 2026
MCQ
Retrieval
COCO
VOC2007
MSRvtt
COCO
MSRytt
Model
R@5
R-Neg@5
R@5
R-Neg@5
CLIP openai
39.3
38.7
32.1
54.8
48.6
50.6
45.8
CLIP laion400M
31.0
31.6
30.0
59.4
51.7
43.1
37.5
NegCLIP
28.7
30.5
27.3
68.7
64.4
53.7
51.0
CLIP
54.4
54.8
44.9
54.2
51.9
46.9
43,9
NegationCLIP
36.0
44.1
34.5
62.2
61.4
43.0
41.6
CLIP
24.7
24.3
27.5
64.8
57.3
49.7
44.5
Ours + CLIP
72.5 (147.8)
78.6
(154.3)
50.0 (122.5)
63.2 (15.9)
49.0 (15.5)
NegCLIP
56.2
59.7
46.2
69.0
67.0
54.0
51.5
Ours+NeCCLIP
69.5 (113.3
75.1 (115.4)
54.1 (t7.9)
67.9 (10.9)
52.2 (10.7)
Table I: Performance evaluation on NegBench. We evaluate our embedding correction method against baseline and fine-tuned CLIP models across Multiple Choice Questions (MCQ) and Retrieval tasks. (*) indicates the models fine-tuned on the CC12M-NegFull. We show improvements over original models in brackets.
We classify the negators into pre-negators and post-negators. The pre-negators are forward-scoped negators which precede the clause being negated. For example, words "empty" and "devoid" activate forward negation scope when followed by the preposition "of". So activate the negation scope starting from the negator until a clause boundary like punctuation or conjunction encountered. Post-negators the other hand are backward-scoper negators, succeeding the clause being negated. For example, words "absent" and "nowhere" trigger backward negation scope when preceded by copular verbs (is/are). Therefore, when such negators are encountered, a backward negation scope is triggered until the clause boundary. We elaborate more on this in the Appendix.
EXPERIMENTS
4.1 BENCHMARKS, EVALUATION TASKS AND BASELINES
In this section we validate proposed approach on the benchmark datasets for negation understanding.
We use NegBench Li et al. 2025) dataset comprising of 79k samples among 18 task variations. Built on top popular datasets like COCO Lin et al. (0014), VOC2007 Everingham et al. (2010) and MSR-VTI VTT al. (016)), this benchmark has two core tasks: Retrieval-Neg, coarse grained retrieval task to test the model's ability to retrieve images based on a text prompt that includes both positive and negative captions; and MCQ-Neg, fine-grained question answering, which tests the model to choose the correct description from closely related captions, including LLM-gennrated positive, negative and hybrid captions. We show MCQ accuracy and the retrieval performance as Recall @5 and Recall-Neg @5. Recall is the performance of the model on the original captions and Recall-Neg 5 the performance for the negated captions. We use 5% of the COCO-MCQ split as validation data to tune A. We also present results on CC-Neg dataset in appendix. We exclude the HardNeg-Syn Li al. (2025) dataset since it is synthetic dataset and not publicly available.
We compare our approach directly with several pretrained (CLJP, NegCLIP) and fine-tuned models (ConCLIIP, and CCi2M-NegFull dataset fine-tuned variants for CLIP* and NegCLIPss. We select ViT-B/32 backbone for all models. We implement our approach on NegBench fine-tuned NegCLIP* and Laion2b pretrained CLIP model from the open open.elip Cherti et al. (2023) implementation.
4.2 RESULTS
Table D shows the comparative analysis for all the models on NegBench. Our method significantly improves the performance across all MCQ datasets. For baseline CLIP model, the original accura- cies COCO, VOC2007 and MSRvtt were chance level, 24.7% 24.3%. 27.5%, respecivvell Us- ing proposed embedding correction, it improved to 72.5%. 78.6% and 50.0%, increasing by 47.8%, 54.3%, 22.5%. Notably, this result outperforms the CLII* and NegCLIP* models which are

---

Published as a conference paper at ICLR 2026
MCQ
Retrieval
COCO
VOC2007
MSRytt
COCO
MSRvtt
Model
R R@5
R-Neg@5
R@5
R-Neg@5
SigLIP
28.9
30.8 66.9
30.7
72.1
64.4
51.4
44.7
+ Ours
61.15 (132.25)
(136.1)
46.8 (116.1)
68.7 (f4.2)
47.6 (12.9)
SigLIP2
27.2
27.1
30.1
72.8
64.0
51.6
45.3
Ours
66.3 (139.1)
70.2 (143.1)
50.1 (120)
70.2 (16.2)
49.4 (14.1)
AlignCLIP
32.7
28.4
23.6
44.8
35.6
35.8
31.1
+ Ours
60.1 (127.4)
69.2 (f40.8)
44.3 (120.7)
41.2 (15.6)
34.7 (13.6)
TripletCLIP
33.8
23.7
30.2
52.3
44.3
42.4
35.8
Ours
61.8 (128.0)
57.2 (133.5) 46.7 (116.5)
47.9 (13.6)
39.9 (14.1)
Table 2: Performance evaluation other backbones show that approach generalizes across different model architectures and training paradigm.
Model
Accuracy
Model
Acc R@5
Ace R@1
NegationCLIP 78.3
CLIP
67.4
34.0
CLIP
71.0
CLIP+Ours
70.4 (1 3.0)
41.3 (1 7.3)
CLiP+Ours
79.5 (1 8.5)
Evaluation on numannannotaan dataset with negation, subset of Flickr30K dataset.
Table
Table 3:
Evaluation on VALSE Existence dataset, subset VALSE dataset.
fine-tuned on CC12M-NegFull dataset with hard-negatives, highlighting that our embedding correc- tion restores the semantic alignment of negated samples without any need of training dataset.
On the negative retrieval task (R-Neg@5) we get an improvement of 5.9% and 5.5% on COCO and MSRvtt retrievals, respectively, achieving 63.2% and 49.0% with CLIP model. When using NegCLIP*, proposed approach achieves SOTA results with 67.9% and 52.2% accuracies. Since our method only corrects the embeddings of captions with negation words, the accuracy on the positive retrieval 5) remains the same as baseline model.
Generalization across different model families: We further evaluate approach some of the popular VLM models such as SigLIP Zhai et al. (2023) & SigLIP2 Tschannen et al. (2025). AlignCIIIP Eslami & de Melo (2024) and TriplerCL.PP Patel et al. (2024b) in Table Our embedding correction method consistently improves the performance across all models showing strong generalization across different backbones.
Generalization across different datasets: We further substantiate the performance of our approach across different negation datasets other than NegBench. We first evaluate our method on a subset of VALSE Parcalabescu al. (022) dataset, the VALSE Existence. This dataset similar in nature to the NegBench with 2 option MCQ task with negations. As shown in Table. 3 our method boosts CLIP model accuracy from 71.0% to 79.5% achieving best results. The evaluations so far focused on benchmarks and datasets which were synthetic and auiimautomatc in nature. To further analyze the robustness of our approach, we extend the evaluation to human-annotated image caption dataset Flickr30K Van Miltenburg et al. (2016). In Table. 4 we show text retrieval accuracy for top-5 and top-1 predictions. Our approach consistently improve baseline CLIP model showing generalization human annotations well.
4.3 WITY DOES CLIP MODEL ACCURACIES IMPROVE SIGNIFICANTLY?
In this section, we visualize the effect of our method on CLIP text embeddings using PCA pro- jections. In Fig. 6 (left), project text embeddings of affirmative and their negated counterparts for different objects using different templates. After applying proposed corrections, the affirmative and negated samples clearly separated along distinct negation axis, leading more coherent
8

---

Published as a conference paper at ICLR 2026
Corrected CLIP Festure Space
Original Space
Feature
Prwerr anfirnttoon
Fllwer
Flwer affrrntton
Obacc
negitios
Both allerularo
Hebre Caplera
A
Negation Anis
Boch regutioaa
Andierr
Ca aftrrudion
altermatono
regalora
Caprinn Tape bo CIMEENN
Sto
Attm
Dcoble Nogazzn
Figure 6: Visualization of the embeddings. (Left): For affirmative and negated captions, corrected CLIP embeddings show separation along negation axis and maintains original semantic grouping. (Center): Original CLIP space with complex captions: single object affirmations/eegation. double allirmations/negaaions and hybrid (Right): Proposed embeddings are well separated and organized.
Model
Positive t
Negative
Caption Type
Confidence Score
ConCLIP
83.8
57.2
CLIP
NegCLIP*
Ours
NegCLIP*
86.8
78.7
CLIP
89.5
89.8
Affirmative
0.45
0.49
0.65
Negated 0.16
0.31
0.64
Ours+CLIP 89.5
10.2
Table 5: Average confidence score between im- age and caption (both affirmative and negated).
Table 6: Zero-shot accuracy on CIFAR1O with positive and negative (distractor) captions.
and aligned clusters. These embeddings are also well organized by class groups, indicating that method is able not only disambiguate negation, but also preserve the semantic class structure similar to affirmative embeddings. In comparison, CLIP* and NegCLIP* embeddings (Fig. (3) for negated captions often lead tight clusters which lead to poor accuracies.
We further extend this analysis to more complex caption types. We show hybrid prompts ("a cat and not flower"), two object affirmations ("a cat and a flower") and double negations ("neither flower"), along with the single object captions. In the original CLIP space (center), the affirmations and negations have very high overlap while all the complex sentences collapse to the same cluster at the center. However, after applying the corrections, all the different categories are separated into semantically interpretable clusters, separating different negations and semantic classes, both. For example, the double affirmations lie between single object affirmations while the double negations lie between single object negations and form cluster along the same virtual negation axis. Further, the hybrid captions lie between affirmative and negated clusters for both classes
These plots highlight the compositional consistency and semantically meaningful structure intro- duced by our method, explaining why proposed approach performs well. It not only corrects the simple negation but also generalizes well to hybrid negations involving multiple objects.
4.4 HOW WELL DO CLIP MODELS ALIGN THE NEGATED CAPTIONS WITH IMAGES?
We analyze the cross-modal alignment between text and images using probability scores between ground truth image-text pairs on COCO NegBench-MCC We show the average score for affirmative and negated ground truth captions separately in Table. 5 We note that CL.IP model achieves the score of 0.45 for positive captions but struggle align negated captions with corresponding images and achieves the score of 0.16. The fine-tuned model NegCLIP achieves the score of 0.31 for negated captions which explains the improved accuracies. When our proposed embedding correction is applied, the CLIP score improves from 0.16 to 0.64 showing improved alignment between negated text and corresponding images. Further, the score for affirmative captions improves from 0.45 to 0.65, due better understanding of incorrect negated captions, thus reducing model confusion.
In Table. 6 we also validate image classification score on CIFAR1O dataset by using negated class template "This is photo of (class)." " distractor to how well the models push away

---

Published as a conference paper at ICLR 2026
distractor text embedding from images. Surprisingly, CLIP with corrected embeddings achieves 10.16% accuracy which is very close to random chance.
These analyses reveal that proposed negation-aware embeddings well aligned with images in the presence of negation, when compared to fine-tuned models, explaining the superior performance.
4.5 ABLATION: COMPARING DIFFERENT EMBEDDING CORRECTION METHODS
We compare the naive formula- tion in Eq. T with formulation in Eq. 2 for different values of on COCO-MCQ split of Neg- We also show results when excluding the anchor term We note that, for Neg- CLIPs, the optimal value for À is 0.3 whereas for CLIP is it 1.9. This result is expected since Neg- CLIP* already fine-tuned and sep- arates negated captions into sep- arate cluster, thus requires only minor adjustment whereas original CLIP needs large offset to correct the embeddings.
70-
Bench in Fig. 7
Accuracy
50
0.0 0.5
1.0
Figure 7: We show accuracies for different values of X. For CLIP we show results for different approaches. Projection of e eng is denoted by Projesss.
LIMITATIONS AND FUTURE WORK
While our training- free,, embedding correction method achieves state-of-the-ard results and success- fully mitigates the catastrophic forgetting associated with fine-tuning approaches, the overall system retains two primary limitations that define clear paths for future research. The most significant is the syntactic brittleness of the negation extraction approach. The reliance on rule-based compo- nent for identifying the precise scope of the negated concept (C"eg) is inherently limited. Specif- ically, this approach struggles with complex syntactic structures (e.g., dist-dist dependencies like "A large, decorative, but clearly not present, table."), implicit negators (e.g.. quantifiers like "few people"). and ambiguous scope arising from multiple negations or complex conjunctions. We emphasize that this is bottleneck of negation extraction step only and flaw in our core vector arithmetic approach. A clear path for generalization is the integration of lightweight, special- ized, sequence tagging NLP model (c.g., fine-tuned DistilBERT) while maintaining the core zero-shot VLM approach. A second limitation concerns the simplicity of current benchmarks (NegBench, VALSE Existence, CC-Neg). These datasets primarily evaluate explicit, simple negation focused single objects. The remarkable success of our training-fir free method suggests these benchmarks chiefly test the disentanglement explicitly marked concepts. To truly test VLM logical reasoning, future benchmarks must incorporate complex logical structures (e.g., double negation) and implicit negation.
ensure robust scope detection
CONCLUSION
We analyze properties CLIP models on the task of negation understanding and note that despite available training datasets, these models struggle to fully understand negations. We show that CLIP text embeddings can be manipulated add or remove semantic information. Using this idea, we present a rule-based approach to explicitly remove semantic information about negated concept from text embeddings, Our approach improves baseline model and outperforms even the best models by wide margin. These results highlight the limitations of current approaches which are largely data driven, and suggest that future works should focus on enforcing compositional properties explicitly into the framework. For example, along with contrastive loss, proach to regularize learned text embeddings for negated captions which may help boost negation understanding. We hope to motivate research community to pursue efforts in this direction.
additionally
ap-
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Inter- preting clip with sparse linear concept embeddings (splice). Advances Neural Information Processing Systems, 37:84298-84328 2024.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker. Debiasing word embeddings, 29, 2016.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Kevin Chen, Christopher Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Asian conference on computer vision, pp. 100-116. Springer, 2018.
Weijing Chen, Linli Yao, and Qin Jin. Rethinking benchmarks for cross-modal image-text retrieval. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pp. 1241-1251, 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive tanguage-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2818-2829, 2023.
Jooyoung Choi, Jungbeom Lee, Chachun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022.
Guillaume Couairon, Matthijs Douze, Matthieu Cord, and Holger Schwenk. Embedding arithmetic of multimodal queries for image retrieval. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 4950-4958, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Matthijs Douze, Matthieu Cord, Holger Schwenk, and Guillaume Couairon. Embedding arithmetic of multimodal queries for image retrieval, 2022. URL
031 62
Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Investigating approaches for improving modal alignment in clip. arXiv preprint orXiv:2406.1763% 2024.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The paseal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Lei Fan, Jianxiong Zhou, Xiaoying Xing, and Ying Wu. Active open-vocabulary recognition: Let intelligent moving mitigate clip limitations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16394-16403, 2024.
Cheng- Yu Hsich, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarerepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural informa- tion processing systems, 36:31096-33 3063--1166 16, 2023.
Yu-Guan Hsieh, Cheng-Ya Yu Hsieh, Shih- Ying Yeh, Louis Béthune, Hadi Pour Ansari, Pavan Ku- mar Anasosalu Vasu, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, and Marco Cuturi. Graph- based captioning: Enhancing visual descriptions by interconnecting region captions. arXiv preprint arXiv:2407.06673, 2024.
Darina Koishigarina, Arna Uselis, and Seong Joon Oh. Clip behaves like a bag- ofg-off-ordds model cross-modally but not uni-modally. arXiv preprint arXiv:2502.03366 2025.

---

Published as a conference paper at ICLR 2026
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- see-ggrrned categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Jungbeom Lee, Sungjin Lee, Jinseok Nam, Seunghak Yu, Jaeyoung Do, and Tara Taghavi. Weakly supervised referring image segmentation with intra-chunk and inter-chunk consistency. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21870-21881, 2023.
Martha Lewis, Nihal V Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H Bach, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.1055 2022.
Fei-Fei Li, Marco Andreeto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022a.
Guohao Li, Yonglong Tian, Philip Torr, Kumail Alhamoud, Marzyeh Ghassemi, Yoon Kim, and Shaden Alshammari. Vision-language models do not understand negation, 2025. URL https:
waw..i.crg/ab///333333. 09425
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-languaee understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Pro- ceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 7086- 7096, 2022.
Ziyang Luo, Yadong Xi, Rongsheng Zhang. GongZheng Li, Zeng Zhao, and Jing Ma. Conditioned masked language and image modeling for image-text dense retrieval. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022, pp. 130-140, 2022.
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10910-109211 2023.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint 1rXiv:1301.331 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixhh Indian conference on computer vision, graphics image processing, pp. 722-729. IEEE, 2008
Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, lacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8253 8280, 2022.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives. Advances in neural information processing systems, 37:3273 1-77771-37600 2024a.
Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale. Chitta Baral, et al. Tripletelip: Improving compositional reasoning of clip via synthetic lasounlanguage negatives. Advances in neural information processing systems, 57:32731-32766
2024b.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565, 2018.
Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. Learn" no" to say" yes" better: Improving vision-language models via negations. arXiv preprint 2:Xi::2403.20312. 2024.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-texi gener- arithmetic. In Proceedings of the IEEE/CVF conference on computer
ation for vision and pattern recognition, pp. 17998- 2022.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint 250. 2025.
Emiel Van Miltenburg, Roser Morante, and Desmond Elliott. Pragmatic factors in image description:
The case of negations. arXiv preprint arXiv: 06164, 2016.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3156-3164, 2015.
Nam Vo, Lu Jiang, and James Hays. Let's transfer transformations of shared semantic representa- tions. arXiv preprint arXiv: 000.00093, 2019.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11886-11695, 2022.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 5296, 2016.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048-2080002 2057. PMLR, 2015.
Sungroh Yoon, Dahuin Jung. Jungbeom Lee, Sangwon Yu, Junsung Park. and Jongyoon Song. Know "no" better: data- approach for enhancing negation awareness in clip, 2025. URL
fictps://arxxr www.///g/////ooooooooooooo/...t/ /tt.o////....99990.... 10913
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like Bags-of-words. and what to do about it? arxiv preprint 10.00:2210.01936, 2022.
13

---

Published as a conference paper at ICLR 2026
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Ruisu Zhang, Yicong Chen, and Kangwook Lee. Zero-shot improvement of object counting with clip. 2023.
A APPENDIX
A.1 EXTRACTING NEGATED CONCEPTS Cmeg
Extracting the negated concepts from given caption is a very crucial step, and therefore, we de- velop rule-based negation scope detection algorithm that identifies negation cues and selects all words within the scope as Cree, Some negators require specific context, for example the words "absent" and "nowhere" trigger backward negation scope when preceded by copular verbs (is/are), whereas "empty" and "devoid" activate forward negation scope when followed by the preposition "of". Therefore, we classify the negators into pre-negators and post-negators. We define a compre- hensive list of negation words in each category as shown in Table. 7 and 8 For a given caption, we process the texts sequentially and check the presence of a negation word from both categories and then select part of the caption Creg accordingly.
The pre-negators are forward-scoper negators which means that they precede the clause they are negating, we activate the negation scope starting from the negator until clause boundary like punctuation or conjunction is encountered. We explicitly define the clause boundary as punctuation marks, coordinating conjunctions, contrasting conjunctions, and subordinating conjunctions. Punc- tuation marks (periods (.), commas ()), semicolons (:)), and contrasting conjunctions (but, although, though, however, nevertheless, yet, even so, while, whereas) typically terminate negation scope, whereas coordinating conjunctions (and, so, for) and subordinating conjunctions (because, since, as, if, unless, until) generally don't terminate the negation scope. Post-negators on the other hand mackward-scoped negators, succeeding the clause they are negating. Therefore, when such negators encountered, backward negation scope triggered until the clause boundary. In this way, we are able to extract the negated concept from given caption for both pre and post negations.
We present the template for extracting Cheg below
Caption Template for processing pre-negators
<caption text>> <pre-negator> <words within the negation scope>> <clause boundary> <remaining Caption.>
Caption Template for processing post-negators
<Caption text>> <clause boundary> <words within the negation scope>> <remaining Caption
<post-negator>
Pre-Negators
does not, lack, no, not, never, nothing none, nowhere, without misses, neither, nor, absencc/devvodd/empty of, hasn't, haven't 't, doesn't 't don't
Table 7: The list of pre-negators terms used.
Post-Negators
is/are/was/were
www.feeelooebeeeiieeeeeeeiiiiiee absence/devvoid/emppy of,
Table 8: The list of post-negators terms used.
14

---

Published as a conference paper at ICLR 2026
Caption C
Negator
Negated Part Chey
Pre-Negation
This image does include any cups
does lacks
include any cups a handbag dining table in sight a backpack
This image features a chair but lacks a handbag. This image features person, with no dining table in sight. The image depicts person, without a backpack.
no without absence of empty of
Note the absence of traffic light in this image.
a traffic light in this image people
This image features chair, but is empty of people.
No pizza is included in this image.
no doesn't
pizza is included in this image include refrigerator
This image shows bowl but doesn't 't include refrigerator.
Post-Negation
Skis are not visible in this image.
are not nowhere not absent
skis sink a person
A sink is nowhere to be in this image
cat is present this image, whereas person not.
cup is absent from this image.
a cup chairs
TV is present in this image, while chairs are not
are not
Table 9: Sample captions and corresponding negators with negated concept Cheg are shown.
A.2 COMPARISON WITH OTHER RULE BASED PARSER:
The NLP community has proposed several deterministic negation detection methods (c.g., NegEx, DEEPEN). Our current rule-based approach is significantly motivated by established clinical NLP tools like NegEx. The evaluation on NegBench-COCO. split for CLIP backbone using our custom VLM-focused rules and off-the-shelf NegEx implementation shown in Table. [O] As be noted, NegEx based caption parsing resulted in lower performance of 61.3% compared to our method 72.5%. We note that we use NegEx simply negated concept from the caption. The NegEx "termsets" does not include some of the common negations natural language and as result fails to parse many captions correctly leading to lower performance, For example, the pre-negation words such as 'lacks', 'excludes', 'won't', t', 'devoid of", "absent', etc. are absent NegEx. Similarly, post-negator words such not', 'is nowhere' and iis absent' are missing. We believe that, incorporating these negator words in the NegEx termsets will lead to better performance.
off-the-shelf component here, parse the
Method Acc.
Ours 72.5
NegFix
61.3
Table 10: We compare our rule based negation parser with off the shelf NegEx library.
B APPENDIX
B.1 RESULTS ON CC-NEG
We compare our approach with other methods on the CC-Neg dataset as shown in Table 11 Our method is easily able to outperform the existing methods, attaining the highest accuracy of 99.79% even outperforming the Con-CLIP model, fine-tuned on the CC-Neg dataset.
B.2 EVALUATION DIFFERENT MODEL PRETRAINING AND MODEL SIZES
To assess the effect of model size model's effectiveness, evaluate range of CLIP models pretrained on the laion dataset Fig. We see that our method yields significant improvement across all model sizes.
15

---

Published as a conference paper at ICLR 2026
Model
CCNeg Acc
CLIP openai
65.70
NegCLIP
62.63
FLAVA
58.93
BLIP
62.31
Con-CLID
99.70
Con-CLIP+Ours 99.79
Table 11: MCQ accuracy (%) comparison on the CC-Neg benchmark. Our zero-shot method outper- forms all baseline models including the fine-tuned Con-CLIP model, specifically trained CC-Neg data. This shows the effectiveness of our approach when compared to the models with domain- specific fine-tuning.
100
Original (COCO)
Corrected (COCO)
Original (VOC)
Corrected (VOC)
60
77.91
71.0
75.03
72.51
73.78
73.52
Accuracy (%)
69.02
64.56
60
31.50
31.8
30.28
25.72
26.32
25.52
25.28
20
400M, 28, Models (ViT)
2B, H-14
Figure 8: Effect of model architectures (B-16, B-32, L-14, H-14, G-14) and pre-training data size (LAION-400M LAION-2B).
Further, to understand the role of pretraining data size, also compare the performance of ViT- B/32 model with two pretrained weights, laion-400M and laion-2B. While the original performance of the model trained on the larger laion-2B dataset is worse (most likely due to the overfitting on af- firmations and lack of exposure to negated samples), our method improves both models consistently and laion-2F 2B model achieves best result indicating learning of richer semantic information.
B.3 EFFECT OF DIFFERENT ANCHOR CHOICES
We compare various choices of our anchor embedding on the NegBench COCO MCQ Table 12 We use few candidate words which are commonly used in natural language to show unbias and impartial opinions. We then compare results when using average of these words embeddings as an anchor. The most ideal choice, mean of all embeddings from text encoder, is also considered. We compute the mean of all captions in COCO MCQ NegBench dataset. We note that, this mean used oracle anchor to compare performance of other words a choice for anchor. We note that as long as we choose semantically netural word as the anchor there is not much variation the evaluation results.
16

---

Published as a conference paper at ICLR 2026
Next, also compare results when using random nouns other words with semantic inclination such as "extreme", "positive", etc. We note that, results drastically drop with such a choice of biased anchor.
Anchor Word
Acc
'neutral'
72.51
"balanced'
72.54
"unbiased"
72,42
'fair'
72.44
Mcan Above
72.52
Mean of NegBench-MCQ
72.53
Table 12: NegBench MCQ accuracy on COCO split. We show results when using different anchor embeddings. When anchor words with semantic meaning of neutralness used, achieve good accuracies.
Anchor Word
Acc
'cat'
50.78
'dog'
50.69
'baasdd'
54.67
positive'
53.93
extreme'
54.31
'right'
52.16
Table 13: NegBench MCQ accuracy on COCO split. We show results when using different anchor embeddings. When using arbitrary noun words or words with specific inclination such as the above words, accuracy dramatically drops.
B.4 RECURSIVELY ADDING ANCHIOR EMBEDDINGS
We analyze the effect of anchor embedding corrected embeddings of cf and c cc. The first half shows that without anchor embedding the corrected embedding e - ef is dissimilar to desired e and after using anchor ef ef + a, its similarity improves. However, further adding anchor terms as in e - e' + 2a and e - ee + 3a makes it more similar to the other embeddings ef i.e. add more and more confusion when repeated anchor embeddings are added into e e This result is expected since anchor embedding is supposed be unbiased and hence more closer to all the other embeddings. Thus, repeatedly adding anchor embeddings will remove semantic information from original embedding and will bring it closer to other embeddings.
the embeddings.
similar trend is observed for e
Target
ec
ef
ecf
en c
-0.07
-0.52
0.10
ec
e ef
0.61
0.38
0.57
ec e + 2a
0.64
0.54
0.54
ecf e
+
3a
0.64
0.59
0.52
eef ec
-0.47
-0.05
0.13
ef
eet 6 +
0.38
0.64
0.56
eff e + 2a
0.52
0.67
0.53
ef c + 3a
0.56
0.67
0.52
Table 14: We analyze effect of anchor embedding the quality of corrected embedding.
17

---

Published as a conference paper at ICLR 2026
Image
Ground Truth Caption
Caption selected by Neg- CLIP
Caption selected by our alg.
Note the absence of chair in this image.
No wine glass is included in this image.
Note the absence of a chair in this image. /
A cup nowhere be found this image.
A cup is present in this im- age, but there is no pizza. X
cup is nowhere be found in this image.
This image contains bowl but lacks cup.
A bowl is not present in this image.
This image contains a bowl but lacks a cup. /
A chair absent from this image.
No fork is present this image. X
A chair is absent from this image. /
No person is present in this image.
A carrot is not included in this image.
No person is present in this image. /
A dining table is not present in this image.
There is no hot dog in this image.
A dining table is not present in this image. /
The image showcases both refrigerator and cup.
This image does not contain refrigerator.
The image showcases both refrigerator and cup. /
A car absent from this image.
No bowl is present this image. X
A car is absent from this image. /
0
This image contains a book, with notable absence a chair.
A chair included this image, but there is no book. x
This image contains book, with a notable absence of chair./
This image contains an orange, but person is nowhere be seen.
No orange included in the image.
This image contains an orange. but person is nowhere to be seen. /
Table Qualitative comparison of caption outputs for NegCL.IP model and our algorithm.
B.5 QUALITATIVE ANALYSIS
In Fig. 15 we show some qualitative examples where NegCLIP* model fails to understand negation in the input captions. For example, in row 1, NegCLIP* model possibly aligned 'wine" text features with image contents and ignored the 'no' completely. Our approach corrected this embedding which resulted in this caption moving away from corresponding image embedding. Similarly, for second row, the word 'no pizza' resulted in incorrect alignment with image embedding but our method explicitly corrected the text embedding.
18


Reviews:
[{"review_id": "sCR80P8hrK", "reviewer": "Reviewer_G9hW", "summary": "This paper proposes a training-free text-embedding edit for CLIP to handle negation: detect the negated span with rules, subtract the projection of the full caption embedding onto the negated-concept embedding, and add an “anchor” vector (Eq. 2). On NegBench, MCQ accuracy jumps from 24.7/24.3/27.5 (COCO/VOC/MSRVTT) to 72.5/78.6/50.0 for CLIP; negated-caption retrieval R-Neg@5 improves modestly (COCO +5.9; MSRVTT +5.5).", "questions": "- It would be great to see if the proposed approach generalizes to similar but more recent models, such as SigLIP families. Is there any particular reason it was not tested on these?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple, training-free mechanism; clear formula and pipeline (Alg. 1, Eq. 2).\n\n- Large MCQ gains on NegBench without changing positive retrieval (R@5).\n\n- Geometry story (negation axis) supported by PCA visualizations (Fig. 3 and 4, Embedding space arithmetic visualization).", "weaknesses": "- Missing relevant related work: prior work [1] shows inference-time steering using CLIP geometry and visual-semantic arithmetic; it should be cited and contrasted (LM-side steering vs. text-side embedding edit). \n\n- Generalization beyond CLIP text towers (e.g., SigLIP/ALIGN, BLIP text encoders, MLLMs) is untested.\n\n[1] Tewel, Yoad, et al. \"Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w5yFi4W1OP", "reviewer": "Reviewer_u7Qg", "summary": "his paper aims to improve negation understanding in embedding-based vision–language models such as CLIP, which struggle to correctly interpret negated textual inputs. Prior approaches have mainly relied on data-centric solutions that fine-tune models on curated negation datasets, but these tend to be resource-intensive and sometimes degrade general performance.\n\nThe authors propose a zero-shot, training-free approach that decomposes a given text query into affirmative and negated parts using a rule-based negation detection algorithm, then applies a linear correction in embedding space to form a corrected “negation-aware” text embedding. The method achieves strong performance improvements on NegBench, boosting CLIP’s retrieval accuracy from 50.9% to 56.1% and multiple-choice (MCQ) accuracy from 25.5% to 67.0%.", "questions": "**Some questions and suggestions**\n\n1- have you considered replacing the rule-based negation scope detection with a learned or LLM-based decomposition approach? It might improve coverage and robustness to natural language variation.\n\n2- the NLP community has proposed several deterministic negation detection methods (e.g., NegEx, DEEPEN, or syntactic-scope models) that go beyond keyword matching. Could you compare performance to some of those?\n\n3- have you evaluated the approach on non-synthetic or human-written captions like real image descriptions to test out-of-distribution generalization?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1- the proposed approach is efficient, zero-shot, and training-free, and easy to integrate into existing CLIP-based pipelines.\n\n2- the paper shows consistent improvements on different model NegBench retrieval and MCQ tasks.\n\n3- the paper is well organized and easy to follow.", "weaknesses": "1- the rule-based negation detection might overfit to the NegBench benchmark, which is synthetically generated using structured templates and paraphrasing; therefore, generalization to natural or free-form text is uncertain.\n\n2- as the authors discussed in the limitation section, the method’s coverage of negation cues is incomplete. This approach misses implicit or compositional negations (e.g. “barely visible,” “few people”, \"alone\"), and may mis-handle nested clauses or double negation.\n\n3- although the improvement on the MCQ task appears large (+50% relative), the absolute task structure (4 options) limits interpretability since a +25% absolute increase is effectively equivalent to eliminating one distractor option.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t1vyhn0eNL", "reviewer": "Reviewer_YhKj", "summary": "This paper addresses VLMs like CLIP’s poor negation understanding (e.g., ignoring \"no\"/\"without\"). Unlike data-centric methods using hard-negative datasets, it proposes a zero-shot, training-free embedding correction approach. In particular, the authors propose a rule-based method to extract negated concepts, then a correction formula removes their semantic info from embeddings (using directional offset and anchor embedding).\n\nEvaluated on NegBench, it boosts CLIP’s MCQ accuracy from 25.5% to 67.0% and retrieval accuracy from 50.9% to 56.1%, outperforming fine-tuned models like NegCLIP.", "questions": "Q1. Only CLIP is discussed. How about applying to SigLIP and SigLIP2? These encoders are more welcomed in practical use. \n\nQ2. MLLMs use CLIPs to encode features. MLLM benchmarks can also be used to evaluate the performance of VLMs. How about using your method to train a LLaVA-1.6 model. Will the general VQA performance boosted?\n\nQ3. In Alg.1, return missing ' '; ’’ should be `` ''\n\nQ4. Will synonyms influence your algorithm? What is the quality and diversity of corrected embedding?\n\nQ5. What is the general retrieval, VQA, classification of the proposed method?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "S1. This paper is well written and is easy-to-follow.\n\nS2. The performance improvement is clear.\n\nS3. Sufficient visualizations are provided.", "weaknesses": "W1. The performance evaluation side is insufficient. Limited methods are compared. VLMs are mentioned but no further study is conducted.\n\nW2. Typos should be corrected.\n\nW3. According to Alg. 1, a set of negator words is utilized. The correctness of these words should be discussed. Besides, the quality of the generated embedding should be studied.\n\nSee questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "chtQdVAABT", "reviewer": "Reviewer_dika", "summary": "The paper points out that visual-language models such as CLIP have weak understanding of negation (no/not/without), often exhibiting affirmative bias. The authors propose a zero-training text embedding correction: rules extract the negated concept, estimate and subtract the \"negation direction\" from the embedding, significantly improving negation understanding without fine-tuning the model, and avoiding the decline in generality caused by fine-tuning the negation data.", "questions": "see the weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The method requires no hard negation data collection or model fine-tuning; it only needs to correct the \"negation direction\" in the CLIP text embedding, resulting in extremely low deployment costs.\n\nOn NegBench, it improved the MCQ of CLIP from approximately 25.5% to approximately 67.0%, and the retrieval rate from approximately 50.9% to approximately 56.1%; it also brought considerable gains to the fine-tuned NegCLIP.", "weaknesses": "The method assumes that \"negation\" is approximately linear and directional in the embedding space, but the semantics of \"missing\" different concepts/attributes are not necessarily collinear; using the global d_global may be ineffective or even lead to incorrect offsets for fine-grained concepts.\n\n\nHave you tried non-CLIP text towers (such as BLIP, SigLIP)? Even small-scale experiments should demonstrate the generality of the architecture.\n\nHow does the average overhead compare to standard CLIP text encoding? Is it easy to batch process to support large-scale retrieval? Please include a small performance table.\n\nIf λ is selected on COCO-MCQ, how would the retrieval be affected by fixing that λ to VOC/MSRVTT MCQ?\n\nRule-based negation scope is fragile. Negation scope extraction (Algorithm 1, page 6; Tables 6–8 in Appendix A.1, pages 12–14) has limited coverage of nested clauses, multiple negations, implicit negations (such as “few/barely”), quantifiers/comparison structures, etc. The main experiments are primarily based on NegBench patterns; generalization to real open-domain instructions remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NEEqlqxc1b", "reviewer": "Reviewer_aYJg", "summary": "- This paper investigates the persistent difficulty of VLMs in handling linguistic negation. Rather than relying on large-scale negation datasets and fine-tuning, the authors propose a training-free embedding correction method that manipulates CLIP’s embeddings directly. By identifying the negated concept via rule-based parsing and removing its semantic component through a projection-based subtraction (plus a neutral anchor addition), the approach aims to improve negation understanding. \n- Experiments on NegBench and CC-Neg show large performance gains over CLIP and fine-tuned NegCLIP baselines. The authors also provide some analyses to show intuitions onto why their proposed method works.", "questions": "- Strange citation bug, e.g. line 143, Similarly TripletCLIP...\n- Consider using an off-the-shelf tool for negation detection. In fact, your algorithm is quite similar to NegEx (https://pypi.org/project/negspacy/).\n- Improvements of 40–50% absolute accuracy for a post-hoc method seem implausibly high. Did the authors reimplement the CLIP baselines and confirm that those results are comparable with previous works of using CLIP (and NegCLIP) on those benchmarks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of improving negation understanding through direct, post-hoc manipulation of text embeddings is novel and unconventional to the more standard data-centric strategies. The approach offers a new angle for thinking about compositional semantics in VLMs.\n- The reported improvements on NegBench are substantial, even surpassing models fine-tuned on specialized negation datasets.\n- The embedding-space visualizations (PCA plots) and ablation studies provide qualitative and quantitative insights into how the correction affects embedding geometry.\n- The approach could inspire further work on embedding-space editing, compositional regularization, and interpretability.", "weaknesses": "- The main concern is the lack of theoretical grounding. The main motivation that negation corresponds to a linear operation in embedding space is not theoretically or empirically justified. CLIP embeddings are known to be highly entangled, and the projection subtraction used here lacks formal motivation beyond analogy to word-vector arithmetic. The choice of \"neutral\" words also seem arbitrary. This is totally fine as the emperical results are good, but the authors should be careful with their claims.\n- The method’s success on simple caption pairs may not translate to more nuanced natural language understanding (more complex negation patterns).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)