What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Authors: Inha Kang
Youngsun Lim
Seonho Lee
Jiho Choi
Junsuk Choe
Hyunjung Shim
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Model
Negation Understanding
Affirmative Bias
Described Object Detection
Chain-of-Thought Reasoning
Token Merging
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Enhances vision-language models' negation understanding in object detection through a novel dataset built with chain-of-thought reasoning and negation-aware token merging techniques.

Abstract: State-of-the-art vision-language models (VLMs) suffer from a critical failure in understanding negation, often referred to as affirmative bias. This limitation is particularly severe in described object detection (DOD) tasks. To address this, we propose two primary contributions: (1) a new dataset pipeline and (2) a novel, lightweight adaptation recipe. First, we introduce CoVAND, a dataset constructed with a systematic chain-of-thought (CoT) and VQA-based pipeline to generate high-quality, instance-grounded negation data. Second, we propose NegToMe, a novel text token merging module that directly tackles the architectural cause of affirmative bias. NegToMe fundamentally addresses the structural loss of negation cues in tokenization, grouping them with attributes into coherent semantic phrases. It maintains correct polarity at the input level, enabling robust negation understanding even with limited data. For instance, to prevent a model from treating the fragmented tokens "not" and "girl" as simply "girl", NegToMe binds them into a single token whose meaning is correctly distinguished from that of "girl" alone. This module is integrated with a parameter-efficient and strategic LoRA fine-tuning approach. Our method significantly improves performance on challenging negation benchmarks with a lowered false positive rate, boosting NMS-AP by up to +10.8 points on OVDEval and demonstrating generalization to SoTA VLMs. This work marks a crucial step forward in addressing negation understanding for real-world detection applications.

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
WHAT "NoT" TO DETECT: NEGATION-AWARE VLMs VIA STRUCTURED REASONING AND TOKEN MERGING
Inha Kang' Youngsun Lim Seonho Lee Jiho Choi Junsuk Choe 3 Hyunjung Shim
'KAIST AI Boston University 'Sogang University
trkswlsj13
glanceyes, jihochoi, kateshim wkwist
youngsunEbu.eduu jschoe@ .c.kr
ABSTRACT
State-of-the-art vision-language models (VLMs) suffer from a critical failure understanding negation, often referred to affirmative bias. This limitation is particularly severe in described object detection (DOD) tasks. To address this, we propose two primary contributions: (1) a new dataset pipeline and (2) a novel, lightweight adaptation recipe. First, we introduce CoVAND, a dataset constructed with systematic chain-of-thouggh (CoT) and VQA-based pipeline to generate high-quality, instance- -rouunddd negation data. Second, we propose NEGTOME, novel text token merging module that directly tackles the architec- tural cause of affirmative bias. NEG fundamentally addresses the structural loss of negation cues in tokenization, grouping them with attributes into coherent semantic phrases. It maintains correct polarity the input level, enabling robust negation understanding even with limited data. For instance, to prevent a model from treating the fragmented tokens not and qirl as simply girl, NEGTOME binds them into single token whose meaning is correctly distinguished from that of girl alone. This module is integrated with a parfmeter-efficient and strate- gie LoRA fine-tuning approach. Our method significantly improves performance challenging negation benchmarks with a lowered false positive rate, boosting NMS-AP by up to +10.8 points on OVDEval and demonstrating generalization SoTA VLMs. This work marks a crucial step forward in addressing negation understanding for real-world detection applications.
INTRODUCTION
Even state-of-the-art Vision-Language Models (VLMs) exhibit a critical failure in understanding negation due to an affirmative bias (Alhamoud et al., 2025). This bias reflects a model's tendency to prioritize nouns while ignoring crucial negation cues. The issue is particularly pronounced in described object detection (DOD) (Xie et al., 2023; Schulter et al., 2023; Yao et al., 2024; Dang al., 2023), a task requiring fine-grained compositional reasoning. As in Figure 1a, this bias causes models to treat phrases like "person with skateboard" and "person without skateboard" as semanti- cally equivalent, leading to identical and incorrect detections. This failure extends to more complex logical structures, such as double negatives (e.g., "not" + "un-") "), Since humans naturally use nega- tion in natural communication (Sarabi & Blanco, 2016; Morante & Blanco, 2021; Beukeboom al., 2020; Morante & Sporleder, 2012), failing to handle negation poses serious barrier to real-world scenarios. This shortcoming can be particularly dangerous in safety-critical domains. For example, in medical imaging, misinterpreting the distinction between "a tumor that is not malignant" and "a ummor that malignant" " can lead to critical misdiagnoses. Therefore, bridging and improving negation understanding is important step toward building robust VLM-based detection systems.
One key reason for the limited negation capability of VLMs is the lack of negated expressions in existing pre-training datasets. For example, large-scale datasets such LAION-400M (Schuhmann et al., 2021) contain about 0.08% negation words (Park et al., 2025). Likewise, Flickr30k (Plummer et al., 2015), widely used captioning dataset, exhibits only 0.04% negation words (Figure lb). In contrast, negation is much more prevalent in real-world language. For instance, 13.76% of words
"Corresponding author

---

What "Not" to Detect: Negation-Awer Aware VLMs via Structured Reasoning and Token Merging
Baseline
Ours
Baseline
Ours
RANAN
"person with skateboard" vs. "person without skateboard'
"uRpeeled banana" vs. "bunanu that not umpooeed"
(a) Qualitative Comparison on Challenging Negation Queries.
Flicka30k Captions (%)
CoVAND Captions (%)
Weught
Original Premcaned
COVAND NoyToMe
Noz
004
22.02
Virh
Avg. Adteation
32.31
Adyettie
9.29
Adwoh
Proponalion
13.93
13:75
9.98
15.93
Pastuation
Negption
3.15
Odbrr
(b) Word Class Distribution
(c) Average Attention Weight by Word Classes
Figure 1: Challenges with Negation Expressions. (a) Standard VLMs exhibit an affirmative bias, failing to distinguish contradictory negation queries. This issue stems from two causes: (b) the scarcity of negation words in standard datasets and (c) the model's tendency to assign low attention to negation cues. Our solutions, COVAND and the NEGTOME, directly address both problems.
in scientific papers (Szarvas et al., 2008) and 22.23% of words in Conan Doyle's stories involve negation (Morante & Daelemans, 2012). This imbalance results in VLMs that poorly equipped to learn or attend to negation semantics.
To mitigate this limitation, we introduce chain-of-thought with VQA alignment for negation detection dataset (CoVAND). It is a negation-focused training dataset constructed via chain-of- thought (CoT) reasoning and VQA-based caption alignment. To construct CoVAND, we first ex- tract both present and absent attributes from object regions. For each region, we then generate matched positive and negative captions using a CoT approach, followed by semantic verification using a VQA module. This process ensures each caption precisely reflects the presence or absence of key attributes, resulting in high-quality negation data pairs. As a result, our dataset provides a rich resource with 9.29% of negation words, a frequency 100x higher than that of typical datasets.
In addition to data- factors, we observe that negation tokens receive notably lower attention weights, suggesting that current VLM detectors architecturally ignore or undervalue negation cues, as shown in Figure lc. To counteract the low attention given to negation cues, the core of our method is NEGTOME, our novel text token merging module. It is designed to solve key problem where standard tokenization often fragments phrases, separating negation cues (e.g.. "not") from the at- tributes they modify (e.g., "lying"). NEGTOME addresses this by first merging these fragmented tokens into a single, coherent phrase. Through this binding, the negated concept of "not lying" can be learned as semantically distinct from "1 y rnn".." This step strengthens the role of the attribute by ensuring it is always interpreted within its negated context. Crucially, this merged representa- boost, explicitly amplifying the negated signal to ensure its
tion enhanced with a negation- polarity is preserved for downstream fusion. To our knowledge, this is the first work to employ a boosted token merging strategy for preserving semantic polarity in VLM-based detection.
To ensure the model effectively uses this enhanced text representation, we combine NEGTOME with a highly targeted application of Low-Rank Adaptation (LoRA). Our layer-wise attention anal- ysis revealed that the negation signal dissipates before reaching the final decision-making blocks. Therefore, we apply LoRA to the deep cross-attention layers, the core of multimodal compositional understanding (Laurençon et al., 2024; Hertz et al., 2022). Together, this strategy modifies less than 0.1% of the model's parameters yet achieves a significant improvement in negation comprehension.
Our approach achieves state-of-the-art performance with 6.6 mAp on 3d dataset, with 7.2 mAP improvement specifically on the challenging absence subset. In particular, our method not only in- creases the NMS-AP metric by 10.8 mAP but also reduces the false positive rate by 19.1%, demon- strating its enhanced ability to distinguish between contradictory queries. Importantly, these re- sults are consistently observed across multiple distinct evaluation datasets, despite the model being
2

---

What "Not" to Detect: Negation-Awarc VLMs via Structured Reasoning and Token Merging
e 3-Step Co'I Negation Caption Generation
VQA-based Caption Alignment
Final Resalts CoVAND
Altiiure lntracinn
Preuen Absar
Captian Conertionn
Ventication
which bbox adth captions?
Vienal Pronpting to the Turget Obiss
(raal without blue belset playing bastball. So not dressed unifomn.
dressed
in red unffrrm.
Prompting Type
Type: jeogge
Apeer [-y "alox Islask", "resching out with räght arm", Away Jor "batt, "reed untifene.. -]
plloing baxcsul
sel snikert
STEP
Cagg" "A boy whithout blue helzet playing baseball."
wathel thaihelree
par "A bny is not dressed in red uniform."
payyng toachal
boy without heloet playing baseball.
STEP 50
Vayy z (1) Contates negation "atthout". . (3) Itaters to "biue
(Gug ):
halmet i (3) The bay clearly saa blue helmet
(Gpu ): B
Figure 2: Dataset Generation Pipeline of the COVAND. Our method first generates negation- focused captions for visually prompted regions using three-step CoT process, then aligns each caption with the correct bounding box via VQA-based reasoning ensure semantic correspondence.
trained solely CoVAND.
This highlights the strength of our approach and its superior general- ization capability to unseen data and negation patterns.
Our work represents initial yet substantial step toward robust negation understanding with the following key contributions:
Our work presents COVAND, a systematically generated dataset focusing on negation, to bridge critical gap within existing multimodal benchmarks.
We propose novel adaptation recipe with NEGTOME, our text token merging module that introduces a negation-awe aware boost to preserve semantic polarity.
We achieve consistent gains across benchmarks, including +7.2 mAP on D absence sub- set and + +10.8 mAp the NMS-AP metric in OVDEval's negation subset, demonstrating effective generalization to real-world negation scenarios.
COVAND: DATASET GENERATION
To address the scarcity of negation data, we present COVAND, a region-grounded negation dataset constructed through multi-stage pipeline. As shown in Figure 2, the curation process consists of CoT caption generation followed by VQA-based alignment. This pipeline generates new high- quality captions that cover not only existence but also diverse attribute- negations. In this way, CoVAND provides -raiirdiddd compositional supervision that trains detectors more robustly than only injecting templated or caption- level negations (Alhamoud et al., 2025; Park et al., 2025).
VISUAL PROMPTING WITH BOUNDING BOXES
Before caption generation, we apply visual prompting (Cai et al., 2024) overlay marker on the image. The marker specifies the region to describe and directs the CoT model's attention to that area. We apply this technique bounding boxes in the Flickr30k Entities dataset (Plummer al., 2015). For each image, we randomly choose two boxes linked to meaningful objects and exclude any box that spans large background area avoid ambiguity. Each selected region is then highlighted with a red bounding box and serves as an input image for region- -rrounde caption generation.
2.2 THREE-STEP THAIN-OF-THOUGHT CAPTION GENERATION
We generate region-groundel paired negation captions through three-step CoT process using GPT- 4o (Hurst et al., 2024). We provide an explicit sequence that ensures consistent quality, rather than leaving it to the model's decision. The design follows the multi-step reasoning strategy of LLMs, where a complex visual query is split into ordered subtasks that improve factual accuracy and transparency. The input prompt for caption generation shows the image with red bounding box, a target phrase such as "a boy" in "person" type. These cues fix the subject within the highlighted area and guide each reasoning step. The three steps detailed below.
3

---

What "Not" to Detect:
Negation-Aware VLMs via Structured Reasoning and Token Merging
Step 1: Present and Absent Attribute Extraction. For each visually prompted region, we extract two sets of attributes: (1) Present Attributes (Apres), consisting of attributes visibly present within the bounding box (e.g., colors, actions, relationships, actions, etc.), and (2) Absent Attributes (Aahs), representing relevant but missing attributes that could reasonably be expected. This rich attribute pool is the key novelty that lets our pipeline create attribute-level negations, which are far beyond the object-level attributes used in prior approaches (Alhamoud et al., 2025).
Step 2:
Negative and Positive Caption Generation. We generate two types of paired captions using the extracted attributes:
Negative Caption (Cney): Incorrectly describes an attribute in Apres as absent (e.g.. "A man without a hat" when "hat" E Apres).
Positive Caption (Cpos) Correctly describes an attribute in Aats absent (e.g., "A woman without a red hoodie" when "red hoodie" E Aaba).
Each caption includes negation cues such "no", "not", "never", "without", the prefix "un-", the contraction "n't". The cue list is open to keep language natural and diverse.
Step 3: Verification. To ensure semantic consistency, we verify that Cpos accurately describes the region while Gneg contradicts it by asking GPT-4o. We also check whether generated captions contain negation words and attributes from step 1. If the pair fails on the test, it discards invalid captions and repeats caption generation until a valid pair appears or the retry limit is reached. This iterative guard preserves semantic integrity and keeps the quality of the overall dataset.
2.3 VQA-BASED CAPTION ALIGNMENT
The CoT stage produces a positive caption Gpos and a negative caption Gueg for each randomly chosen target box. However, label noise may still occur since another object of the same phrase type can also fit the captions. In Figure 2, for example, a person marked with "A" in the image could satisfy Gneg- even though it is not the designated target, which causes label noise. To eliminate this ambiguity, we add a dedicated region-level VQA alignment step.
First, we draw alphabetical labels on every box that shares the phrase type of the target. The tar- get box stays unlabelled because it has already passed the in-context verification step. To deter- mine the final alignment, we ask VQA model two separate questions: "Which labelles aalinns with Coos/Casp ?". Then, the VQA model simply answers with overlayed letters on the input images. While prior work used VQA for coarse, image-level validation (Park al., 2025), their approach fails to resolve which specific instance a caption refers to. Our region-level align- ment stage solves this ambiguity by requiring the VQA model to match each caption to a specific, visually-labeled bounding box, thereby delivering more region-level ground truth.
box
Through this multi-stage process combining CoT reasoning and VQA alignment, COVAND pro- vides rich training signals for negation understanding. We generate 91,110 captions with 23,876 images. In particular, our dataset exhibits approximately 9.29% negation word frequency, signifi- cantly higher than existing datasets like Flickr30k (0.04%). Detailed examples in Appendix A.
3
WITH NEGATION- NEGATION-SENSITIVE SENSITIVE TEXT TOKEN MERGING
Our method addresses the two root causes of negation blindness: token fragmentation and low attention on negation cues. We propose a lightweight adaptation recipe that integrates our novel text token merging module, NEGTOME, with a targeted application of LoRA as in Figure 3.
3.1 NEGATION LORA ADAPTER
We apply LoRA following (Hu et al., 2022) with two key enhancements for vision-languee fusion.
Given frozen base weights Wg. g. W, E d in cross-aties aoossatteentonn layers, we inject parallel adapters with an activation layer. Let a--) denote ReLU (Agarap, 2018) and let Aq. Av gxxd and By. B, e pdxr be the trainable low-rank matrices. For an input IRd we obtain
q = Wg + (B,z( v = W. + aB (Ag
(1)
where We, W., dxd the frozen base weights and a scales the LoRA update.
4

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
NEGTOME
cut not tying on skateboard
(CLS]
call
not
tring
the
Allatoo
[SEP]
Modbiltt
Neganon Adapter
(C.S)
cat
B
sicatee boardd
[SEP]
Layens
Image
Figure 3:
Overview of Training Pipeline. The input image and captions of COVAND are encoded by frozen backbones. NEGTOME assigns higher importance to negation cues in the text, and the LoRA adapter enables accurate localization of objects described by negated queries.
3.2 NEGTOME: SEMANTIC TEXT TOKEN MERGING FOR NEGATION UNDERSTANDING
Motivation. While fine-tuning with negation-rich data can partially alleviate affirmative bias, it does not address more fundamental flaw embedded in the model's tokenization process. Standard tokenizers inherently fragment phrases, separating negation cues (e.g.. "not"") from the words they modify (e.g.. "Lying"). This structural separation effectively causes the model to treat the phrase "not lying" as semantically equivalent to "lying", the attention weight of the isolated nega- tion tends to be ignored. To rectify this intrinsic information loss, we introduce NEGTOME. It moves beyond data-level fixes to structurally ensure that negated concept like "cat not lying" is represented as a single semantic unit, fundamentally distinct from ("cat." "not", "lying").
Text Token Merging. The caption is first split into sub-tokens T [t...... fn by standard tokenizer. To merge the tokens, an off-the-shelf parser then groups these tokens into disjoint phrase sets P = {PI,-- ..... Pm) where < For every phrase P. T, compute representative embedding by taking the normalized weighted average using fixed importance weights 7; of the sub-token vectors inside the phrase and replacing the original vectors with this average.
Negation-aware Boost.
After merging, let P be the phrase containing a cue (not, no, wi t hhout, J its index set. We assign larger weight to the negation cue:
un- enc.)tc.) and Ineg = { tj Pneg]
Eneg EjeZ,, T5 Ejet
Y = [B if is the negation cue,
B > 1.
(2)
otherwise,
The negation boosting factor 3 amplifies the cue that the merged embedding explicitly retains the negated meaning, improving polarity reasoning without increasing sequence length.
Effect of Negation Boost Representations. Suppose the encoder maps a caption of sub-tokens to vectors hi....,h,, hn E Rd We write he for the vector of the negation cue (e.g. "nor") and hp for the vector of the predicate modifies (e.g. "moving"). With vanilla mean pooling, the sentence embedding is h 1 E h, so the cue contributes only Ssingle (u, ha) /n n. to any linear probe Rd. After applying NEGTOME, the merged representation of the negated phrase becomes the and the pooled vector gives Smerge
hmeg B+1
B 3+1 (v, he) Hence
Smerge
B n
15 m <
Ssingle
B
++1
m
so the cue's influence is amplified by at least the factor 3+1 This gain aligns with the larger attention weights observed in Figure Ie and Figure S18, and experimentally show higher mAP.
4
EXPERIMENTS
4.1 EXPERIMENTAL. SETUPS
Datasets. DOD requires resolving compositional descriptions as in Figure 4a. To rigorously assess model's ability to overcome the affirmative bias inherent in VLMs, we select benchmarks
5

---

What "Not" to Detect: Negation-Awee Aware VLMs via Structured Reasoning and Token Merging
Described Object Detection
Sockel Bay
(DOD)
Supprnet of and Aite
class-gnnoed NMS
prosunt afoont
OVD
"a giont pands"" OVV, REC, DOD
glant parnt matien" REC. DOD
cangailaa
"a OVVN,, DUB
image
glone "lore pando nor eatting DOD
(a) Definition of DOD
(b) Process of NMS-AP score (Yao et al., 2024).
Figure 4: Definition of Task and Metric.
specifically designed to challenge negation understanding. We evaluate our method on two challeng- ing DOD benchmarks for negation detection in VLMs. Described Object Detection (D²) (Xie et al., 2023) introduces three evaluation protocols. Pres is a subset of 316 presence descriptions, ABS is 106 absence descriptions, and Full is an evaluation across all 422 descriptions. For OVDEval Nega- tion Subset (Yao et al., 2024), we report both standard AP and the NMS-AP. The standard AP score be misleadingly inflated when model, confused by fragmented tokens, predicts overlapping boxes for contradictory pairs like "black dog" and "dog that is not black". In contrast, NMS- AP (Yao al., 2024) applies stricter filtering by removing overlapping predictions on contradictory pairs with I055 effectively penalizing affirmative bias and accurately measuring negation under- standing (Figure 4b). Additionally, we employ a practical yet challenging evaluation by performing class-ignored NMS separately after predicting each caption individually. (see the Appendix F.I.)
Implementation Details. We implement parameter-efficient fine-tuning through LoRA (Hu al., 2022) applied to the deep cross-attention layers in the vision-language fusion module with 4. VLM-based detectors are trained for 000 iterations with a batch size of 24 for the Grounding DINO model, and 6, 000 iterations with batch size of 4 for the APE-Ti model. Training is con- ducted on two NVIDIA A6000 GPUs with mixed precision with learning rate of 5 10 Qwen- 2.5-VL (Bai al., 2025) is trained for I epoch batch size of 32 with learning rate of 5 x 10 All models are only trained with the CoVAND dataset using the Adam optimizer (Loshchilov & Hutter, 2017), freezing all backbone parameters except the LoRA layers. For NEGTOME, spaCy for the parser and set the negation boost factor 3 = 2.0. More details in the Appendix B.
4.2 EXPERIMENTAL RESULTS
Quantitative Results. As shown in Table even powerful Multimodal Large Language Models (MLLMs) struggle with the D benchmark. SoTA models like SPHINX-7B (Lin et al., 2023) and Qwen-2.5-VL-3B (Bai et al., 2025) achieve low performance on the full set (10.6 and 18.6 mAP,
Table 1: Evaluation on the Dd benchmarks. Descriptions categorized by length; S for 1-3, M for 4-6, L for 7-9, and XL for 10+ words. Pres refers to present and Abs refers to absence subset.
Architecture
D (default)
D (by length of texts)
Method
Backbone DesNeet-1044ii1
Full
OFA-I
BART
Sey2Seq OWL-VIT
4.2
4.1
4.6
4.9
5.4
3.0
2.1
OWL-ViT-L
VIT-L
CLIP
9.6
10.7
6.4
20.7
9.1
5,3
SPHINK-BB
CLIP.DINO-V,, Q-Former 1oNNet-1Ot4ViI
LLaMA-2
10.6
11.4
OFA-DOD
BART
Seg2Seq
21.6
23.7
15.4
23.6
22.6
20.5
18.4
GLIP-T
19.1
18.3
21.5
22.4
22.0
16.6
10.6
Swin-T
DyHlaa
21.4
20.6
W75
26.0
25.6
27.1
FIBER-B
22.7
21.5
26.0
30.1
25.9
17.9
13.1
GEN
Swin-B
RoBERTs-B
DyHead
26.0
25.2
28.1
35.5
29.7
20.5
14.2
W25
26.5
26.0
27.7
G-DINO.B
20.7
20.1
22.5
22.6
22.5
18.9
16.5
Ours
Swin-B
BERT
DINO
27.3
26.4
29.7
29.9
29.5
25.2
21.3
(t-A)
[+6.0)
(45.3)
(44.5)
APE-TI
29,1
29.9
26.9
Ours
VIFTI:
CLIP
DETA
32.5
32.9
31.5
33.2
35.3
31.3
25.4
AI
[+3.41
(43.3)
[+2.31
Qwea-225-VL-3B
18.6
18.5
19.2
Ours
WTT
Q.en:2.5
22.2
22.8
20.6
19.8
25.8
20.2
17.8

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
respectively). and their slow inference makes them impractical for many detection scenarios. In contrast, our lightweight adaptation recipe significantly boosts the performance of strong detector baselines. When applied to Grounding-DINe with a notable gain of +7.2 mAP on the challenging absence subset. This performance gain is di- rect evidence of more robust understanding of semantic polarity. Baseline models often generate false positives because they fail to distinguish between conceptually opposite phrases like "with hat" and "without hat". As specific absence scenario, when prompted with "a person without har" in an image where everyone is wearing one, they would incorrectly detect person. Our to- kenizer modification, NEGTOME, resolves this by foreing the model to process the negated phrase as single semantic unit with distinct polarity, enabling it to correctly reject such invalid instances. Similarly, on APE-Ti, we achieve a +4.6 mAP improvement on the absence subset, demonstrating an enhanced ability to reject non-existent objects. Notably, these gains are comparable to compu- tationally expensive, large-scale fine-tuning methods (Zhao et al., 2024a; Park et al., 2024b) while updating less than 0.1 % of the model's parameters only with our CoVAND dataset. The improve- ments are also consistent across all description lengths, validating the robustness of our approach. Furthermore, preliminary experiments demonstrate the generalizability of method MLLMs, with an improvement of +3.6 mAP on Qwen-2.5-VLL-
method improves the overall mAP by +6.6 points,
Even powerful SoTA MILMS struggle the chal-
Table 2: Results on OVDEval-Negation.
lenging OVDEval-Negation subset, demonstrating that simply applying large-scale model is not suf- ficient solution for negation. Notably, as shown in Ta- ble 2, the powerful Qwen-2.5-VL-7B underperforms the much smaller Grounding-DDin DINO baseline, high- lighting the difficulty of the task. In contrast, our lightweight adaptation recipe yields significant per- formance gains across tested architectures, partic- ularly the stricter NMS-AP metrie. Our method boosts the Grounding-DIne DINO by a substantial +10.8 mAP in NMS-AP and improves the Qwen-2.5-VL- 3B by +7.3 in mAP and +3.8 in NMS-AP. For the MLLM, the substantial AP gain is significant because it enhances both negation reasoning and foundational localization, a typical weakness of such mod- els. Further results, including detailed comparison with two-stage post-hoc VQA with MLLM and a full evaluation across all OVDEval subsets, are available in Appendix E and F.
means reproduced AP.
AP
NMS-AP
G-DINO-B
54.0
36.8
+ +Ours
57.2
47.6
A)
(+3.2)
(+10.8)
APE-Ti
50.5
32.3
+ +Ours
54.1
33.5
( (1))
(+3.6)
(+1.2)
Qwen-2.5-VL-777
37.8
35.9
Qwen-2.5-VL-33
34.6
31.3
+ Ours
41.9
35.1
( A)
(17.31
(13.8)
Images Captions
110.8
Jodartic
119.1
19.2
63.2
11.77
116.0
17.7....
COVAND-S
20.6k
41.5
46.0
COVAND-M
16k
60.4k
47.2
32-
36.8
44.1
COVAND-L
24k
91.1k
Boseline
DoVAND- CoVAND-M CoWAND-
Dazeltne
CoVAND-S COVAND- COVAND-L
(a) CoVAND splits.
(b) Dataset Scalahility. NMS-AP (left) and FPR (right).
Figure 5: Dataset Statistics and Performance Scaling, (a) Statistics for our three CoVAND splits. (b) Bar plots with blue refer to NMS-AP and pink refer to FPR (lower is better).
Dataset Scalability. Figure 5 presents our scalability analysis of the dataset on the OVDEval- Negation subset. We observe a consistent improvement as we scale the CoVAND dataset from small to large. Specifically, NMS-AP improves from 44.5 to 47.6, while the FPR decreases from 48.5% to 44.1%, which is total reduction of 19.1 points from the baseline. This trend of simultancously improving NMS-AP, a metric that penalizes contradictory predictions, while lowering FPR, which measures the failure reject absent objects, shows the effectiveness of our approach.
Qualitative Results. Figure presents qualitative results from the OVDEval dataset comparing our fine-tuned Grounding DINO model against the baseline. The baseline model often exhibits strong affirmative bias, frequently collapsing contradictory captions into the same prediction. Our model, however, successfully handles these complexities across various patterns. For instance, it accurately
7

---

What "Not" to Detect: Negation-Awann VLMs via Structured Reasoning and Token Merging
bome with aiinein Eeete wilkent La
COmpleY pama
HAST mihk BAST KUNAAr
2008 miNroa Sooking
shar AARACNAANE
GT
Roxeline
Ours
Figure 6: Qualitative Comparison the OVDEval Negation Subset. Our model correctly dis-
tinguishes the polarity of contradictory caption pairs, overcoming the baseline's 's affirmative bias.
Table 3:
Ablation Study. Best in blue and worst in red LoRA adapters are inserted at three fusion-blos block depths: shallow (blocks 0 2), strided (1, 3, 3)5, and deep (3-5).
Settings
OVDEval (Negation Subset)
Training Data
LoRA Placement
NEGTOMB
3
AP
NMS-AP
AR
NMS-AR
.FPR
Full
Pres
Abs
JFPR
Pretrained Weight
54.0
36.8
20.5
14.7
63.2
20.7
20.1
22.5
67.2
Flick33kk
shallow
21.7
61.3
Flickr30k
strided
X
54.8
36.5
20.5
14.1
62.6
20.9
19.9
24.0
68.2
deep
COVAND-S
shallom Tow
X
46.8
31.5
21.9
14.8
56.0
18.5
17.6
21.0
63.9
COVAND-S
sL:ided
52.8
43.9
20.0
17.1
49.0
20.1
19.2
22.9
63.4
COVAND-S
deep
55.4
41.8
21.4
18.0
48.6
24.2
23.0
27.0
64.0
COVAND-S
deep
1.0
57.8
43.8
24.0
19.6
50.8
25.7
25.1
27.3
63.7
COVAND-5
2.0
58.7
44.5
24.1
19.2
48.5
26.2
25.4
28.2
63.3
identifies the "cow without looking at the camera" and the "horse that is not urinating", proving it ground negation in complex contexts. Moreover, for "banana that is not unpeeled", it cor- rectly identifies the peeled banana by resolving the "not" + "un-" double negative as in Figure la. Our model sometimes fails detect every target instance, for example "pizza that is not complete", its predictions are a marked improvement over the baseline, which provides completely unreliable com-
detections for both queries. Together, these examples show that method achieves positional understanding of negation. Further qualitative results on OVDEval and D are presented Figure S23-S24 and Figure $25-S27, respectively.
4.3 ABLATION STUDY
Our ablation study, summarized in Table 3, reveals the impact of each component, with attention diagnostics in Figure S18 in the Appendix providing a clear mechanism for the improvements. Plac- ing LoRA adapters in the deep fusion blocks consistently outperforms shallow This is because deep placement maintains elevated attention on negation tokens in the later blocks where deci- sions are formed, whereas the effect of shallow placement dissipates too early. Furthermore, training with CoVAND dataset yields substantial gains over generic captions, demonstrating its value for both accuracy and generalization. Finally, adding NEGTOME with its negation boost fac- tor provides large gains, such as +2.7 improvement in NMS-AP. This trend is mirrored on the D benchmark. While using our COVAND dataset alone yields a +2.2 mAP improvement over the baseline, NEGTOME adds further +2.0 mAP top. This near-equal contribution highlights that our token merging strategy is as impactful as the dataset itself. The attention analysis further confirms that NEGTOME directly causes this improvement by increasing attention to the negated phrase. Together, these results motivate our final design that locates adaptation late in the fusion stack and explicitly increases negation counteract affirmative bias.
8

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
positive
negative
hybrid
(1) This depicts
bovt.
(3) is not thiN Snage:
with no truck in night:
This features
(3) dining table is sousent image.
This image Features dintng table.
visible.
This image containe botcle, Aithout fack.
bottle
but there is person.
image
this
banana appears this
Fook is
this Image
(4) This image doss not have a paeson in 1t.
Figure 7: Qualitative Comparison on the NegBench MCQ Benchmark. Captions with green
checkmark GT. pink refer to Baseline, and blue refer to Ours.
4.4 ZERO-SHOT DOWNSTREAM EVALUATION OF SEMANTIC COMPREHENSION.
To verify our method achieves seman- tic understanding of negation that general-
Table 4: Results on the NegBench Multiple
Choice Question (MCQ) benchmark.
izes beyond detection, we evaluate it on the NegBench COCO subset of Multiple Choice Question (MCQ) benchmark (Al- hamoud et al., 2025). This task requires the model to select the most accurate caption for an image from four options. These options include three subsets: 'Positive' correctly af- firming present objects (e.g.. "A and B")), 'Negative" correctly negating absent ones (e.g.. "nol B"), and 'Hybrid' that combine both types (e.g.. "A but not B"). In zero-shot setting, we select the caption that produces the highest max-logit score when grounded in the image. shown in Ta- ble 4, our method improves accuracy over the baseline with a +10.86% improvement. This result provides strong evidence that our approach enhances a robust understanding of negation. We present qualitative examples in Figure 7 and in Appendix H.
Model
Overall Acc.
Positive
Negative
Hybrid
CLIP-Opendh NegCLIIP
16.27 %
10.21 %
G-DINO-B
21.69 %
27.36 %
13.37 %
23.71 %
+ +uurs
32.55 %
46.85 %
23.37 %
26.64 %
07 A1
(-10.86)
(+19.49)
(+10.00)
1+2.931
4.5 ZERO-SHOT GENERALIZATION ON BIOMEDICAL DOMAIN
To validate that our method learns a robust nega- tion mechanism rather than merely memorizing the training data, we conducted a zero-shot evalua- tion on the biomedical domain using the FG-CXR dataset. This domain presents significant general- ization challenges due to its distinct visual features (e.g.. grayscale X-rays) and unique negation taxon-
Table 5: Zero-shot Results on FG-CXR.
Method
Accuracy
Baseline (G-DINO-B)
54.86% 62.55 %
+ Ours (NEGTOME)
omy. We formulated zero-shot binary discrimination task by generating hard negative contra- dictions for each GT diagnosis through rule- based polarity flipping (e.g., presence vs. absence of disease). The model was evaluated on its ability to assign a higher matching score to the GT caption than to the hard negative. While the baseline model (G-DINO-B) struggled with near-random curacy of 54.86%, our NEGTOME method achieved 62.55% (+7.69%). Since our model was never exposed to medical data during fine-tuning, this substantial improvement demonstrates that NEG- ToME effectively structuralizes the binding between negation cues and their targets, enabling robust generalization to entirely unseen domains. Details are in
RELATED WORK
5.1 OBJECT DETECTION
OVD extends classical detectors to arbitrary text labels (Zareian et al., 2021; Yao et al,, 2022; Kim et al., 2023a; Chen et al., 2025d). Methods such as GLIP (Li et al., 2022), FIBER (Dou et al., 2022), and APE (Shen et al., 2024) fuse language either in the detection head, in the backbone, or in a fask-genent
prompt module, and achieve strong zero-shot performance. REC adds composi- tional phrases. Grounding DINO (Liu et al., 2024b) proposes stTR- decoders that localize the described object without category supervision. Despite this progress, REC models still assume the target exists and therefore struggle to reject absent negated descriptions. DOD (Xie et al.,

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
2023) generalizes OVD and REC by requiring the detector to decide both existence and location. Benchmarks such as D and OVDEval (Yao et al., 2024) reveal low in accuracy on absence or negation subsets. It confirms that current VLMs often have affirmative bias on negation cues. MILM (Lin et al., 2023; Bai et al., 2025) have recently been applied to DOD, but their accuracy fails to surpass that of VLMMbasedd detectors, their performance inference speed is incompatible with real-time detection scenarios. We tackle negation explicitly by (I) introducing CoVAND, high-coverage negation dataset, and (2) proposing lightweight LoRA and NEGTOME recipe that plugs into VLM decoders. This design yields higher mAP and lower FPR on both D and OVDEval, thereby overcoming the limitations of prior approaches (Zhao al., 2024a; Park et al., 2024b).
negation remains low, and their
5.2 NEGATION UNDERSTANDING IN VISION-LANGUAGE MODELS
CLIP-based studies such as NegBench (Alhamoud et al., 2025) reveal the affirmative bais that state-of-the-arr VLMs often treat "dog" and "not dog" identically; subsequent fixes like Negation- CLIP (Park et al., 2025) simply augment pre-training with template-level negation pairs and thus miss context deeenndet or region-groundd reasoning and VQA alignment, producing positive and negative caption pairs that are grounded to target boxes, and show that this richer supervision transfers to multiple architectures beyond CLIPP.
cases. We instead build a fine-grained dataset with CoT
5.3 TEXT TOKEN-LEVEL MERGING
Token Merging (ToMc) (Bolya et al., 2022) merges similar image tokens to accelerate inference without sacrificing accuracy. It is a technique originally proposed for Vision Transformers (ViTs), where similar image tokens are merged to accelerate inference without sacrificing accuracy. ToMe extended to diffusion and grounding models, where token merging based on semantic phrase introduced to mitigate the loss of modifier information (Hu et al., 2024; Li et al., 2024b). In the context of OVD, there have been attempts to merge image tokens (Su et al., 2024; Norouzi et al., 2024), but the merging of text tokens has been unexplored. Previous studies text token merging have primarily focused on diffusion models, particularly in text-to-image generation (Hu the first explore text token merging in detection models and
al., 2024). In this work, empirically demonstrate its feasibility and effectiveness.
6 CONCLUSION
This work presents comprehensive solution to the affirmative bias that hinders negation under- standing in VLMs by addressing its two root causes. To resolve data scarcity, we introduce Co- VAND, systematic pipeline using CoT reasoning and VQA-based alignment to generate high- quality, instance-grounded negation data. To counteract the model's architectural tendency to ignore negation cues, we propose NEGTOME, novel module that, to our knowledge, is the first to use negation-award -wwae boost to preserve semantic polarity in detection tasks. Our parameter-efficient recipe integrates these contributions to achieve substantial gains on challenging negation benchmarks and demonstrate strong generalization across VL-M-aseed detectors and MI MI.LMs, marking significant step towards VLMMs that understand not only what is present, but also what is absent.
ACKNOWLEDGMENTS
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (No. RS-2025-00520207 the Institute of Infor-
mation & Communications Technology Planning & Evaluation (IIPP) grants funded by the Korea government (MSIT) (Nos. 2022-0-00680 2022-0-01045. RS--R2---0000 2024-00457882. RS-2025-02217259, RS-2019
1-20019-I11 III 990075), including the National AI Research Lab Project and the Artificial Intelligence Graduate School Support Program (KAIST); and the Korea Evaluation Institute of Industrial Tech- nology (KEIT) grants funded by the Korea government (MOTIE) (Nos. 2022-0-00680, 2022-0- 01045, RS- RS-2025-02217255
10

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
BIBLIOGRAPHY
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.0955 09540, 2023.
Abien Fred Agarap. Deep learning using rectified AXii:/803.0837, 2018.
linear units (relu).
arXiv preprint
Paul Albert, Frederic Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton van den Hengel, and Ehsan Abbasnejad. Randlora: Full-rank parameter-efficient fine-tuning of large mod- cls. arXiv preprint SKGi::2502.0098877 2025.
Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. Vision-language models do not understand negation. arXiv preprint arXiv:2501.09425 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
URL https::////// /awpiv.xvi../////// orgg abs/ 20os/2000 2502.502..00 13923.
Camiel J Beukeboom, Christian Burgers, Zsolt P Szabó, Slavica Cvejic, Jan-Erik M Lönnqvist, and Kasper Welbers. The negation bias in stereotype maintenance: replication five languages. Journal of Language and Social Psychology. 39(2):219-236, 2020.
Franziska Boenisch, Kamil Deja, Adam Dziedzic, et al. Precise parameter localization for textual generation in diffusion models. arXiv preprint arXiv:2502.09935, 2025.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:22 2022.
Rui Bu, Haofeng Zhong, Wenzheng Chen, and Yangyan Li. Value-state gated attention for mitigating extreme-token phenomena in transformers. arXiv preprini 1:Xiv:2510.09011 2025.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
Declan Campbell, Sunayana Rane, Tyler Giallanza, Camillo Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven Frankland, Tom Griffiths, Jonathan D Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37:1 13333-1134666 2024.
Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann. A comprehensive survey of scene graphs: Generation and application. IEEE Transactions Pattern Analysis and Machine Intelligence, -5 2021.
Chongyan Chen, Samreen Anjum, and Danna Gurari. Vqa therapy: Exploring answer differences by visually grounding answers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15315-15325, 2023.
Mu Chen, Liulei Li, Wenguan Wang, and Yi Yang. Diffvsgg: Diffusion--riven online video scene graph generation. Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 001229112219777 2025a.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678. 2022.
Wenxiang Chen, Wei He. Zhiheng Xi. Honglin Guo, Boyang Hong. Jiazheng Zhang, Nijun Li. Tao Gui, Yun Li, Qi Zhang, et al. Better process supervision with bi-directional rewarding signals. In Findings of the Association for Computational Linguistics: ACI. 2025, pp. 14471-14485, 2025b.
11

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
Xi Chen, Aske Plaat, and Niki van Stein. How does chain of thought think? mechanis- tic interpretability chain-of thouuutt reasoning with sparse autoencoding. arXiv preprint arXiv:2507.22928, 2025c.
Yuming Chen, Jiangyan Feng, Haodong Zhang, Lijun Gong, Feng Zhu, Rui Zhao, Qibin Hou, Ming- Ming Cheng, and Yibing Song. Re-aligning language to visual objects with an agentic workflow, 2025d. URL https:: /arxivv.v./..////////aa//aaaaav/ org/ abs abs/2503. 23508.
Ziyang Chen, Erxue Min, Xiang Zhao, Yunxin Li, Xin Jia, Jinzhi Liao, Jichao Li, Shuaiqiang Wang, Baotian Hu, and Dawei Yin. A question answering dataset for temsoral-sensitive retrieval- augmented generation. Scientific Data, 12(1))1855, 2025e. ISSN 2052-4463. doi: 10. 10.1038/ $41597-025-06098- URL https:/ http://oooooogg/.//////// 10.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXivv2507.06261 2025.
Ming Dai, Jian Li, Jiedong Zhuang, Xian Zhang, and Wankou Yang. Multi-task visual grounding with coarse-to-fine consistency constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 2618-2626, 2025.
Ronghao Dang, Jiangyan Feng, Haodong Zhang, Chongjian Ge. Lin Song, Lijun Gong, Chengju Liu, Qijun Chen, Feng Zhu, Rui Zhao, et Instructdet: Diversifying referring object detection with generalized instructions. arXiv preprint arXiv:2310.051300 2023.
Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang, Afshin Dehghan, et al. Mm-spatial: Exploring 3d spatial understanding in multimodal llms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7308-7408, 2025.
Jiajun Deng, Zhengyuan Yang. Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to- end visual grounding with transformers. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pp. 1769-1779, 2021.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang. Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse-to- to-fine assion-language pre- pre-training with fusion in the backbone. Advances neural information processing systems, 35:32942-32956. 2022.
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1):110-120, 2023.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 1051105,,,, 2024.
Shaoxiong Feng, Xuancheng Ren, Kan Li, and Xu Sun. Hierarchical inductive transfer for continual
dialogue learning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 693-699, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 20118653/11/222 1/2002.findnngg---emee https: //acl antho ogy. or2/2022. fi indingg-acll.... 57/.
fwwdings--cll.555 URL
12

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Arduin Findeis, Floris Weers, Guoli Yin, Ke Ye. Ruoming Pang, and Tom Gunter. Can external validation tools improve annotation quality for llm-as con-as-a-jja judgge-- In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15997-
16020, 2025.
Zhizhang FU, Guangsheng Bao, Hongbo Zhang, Chenkai Hu, and Yue Zhang- Correlation or causation: Analyzing the causal structures of Ilm and Irm reasoning process. arXiv preprint arXiv:2509.17380, 2025.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint arXiv:2402.00 08562, 2024.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, and Vs Subrahmanian. MoLA: MoE LoRA with layer-wise expert allocation. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACI. 2025, pp- 5097-5112, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 111788889976-19995 URL https: http:///aaaaatttooooo org/2025 findings naaci. 284/.
Walter Gerych, Haoran Zhang, Kimia Hamidich, Eileen Pan, Maanas K Sharma, Tom Hartvigsen, and Marzyeh Ghassemi. Bendvlm: Test-time debiasing of vision-language in Neural Information Processing Systems, 77762480-62500 2024.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2918-2923 2021.
Yongbin Guo, Shuzhen Li, Zhulin Liu, Tong Zhang, and C.L.Philip Chen. A parameter-efficient and fine-grained prompt learning for vision-language models. In Wanxiang Che, Joyce Nabende, Eka- terina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 11346-31355 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 1/26655///222222lllloooo acl-long. 1514/.
URL
ortp:/org/2200 2025.
Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of -f-thhughttt dataset for fine-grained video reasoning via core frame selection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26181-26191, 2025.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficien fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Michael Hanna, Mateusz Piotrowski, Jack Lindsey, and Emmanuel Ameisen. Circuit-tracer: A new library for finding feature circuits. Yonatan Belinkov, Aaron Mueller, Najoung Kim, Ho- sein Mohebbi, Hanjic Chen, Dana Arad, and Gabriele Sarti (eds.), Proceedings of the 8th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 239-249, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-344 2025. URL https: /tcl//aclllaa wwt.ccaaathoooggooooooo
3. doi: 10. 10.18653/v1/2025.11 121166531///22555lakbmmmllll/l/ blackboxnlp-1.14/
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompe image editing with cross attention control. arXiv preprint arXiv:2208.0166 016666 2022.
Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. An analysis of negation in natural language understanding corpora. arXiv preprint arXiv:2203.08929 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
13

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Taihang Hu, Linxuan Li, Joost van de Weijer, Hongcheng Gao, Fahad Shahbaz Khan, Jian Yang, Ming-Ming Cheng, Kai Wang. and Yaxing Wang. Token merging for training-fres semantic binding in -mxa-to-image synthesis. Advances in Neural Information Processing Systems, 37: 137646-137672, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2127,, 2024.
Amirmohammad Izadi, Mohammad Ali Banayeeanzade, Fatemeh Askari, Ali Rahimiakbar, Mo- hammad Mahdi Vahedi, Hosein Hasani, and Mahdich Soleymani Baghshah. Visual struc- tures helps visual reasoning: Addressing the binding problem in vims. arXiv preprint arXiv:2506.22146 2025.
Zeyu Jia, Alexander Rakhlin, and Tengyang Xie. Do we need to verify step by step? rethinking process supervision from theoretical perspective. arXiv preprint arXiv:2502.10581, 2025.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1780-1790, 2021.
Evangelos Kazakos, Cordelia Schmid, and Josef Sivic. Large-scale pre-training for grounded video caption generation. arXiv preprint arXiv:2503.11771 2025.
Fucai Ke, Joy Hsu, Zhixi Cai, Zixian Ma, Xin Zheng. Xindi Wu, Sukai Huang, Weiqing Wang, Pari Delir Haghighi, Gholamreza Haffari, et al. Explain before you answer: A survey on compo- sitional visual reasoning. arXiv preprint arXiv:2508. /7298, 2025.
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open-vocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 11144-11154, 2023a.
Dahun Kim, AJ Piergiovanni, Ganesh Mallya, and Anelia Angelova. Videocomp: Advancing fine- grained compositional and temporal alignment in video-text models. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, pp. 29060-29070, 2025.
Yeongbin Kim, Gautam Singh, Junyeong Park, Caglar Gulcehre, and Sungjin Ahn. Imagine the unseen world: a benchmark for systematic generalization in visual world models. Advances in Neural Information Processing Systems, 26:27780-27886, 2023b.
Jian Lan, Yifei Fu, Udo Schlegel, Gengyuan Zhang, Tanveer Hannan, Haokun Chen, and Thomas Seidl. My answer is not fair': Mitigating social bias in vision-language models via fair and biased residuals. arXiv preprint 250.::5505.23798 2025.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87774-87907, 2024.
Nicholas Lee, Thanakul Wattanawong, Schoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting llms with novel iterative data enhancement. In Findings of the Association for Computational Linguis- fics: ACL 2024, pp. 6498 0026,5266622 2024.
Alexander Cong Li, Ananya Kumar, and Deepak Pathak. Generative classifiers avoid shortcut solutions. In International Conference on Learning Representations (ICLR), 2025a. URL https://
net htte:pddd2c----CCCBBB.-
Poster Presentation.
Chuanhao Li, Chenchen Jing, Zhen Li, Mingliang Zhai, Yuwei Wu, and Yunde Jia. In-context compositional generalization for large vision-language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 17954-17966, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v h/222/.cmmllmmaaiiii main.996. URL https://acll h/thh//hlllaahhacca wog..orrg/a... emnlp-ma i n. 996/.
14

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Chuanhao Li, Wenbo Ye, Zhen Li, Yuwei Wu, and Yunde Jia. Multi-soureed compositional gener- alization in visual question answering. arXiv preprint wrXiv:2505.23044 2025b.
Liunian Li, Zi-Yi Dou, Nanyun Peng, and Kai-Wei Chang. Desco: Learning object recognition
with rich language descriptions. Advances in Neural Information Processing Systems, 36:37511-
37526, 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10965-10975, 2022.
Tianle Li, Jihai Zhang, Yongming Rao, and Yu Cheng. Unveiling the compositional ability gap in vision-language reasoning model. arXiv preprint arXiv:250. /9406, 2025c.
Wentong Li, Yuqian Yuan, Jian Liu, Donggi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.0233 2024b.
Yanjun Li, Zhaoyang Li, Honghui Chen, and Lizhi Xu. Unbiased video scene graph generation via visual and semantic dual debiasing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19047-19056, 2025d.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint 1rXiv:2311.07575. 2023.
Shih- Yang Liu, Chien-Y Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- low-rank adaptation. In Forty-first
Ting Cheng, and Min-Hung Chen. Dora: International Conference on Machine Learning, 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2024b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17Xiv: 1.051001, 2017.
Roser Morante and Eduardo Blanco. Recent advances in processing negation. Natural Language
Engineering, 22(2):121-130, 2021.
Roser Morante and Walter Daelemans. Conandoyle-neg: Annotation negation in conan doyle sto- ries. In Proceedings of the eighth international conference on language resources and evaluation, istanbul, pp. 1563 1563-1568. Citeseer, 2012.
Roser Morante and Caroline Sporleder. Modality and negation: An introduction to the special issue. Computational linguistics, 38(2):223-260, 2012.
Aashiq Muhamed, Mona Diab, and Virginia Smith. Decoding dark matter: Specialized sparse au- tocncoders for interpreting rare concepts in foundation models. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1604-1635, 2025.
Nilay Naharas, Dang Nguyen, Nesihan Bulut, Mohammadhossenn Bateni, Vahab Mirrokni, and Baharan Mirzasoleiman. Data selection for fine-tuning vision language models via cross modal alignment trajectories. arXiv preprint arXiv:25 2025.
15

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Narges Norouzi, Svetlana Orlova, Daan De Geus, and Gijs Dubbelman. Algm: Adaptive local- then-global token merging for efficient semantic segmentation with plain vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15773-15782, 2024.
Bo Pang, Tingrui Qiao, Caroline Walker, Chris Cunningham, and Yun Sing Koh. Cabin: Debiasing vision-language models using backdoor adjustments. In James Kwok (ed.). Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IICAI-25, pp. 484-492. International Joint Conferences on Artificial Intelligence Organization, 8 2025. doi: 10.24963/ URL htttpp:///.i.rrrtppeeeett wtt.r//...499..0 24963/ jiaaiiiiiiii/i 2025/5555555 Main Track.
ijeai. 2025/555
Junsung Park, Jungheom Lee, Jongyoon Song, Sangwon Yu, Dahuin Jung, and Sungroh Yoon. Know" no" better: A data-driven approach for enhancing negation awareness in clip. arXiv preprint arXiv:2501.10913, 2025.
Junyoung Park, Jin Kim, Hyeongjun Kwon, Ilhoon Yoon, and Kwanghoon Sohn. Layer-wise auto- weighting for non-stationary test-time adaptation. Proceedings of the IEEE/CVF Winter Con- ference Applications of Computer Vision, pp. 1414-1423, 2024a.
Kwanyong Park, Kuniaki Saito, and Donghyun Kim. Weak-to-strongg compositional learning from generative models for aunuuge--bsed object detection. In European Conference on Computer Vision, pp. 1-19. Springer, 2024b.
Trong Thang Pham, Ngoc-Vuong Ho, Nhat-Tan Bui, Thinh Phan, Patel Brijesh, Donald Adjeroh, Gianfranco Doretto, Anh Nguyen, Carol Wu, Hien Nguyen, al. Fg-cxr: radiologist-aligned gaze dataset for enhancing interpretability in chest x-ray report generation. In Proceedings of the Asian conference on computer vision, pp. 241-958. 2024.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to- segion-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Daniel Reich and Tanja Schultz. Uncovering the full potential visual grounding methods in VQA. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4416,4419, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653vvvv 2024.acl-long 241. URL https:tp / //aacclaa aaclantholooyy wttoo/////2/1/111oooor.oooo 241/.
Zahra Sarabi and Eduardo Blanco. Understanding negation in positive terms using syntactic de- pendencies. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1108-1118, 2016.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114. 2021.
Samuel Schulter, Vijay Kumar B G. Yumin Suh, Konstantinos M. Dafnis, Zhixing Zhang, Shiyu Zhao, and Dimitris Metaxas. Omnilabel: A challenging benchmark for language-based object detection. In ICCV, 2023.
Dominykas Seputis, Serghei Mihailov, Soham Chatterjee, and Zehao Xiao. Multi-modal adapter for vision-language models. arXiv preprint arXiv:2409.02998, 2024.
Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEF/CVF Conference on Compuler Vision and Pattern Recognition, pp. 6820-6829, 2023.
Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13193-13203 13203, 2024.
16

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Robik Shrestha, Kushal Kafle, and Christopher Kanan. A negative case analysis of visual ground- ing methods for VQA. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8172-8181, Online, July 2020a. Association for Computational Linguisties. doi: 10. 18653v1// 2020.acl-main. 727. URL www.t:tttttt/eectm http:p/:////aaaaahlloooooooooo/
org/ /200 acl-main 727/-
Robik Shrestha, Kushal Kafle, and Christopher Kanan. Visual grounding methods for vqa are work- ing for the wrong reasons! arXiv preprint arXiv:2004.05704, 2020b.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment tree- bank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.). Proceedings of the 2013 Conference on Empirical Methods in Natural Language Process- ing. pp. 1631-1642 Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https:::p//...
Jiajun Song, Zhuoyan Xu, and Yiqiao Zhong, Out-of-distribution generalization via composition: lens through induction heads in transformers. Proceedings of the National Academy of Sciences, (22(6))2417182122 2025.
Wei Su, Peihan Miao, Huanzhang Dou, and Xi Li. Scanformer: Referring expression comprehension by iteratively scanning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11448, 2024.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 2023.
György Szarvas, Veronika Vineze, Richárd Farkas, and János Csirik. The bioscope corpus: annota- tion for negation, uncertainty and their scope in biomedical texts. In Proceedings of the workshop current trends in biomedical natural language processing, pp. 38-45, 2008
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume I (Long and Short Papers), pp. 4149-4158, 2019.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey, arXiv preprini arXiv:2402. 13446, 2024.
Jean-Francois Ton, Muhammad Faaiz Taufiq, and Yang Liu. Understanding chain-of-thought in Ilms through information theory. arXiv preprint arXiv:2411.11988 2024.
David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. In European Conference on Computer Vision, pp. 198-215. Springer, 2024.
Shijie Wang, Dahun Kim, Ali Taalimi, Chen Sun, and Weicheng Kuo. Learning visual grounding from generative vision and language model. 2025 IEEF/CV Winter Conference on Applica- tions of Computer Vision ( (WACV), pp. 8057-8067. IEEE, 2025a.
Yudong Wang, Zixuan Fu, Jie Cai. Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, et al. Ultra-fineweb: Efficient data filtering and verification for high-quality Ilm training data. arXiv preprint arXiv:2505.088 005427 2025b.
Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. Towards visual grounding: A survey, arXiv preprint arXiv:2412.20206. 2024.
Teng Xiao, Zhen Ge. Sujay Sanghavi, Tian Wang, Julian Katz-Samuels, Mare Versage, Qingjun Cui, and Trishul Chilimbi. InfoPO: On mutual information maximization for large language model alignment. In Luis Chiruzzo. Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis- fies: Human Language Technologies (Volume 1: Long Papers), pp. 11699-1 1 1711, Albuquerque,
17

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189 6. doi: 10.18653/w1/2025 http://ww1122mmmmoooooo....5. maacl-long..85. URL naacl-long 585/.
Chi Xie. Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. Described object detection: Liberating object detection with flexible expressions. Advances in Neural Information Processing Systems, 36:79095--91077 2023.
Huihui Xu, Jiashi Lin, Haoyu Chen, Junjun He, and Lei Zhu. Eventrr: Event referential reasoning for referring video object segmentation. arXiv preprint arXiv:2508.07177, 2025.
Yi Yang, Hanyu Duan, Ahmed Abbasi, John P. Lalor, and Kar Yan Tam. Bias a-head? analyzing language model attention heads. In Trista Cao, Anubrata Das, Tharindu
bias in transformer- Kumarage, Yixin Wan, Satyapriya Krishna, Ninareh Mchrabi, Jwala Dhamala, Anil Ramakrishna, Aram Galystan, Anoop Kumar, Rahul Gupta, and Kai-Wei Chang (eds.), Proceedings of the Sth Workshop on Trustworthy NLP (TrustNLP 2025), pp. 276-290, Albuquerque, New Mexico, May 2025. Association for Computational Linguistics. ISBN 979-8-89176-2334 doi: 18.15vv// 2025. trrp..rruunlpmmai... URL https:////// / ac www.llannhoooyyoooooooooooo http://222....tssn.1ee trustnlp-main. 18/.
Yu Yang, Eric Gan, Gintare Karolina Dziugaite, and Baharan Mirzasoleiman. Identifying spurious biases early in training through the lens of simplicity bias. In International confenctionaacctffeenc on artificial intelligence and statistics, pp. 2953-2961. PMLR, 2024.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dietionary-enriched visual-concept paralleled pre-training for open- world detection. Advances in Neural Information Processing Systems, 35:9125-9138, 2022.
Yiyang Yao, Peng Liu, Tiancheng Zhao, Qianqian Zhang, Jiajia Liao, Chunxin Fang. Kyusong Lee, and Qing Wang. How to evaluate the generalization of detection? a benchmark for comprehensive open-vocabular detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6630-6638, 2024.
Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1593-1603, 2024.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14393-14402, 2021.
Hanzhi Zhang, Heng Fan, Kewei Sha, Yan Huang, and Yunhe Feng. Dam: Dynamic attention mask for long-context large language model inference acceleration. arXiv preprint arXiv:2506. 11104, 2025.
Shiyu Zhao, Long Zhao, Yumin Suh, Dimitris N Metaxas, Manmohan Chandraker, Samuel Schulter, et al. Generating enhanced negatives for training language-basee object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13592-13602, 2024a.
Tiancheng Zhao, Peng Liu, and Kyusong Lee. Omdet: Large-scale vision-language multi-dataset pre- pre-training with multimodal detection network. IET Computer Vision, (8(5):626-639) 2024b.
Zheng Zhao, Yeskendir Koishekenov, Xianjun Yang, Naila Murray, and Nicola Cancedda, Verifying chain-of thothut reasoning via its computational graph. arXiv preprint arXiv:2510.09312, 2025.
Guangtao Zheng, Wenqian Ye, and Aidong Zhang. Shortcutprobe: Probing prediction shortcuts for learning robust models. arXiv preprint arXiv:2505.1 13910, 2025.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 10.44:2010..41159 2020.
Wei Li Zuwei Long. Open grounding dino:the third party implementation of the paper grounding dino. https:://ithhtt wttpo/...oooo///gg/o////a 997 http////ppeeenooonnnnnnniiiiiii 2023.
18

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
SUPPLEMENTARY MATERIALS
We provide supplementary materials the following order:
Section A: COVAND Details describing our negation-focused dataset construction pipeline, including the three-step Chain-of-Thought prompt design, VQA-based caption alignment, negation cue distribution, and a human-in-tie -inaa-in-thh-loop data error analysis.
Section B: Implementation Details of all backbone architectures and our LoRA placement strategy, together with additional attention visualizations.
Section C: Extended Related Work covering CoT-based dataset construction, visual grounding and region-level alignment, parameter-efficient fine-tuning for VLMs, composi- tional reasoning. and bias mitigation.
Section D: Additional Ablations: Negation- and Noun-Only Boosting, which compare
simple token-level boosting and attention-bias variants against our NEGTOME.
Section E: Comparison with Post-hoc VQA Methods analyzing two-stage detector+VQA pipelines and their iccuracy-latents trade-offs.
Section F: Evaluation on Full OVDEval Subsets reporting results on all OVDEval subset.
Section G: Analysis on RPN-based Detectors contrasting RPN-based and DtTR-style
architectures under negation.
Section H: Zero-shot Downstream NegBench MCQ presenting a detailed breakdown of the multiple-choice subsets and characteristic error patterns.
Section I: Zero-shot Generalization on the Biomedical Domain evaluating our method on the FG-CXR chest X-ray dataset and analyzing cross-domain.
Section J: Qualitative Results displaying additional examples OVDEval and D as well as representative failure cases on complex negation and event-level reasoning.
Section K: Declarations summarizing LLM usage, ethics, and reproducibility.
A DETAILS ON COVAND
A.1 PROMPT FOR THREE-STEP CoT CAPTION GENERATION
We employ a systematic three-step CoT reasoning approach using GPT-4o (Hurst et al., 2024) to generate high-quality rggation-ocussed captions. As shown in Figure S11, the prompt structure is carefully designed to elicit temporally coherent reasoning that produces semantically valid negation captions grounded in the visual content.
Our prompt begins by informing the model that it will be provided with an image containing highlighted bounding box, along with target phrase describing the main subject the region. The model is then guided through three distinct reasoning steps:
A.1.1 STEP 1: ATTRIBUTE EXTRACTION
The model first generates two comprehensive lists of attributes:
Present Attribute (Apres): At least three attributes or keyword items clearly visible within the bounded region.
Absent Attribute (Aabs): At least three attributes keyword items that are contextually relevant but clearly not present in the bounded region.
A.1.2 STEP 2: CAPTION GENERATION
Using the attributes from Step 1, the model produces two types of captions:
Negative Caption (Cneg): Creates factually incorrect statement by falsely claiming existing attribute is absent. This caption must contain a negation expression (c.g., "no", "not", "without") coupled with an attribute from the existing contents list.

---

What "Not" to Detect: Negation-Awarc VLMs via Structured Reasoning and Token Merging
Positive Caption (Cpos): Creates factually correct statement by accurately describing an absent attribute as absent. This caption pairs a negation expression with an attribute from the absent contents list.
This approach yields contrastive pairs where the negative caption contradicts the visual evidence while the positive caption aligns with it, creating training data that specifically targets negation understanding.
A.1.3 STEP 3: SEMANTIC VERIFICATION
For quality assurance, each generated caption undergoes verification:
Confirms the caption (1) contains a negation expression, (2) refer- ences existing attribute from Step 1, and (3) factually mismatches the actual content of the bounded region.
Negative Verification:
Positive Verification:
Confirms the caption (1) contains negation expression, (2) refer- ences an absent attribute from Step I, and (3) correctly describes the absence of the attribute in a way relevant to the context.
This verification step ensures semantic integrity and prevents generation artifacts by applying ex- plicit logical checks. either caption fails verification, the process iteratively regenerates captions until valid pairs produced or the retry limit is reached.
The prompt enforces concise, natural language expressions with a single-sentence structure. As examples in Figure S12 and Figure $13, it requires the model to focus exclusively on the bounded region, preventing semantic drift to other parts of the image. The entire process outputs a structured ISON format containing the attribute lists, caption pairs, and verification rationales, facilitating downstream dataset creation and quality control processes.
A.2 VQA-BASED CAPTION ALIGNMENT
To address a critical challenge in negation-ave aware detection, ensuring generated captions reference exclusively the intended bounding box rather than other visually similar regions, we implement structured verification pipeline with VQA alignment.
First, we apply alphabetical region labeling to all bounding boxes that share the target phrase type (e.g.. "person") by assigning distinct markers (A, B, C, ...) to each instance. The originally prompted region remains unlabeled to avoid biasing the verification process. As shown in Fig- ure S14, our visual prompting approach carefully considers label placement to maintain visual clar- ity. When labeling multiple instances of the same type (e.g., multiple "person" boxes), we position alphabetical markers outside the top-left corner of each bounding box to avoid occluding the object itself. This placement strategy preserves the visual integrity of the object while providing clear refer- ence points for the VQA model. In cases where objects appear near image boundaries, we adaptively place labels inside the top-left corner of the bounding box to ensure they remain visible within the frame. This adaptive positioning is crucial for maintaining consistent label visibility across diverse image compositions.
Then, for each caption pair (Cpos, Cogg), we query a multimodal VQA model with two precisely formulated questions in Figure $15. The VQA model analyzes the image and captions to produce structured JSON responses specifying matching box labels. A valid alignment requires that Cpos matches exactly the original unlabeled region, while Casg either matches no regions (" 'None'') or incorrectly matches another box. This process effectively eliminates label noises: false nega- tives, where Cney accidentally describes another instance, and ambiguous groundings, where cap- tions generically describe multiple regions.
Figure S16 showcases several successful examples from our complete caption generation pipeline. In these examples, we can observe how the three-step CoT process first generates attribute-based negative and positive captions for the target region, followed by the VQA aligmment step that verifies caption-region correspondence. Despite the effectiveness of our approach, we encountered certain limitations complex scenes, as illustrated in Figure S17. When multiple instances the same type are densely clustered, the visual prompting become ambiguous, making it difficult for the
2

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
VQA model to determine precise correspondences. To maintain dataset quality, implemented a filtering mechanism that excludes images containing more than five instances of the same type from the caption generation process. This threshold diversity of the dataset with the precision of the annotation, ensuring that our training data provides unambiguous supervision signals for understanding the meaning of negations.
empirically determined to balance the
A.3 DATASET DISTRIBUTION
Type of Negation Cues
Distributions of Flexible Negation
K366
8,000
16.59%
6,000
Count
4,000
83.41%
2,000
611
142
201
Regularized
Flexible
lack Ammuy
suffer
heve Amauly
deread
and
life
attrave
lack
dtom
ordbrr
4.5oc
alt
Figure S8: Distributions of Negation Type. Analyzing all 48,761 captions in CoVAND and identi- fied 57,874 negation instances. Following standard linguistic taxonomies, we categorized them into Regularized (explicit syntactic markers) and Flexible (exical/morphologica) cues. Regularized Cues means frehffryyyency surface markers, including not, no, without, never, and contractions like n't. Flexible Cues means diverse long-tail of expressions including lack, un- in-/im-/ir-, dis-, non-, and -less.
We separate surface regularized cues versus flexible cues as below:
Regularized cues are short, high-frequency surface markers: not, without, no, never, and clitic contractions n t.
Flexible cues cover lexical and morphological forms that naturally occur open text: lack- family (lack, lacks, lacking, lack of), devoid of, absence of/absent, coordinations (neither/nor, but not, rather than, instead of), and pro- ductive morphology such negative prefixes/suffixs (in-/im-/il non-), and -less, well as X-free/ ree of.
un-, dis-,
We analyze all 48,761 captions (24,381 positive 24,380 negative). Across all captions, detect 57,874 negation cues in total: 48,275 regularized (83.41%) and 9,599 flexible (16.59%).
Prior analyses negation in natural language understanding corpora show that explicit markers such as not, no, and n' t account for the large majority of negation instances. Hossain et al. (2022) found that syntactic negation (regularized cues) constitutes 88.6% in CommonsenseeAA (Talmor et al., 2019) and 71.9% in SST-2 (Socher et al., 2013), compared to morphological negation (1 111.4% and 28.1%, respectively) as in (Hossain et al.. 2022). While these figures come from specific NLU datasets rather than unrestricted natural language, they suggest that regularized forms are prevalent realistic language tasks. Thus, our dataset's 83.41 % regularized distribution not solely an artifact of GPT-4o's generation bias, but rather aligns with patterns observed in cxisting negation- annotated corpora for downstream applications.
Flexible forms provide meaningful diversity. While regularized cues dominate, the presence of 9,599 flexible cues (16.59%) ensures the dataset includes a non-trivial variety of negation expres- sions. This diversity is essential for evaluating whether models generalize beyond sigh-frequnncy patterns. Flexible negations, though less common, are critical in compositional reasoning tasks such DOD, where attribute-level and relational negations often require nuanced understanding. By

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
including both regularized and flexible forms, CoVAND provides more comprehensive training signal than datasets relying solely on template-bast based augmentation.
Future work: Mitigating prompt bias for richer flexibility. We acknowledge that the current distribution may still reflect prompt-design bias inherent to GPT-4o's 's training data. To further en- hance the diversity of flexible negation forms, future iterations of CoVAND could employ targeted prompt engineering strategies- such as explicitly requesting diverse negation structures (e.g., "de- scribe the absence using lexical negation such lack or devoid of") post-hoc augmentation techniques to rewrite regularized negations into their flexible counterparts. Such refinements could yield more balanced distribution while preserving the semantic integrity established by our CoT and VQA pipeline.
A.4 DATA ERROR CASE ANALYSIS
To quantify residual annotation emrors in CoVAND, perform two-stage human-in- -n-the-loop audit combining an independent multimodal language model with manual inspection.
Stage Automated cross-model audit. We first randomly sample 1,000 image-caption pairs from the training split of CoVAND. Each sample consists of an image, target bounding box, and a pair of captions describing the same region: a negative caption (hard negative, c.g., "a boy without helmet") and a positive caption (true description, e.g., "a boy without backpack"), together with the key attribute mentioned in the caption ("helmet", "backpack", etc.).
For each sample, we generate visual prompt by overlaying a red rectangle on the target bounding box and feed the resulting image, the caption, and the attribute to an off-the-shelf multimodal LLM, Gemini-2.5 (Comanici et al., 2025), that is architecturally and training- model used in our data generation pipeline. The model is instructed to act as an objective judge and to return a structured JSON answer indicating whether the attribute visually present in the red box:
independent from the
"is dis_attribute__e. ptw.sbbuttepppeettt:..e boolean, "reasoning": "short explanation"
We then apply a deterministic decision rule to compare the dataset label with the model's prediction:
For a negative caption (intended hard negative of the form "X without Y"), the example is considered valid if the attribute Y is in fact present in the region true).
For positive caption (intended true caption of the form "X without Y"), the example is considered valid if the attribute indeed absent (is.attribute.press false).
For each caption, we log the full record (image path, bounding box, caption type, attribute, model verdict, and free-form reasoning) in a JSON file for subsequent human analysis.
Over 1,000 sampled pairs, the independent model disagrees with the CoVAND label in 78 cases (7.8%). These disagreements define the pool of potentially erroneous annotations.
Stage 2: Manual verification of disagreements. In the second stage, we load the logged results and focus on the disagreement set. A lightweight visualization tool displays. for each case, the original image with the red bounding box plus textual metadata (caption, attribute, model verdict, and reasoning). Annotators then categorize each disagreement either:
Dataset error: the CoVAND label is incorrect and the independent model's judgment is correct, or
Model error: the CoVAND label is correct and the independent model misinterprets the visual evidence.
4

---

What "Not" to Detect: Negation-Awann VLMs via Structured Reasoning and Token Merging
Gueg "The caricature of young women ES not hoiding drink."
Cugyi The jacket is not blue.
Gomtns, does not Abee Sholding drink" The noman in the rnd bounding bos is not holeiing drank
Gended does not san talue jackets. Person in the hourding has in unartng yellow 1ife over grey or Light-eatared farkert.
table reent ber, but
Figure S9: Representative dataset errors for negative captions. Each panel shows an image with the target region highlighted and the corresponding hard-negative caption ("X without ""). Many mislabels arise in visually subtle cases (e.g. barely visible or skin-colored attributes) where "absent" attribute Y is in fact present but difficult to perceive even for humans.
Cuee' "The heimet does have a comouf lage pottern."
Gomind sam Aue #canouFlage pattere". bounding
displays #flitary- style canouft age
'Al male surgeon perf surgery without gloves his hunds."
aplotches
surgnen Aus "glaves". bata his
wearing
Figure S10: Representative dataset errors for positive captions. Examples where the caption intends to describe true absence of an attribute, but small objects, occlusions, cluttered scenes make the decision borderline. These cases illustrate that the residual annotation noise COVAND is dominated by inherently ambiguous instances.
Among the 78 disagreements, 23 are judged as true dataset errors and 55 as model errors. This yields an estimated annotation error rate of 2.3% on the audited sample, corresponding to 97.7% factual accuracy for CoVAND. Most errors occur in visually ambiguous situations, such as fine-grained appearance attributes or partially occluded objects, rather than systematic failures of the generation pipeline.
Qualitative patterns. Fig. S9 and Fig. S10 illustrate typical error cases discovered by this audit. For negative captions, errors often arise when the "absent" attribute is present only in subtle prototypical form (e.g.. skin-toned medical gloves that are hard to distinguish from bare hands), or when the bounding box tightly crops context that would disambiguate the attribute. For positive captions, errors typically involve borderline cases where small accessories, distant objects, or strong reflections make difficult determine whether the attribute truly absent in the marked region.
Overall, this analysis indicates that the remaining noise in CoVAND is both quantitatively small and qualitatively concentrated in genuinely ambiguous instances, rather than reflecting systematic coff-snsiitency bias of the underlying generation procedure.
5

---

What "Not" to Detect: Negation-Awar
VLMs via Structured Reasoning and Token Merging
provided with bounding box. are viston-language model advanced chatn-of- thought reasoning. negotive and positive captions referencing the sate main subject, "<TARGET "TARGET_PHIRAS>"" PHIASSE>..
which the object "cTARGET PHRASE>" is highlighted red contoured produce
Step 1) Sumnarize the highlighted bbox existing/missiig contents (color, action, location, relationship, shape, texture, etc.):
[Existing Contents] Provide least short attribute or keyword itens that describe SHOW within the red bounding box.
All contents should be ELEARLY CHECKED image.
Example: the region corresponds to "woman", , you could include Itens like ('running at left lane", 'brown hair', 'blue shirt', 'juuping', 'holding bat'].
[Absent Contents] Provide least shart attribute sonewhat relvant the situation.
the red bounding Example: If the region corresponds to 'A wonan in blue shirt rides bicycle', you could include itens like ["helnet', "glasses', 'red hoodie")
keyward iters that describe
All contents should CLEARLY MISSING image,
all itens
in the inage.
Step 2) For selected content iters from step 1, produce exactly ONE negalive caption and ONE positive caption with (e.g. 'no', 'not', "never', 'without' "un-', ...). Each caption should about the bounding subject ("CTARGET_ PHRASEy"
bbox)
combining negation expression existing Facus.
[Negative caption]: Caption that misnatched with the
content
(1) Must contain negation expression with Existing Contents. santence
dascriptive target region. (3) Exarple: existing contents are ['nan", "blue shirt', "hat'J select "hat"
it
=> "A nan Mithout hat on his head. ('hat' with 'vithout')
existing
'black', 'near the woran'] select "near the wonan" es black plate is not located near the wosan."
('plate',
the
[Positive caption]: Caption natch with
region Absent Contents.
absent
with negation expressions,
(1) contain negation expression
(2) Keep it single sentence or phrase, which is actually present or relevant.
(3) Example: If contents ('helnet', 'glasses', 'red hoadie'] select 'red hoodie', you could say wonan
red
bicycle.'
Provide verification for each caption:
After each negative or positive caption, include a short "verification" string that clarifies why It is truly negative positive, focusing the of the negation.
Negative check: (1) Does contain negation expression? (2)
the existing from Step (3)
boes it et statch with the hounding hox contents?
Fositive check: (1) Does it contain negation expression? (2) Does it contain the absent iten from Step 1? (3)
negation absent
bounding box, but theratically relevant?
Keep each caption to one sentence. Natural, fluent English with bit of descriptive detail is encouraged.
bbox contents and subsequent captions should provide unique distinguishing details specifically about the objects or attributes that He
objact target region, ensuring that they not unintentionallyy
outside of this indicated region.
Return your final
in structure with the following schena:
"steps": [ "explanation": "...", "output": "..." "absent": [ ]
contents": "pairs": [
"existing":
content iten": {
"existing": "cone existing Stem>",
"absent":
*cone absent item>"
}.
'negative caption":
"negative_ pengative_verficcee
"...",
"positive caption":
positive veriiicttionn "..."
You reveal. your chain-of thought in st,)s keep concise nention about visual pronpt in the final output sentences. Please identify at least existing/wissing itens (other than the nain subject) in that repion, then select for generating regative/positivee caption pairs with verification. Use the 350N schena above.
Figure S11: Prompt for Three-step CoT Negation Caption Generation. Our prompt guides the model to systematically (1) extract present and absent attributes from visually highlighted regions, (2) generate complementary negative and positive captions with explicit negation markers, and (3) verify semantic alignment through logical validation.

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
STEP
hightigghed selstinggntising
Input Image:
Apres ("darit silbouette,, "still water reflection', "sultiple anternas")
Aate (briaht decerstionn", "passengar activity", "painted detalls")
Target phrase: "The boat"
STEP2
selected content regative posstive
stop 1,
Cap!
aspuar silhoustte against sunrise." any bright aling
STEP3
verification cach caption
Vay caption
"dark silhouetie, nisrepresenting
boot's 's visuel attribute." Ypu "This caption 'without" with "bright decorations, confireing its assence in the bounding box."
Summarize the häghlighted tbox exiating/nissing contents
Input Image:
S
Aprac? ('hiitt beord", "blae cag", "stonding")
Targel phrase:
USED TIRES
Ause ("gloves", "Kissses", "apron"]
"A nan"
STEP produce selected
posttive captiion oegatiion expressions
Phone
Guy! without blue on his head," " starding doorway."
Spp
STEP3
Provide verification for each caption
Var "The cortxins negatiin ("without") and the rcisting stes ('blur rap'), misratches
Ya- "It contains a negaciion ('kithout') 1 and the absent itee (' ('plasses"),
which from Lmage relevant to the setting."
STEP1 Sumeriice the highlighhed beue esisting/nissing contents
Input Image:
Asrest ("round", "black and whiite", "Laze")
Aate ["ret",, "grass", "Fact touching"))
STEP2
For selected contant isems fres stap 1, produce regativve and posstive caption with negation expressions
Gug! "The ball has no visible logo on it." Gam 2 "The boll shows no foot touchiig by any player on the field."
Target phrase "The ball"
STEP3 3
Provide verification for each caption
it contatos negation. Yes, contains 'Jogo', ' (B) Yes, st alssatches becouse the boll does have a logo." * Vou "Posteive Vertiication: (1) Yes, contains negatioon. (2) Yes, contains "foot touching'. (3) Wes, it is absent but relevant to ball."
Mag F "Negative
STEP
Sumnorize the highlighoedd bbox extsting/atssing contents
Input Image:
Apros
("betge hat", "pattern", helar]]
Target phanisos "a hat"
Aate ("lctters"; "passongen sctivity", "pointed details"]
STEP2
For selected content Ioews from stop 1, produce negative and positive caption with negation expressions
Gay "The wrean is not weaning beige stop her head," Gam "The hat on her head is without a lettors."
STEP 3
Provide verificaaton for each caption
Very z "(3) Contains 'not' for negationn (2) Refers to 'belge hat", (3) Contradicts visual
negotive."
Fyur "(1) Contains 'without' for regationn (2) Refers to 'letters' 35
absent. (3) absenre lagn true and contestually relevest."
Figure S12: Examples of COVAND with 3-step CoT Caption Generation (1). Example im- ages and corresponding captions. Text with blue is present attribute(A and pink is absent attribute( In detail, <negation + can generate negative caption(Cneg) and <negation word>+<Ambe> can generate positive caption(Cnns).
7

---

What "Not" to Detect: Negation-Awer Aware VLMs via Structured Reasoning and Token Merging
STEP1
highti selsting/nissing
Input Image:
Apro ["wet fart, "veariing col collant, "Lockiin upward") MMlathing
STEP2
negative
expressions
upward,"
Cpee dog
standing
STEP3
Provide verification for eath caption
Target phrasse "A clog"
expression existing attrihute, stswstchingg the abserved ontentation the dep." "This caption positive
"This caption
negation exprission for absent
attribute where no leash visthle, matching
an thee."
Input Image:
STEP Suanorize the highlighoed existing/nissig contents
Agres? ("blond heirf, "slue clog sundall", "orange trufficc conee]
Aate ("hat", "glasses", "red hoodie"!
phrase: "Infant Boy"
1, STEP2 produce negative positivee caption nogation expressiion
selected
boy
traffic
Veay which is present, making this factually incorrect." negecion indoed absent in the imago."
'sat'
"hat',
Figure S13: Examples of COVAND with 3-step CoT Caption Generation (2). Example im- ages and corresponding captions. Text with blue is present attribute(Aprrat and pink is absent attribute(Aatt)) In detail, <negation word> + <Apres>> > can generate negative caption(Cneg) and <negation pord>+<Aal can generate positive caption(Cpod
Sharp
Figure S14: Examples of Visual Prompt VQA Alignments. We apply alphabetical region labeling to all bounding boxes that share the target phrase type by assigning distinct markers (A, B, C, ...) to each instance with red bounding boxes.
8

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
You provided with inage where each bounding box labeled with letter, ['A', ',, -].
('A': "<PHRASE>", "B': <<HRRAS>>>
Additionally, the fol lowing captions are given: Caption <POSITIVE CAPTION> Caption 2: <NEGATIVE CAPTION
Your task is determine which bounding box aligns with Caption and which one aligns with Caption 2, based the context of the image. For each caption, please provide the label(s) of the bounding box boxes that natch its description. caption does not align with any bounding box, respond with 'None'.
Example:
labeled B, Let bbox 'A" and 'C" show black dog with red collar, 'g' shows 'small
Suppose
bounding
dog' NITH collar,and bbox shows 'cat' without collor. Caption I: 'A black dog wearing red collar'
Caption 2: "A small white dog without a collar'
Captian 1 .> semantically align with bbox "A' and 'c.. Caption is not a dog.
bboxes perfectly aligned since snall white NEARING red collor and CAT
Hence, the final answer would look like: 'captionl's ['A", 'c'), 'caption2": ('None'])
Now, please return your final answer in 350N structure with the following format: 'caption1'; [...],
..., "None"
"caption2': [...], A, B, ..., 'None')
Figure S15: Prompt for VQA Alignment. Our alignment process with (1) labeling all candidate bounding boxes with alphabetical markers, and (2) querying the VQA model to determine precise correspondences between generated captions and visually annotated regions.

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Target phrase:
3-step CoT Negation Caption Generation
"Fornal
Frog
without
VQA-based Caption Allgnment
Witch hñox al cantloon??
Guzy!
Gper
3-step CoT Negation Caption Generation
brown
ind collan
1ight brown
without
VQA-bmsdd Caption Alignment
vithh
Target plrose: "light brown dog"
Gper
3-step CoT Negation Caption Generation
Target phmase: "dog"
Guzy! dog Jurping pose." dog playing without
Cyu
VQA-bued Caption Alignment
Mhtch with
3-step CoT Negation Caption Generation
bp
sidewalk."
ta kts without jarket the sidewcik."
VQA-bised Caption Alignment
with
Target phrase: "kids"
Gye
Target plaasse Pyyuung boys"
3-step CoT Negation Caption Generation
"The quitaan."
"The perforn without visible music sheets,
VQA-bused Caption Alignment
Mhich with
Gyu
Figure S16: Examples of COVAND. Example images for the 3-step CoT Negation Caption Gener-
ation and the VQA aligmment are needed. The VQA alignment step is only executed when there multiple instances with the same phrase type.
10

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Target plinse:
3-step CoT Negation Caption Generation
groug-ott
Guzy "A group of childran not engaging with an woult or the beach,"
Gyu "A group of children vithout beach bells on the sard."
VQA-based Caption Alignment
aligns given captions?
Coyg Gper
3-step CoT Negution Caption Generation
Gury "The entertaincrs are not wearing Roman style outfits." Gyu "The ontertairers are without soders attire, instead shoveasing Raman- style outfits."
YQA-baaed Caption Alignment
Q: Which bbox aligns with givenn coptions?
Ges
Figure S17: Error on COVAND. VQA alignment occasionally fails when instances are densely clustered, making it difficult to determine which instance each visual prompt references.

---

What "Not" to Detect: Negation-Awer Aware VLMs via Structured Reasoning and Token Merging
B IMPLEMENTATION DETAILS
B.1 GROUNDING DINO MODEL
Our implementation is built upon the Grounding DINO architecture (Liu al., 2024b; Zuwei Long, 2023), which employs a duall-encoeer-siglle-eooooo design for vision-language understanding, For efficient fine-tuning towards negation understanding, we apply LoRA (Hu al., 2022) specific layers of the cross pross-modality decoder. The Grounding DINO consists of several key components:
An image backbone (Swin Transformer (Liu al., 2021)) for visual feature extraction
A text backbone (BERT (Devlin al., 2019)) for textual feature encoding
A feature enhancer with self-attention and cross-attention mechanisms
A language-guin guggged query selection module that initializes query embeddings
A cross-modalit decoder that refines object detection based on both visual and text
We implement parameter-efficien fine-tuning by applying LoRA to deep layers (the final three cross-attention layers in the cross-modalit, decoder). This strategic placement allows to modify how the model integrates negation cues from text with visual features while preserving pre- pre-trained knowledge in earlier layers. Specifically, projections of the text cross-attention; the image deformable cross-attention and the self-attention blocks remain unchanged. The addition of ReLU activation between the cown-projection and up- projection matrices, similar to (Chen et al., 2022), enhances the model's 's ability to capture non-linear relationships between negation cues and visual features. Grounding DINO's cross-attentis the interactions operate as follows:
insert LoRA only into the query (Q) and value (V)
Image Cross-Attention:
Query (Q): the updated cross-modality query from the preceding self-attention layer Key (K) and Value (V): the image features processed through the feature enhancer
Text
-Query (Q): the output from the image cross-attention layer
- Keyy (K) and Value (V): text features encoding language information
Figure S18 reveals critical insights into the optimal placement of LoRA modules (Boenisch et al., 2025) for negation understanding. The bascline model (Figure S18a) shows strong bias toward Special tokens across all decoder blocks, with negation cues receiving minimal attention. When apply LoRA shall low blocks (Figure $18h), negation tokens initially receive higher atten- tion weights in blocks 0-2, but this effect rapidly diminishes in the later blocks where attention to negation drops.
In contrast, when we apply LoRA to deep blocks (Figure S18c), the model maintains consistent attention negation tokens through blocks. This pattern persists through the final detection heads, explaining the superior negation-awan aware detection performance. Some works (Gao et al., 2025; 2024; Seputis et al., 2024) further validate our approach by demonstrating that allocation of adaptation capacity to mid-to-late transformer layers yields optimal results for complex semantic tasks.
With the addition of NEGTOMe (Figure S18d), attention to negation tokens increases consistently across all blocks, with particular amplification in the final blocks where detection decisions are made. This confirms that our token merging strategy effectively preserves negation signals through- out the entire network, even in early blocks that did not receive LoRA adaptation. The combined effect creates a consistent processing path for negation cues from text encoding through to final detection, explaining the significant performance improvements observed in the OVDEval and D benchmarks.
Together, these adaptations enable our model effectively capture the semantics of negation by enhancing the cross- trosl integration of negation cues with their corresponding visual attributes, resulting in more accurate detection under negation scenarios.
Compared with the tiny model of Grounding DINO baseline, we need merely 0.005% trainable parameters capture negation cues effectively, in Table S7. To keep the tiny model within the
12

---

What "Not" to Detect: Negation-Awer Aware VLMs via Structured Reasoning and Token Merging
Mick
Bick
Blokk2
Block
block
Block
(a)
Baseline Model
Meek
Blck
Blok
Block
Blok'
(b) Baseline Model CoVAND Fine-tuning (LoRA shal shallow layers)
Bioc
Blocct
Block
(c) Baseline Model + CoVAND Fine-tuning (LoRA on deep layers)
(d) Baseline Model CoVAND Fine-tuning (LoRA on deep layers) + NEGTOME
Figure S18: Average Attention Weights by Decoder Blocks. We only update the LoRA modules while freezing other layers for fine-tuning. Placement LoRA, shallow means LoRA located on early decoder blocks (0-2) and deep means LoRA located on latter decoder blocks (3-5).
Table S6: OVDEyal-Negation Evaluation. Performance on Grounding DINO tiny model.
AP
NMS-AP (Yao et al., 2024)
FPR
G-DINO-T (Liu al, 2024b)
48.5
22.8 23.3
$4.0
Ours
51.1
42.5
(2.66)
(10.5)
11.5)
same 0.005% budget, we attach LoRA adapters to the and 5 text cross-attention blocks of the decoder. Our lightweight adaptation yields a consistent performance gain: +2.6 AP and +0.5 NMS- AP, while slashing the Positivve Rate (FPR) by 11.5% as in Table S6. Although AP and NMS- AP improvements are moderate, they are achieved without sacrificing any metric; in fact, every reported score on par with, or better than, the baseline, indicating that our negation-centric tuning does not degrade the detector's 's general ability.
13

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Table S7: Trainable Parameter Ratio. The table compares the total model size with the number of LoRA-tuned parameters for each detector and backbone pair. During fine-tuning CoVAND, only the LoRA layers are trainable, with all other layers kept frozen with their pretrained weights.
Image Backhone
Total Param.
LoRA Param.
Ratio
G-DINO-T (I al,, 2024hb
Swin-T (28.8M)
173M
8.2k
0.005
G-DINO-B B (Liu et al., 2024b)
Swin-B B (88M)
233M
12.3k
0.005
APE-Ti (Shen al., 2024)
ViT-Ti (5.8M)
771M
129k
0.017
APE-L (Shen et al,, 2024)
VIT-L (307M)
IB
129k
0.012
Owen-22.5VLL-31 (Bai et al,, 2025)
VIT-H (632M)
3.8B
77k
0.002
B.2 APE MODEL
Our implementation builds upon the APE framework (Shen et al., 2024), a universal visual per- ception model that unifies detection, segmentation, and grounding through instance-level region- sentence alignment. The architecture features several key innovations:
A vision backbone (ViT-L (Dosovitskiy et al., 2020)) pretrained with EVA-CLIP (Sun et al.,
2023) for visual feature extraction
A text encoder (EVA02-CLI) (Fang al., 2024)) processing both categorical vocabularies and free-form descriptions
gated cross-modality interaction module that fuses visual and text features
A transformer decoder with deformable attention (Zhu al., 2020) for joint reasoning
APE introduces novel gated fusion mechanism that efficiently handles thousands of prompts per forward pass. Unlike previous approaches that directly fuse all text features (Li et al., 2022), APE
implements conditional interaction paths:
V + (V, Pooc) (V, Psen)
for vocabulary prompts for sentence descriptions
where V denotes visual features and P represents text embeddings. This gating strategy reduces FLOPs compared to GLIP-style fusion (Li et al., 2022). The model processes inputs at 1,024 pixel resolution using AdamW optimization (Loshchilov & Hutter, 2017) with learning rate 0.0005 and weight decay 0.05. We employ large-scale jittering augmentation (Ghiasi et al., 2021) with random scales from 0.1 to 2.0. We train APE-Ti Ti models with four A6000 GPUs with batch size of 4.
We apply LoRA exclusively to the encoder's cross-attention layers where visual and text features interact. This targeted adaptation modifies only 0.017% APE's parameters as Table S7. Despite APE-L's strong theoretical performance. its IB parameters exceed the 48GB memory capacity of NVIDIA A6000 GPUs during training. We therefore focus on APE-Ti, which achieves 32.5 AP on D while maintaining practical deployability.
B.3 QWEN-2.5- LL MODEL
In addition to dedicated detectors, we test our method's generalizability on a powerful Multimodal Large Language Model (MLLM), Q.en-2.5-VL (Bai al., 2025). Unlike dual-encoder architec- tures, Owen-22.5-LL is an end-to-end model that directly processes interleaved image and text data. As detailed in its technical report, the architecture consists of three main components:
A Vision Transformer (ViT-H) that is redesigned and trained from scratch handle na- tive resolution inputs. For efficiency, it incorporates windowed attention in most layers, with full self-attention only in specific blocks. The ViT architecture is also updated with RMSNorm and SwiGLU activations to align with modern LLM design principles.
An MLP-based Vision- LanguuLe Merger that compresses spatially adjacent patch features before feeding them into the language model, enhancing computational efficiency.
14

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
A Large Language Model decoder based the Qwen2.5 architecture, which performs unified reasoning over the combined multimodal input and generates responses, including object coordinates for detection tasks.
For parameter-efficient fine-tuning, we again employ LoRA, strategically targeting the deep layers of the LLM decoder to enhance its negation reasoning without disturbing its foundational knowl- experimental setup, LoRA adapters are specifically injected into the query (q-proj) and value (v-proj) projections of the self-attention modules within layers (15, 24, 30). This targeted placement is designed to modulate how the model integrates visual information with textual negation cues in its higher-level semantic reasoning stages. We configure the LoRA adapters with a rank of 4 and a dropout probability of 0.05.
edge. Based
Following the execution script, the Qwen-2.5-VI-33B model is fine-tuned for 1 epoch on our CoVAND dataset on H200 GPU. We use learning rate of 5e 5 with the AdamW opti- mizer (Loshchilov & Hutter, 2017) and a per-device batch size of 32, with 2 gradient accumulation steps, totaling an effective batch size of 64. The model is trained using b mixed-precisiog the ViT, MLP merger, and LLM back-
During this process, all original model parameters-includin bone are kept frozen; only the injected LoRA adapter weights are updated.
C EXTENDED RELATED WORK
C.I DATASET CONSTRUCTION WITH CHAIN-OF-THOUGHT REASONING
Chain-of-Thoure (CoT) based data generation has emerged critical methodology for construct- ing high-quality datasets, particularly for tasks requiring multi-step reasoning. Early work in this domain relied heavily on manual annotation or rule-based generation, but recent advances have au- tomated and scaled CoT-based dataset construction through Large Language Models (LLMs).
Systematic CoT Pipeline Design The construction of CoT datasets requires careful consideration of quality and scale trade-offs. VideoEspresso (Han et al., 2025) demonstrates a systematic approach to automatic CoT dataset generation, employing a stree-stage pipeline: (1) semantic-aware key information extraction using frame-level captioning (Kazakos et al., 2025), (2) multi-frame question- answer pair construction guided by carefully designed prompts, and (3) multimodal CoT annotation
with spatial and temporal grounding (Kim al., 2025).
This pipeline bridges the gap between manual annotation and fully automated generation by leverag- ing LLMs to produce reasoning chains (Lee et al., 2024; Tan al., 2024) while maintaining consis- tency and factual accuracy through iterative quality filtering with an auxiliary LLM validator (Wang al., 2025b; Chen al., 2025c). The approach implements a redundancy removal mechanism using semantic similarity (Abbas et al., 2023) to filter out low-quality data, achieving 203,546 high-quality QA pairs with fine-gran -raiind CoT annotations. Furthermore, external validation tools (Findeis al., 2025) are employed ground generated annotations in visual evidence, ensuring both consistency across temporal boundaries and factual accuracy of intermediate reasoning steps.
Framework for CoT Evaluation Understanding the quality of interme- diate reasoning steps in CoT chains is essential for constructing reliable datasets. Recent works (Ton et al., 2024) propose an information-theorett framework that formalizes CoT reasoning through the lens of information theory (Xiao et al., 2025), enabling the identification of failure modes without an- notated CoT data. This theoretical foundation is grounded in process supervision paradigms (Light- man et al., 2023; Jia et al., 2025).
Their key contribution the concept of information-gain at each reasoning step: a correct step should provide meaningful information toward predicting the final answer. By training supervisor model to estimate conditional mutual information between intermediate steps and the final answer, they can quantify the contribution of each step without expensive human annotation. This approach outperforms outcome- based reward models (Lightman et al., 2023) and Math- aath-Shepherd on detecting erroneous reasoning steps (Chen et al., 2025b).
The framework extends toward mechanistic interpretability through sparse autoencoders and activa- tion patching (Chen al., 2025c). These techniques reveal that CoT induces interpretable compu-
15

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
tational structures in larger models, with domain- -ppecffc causal patterns (Zhao et al., 2025). This mechanistic analysis validates that intermediate steps reflect genuine internal computation (FU et al,, 2025).
The framework extends beyond simple accuracy metrics to provide step-wise interpretability (Hanna et al., 2025) and identifies unidentifiable sub-tasks- those which the model has not learned from training data (Muhamed et al., 2025). By combining irfommaiion-teereet analysis with causal structure verification, the framework creates a comprehensive evaluation pipeline grounded in both statistical and mechanistic principles.
Synthesis for COVAND Dataset Construction Drawing from these insights, our COVAND dataset construction pipeline incorporates: (1) systematic 3-step CoT prompts guided by information-theoretie principles to ensure each step contributes meaningful information, (2) multi- level quality verification using both LLM-based filtering and human review to ensure faithfulness, (3) validation through external VQA models to confirm region-level alignment accuracy, and (4) careful consideration of model scale effects by selecting sufficiently large models for CoT genera- tion. This comprehensive approach ensures that CoVAND provides high-quality, instance- groundddd negation data where the reasoning process genuinely reflects the model's internal understanding.
C.2 VISUAL GROUNDING AND REGION- ALIGNMENT IN VQA
Visual grounding, which is the task of localizing image regions corresponding to textual descrip- tions, plays a crucial role in connecting language-based reasoning to visual content. This capability is particularly important for our NEGTOME module, which grounds text tokens to image regions to implement negation-awer aaarre token merging.
Visual Grounding Methods and Evaluation Visual grounding has evolved from CNN-based two-stage pipelines to unified transformer-based end-to-end frameworks. TransVG (Deng et al., 2021) pioneered transformmer-based visual grounding by performing intra- and inter-modality rela- tion reasoming homogeneously. Recent advances further improved this by moving toward multi- task grounding architectures that jointly perform localization and segmentation (Dai al.. 2025) and leveraging coarse-to-fing fine consistency constraints. A comprehensive survey on visual ground- ing (Xiao et al., 2024) systematizes this evolution and identifies critical research directions, includ- ing grounding multimodal LLMs and generalized visual grounding.
A key insight that visual grounding can be performed efficiently without fine-tuning Contrastive Region Guidance (CRG) (Wan et al., 2024) introduces a training-free guidance method that enables open-source VLMs to respond to visual prompts by contrasting model outputs with and without region masking. CRG achieves up 11.1% absolute accuracy improvements across diverse region- based tasks, demonstrating that pre-traine trainrr VLM representations already encode spatial informa- tion. This aligns with our design choice of using lightweight region guidance rather than expensive fine-tuning, and CRG complements our meening approach by providing model-driven region importance weighting.
-QA-Based Visual Grounding Visual Question Answering (VQA) provides a natural framework for generating region-level annotations. Some works (Shrestha al., 2020a;b) show that visual grounding mechanisms are essential for explaining VQA predictions, bridging the gap between model confidence and spatial localization. More recent work (Reich & Schultz, 2024) uncovers the full potential of visual grounding methods in VQA, revealing that region-level understanding significantly improves multi-hop reasoning capabilities. Learning visual grounding from generative VLMs (Wang et al., 2025a) demonstrates that modern generative models can implicitly learn spatial correspondences, which our approach leverages by using model-generated region descriptions.
Spatial vs. Semantic Matching A fundamental distinction in visual grounding exists between spatial matching (geometric alignment) and semantic matching (concept alignment). Recent works (Chen al., 2023; Daxberger et al., 2025) demonstrate that semantic alignment based on visual- linguistii correspondence often outperforms purely spatial approaches. Our VQA-based val- idation mechanism employs semantic matching by querying the model about object presence and
16

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
roles in specific regions, effectively combining spatial information (bounding boxes) with semantic understanding (question-ami -ansoorr--wwwrreeeee consistency).
Grounding for Instance-Level Negation For instance-grounded negation understanding, precise visual grounding is critical. The CoVAND dataset construction process uses region-level VQA to ensure that when we assert "not X," the region boundaries are accurately localized and the ab- sence claim is semantically verified. This region-centric approach differs from image-level negation understanding, which cannot distinguish between objects appearing in different spatial contexts.
C.3 PARAMETER- ERRIMETER-EFFICIENT FINE- -UNING FOR VISION-LANGUAGE MODELS
Large vision-language models contain billions of parameters, making full fine-tuning computation- ally prohibitive. Parameter-efficien fine-tuning (PEFT) techniques enable adaptation with minimal additional parameters, critical requirement for our NEGToME adapter design.
Low-Rank Adaptation (LoRA) and Variants LoRA (Hu et al., 2022) introduced a breakthrough approach decomposing weight updates into low-rank matrices W = W. + +AB" where A E R and B IRrx with rank r < This reduces trainable parameters from (dd)) to (dr).. For CLIP- like models, LoRA has proven highly effective in few-shot scenarios. Low-rank few-shot adaptation of vision- rision-language models (Zanella & Ben Ayed, 2024) empirically demonstrates that CLIP-LORA achieves substantial improvements over prompt-learmn and adapter- adapter-based approaches across 11 datasets while maintaining consistent hyperparameters across all tasks, effectively democratizing few-shot VLM adaptation without task-specific tuning.
A comprehensive PEFT survey (Han at,,, 2024) provides systematic evaluation of PEFT algorithms and their computational overhead, revealing that LoRA remains competitive against more complex PEFT variants (prefix tuning, adapters. prompt tuning) for vision-language adaptation, particularly when strategically applied specific layer types. However, the low-rank constraint limits expres- sivity on complex tasks. DoRA (Liu et al., 2024a), a weight-deccoposse variant, decouples weight matrices into independent magnitude and direction components, emulating full fine-tuning dynam- ies more faithfully. DoRA consistently outperforms LoRA on vision-language tasks including visual instruction tuning (Liu et al., 2024a).
Strategic Layer- Wise Adaptation Not all layers benefit equally from fine-tuning. Analysis of vision-language architectures reveals that cross-modal attention layers are critical for semantic align- ment. Full-rank parameter-efficient fine-tuning (Albert et al., 2025) proposes RandL oRA,, which performs full-rank updates via learned linear combinations of low-rank random matrices. Rand- LoRA significantly reduces the performance gap between LoRA and standard fine-tuning, particu- larly on vision-language tasks, demonstrating that when higher ranks are required, full-rank updates substantially outperform low-rank approximations.
Layer-wise learning strategies further refine adaptation. Layer-wise auto-weighting (Park et al., 2024a) employs Fisher Information Matrix (FIM) autonomously identify which layers require preservation or concentrated adaptation, enabling more efficient non-stationary test-time adapta- tion. For our approach, we strategically apply LoRA to deep cross-attention layers where negation- specific reasoning is encoded, achieving a balance between expressiveness and efficiency. Addi- tionally, data-efficient instruction tuning (Naharas et al., 2025) proves that examples with similar cross-modal attention matrices have similar gradients, informing our layer selection strategy: layers exhibiting high cross-modal attention variance are most critical for negation grounding.
Fine-Grained Semantic Alignment Task-specific semantic alignment requires minimal parame- ters but careful design. A parameter-efficient prompt learning method (Guo et al., 2025) achieves fine-grained semantic alignment using only a few additional parameters. Their key insight is that semantic- objectives require less parameter capacity than low-level pixel matching, particularly when combined with well-designed prompt templates and explicit semantic constraints.
For semantic gating in transformers, Value-State Gated Attention (Bu et al., 2025) introduces learn- able, det-ndepn gates computed from value vectors to modulate token contributions based on semantic importance. This mechanism directly informs NEGTOME's token importance weighting: by learning task-specific gates that suppress irrelevant tokens and amplify negation-bearing tokens,
17

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
achieve semantic specialization with minimal overhead. Dynamic Mask Attention (Zhang all, 2025) extends this by using soft-gating masks and content-aware mask generation based on value representations, enabling fine- tie-ggrined token importance capture without binary keep-or-drop deci- sions.
Negation-Speciffe Adapter Design While most PEFT work addresses general-purpott adapta- tion, negation understanding requires specialized reasoning. Our NEGTOME module combines LoRA's parameter efficiency with negation-aware token merging, creating compact yet semanti- cally rich adaptation.
The negation-boos' mechanism in token merging acts as a learned semantic gating function, allo- cating more computational resources to tokens representing negated concepts. This semantic spe- cialization follows principles from hierarchical inductive transfer for continual learning (Feng al,, 2022), which demonstrates that separating base adapters (capturing general knowledge) from task- specific adapters (capturing task-specialized reasoning) prevents interference and improves perfor- mance on specialized tasks. By analogy, NEGTOME'S selective token enhancement prevents general cross-modal attention from suppressing negation signals.
Synthesis: Efficient Negation-Aware Adaptation Our PEFT strategy for NEGTOME combines: (I) LoRA efficiency: strategic application to cross-modal attention layers, (2) full-rank expressivity: capturing complex regation-direction mappings, (3) semantic gating: value-state aware token im- portance weighting, and (4) hierarchical structure: base cross-modal adaptation and task teec-spec negation enhancement. This integrated approach ensures parameter efficiency while providing suf- ficient expressivity for negation-aware semantic reasoning, critical for learning instance-grounder negation in vision-language tasks.
C.4 COMPOSITIONAL REASONING IN VISION-LANGUAGE TASKS
Compositional reasoning understanding how simple concepts combine to form complex mean- ings-it fundamental negation understanding. Negation inherently compositional: "not red" requires first identifying "red" then applying the negation operator.
Compositional Visual Reasoning Fundamentals A comprehensive survey on compositional vi- sual reasoning (Ke et al., 2025), establishing core definitions and theoretical foundations. The sur- vey identifies key requirements for compositionality: (1)
indntiii-iiinniiiiinnn atomic concepts, combining concepts, and (3) tittemmticiiy--apllyii compo- sitional rules consistently across novel scenarios. The survey emphasizes that compositional reason- ing provides advantages in cognitive alignment, semantic fidelity, robustness, interpretability, and data efficiency. Notably, it highlights that vision-language models struggle with compositionality due to training data bias toward positive instances and the parallel feature processing architecture, challenge directly addressed by COVAND.
(2) http://iiiiinllyy--yyttttteeeeee
Recent analysis reveals that VLMs exhibit the "binding problem" (Campbell al., 2024) mental failures in reliably associating perceptual features with correct visual referents, particularly multi-object scenarios. This limitation mirrors cognitive science findings on rapid feedforward processing in human brains. Addressing this binding problem through structured spatial reasoning (e.g., horizontal lines and sequential scanning prompts) yields substantial improvements across sual reasoning tasks (Izadi et al., 2025), demonstrating that explicit compositional structure enhances multi-object understanding.
Compositional Generalization in Vision-Language Models A critical gap exists between train- ing and test distributions: models trained on one compositional split often fail on others, reveal- ing weak compositional generalization. An empirical study on compositional generalization in VLMs (Li et al., 2024a) shows that current models rely primarily on linguistic priors rather than visual information, causing benchmarks to favor pure language models. To overcome this limita- tion, researchers propose evaluation frameworks without linguistic priors, forcing models to ground compositional reasoning in actual visual content.
Systematic compositional probing studies (Li et al., 2025c) evaluate whether RL-trained VLMs in- herit compositional capabilities from LLMss. Key findings reveal: (1) RL-trained models outperform
18

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
SFT-only models on compositional generalization, (2) VLMs struggle to generalize compositionally under cross-modal and cross-task scenarios despite strong individual-task performance, and (3) en- forcing models to explicitly ground visual content before reasoning (e.g.. peption-before-thnnkeee with visual grounding) yields notable gains. These findings directly motivate our explicit chain- of-thought (CoT) dataset construction: by forcing models descrihe what negation means in spe- cific regions before making predictions, we encourage true compositional understanding rather than shortcut learning.
Out-of-distribun generalization in LLMs is fundamentally tied to compositional structures inter- nally achieved through aligned principal subspaces in self-attention layers (Song al., 2025). This analysis suggests that models composing two self-attention layers can learn rules and generalize to novel tasks-a principle we leverage by designing NEGTOME to operate intermediate layers where compositional negation reasoning naturally emerges.
From Objects to Events: Temporal Compositional Reasoning Event-level reasoning extends compositional understanding beyond static attributes to temporal dynamics. Compositional event reasoning requires understanding object interactions and temporal sequences. Recent work on diffusion-driven video scene graph generation (Chen et al., 2025a) demonstrates that detecting com- plex temporal relations requires compositional reasoning about object trajectories and inter-frame dependencies, achieved through iterative refinement of spatial-lemporal embeddings. Video refer- ring object segmentation (Xu et al., 2025) decomposes referring expressions into structured event graphs with objects, relations, and temporal constraints, revealing that video-level negation requires reasoning about event-level compositional patterns.
Our 3-step CoT generation aligns with this event decomposition: each step performs a primitive operation (object detection -> region analysis negation verification), mirroring the hierarchical event reasoning paradigm. By explicitly grounding each reasoning step in concrete regions, Co- VAND teaches models compose negation patterns systematically rather than relying on spurious correlations.
Attribute Binding and Multi-Object Reasoning A central compositional challenge is correctly binding attributes to objects (the binding problem). Scene graph generation literature (Chang al., 2021; Yang et al., 2024), identifies systematic failures in relational understanding. Video scene graph generation with unbiased training (Li et al., 2025d) addresses these failures through visual-semantic dual supervision, explicitly enforcing consistency between visual features and semantic relations. For negation, this principle translates enforcing consistency between negated object descriptions and their spatial absence in images.
Multimodal Compositional Generalization Retrieving semantically equivalent primitives across modalities enhances compositional generalization. Multi-sourced compositional generalization in training where equivalent primitives from dif-
VQA (Li et al., 2025b) proposes ferent modalities are aggregated refine representations, improving generalization to novel compo- sitions. For negation, this principle suggests that enforcing consistency between linguistic negation descriptions and visual absence patterns (via VQA-based validation) teaches models to reason about negation unified cross-modal compositional operator.
Systematic Generalization in Visual Reasoning A benchmark for systematic generalization in visual world models (Kim et al., 2023b) evaluates whether models perform visual imagination and measure compositional generalization systematically. The benchmark reveals that even state-of-the- art models fail on out-of-distribution compositional scenarios. Our approach mirrors this systematic evaluation: by constructing COVAND with systematic variations across objects, attributes, and negation types, we provide training data that encourages systematic rather than accidental general- ization.
Synthesis: Compositional Negation as Structured Visual Reasoning Our approach positions negation as a compositional operator within a broader framework of structured visual reasoning. By combining: (1) explicit event decomposition through 3-step CoT reasoning, (2) visual-semantic binding enforcement via VQA validation, (3) spurious correlation mitigation through systematic data variation, and (4) out-of-distribution generalization via consistent cross-modal mappings, Co-
19

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
VAND enables models to learn compositional negation patterns grounded in both visual evidence and semantic structure. This compositional foundation, combined with NEGTOME's semantic- aware token merging that respects the hierarchical structure of compositional reasoning, creates framework where negation understanding emerges from systematic compositional principles rather than shallow surface-level negation cues.
C.5 BIAS MITIGATION IN VISION-LANGUAGE MODELS
Affirmative bias- the tendency to misinterpret negations as affirmations represents a systematic bias in VLM training data and architecture. Understanding and mitigating this bias is central to our work.
Understanding Affirmative Bias in VLMs The fundamental source of affirmative bias lies in training data distribution. Most -inguisic corpora contain predominantly positive image-text pairs: "dog park," "person running," " etc. Negative examples are rare, creating class imbalance that models exploit through shorteut learning. DeAR (Seth et al., 2023) demonstrates that VLM bias can be partially mitigated by learning additive residual corrections to visual representations without retraining. However, their approach targets shallow social biases; negation bias requires deeper structural understanding.
Recent work reveals that VLMs exhibit social biases including gender and racial stereotypes in their generative responses (Lan et al., 2025). More critically, fair-response reliability differs from accuracy: models may achieve high accuracy while exhibiting significantly lower faimesss scores. This decoupling between performance and fairness motivates our multi-level debiasing strategy, where data-level, method-level, and evaluation-level components collectively address negation bias.
Architectural Sources of Aftirmative Bias Bias is not merely a data problem but deeply embed- ded in architectural choices. Bias analysis in transformer attention heads (Yang al., 2025) reveals that specific attention heads are responsible for encoding stereotypical biases; by identifying and masking high-bias heads, models achieve significant bias reduction without affecting language un- derstanding. This attention-head level analysis directly informs NEGTOME's design: our selective token merging operates at the semantic level, effectively masking attention patterns that suppress negation signals.
Shorteut Learning and Spurious Correlation Models exploit spurious correlations- -eattursss-esuurss coincidentally correlated with labels in training data-to achieve high training accuracy while gen- cralizing poorly to test data. Shortcut learning in LLMs (Du et al., 2023) identifies that models learn to associate demographic attributes with spurious features rather than learning genuine causal rela- tionships. For negation, this manifests as: models may learn that "absence of red" correlates with "dark backgrounds" rather than learning true negation semantics.
(Zheng et al., 2025) introduces post-hoc framework for identifying and mitigating spurious biases via prediction shortcuts in the model's 's latent space, without requiring group labels. The key insight is that spurious correlations create non-generalizable prediction shortcuts that fail under distribution shift. Our CoVAND addresses this by systematically varying objects. attributes, and spatial configurations, forcing models to learn generalizable negation patterns rather than spuri- ous shortcuts. Furthermore, generative classifiers naturally avoid shortcut learning (Li et al., 2025a) by modeling all features rather than mainly spurious ones, suggesting an alternative avenue for ro- bust negation understanding.
ShortcutProbe
Synthetic Data Augmentation and Its Pitfalls While data augmentation seems natural solution, naive approaches can inherit and amplify existing biases. Decoupling augmentation bias in prompt learning for VLMs (Gerych et al., 2024) demonstrates that random negation (c.g.. "not red" gener- ated from "red") can create inconsistent training signals due to augmentation distribution mismatch. CoVAND addresses this by employing LLMs to generate semantically coherent negation examples e with multi-step Chain-of-thougle reasoning verification, ensuring both linguistic consistency and visual grounding.
Handling imbalanced pseudolabels in VLMs (Pang et al., 2025) reveals that models exhibit class- preference biases when generating pseudolabels for unlabeled data. Their proposed solution com-
20

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
bines concept alignment with confusion-awee aware calibrated margins. Our DWWAND-derived dataset similarly enforces consistency through VQA validation and avoids concept confusion via systematic negation type variation.
Bias Inheritance in LLM-Based Data Augmentation LLM-based augmentation introduces new challenges: models inherit biases from their training distribution even when generating synthetic data (Li et al., 2025a). When an LLM generates "not sitting," it may unconsciously preserve biases about typical sitting scemarios and demographic associations. Our solution combines LLM genera- tion with human review and VQA validation, creating a hybrid pipeline resistant to bias inheritance. This stage validation mirrors fairess--awaee domain adaptation for VLMs (Pang et al., 2025), which uses attribute-aware strategies to dynamically adapt to diverse demographies for equitable outcomes.
Negation-Aware Test-Time Adaptation Beyond training time mitigation, test-time debiasing of- fers complementary benefits. BEND-VLM (Gerych et al., 2024) proposes nonlinear, fine-tuning- free debiasing that tailors debiasing each unique input without prior knowledge of the test set. While their approach targets social biases, the principle of input-specific bias correction directly ap- plies to negation: negation scopes and interpretations vary contextually, so adaptive, query-specific debiasing enhances robustness.
Debiasing VLMs using backdoor learning (Pang et al., 2025) proposes learning backdoor patterns that trigger debiased representations, enabling inference-tines debiasing without model modifica- tion. This technique complements our approach: while NEGToME merges tokens based on learned negation patterns, backdoor triggers could provide additional control signals for negation-aware in- ference.
Synthesis:
Multi-Level Affirmative Bias Mitigation Our comprehensive approach targets affir- mative bias across multiple levels:
1. Data-level: Systematic COVAND dataset construction with multi-modal chain-of-thougee reasoning and VQA validation, explicitly addressing spurious correlations and shortcut learning through systematic data variation (Zheng et al., 2025; Gerych al., 2024).
2. Architecture-level: NEGTOME's negation-aware token merging acts as selective attention-head debiaser, amplifying negation-bearing tokens while suppressing spurious features. This semantic gating follows attention-heds head masking principles, but operates at the token level with learned importance weighting.
3. Method-level: Strategic LoRA application to cross-modal attention layers (Sections C.3) provides fine-grained control over negation reasoning without full model retraining, miti- gating catastrophic forgetting associated with fine-tuning-bar
debiasing.
4. Evaluation-level:
Instance-grounded metrics that measure negation understanding sepa- rately from general object detection, preventing bias amplification in evaluation and en- abling fair assessment minority negation patterns.
By combining these complementary strategies, we provide stronger affirmative bias mitigation than any single technique, addressing shortcut learning, spurious correlations, and architectural bias si- multaneously. Furthermore, this multi-level approach generalizes beyond negation to other linguistic phenomena requiring fine-grained compositional understanding.
D ADDITIONAL ABLATIONS: NEGATION- AND NOUN-ONLY BOOSTING
This section provides implementation details and empirical analysis of negation- and noun-only boosting variants that test whether simply increasing the emphasis on certain tokens (without struc- tural merging) is sufficient improve negation understanding in VLsebassed detectors.
Implementation details. sequence = (w)...., w.,.)) and phrase-level groupings P Par as described in the main paper. We then tokenize the same string with the model's text tokenizer (BERT/CLIP) to obtain
Given caption string first run spaCy parser obtain token
21

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
subword tokens (#1, (.1..... (Lw) and use the alignment procedure to map each spaCy token to its set of subword indices.
For each phrase P. we obtain index set I, {L...., Lar) of subword positions and compute merged embedding t; using the normalized weighted- -vverage merging operator:
f Ejez, 7/4
(5)
ΣJEZ, 75
where fy is the original text embedding at subword index j and 7j > is an importance weight that depends the phrase type.
In all variants, the phrase replacement applied once per phrase: the original subword embed- dings {ts Jyez, are removed and replaced by a single merged embedding i; at that position. The different ablations differ only in how we choose the weights "/ and which tokens are boosted by multiplicative factor 1:
Noun-only Boost. We identify head nouns inside each phrase P. using spaCy dependency tags (see collect noun spacy indices and spacy indices to hf indices in the code), map them to subword indices, and form set BRomn of boosted positions. For all phrases we use the merging operator in Eq. equation with
Y = B if E B
otherwise.
This variant is agnostic to negation and only amplifies nouns/heads.
Neg-only Boost. We first collect spaCy indices of negation cues ("not", "no", "without", "never", "n't", etc.) using col ect negati on cue spacy indi ces, then map them to subword indices to obtain a set Baey. We again use Eq. equation 5 with
3 if Baeg [1
otherwise.
Importantly, we do not structurally bind the negation token the attribute; the mains an isolated token (or part a short phrase) that competes with other tokens in cross- attention.
Merge Head Boost. For this variant we apply phrase merging to all phrases as in Eq. equa- tion 5, but we treat negation cues and non-negation phrases uniformly. Inside each phrase, only the head noun is boosted:
if E Brend; 3 = l otherwise.
where Breal contains subword indices of phrase heads. This tests whether phrase-level structural unification alone (without explicit negation-aware weighting) is sufficient.
Attn Bias. Instead of modifying the text embeddings, this variant inserts per-token bias into the cross-attention logits. For each negation cue index j E Breg we add fixed bias 5 >> 0 to the attention score for that token, leaving the tokenization structure unchanged. Concretely, if A E Qx the pey dot product matrix in decoder block, we replace it by A' A + Bneg, where Baeg is matrix whose j-th column is shifted by å for negation indices and zero elsewhere. This variant corresponds to the Attn Bias row in Table S8.
Ours (NEGTOME). For non-negated phrases P: we use Eq, equation 5 with uniform weights ?/ 1, i.e., a simple average. For negated phrases Paeg that contain negation cue and modified attribute (e.g., "not lying", "without hat")) we set
{ if corresponds to the negation cue, Yj
for the other tokens in Powg-
Thus, Eq. equation 5 simultaneously performs phrase merging and negation- -wwaee boost: all subwords in the negated phrase are collapsed into single merged token faeg. whose representation is dominated by the cue contribution but remains conditioned on the attribute it negates.
22

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Method
AP1
FPR +
Noun-only Boost
50.5
66.1
Neg-only Boost
49.7
62.0
Merge Head Boost
56.2
62.4
Attn Bias
46.2
59.6
Ours (NEGTOME)
$7.3
47.7
Table S8: Negation- vs. noun-only boosting ablations on OVDEval-Negation, All models are fine-tuned on COVAND-L for epoch with identical LoRA configuration. NEGTOME is the only variant that simultaneously improves AP and substantially reduces FPR.
All variants share the same training and evaluation protocol: we fine-tune Grounding DINO-B with LoRA on the CoVAND-L split for one epoch and evaluate on OVDEval-Negation, using COCO- style AP and the False Positive Rate (FPR) computed after NMS.
Quantitative results. Table S8 summarizes the results.
We observe three key trends:
1. Negation-only boosting is insufficient. The Neg-only Boost variant (boosting only nega- tion cue embeddings without changing structure) yields AP 49.7 and FPR 62.0. Similarly, the Atn Bias variant, which adds attention bias toward negation tokens while preserving the original tokenization, collapses to the lowest AP (46.2) despite small FPR reduction (59.6). This confirms that simply increasing the magnitude or attention weight of negation tokens does not translate into reliable negation understanding: can even disrupt compositional semantics.
2. Phrase merging alone is necessary but not sufficient. The Merge Head Boost variant leverages the same phrase-merging operator as NEGTOME but boosts only the head noun in each phrase and does not treat negation cues specially. This already improves AP to 56.2, suggesting that structurally unifying multi-token phrases into single semantic units helps detection. However, FPR remains high (62.4), indicating that the model still tends to fire on both "with" and "without" queries (i.c., detects the object but fails respect the polarity).
3. NEGTOME (merging + negation-aware boost) required. Only the full NEGTOME module, which both merges negation phrases and assigns higher weight to the negation cue inside the merged representation, achieves the desired behavior: AP = 57.3 (best among all variants) and FPR 47.7 (a reduction of 14.3-18.4 absolute points compared to the other ablations). These results support the view that the main bottleneck is not merely low attention on negation tokens, but the structural separation between cues and the attributes they modify.
Taken together, these controlled ablations provide converging evidence that (i) boosting the nega- tion cue in isolation is not enough, and (ii) negation- aware phrase merging, as implemented by NEGTOME, crucial for improving both localization (AP) and negation discrimination (FPR) on OvDeval-Neeation.
E COMPARISON WITH POST-HOC VQA METHODS
Motivation. An alternative to enhancing a detector's internal negation understanding two- stage pipeline, where standard detector generates initial proposals and powerful Multimodal Large Language Model (MLLM) then acts as a post-hoe filter to remove erroncous detections. To implemented two post-hoc
investigate the viability and trade-offs of this common alternative, VQA variants. We built these on top of the same baseline detector used in our main experiments and report results the OVDEval Negation subset using both AP and class-ignored NMS-AP.
Two pust-hoe settings. (A) Crop & Verify. For each image, we take the detector's top-k boxes, crop each region, and query MILM with yes/no question about whether the crop satisfies the
23

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Table S9: Post-hoc VQA on OVDEval Negation. Numbers are AP/ NMS-AP (1). "Ours" is the single-stage detector fine-tuned with deep-layer LoRA + NEGTOME. Crop & Verify improves the baseline but requires & MLLM calls per image; Coordinate Prompting is faster but brittle. Stacking the expensive verifier on top of our improved detector yields the best overall numbers.
Detector
Post-hoe Verifier
AP
NMS-AP
(A) Crop & Verify (top-k crops = k MLLM calls)
G-DINO-B
54.0
36.8
G-DINO-B
+ Qwen-2.5-VL-33
59.2
54.4
G-DINO-B Ours
58.7 63.8
44.5
G-DINO-B Ours
Qwen-2.5-VIL-3B
58.4
(B) Coordinate Prompting (single MLLM call with all boxcs)
G-DINO-B
54.0
36.8
G-DINO-B
Owen-2.5.VD.
48.6
34.1
G-DINO-B
+ Qwwe-2.5-VL-7B
54.0
36.9
G-DINO-B Ours
58.7
44.5
G-DINO-B Ours
+ Owen -wen-2.55VL
2.5-B
49.6
37.0
G-DINO-B Ours
+ Qwen-2.5-
58.6
44.4
input description. This yields & separate MLLM calls per image. (B) Coordinate Prompting. We avoid cropping and instead pass all top-k box coordinates and the description to the MLLM at once, asking it to indicate which boxes are inconsistent.
Results. As shown in Table S9, the Crop & Verify method substantially increases the bascline de- tector's NMS-AP from 36.8 to 54.4, confirming that strong VQA filter can reduce contradictory detections. However, this accuracy gain comes with heavy latency cost due to the O(k) MLLM calls required per image. In contrast, the faster Coordinate Prompting method is unreliable for fine- grained reasoning and often degrades performance; for instance, the baseline's 's NMS-AP drops from 36.8 to 34.1 when paired with the 3B verifier. Notably, our single-stage method already achieves an NMS-AP of 44.5, closing much of this performance gap without any added latency. When the ac- curate but slow Crop & Verify filter is applied on top of our already-improved model, it achieves the highest NMS-AP of 58.4, indicating that our method and post-hoe verification are complementary rather than redundant.
Conclusion. These experiments demonstrate that while a two-stage VQA pipeline can be effective, it presents a clear trade-off between accuracy and speed. The rrsp-based verifier is accurate but slow, whereas coordinate prompting fast but brittle. Our single-stage approach, by contrast, instills negation sensitivity directly within the detector, improving the stricter NMS-AP and reducing false positives in single, efficient pass. This confirms that post-hoe filtering does not obviate the need for negation- aware detector. For practical, real-time settings, integrating negation reasoning directly into the model's fusion layers remains the most effective path. If latency is not a concern, our work also shows that costly verifier can be used to further refine the outputs of our model.
F EVALUATION ON FULL OVDEVAL SUBSETS
OVDEval (Yao et al., 2024) a comprehensive benchmark designed to evaluate the generalization capability of open-vocabulary detection (OVD) models across diverse linguistic aspects. The dataset includes 9 sub-datasets that test 6 distinct aspects: object, proper noun (landmark, logo, celebrity), attribute (color, material), position, relationship, and negation. Each subset features meticulously curated hard negative samples that challenge models to demonstrate true understanding of fine- grained linguistic descriptions rather than exploiting dataset biases. For instance. the color subset includes negative labels with the same object category but different colors, while relationship subsets maintain identical subjects and objects but alter the connecting verbs.
24

---

What "Not" to Detect: Negation-Awer Aware VLMs via Structured Reasoning and Token Merging
Urio, Fred
Only, Fred
G7
Orlo, Fred
dlandin langaes
rapes cemaad
ragee not zspes outade""
Waning Aapaaro
"Kagaao stonding
Figure S19: Failure Cases of Prior Models on Negation Descriptions
F.1 THE INFLATED AP PROBLEM AND NMS-AP METRIC
Standard Average Precision (AP) metrics face limitations when evaluating fine-grained described object detection due to what OVDEval terms the Inflated AP Problem. This issue occurs when model predicts multiple bounding boxes for the same object with different labels, including mutu- ally exclusive ones in Figure S19. For example, model might predict both "outdoor dog led by rope" and "dog not led by ropes outside" for the same dog, artificially inflating its AP score. Mathematically, this manifests
TP
TP
1 = 1.0
Precision
TP+FP 1+1 = 0.50, Recall
(6)
GT
1
Where model with no actual understanding of attributes can still achieve mAP of 0.50. To address this, we follow OVDEval's Non-Maximum Suppression Average Precision (NMS-AP) metric (Yao et al., 2024), which applies class-ignored NMS to remove redundant predictions for the same object before AP calculation. This provides more accurate assessment model's ability to understand fine-grained descriptions of contradictory pairs.
F.2 GENERALIZATION TO NON-NEGATION SUBSETS
Table S10 demonstrates that our model maintains robust performance across all OVDEval subsets despite being trained exclusively the regation-focused CoVAND dataset. Notably, approach shows improved NMS-AP scores for Logo (+0.2), Landmark (+4.8), Color (+0.7), and Relation- ship (+3.8) subsets compared the baseline. This broad generalization suggests that our negation- sensitive adaptations enhance the model's 's overall reasoning capabilities for complex descriptions. These results confirm that our LoRA-based parameter-efficien fine-tuning and NEGTOME token merging strategy provide benefits beyond negation understanding, enhancing the model's capability process compositional descriptions across multiple semantic aspects.
Table S10: Evaluation Results on Full OVDEval. Performance on OVDEval subsets, except for the Negation. Even though we only trained with regeation-foussee CoVAND dataset, our models show robust results for other subsets.
Logo
Landmark
Culebrity
Color
Material
Positson
Relationship
Average
AP
NMS-AP
AP
NMS-AP
AP
NMS-AP
AP
NMS-AP
AP
NMS-AP
AP
NMS-AP
AP
NMS-AP
AP
NMS-AP
20.5
15.2
4Oars
11.5
78
22.4
21.3
6.6
03
79
63
15.8
5,3
70.5
54.6
42.3
22.1
25
168
G ANALYSIS ON RPN-BASED DETECTOR
G. LIMITATIONS OF RPN-BASED DETECTORS UNDER NEGATION
Marginal or negative gains with LoRA. Two-stage region proposal detectors such as GLIP (Li al., 2022) and FIBER (Dou al., 2022) obtain slight improvement negation focused bench- marks after attaching LoRA adapters as in Table S11. For GLIP, whose backbone consists of stacked heati-head attention blocks, inject LoRA only into the FFN layers of the last two transformer
25

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
blocks, leaving all attention projections frozen. Even with this targeted fine-tuning, the gains main marginal. These findings indicate that low-rank fine-tuning brings far smaller gains to GLIP than DETR-style detectors. The gap can be traced to their attention layouts: GLIP employs self-MHA over a mixed token pool, whereas GroNndingg-INNO uses two modality-specific cross- attention blocks driven by compact query set, design that lets LoRA and NEGTTMME act directly on phrase-level cues and thus respond much more strongly to negation.
Affirmative bias and context insensitivity. Recent negation benchmarks reveal a sharp drop in detection accuracy whenever query expresses absence negation (Alhamoud et al., 2025). GLIP and FIBER often treat a negated phrase ("not x") as if it were "X", triggering on object names while ignoring context qualifiers. Consequently, GLIP still localizes "microphone" when the description states "a person with no microphone", producing hallucinated objects. LoRA-adapted RPN detectors exhibit diminishing returns on negation-centric tasks because their proposal stage detects any region matching a noun, leaving little capacity to encode absence semantics.
Performance gap between AP and NMS-AP on the Negation subset. Table S11 further shows that model capacity alone does not resolve the issue: even the larger GLIP-L still exhibits a gap between AP and class-ignored NMS-AP, substantially wider than the gap of smaller DETR coun- terparts. The gap quantifies how many redundant, mutually exclusive boxes each model produces. A large drop after class-ignored NMS indicates that the detector continues to fire on the noun even when the query contains negation cue, confirming the affirmative bias analyzed in the main paper.
Effect of NEGTOME. DETR keeps token granularity throughout the vision-language stack, al- lowing a merged phrase embedding to dominate (g, kj) for its specific key k; while leaving other keys unaltered. By contrast, GLIP FIBER fuse language either by (a) global pooling of the entire caption ([cl s]), }) or (b) class-name pooling plus a separate visual prompt. Both strategies erase intra-sentence polarity (dog vs. not dog) before the detector sees it. Token merging cannot recover that lost contrast; at best it shortens sequence that will be pooled anyway.
Table S11: OVDEval-Neeation Evaluation on Additional Architectures. RPN-based detectors show a large gap between AP and NMS-AP. FT denotes fine-tuning of LoRA parameters only, with all other pretrained weights kept frozen as in the main paper. Results marked with are reproduced.
Image Backbone
Total Param.
AP
NMS-AP (Yao et al,, 2024)
Non Detector
MDETR (Kamath et all, 2021)
ResNet-101 101 Con-Next-B Swin-B
185M
41.1
28.3
OmDet (Zho et 20246)
242M
55.9
35.1
Grounding DINO' (Lin et al., 2024b)
233M
54.0
36.8
RPN-based Detector
FIBER DDu et 2022)
Swin-B Swin-L
252M
57.2
28.7
GLIP-L (Li et al., 2022)
430M
51.8
GLIP-T (Li et al., 2022)
47.7
25.4
FT w.COVAND
Swin-T
232M
47.8
26.1
+FT COVAND + NEGTOME
48.3
26.0
G.2 ADVANTAGES OF
DETECTORS WITH LORA
Compositional reasoning. DETR- detectors with transformer decoders (Kamath et al., 2021; Liu et al., 2024b; Li et al., 2023; Shen al., 2024) perform joint text-image reasoning through cross-attention in decoder blocks. This design enables natural handling of relations such as "X but not Y".
Effectiveness of LoRA. Injecting LoRA adapters into the decoder cross-attention layers of Grounding DINO and fine-tuning on regatioo-focused dataset improves mAp by +2.6 and cuts the false positive rate by 11.5%. The same lightweight adaptation reduces spurious detections on the Negation subset of OVDEval by nearly half, while preserving general detection accuracy.
Why the architecture helps. Each decoder layer attends to textual tokens; negation words there- fore, modulate visual attention directly. In RPN pipelines, language supervision is applied only after
26

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
proposals are fixed, limiting early rejection of forbidden objects. A fully fused DETR decoder yields a contextual representation "what not to detect," which a small LoRA module can efficiently re- fine.
Advantage of NEGTOME. Both Grounding DINO and APE inherit the sub-token fragmentation of their text backbones with BERT and BPE in CLIP. NEGTOME merges those fragments into one polarity-aware phrase embedding and re-weights it by a boost factor /. In Grounding DINO, this merged vector is fed intact through token-level cross-attention, so every decoder layer receives sharper gradient signal for the absence condition; the result is +00.8 rise NMS-AP and 19.1% drop in false positives on OVeeval-Negation. APE employs CLIP, whose encoder pools all tokens into a single sentence vector before fusion. Here NEGTOME acts pre-pooling: by assigning larger softmax weights the merged negation phrase it skews the sentence representation toward the correct polarity, yet does not increase sequence length. Consequently, the lightweight merger lifts APE-Ti by +1.2 in NMS-AP and reduces absent-object errors by 8.3%, despite updating only 0.017% of parameters. NEGT aligns with the inductive bias of both encoders: it supplies BERT-based decoders with an explicit token for cross-modal attention, and it biases CLIP's global pooling toward the correct semantic polarity. The mechanism is encoder-agnostic and therefore complements LoRA across heterogeneous DETR frameworks.
BETR-basede detectors fine-tuned with LoRA and NEGTOME achieve larger and more reliable gains on negation and other compositional queries than RPN counterparts. Their set- set-prediction decoder offers single, expressive locus for parameter-efficieet language adaptation.
H ZERO-SHOT DOWNSTREAM TASKS: MULTIPLE CHOICE QUESTIONS
To further analyze our model's semantic comprehension of negation, we evaluate it on the NegBench Multiple Choice Question (MCQ) benchmark (Alhamoud et al., 2025). This benchmark is specif- ically designed diagnose a VL.'ss ability to handle negation by requiring it select the most accurate caption for an image from four options. These options are structured into three challenging categories as detailed below, providing a fine-grained analysis of a model's capabilities.
H.1 STRUCTURE OF THE NEGBENCH MCQ
The NegBench MCQ task (Alhamoud et al., 2025) generates multiple-choice questions where one answer is correct and the other three serve as hard negatives, designed to mislead models that do not properly understand negation. The questions are categorized into three distinct types based on the linguistic structure of the correct answer:
The correct caption is a simple affirmation that accurately describes ob- jects present in the image (e.g.. "This image shows baseball bat and baseball glove"). This subset tests the model's fundamental visual grounding capabilities, as shown in Fig- ure $20. Incorrect options often involve falsely negating a present object.
Positive Subset:
Negative Subset: The correct caption accurately negates the presence an object that is contextually relevant but absent from the image (e.g.. "A bowl is not present in this image"). This directly tests the model's ability to comprehend explicit negation, as illustrated in Figure $21.
Hybrid Subset:
The correct caption combines both an affirmation and a negation within a single sentence (e.g.. "This image features a person, with no truck in sight"). As shown in Figure S22, this is the most challenging subset as it requires compositional reasoning and understanding of complex sentence structures that assign different polarities different objects.
H.2 ERROR PATTERN ANALYSIS OF BASELINE MODELS
Our qualitative analysis reveals that baseline models exhibit consistent and fundamental error pat- terns on the NegBench MCQ task, primarily stemming from severe affirmative bias as below:
27

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
1. Blatant Contradiction of Visual Facts: The most common failure is choosing caption that directly contradicts the visual evidence. For example, in Figure S21, the baseline model selects "There is no horse in this image" for image clearly depicting horse. This indicates that the model heavily weighs the noun ("horse") while effectively ignoring the negation cue, treating both affirmative and negative statements semantically similar.
2. Polarity Confusion in Hybrid Sentences: In the Hybrid subset (Figure S22), bascline models systematically fail to parse sentences containing both positive and negative clauses. For instance, given the ground truth "This image features refrigerator, but lack of bottle," the baseline chooses "This image features a bottle, but does not include a refrigerator." This shows a critical failure in compositional reasoning, where the model cannot correctly assign presence and absence to different objects within the same logical construct.
3. Selection of Suboptimal Negatives: In some cases on the Negative subset, the baseline avoids direct contradiction but fails to select the most accurate description. As seen in Figure S21, when the ground truth is "A bowl is not present, the baseline chooses "no cake is present." While factually correct, this choice suggests the model lacks a deeper contextual understanding to identify the most salient absent object among multiple true negative options.
These error patterns underscore that many state-of-the-art VLMs do not understand negation. In- stead, they rely on shortcut strategies that collapse the semantic meaning of affirmative and negative statements. This motivates need for methods that can fundamentally address this architectural limitation.
(1) This inage shows baseball bat and baseball glove
(2) This inage features a dining table, but does not include baseball bat.
(3) A dining table is present in this inage.
(4) This inage doesn't feature a baseball bat
This image features both zebra and giraffe.
is present in this inage, but there no zebra.
(8) A is present in inage. No zebra present this inage.
(1) inage features knife dining table.
(2) This inage contains bowl, with knife in sight.
(3) This inage shows bowl.
(4) Noticeably absent fron this inage is knife.
Figure S20: Qualitative Results on the Positive subset of the Multiple Choice Question bench-
mark. Captions with green checkmark is GT, pink refer to Baseline, and blue refer to Ours.
28

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
(1) present in this inage.
(2) This image features boul, but cake present.
(3) boul is shown in this image.
(4) included inage.
(1) This image contains ne car.
(2) This image shows car, but no horse is present.
(3) This inage features car.
(4) There is na horse this inago.
There is chair in this image.
(2) This inage features chair, but there's no cat.
( chair is present in this inage.
There is cat in this inage.
There is traffic Light this inage.
This inage features traffic light, present.
(3) traffic light is included in this inage.
(4) No car is present in this inage.
Figure S21: Qualitative Results on the Negative subset of the Multiple Choice Question bench-
mark. Captions with green checkmark is GT, pink refer to Baseline, and blue refer to Ours.
29

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
This image features refrigerator, but Lack of bottle.
This inage features bottle, but does not include refrigerator
(3)A bottle is included in this image.
(4) refrigerator is not present in this image.
(1) This inage includes sheep but truck.
(2) This inage shows truck, but there sheep.
(3) inage
(4) There is no sheep in this image.
(1) This image features dining table, but no car is present.
(2) This image features but does not include dining table.
(3) A car is included in this inage.
(4) dining table included this inage.
(1) inage features frisbee, but there's no bench in sight.
(2) This inage features bench, but no frisbee is visible.
(3) This inage shows a bench.
(4) frisbee is included in this inage.
Figure S22: Qualitative Results the Hybrid subset of the Multiple Choice Question bench-
mark. Captions with green checkmark is GT. pink refer Baseline, and blue refer to Ours.
30

---

What "Not" to Detect: Negation-Awart VLMs via Structured Reasoning and Token Merging
1 ZERO-SHOT GENERALIZATION ON BIOMEDICAL DOMAIN
To validate that our method learns robust mechanism of negation processing rather than merely memorizing specific object-negation pairs from the COVAND dataset, conducted zero-shot evaluation on the Biomedical Domain. We utilized the FG-CXR dataset (Pham et al., 2024), a fine- grained chest X-ray dataset that aligns radiologist gaze with diagnostic reports. This domain poses a severe generalization challenge due to the drastic shift in visual features (grayscale X-rays) and a distinct linguistic taxonomy of negation (e.g.. "normal", "clear", "no findings").
1.1 EXPERIMENTAL SETUP
Unlike standard captioning benchmarks, FG-CXR maps diagnostic sentences to specific anatom- ical regions (e.g., heart, upper left lung). We formulated the evaluation Zero-shot Binary Discrimination Task:
Data Processing:
We flattened the dataset evaluate every valid anatomical region indi- vidually. For each region in an image, we extracted the ground truth (GT) diagnosis.
Hard Negative Generation: To strictly test negation understanding, we generated a con- tradiction for every GT sentence by syntactically flipping its polarity using a rule-based approach.
State Negation: "The left lung is possibly normal" + "The left lung is possibly ab- normal".
Absence of Findings: "No pleural effusion"" > "Pleural effusion is present".
Disease Assertion: "Opacity in the right lung" "No opacity in the right lung".
Metric: We report Accuracy. The model is considered correct if it assigns a higher match- ing score (max logit) to the GT caption than to the Hard Negative caption given the visual region.
I.2 RESULTS AND ANALYSIS
The results of this cross-domain evaluation are presented Table $12,
Table $12: Zero-shot Generalization Results on FG-CXR.
Method
Domain
Accuracy
Baseline (G-DINO-B)
Biomedical (Zero-shot) 54.86%
+ Ours (NEGTOME)
Biomedical (Zero-shot)
62.55%
The baseline model achieves an accuracy of 54.86%, which only marginally above random chance (50%). This confirms that standard VLMs struggle deeply with medical negation, often failing to distinguish "normal" from "abnormal" even when visual features are distinct.
Analysis:
In contrast, method achieves 62.55% (+7.69%) substantial improvement. Since model was never exposed to medical images or jargon during fine-tuning, this gain cannot be attributed memorizing in-domain data. Instead, demonstrates that NEGTOME effectively structuralizes the binding between negation cues (e.g., "no", "normal") and their targets, allowing the model to generalize this reasoning mechanism to entirely new domains.
J QUALITATIVE RESULTS
We present additional qualitative examples from the OVDEval and D datasets to further demon- strate the effectiveness of our negation understanding approach. Figure S23 and Figure S24 show our model's ability to distinguish between contradictory attribute pairs such as "horse urinating" versus "horse that is not urinating" and "complete pizza" versus "pizza that is not complete". The baseline model often detects identical regions for both negative and positive descriptions, demon- strating significant affirmative bias. contrast, our method successfully differentiates between these contradictory descriptions by correctly emphasizing negation cues.
31

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
Figures S25 to S27 illustrate our model's performance the D dataset. For descriptions such as "hanger without clothes" and "a bed without patterns", our model correctly identifies only the objects that satisfy these negated constraints. The bascline frequently exhibits false positives by detecting objects regardless of negation markers. Our approach demonstrates particular effectiveness for simple negation cases involving physical attributes and object presence.
Despite these improvements, our method still exhibits limitations in scenarios requiring highly com- plex reasoning, shown in Figure S28. These challenges often involve multi-step relational logic combined with negation, such as in the query "a woman in white wedding dress not beside any men in suits", or understanding negated states, as in "a volley ball in the middle of the air untouched". Furthermore, resolving ambiguous or implicit negation cues like "unlike" in "origami unlike bird" remains a difficult problem. common failure pattern these cases that when a complex event or state is entirely absent from the image (e.g.. "the person who was proposed to one knee"), the model defaults its affirmative bias, detecting the main subject of the query ("person") rather than correctly identifying that object matches the full description. Crucially, the bascline model faces identical challenges in these cases, demonstrating that these are open problems for the current generation of VLM detectors. This confirms that method, while such intricate reasoning, does not degrade performance on these hardest examples. These limita- tions highlight important areas for future research in handling complex linguistic constructions and multi-step negation scope resolution.
complete solution for
K. DECLARATIONS
LLM usage. A large language model (LLM) was used during the preparation of this paper to proofread and refine the writing, including correcting grammar and improving sentence structure.
Ethics Statement. Our work adheres to the ICLR Code of Ethics. The primary goal of this re- search is improve the reliability and safety of vision-language models by addressing a fundamen- tal flaw in their reasoning the failure to understand negation. By reducing "affirmative bias," we closely with human language and intent, which prevent
aim create models that align critical errors in real-world applications (e.g.. medical imaging or autonomous systems). Our new dataset, CoVAND, is built upon the public Flickr30k Entities benchmark. The new captions generated using a large language model (GPT-4o) with a systematic, multi-step pipeline designed to ensure high-quality, relevant, and grounded annotations. While our method improves model's lin- guistic comprehension, it does not inherently address or remove societal biases that may be present the underlying web-scale pre-training data or the baseline models themselves. We believe the contribution is net positive, leading to more robust and predictable Al systems.
Reproducibility Statement. We are committed to ensuring the reproducibility our research. To this end, will make source code, including the implementation of the NEGTOME module, and the complete CoVAND dataset publicly available upon publication.
32

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
GT
Baseline
Ours
"horse urinating vs. horse that is not urinating
"orange without leaves vs.
orange with leaves
"pizza without green peppers vs. pizza with green peppers
oottpitt pisa
comple
"complete pizza vs. pizza that is not complete
"cow looking the camera vs. Cow without looking at the camera
Figure S23: Qualitative Results on OVDEval Datasets (1), Prediction results on contradictory caption pairs (yellow box green box) from the negation subset of OVDEval dataset. Each row displays (left) ground-truth boxes, (middle) baseline predictions, and (right) our predictions. Our model effectively reduces affirmative bias, no longer returning identical bounding boxes for captions that express opposite meanings.
33

---

What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
GT
Baseline
Ours
"person with skateboard vs. person without skateboard
"bread with vegetables vs. bread without vegetables
"horse with white fur vs. horse without white fur"
baldheer that
"hairdryer that is not pure white vs. pure white hairdryer
hent
"cup with handle VS. cup without handle
Figure S24: Qualitative Results on OVDEval Datasets (2). Prediction results contradictory caption pairs (yellow box vs. green box) from the negation subset of OVDEval dataset. Each row displays (left) ground-truth boxes, (middle) baseline predictions, and (right) our predictions. Our model effectively reduces affirmative bias, no longer returning identical bounding boxes for captions that express opposite meanings.
34

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
GT
Baseline
Ours
"clothed dog" "
"hanger without clothes" "
"a bed without patterns on it
"horseman without helmet"
person the conference table who is not seated
Swan not in the water
Figure S25: Qualitative Results on 3 Datasets (1), Absence of bounding box shows the model has determined that instance in the image matches the input description. By filtering such invalid predictions, our approach reduces affirmative bias and lowers the false-positive rate.
35

---

What "Not" to Detect: Negation-Awar VLMs via Structured Reasoning and Token Merging
GT
Baseline
Ours
"surfboard not on the beach"
"surfboard not on the beach" "
Air
"helicopter not flying in the air"
CATHAR PACIFIC
CATHAS PACIFIC
CAIHAR PACYFIC
"aircraft not on the ground" "
hippo whose mouth are not open
hippo whose mouth are not open
Figure S26: Qualitative Results D Datasets (2). Absence of bounding box means the model has determined that no instance in the image matches the input description. Our model effectively reduces the affirmative bias while keeping the correct predictions.
36

---

What "Not" to Detect: Negation-Awarc VLMs via Structured Reasoning and Token Merging
GT
Baseline
Ours
"parrot not on human body "
"parrot not on human body" "
"a bed without patterns on it"
"a bed without patterns on it"
"otter not in the water"
Figure S27: Qualitative Results on 3 Datasets (3), Absence of bounding box means the model has determined that instance in the image matches the input description. Our model effectively reduces the affirmative bias while keeping the correct predictions.
37

---

What "Not" to Detect: Negation-Awarc VLMs via Structured Reasoning and Token Merging
GT
Baseline
Ours
"a woman white wedding dress not beside any men in suits."
"origami unlike bird"
"a volley ball in the middle of the air untouched on the beach" "
"the person who was proposed to on one knee" "
Figure S28: Qualitative Analysis of Limitations on Complex Negation. Despite overall improve- ments, our method, like the baseline, still struggles with highly complex linguistic constructions involving negation. The examples show failures in: (i) multi-step relational reasoning ("not beside any men in suits"), (ii) abstract or implicit negation ("unlike bird"). (ii)) understanding negated states ("untouched"", and (iv) recognizing the absence of complex event ("proposed to on one knee"). In these challenging cases, both models tend to default to their affirmative bias, detecting the main subject of the query rather than correctly concluding that nothing in the image matches the full de- scription. These limitations highlight the need for more sophisticated compositional reasoning to ground complex negative constraints.
38


Reviews:
[{"review_id": "WUtyye0QpU", "reviewer": "Reviewer_gNbc", "summary": "This paper addresses the challenge of negation descriptions in the described object detection (DOD) task. To tackle this issue, the authors construct a large-scale training dataset COVAND that significantly raises the occurrence probability of negation semantics. Simultaneously, they propose NEGTOME which employs a novel token-merging module to enhance the model’s attention to negation semantics. Experimental results on the D3 and OVDEval datasets demonstrate the superior performance of the proposed approach.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Understanding negation is a significant and critical weakness of current Vision-Language Models (VLMs). This paper provides a rational and important perspective for addressing this issue by analyzing it from two key aspects — the insufficiency of existing datasets and the limitations of model tokenizers.\n- This paper proposes a large-scale training dataset, COVAND, specifically designed for negation semantics, along with its corresponding construction pipeline, providing an important data contribution toward addressing the negation problem.\n- This paper proposes NEGTOME, which enhances model's attention on negation at the tokenizer-level, offering an interesting and novel perspective for addressing the negation problem.\n- The paper is clearly written and easy to read.", "weaknesses": "- This work lacks comparision with SOTAs on the DOD task, for example, Real-Model (Re-Aligning Language to Visual Objects with an Agentic Workflow, ICLR 2025) which achieves 34.1 on D3 benchmark. The citations to such recent SOTAs are also missing. The compared methods in the paper are not sufficiently strong.\n- The detailed designs of NEGTOME are somewhat unclear. (1) It's not clear how \"softmax-weighted average\" (line 240) is performed. Specifically, what's the logits before softmax? (2) The description of \"Negation-aware Boost\" is somewhat confusing. Since the original vectors have been replaced by  \"softmax-weighted averaged\" ones (line 240), would equation (2) make any changes to tokens?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oRSS5frGBZ", "reviewer": "Reviewer_pMrC", "summary": "This manuscript identifies a critical flaw in state-of-the-art Vision-Language Models (VLMs): their inability to understand negation, often referred to as \"affirmative bias.\" For instance, VLMs frequently treat semantically contradictory phrases (e.g., \"person with skateboard\" vs. \"person without skateboard\") as equivalent. The authors quantitively validate this motivation by showing that negation words occupy only 0.04% of Flickr30k and 0.08% of LAION-400M—far lower than their prevalence in real-world language (13.76% in scientific papers, 22.23% in Conan Doyle’s stories).\nThe authors attribute this bias to two root causes: scarcity of negation data and architectural defects (low attention weights on negation tokens). To address these, they propose two core solutions: (1) COVAND, a synthetic negation-enhanced dataset built via Chain-of-Thought (CoT) reasoning and VQA-based alignment; and (2) NEGTOME, a text token merging module that binds negation cues (e.g., \"not\") with modified attributes (e.g., \"lying\") into coherent semantic phrases. This method achieves state-of-the-art (SOTA) performance on two benchmark datasets.", "questions": "1.\tRegarding Weakness 2: Have you compared this distribution to real-world corpora (e.g., COCA, biomedical literature) to validate its alignment with natural language? \n2.\tPlease refer to Weakness 3\n3.\tThe manuscript applies LoRA only to deep cross-attention layers. Have you tested LoRA placement in shallow layers or all layers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tCompelling Motivation with Practical Implications: The authors clearly articulate the real-world risks of affirmative bias (e.g., misdiagnosing \"a tumor that is not malignant\" as malignant in medical imaging) and back it with quantitative data on negation scarcity in popular datasets. This not only highlights the urgency of the problem but also provides a clear roadmap for future research.\n2.\tReasonable and Targeted Method Design: COVAND’s three-step CoT pipeline and VQA-based region alignment directly solve \"attribute-level negation\" and \"label ambiguity\"—limitations of prior template-based negation data (e.g., NegCLIP). With 9.29% negation word frequency, COVAND effectively balances data quality and negation coverage. While NEGTOME’s token merging (binding \"not\" + \"lying\" into a single unit) and negation address the structural loss of negation cues in tokenization, deep-layer LoRA ensures parameter efficiency—aligning with real-world deployment needs.\n3.\tRobust Experimental Results: The method consistently improves performance across architectures.", "weaknesses": "1.\tLack of Transformative Innovation: The core components (CoT for data generation, Token Merging for text processing, LoRA for fine-tuning) are all mature technologies. The manuscript focuses on \"combining and adapting\" these tools rather than proposing a novel framework. This incremental design may limit its impact on the broader VLM community.\n2.\tOver-Reliance on GPT-4o Leads to Normalized Negation Expressions: GPT-4o’s generation is constrained by its training data, leading to over-reliance on \"regularized\" negation cues (e.g., \"without,\" \"not\") while underrepresenting flexible, real-world negation types (e.g., \"lacks,\" \"absent of,\" \"non-\"). For example, COVAND likely prioritizes attribute negation (e.g., \"a table without drawers\") over relational negation (e.g., \"drawers not beside the table\") due to prompt biases. This mismatch may hurt the model’s generalization to unstructured scenarios.\n3.\tSelf-Consistency Bias in GPT-4o-Based Validation: The third step of COVAND’s pipeline uses GPT-4o to verify the same captions it generates. This \"generator-as-verifier\" setup creates a loop: if GPT-4o misinterprets visual attributes (e.g., mistaking \"pink\" for \"red\") during generation, it will likely repeat the error during validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dK1x0tIfv2", "reviewer": "Reviewer_6vMb", "summary": "This paper addresses an under-explored limitation of VLMs: the inability to correctly handle negation. The authors introduce two key contributions:\n\n+ COVAND, a systematically generated dataset focusing on negation through a chain-of-thought (CoT) reasoning and VQA-based alignment pipeline, producing region-grounded caption pairs that explicitly encode presence/absence attributes.\n\n+ NEGTOME, a token merging method that binds negation cues  to their associated scope (usualy nouns), to increase model attention to negation.\n\nUsing LoRA fine-tuning, the approach significantly improves negation understanding across multiple negation-focused object detection benchmarks (e.g., +10.8 NMS-AP on OVDEval, +7.2 mAP on the D3 absence subset).\n\nOverall the paper is well-written, method is intuitive, and experiment is sound, but may not have a broader impact beyond negation understanding (or similar types of linguistics phenomenon) as it is hard to apply this method to the default training pipeline of VLM/LLM. Below I detailed some concerns that I hope the authors can clarify further.", "questions": "- Q1: VLM detectors architecturally ignore or undervalue negation cues => I think using attention scores is not robust as models usually encode information into the later token (like punctuation), so lower attention score on negation token does not mean that it does not understand negation, but negation information maybe encoded in subsequent token. \n\n- Q2: Follow up on W1 and W2, it would be nice to see an analysis of the results on unseen negation cues (no need , and results on a different domain/task.\n\n- Q3: Follow up on Q1, did the authors consider boosting the attention score for the negation cue token only, rather than merging it with the following noun tokens?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper tackles an underexplored yet fundamental issue in VLM. The proposed method include both data (COVAND) and architectural (NEGTOME) innovation, with clear ablation to show the contribution of each. \n- Both the dataset creation pipeline and token merging method can potentially be extended to other linguistics phenomena (quantifier, hedging).\n- Experiments are well conducted with good empirical results showing substantial improvement over baselines.", "weaknesses": "A general concern is on overlap between COVAND and the evaluation datasets:\n+ W1: If I understand correctly, COVAND and D3, OVDEval follow similar pattern in their captions (e.g. paired caption of X with Y/ X without Y), if they also have the same distribution of object types, would the gains be from training from more in-domain data rather than actually understanding negation? The task evaluated in 4.4 is image captioning from COCO and again may overlap with Flickr30k. \n\n+ W2: ...Each caption includes negation cues such as “no”, “not”, “never”, “without”, the prefix “un-”, or the contraction “n’t”. The cue list is open to keep language natural and diverse...: statistics on negation cue distribution is needed to show that the generations are diverse. I'm skeptical that the LLM is able to generate diverse patterns of negations if we don't explicitly include a list of cues. A small suggestion is to use existing taxonomies of negation to ensure we generate a diverse and systematic dataset.\n\nRelated work seems limited. Perhaps more coverage of the literature should be included in the appendix", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)