CF-VLM:CounterFactual Vision-Language Fine-tuning
Authors: Jusheng Zhang
Kaitong Cai
Yijia Fan
Jian Wang
Keze Wang
Venue: NeurIPS 2025 poster ()
Keywords: VLM
causal
clip
Primary area: applications
TL;DR: VLMs lack deep causal reasoning. CF-VLM, via counterfactuals & novel training, boosts causal logic. It surpasses SOTA in reasoning/generalization, cuts hallucinations, aids real-world VLM use.

Abstract: Recent advances in vision-language models (VLMs) have greatly improved cross-modal semantic understanding, yet significant limitations remain in fine-grained discrimination and deep causal reasoning tasks. Existing VLMs often rely on superficial statistical correlations, lacking the ability to capture the underlying causal logic between visual and textual content. To address this, we propose the **CounterFactual Vision-Language Fine-tuning Model (CF-VLM)**, a novel framework that enhances the causal reasoning capabilities of VLMs through the targeted use of counterfactual samples. CF-VLM introduces three complementary training objectives: maintaining foundational cross-modal alignment, reinforcing the uniqueness, and stability of factual scene representations against coherent counterfactuals, and sharpening the model’s sensitivity to minimal but critical causal edits. Extensive experiments demonstrate that CF-VLM consistently outperforms strong baselines and state-of-the-art methods on compositional reasoning and generalization benchmarks. Furthermore, it shows promise in mitigating visual hallucinations, indicating improved factual consistency. Our CF-VLM provides a robust foundation for deploying VLMs in high-stakes, real-world scenarios requiring reliable reasoning and interpretability.

CF-VLM: Counterfactual Vision-Language Fine-tuning
Jusheng Zhang*l, Kaitong Cai*!, Yijia Fan', Jian Wang", Keze Wang
Sun Yat-sen University
Snap Inc.
'Corresponding author: kezewangogmail.co com
Abstract
Recent advances in vision-language models (VLMs) have greatly improved cross- modal semantic understanding, yet significant limitations remain in fine-grained discrimination and deep causal reasoning tasks. Existing VLMs often rely on superficial statistical correlations, lacking the ability to capture the underlying causal logic between visual and textual content. To address this, we propose CounterFactual Vision-Language Fine-tuning (CF-VLM), novel framework that enhances the causal reasoning capabilities of VLMs through the targeted use of counterfactual samples. CF-VLM introduces three complementary training objec- tives: maintaining foundational cross- modal alignment, reinforcing the uniqueness, and stability of factual scene representations against coherent counterfactuals, and sharpening the model's s sensitivity to minimal but critical causal edits. Ex- tensive experiments demonstrate that CF-VLM consistently outperforms strong baselines and satte-of-the-aee methods on compositional reasoning and generaliza- tion benchmarks. Furthermore, it shows promise in mitigating visual hallucinations, indicating improved factual consistency. Our CF-VLM provides a robust founda- tion for deploying VLMs in high-stakes, real-world scenarios requiring reliable reasoning and interpretability code.
1 Introduction
With the rapid advancement of vision- language models (HZ such CLIP [8), the capability of cross-modal semantic understanding has significantly improved, laying a solid foundation for diverse artificial intelligence applications [1]. However, current VLMs still exhibit fundamental limitations in complex visual-language tasks that demand fine-grained discrimination and deep reasoning 16] These models tend rely superficial statistical correlations, struggling to capture the underlying causal logic between visual and textual content [
17 8 Such limitations are particularly evident fine-grained recognition and causal reasoning scenarios, constituting core bottleneck that hampers the applicability of VLMs in high-stakes, real-world environments requiring robust reasoning 19].
To improve fine- grained alignment, recent efforts have extended contrastive learning strategies 20 22 commonly optimizing objectives like the triplet loss L == [m + S(A, S(A, P)]+, where ((.)) denotes similarity and m is a margin. These methods aim to enforce 5(A, P) > (SA, N) + m in the representation space [23]. improving discriminative performance. Nevertheless, their primary focus lies in binary distinction- whether pairs
than in understanding the causal attributes that explain how minimal differences in counterfactuals may lead to semantic shift 24. 26). Consequently, these models often fail to attend to the decisive attributes or relations that govern semantic matching. leading misinterpretation when faced with counterfactual perturbations.
To overcome these limitations, we propose CF-VLM (CounterFactual Vision- Laiiu-Le Fine-tuning), novel fine-tuning framework designed to enhance the causal reasoning capability of VLMs. Unlike
39th Conference on Neural Information Processing Systems (NcurIPS 2025)

---

prior approaches focused solely improving general discrimination, CF-VLM centers the learning process around counterfactual samples [27]. Specifically, counterfactual samples are crafted by applying minimal yet semantically crucial edits to an original image or its associated causal relations.
These edits alter the logical meaning scene while preserving proximity to the factual base. For instance, changing the color, category, status, or spatial configuration of core object, or reversing an action-outcome relation, can gen- erate counterfactual scenario. Though seemingly subtle, these modifications result in fundamental changes in the semantic match with the accompanying text. CF-VLM thus encour- ages the model to detect and reason about such causal decision points-the the critical attributes or relations that determine whether visual-textual pair truly matches. Building upon cross-modal alignment, CF-VLM introduces two progressive and complementary training objectives to model the contrastive relationships between factual and counterfactual data:
the
Please describe the image.
Trippett Clip
group of picture
Img
Triplet Clip
Training Strategy
Mon + Weren
Fielé + banch
On the field, mnch in blue locked the ball, and seared through
Triplet
On the besch, Noron licks
Too mony elements change??
Whichhi is I need?
CF-VLM
First, CF-VLM reinforces the unique- ness and stability of the anchor factual scene (Tanchur, Tanchor) by contrasting it with complete counterfactual scenarios (Tei, Tot.) ). where both image and text are jointly modified to represent logically consistent yet semanti- cally distinct alternative. The training objective enforces ([ Tanchor) > S(Iat, Ta:) for all thus establishing a clear reference and pre- venting representation confusion caused by the presence of numerous "parallel realities".
men and wemen pasing for a picture with hermil rackets
of
Img
CF-VLM
Training Strategy
weman lecke boll and doans
blut-
CP VLM
One change
helps
On the field, max blue kicke the ball and sears.
Freldre beach
recegnize 2e
Second and more critically, our CF-VLM fo- cuses enhancing the model's sensitivity to minimal causal edits-those that singularly af- feet the semantic validity of factual descrip- tion. In this setting, the factual image Jnat is paired with its original text Tiss (with similar- ity Staac Tia)),, while the same text is also paired with a minimally edited image Ler (with similarity S(er,, Tise)). CF-VLM aims to max- imize the margin S ((Ttaa, Tima) > S(Ta Tua), thereby guiding the model to focus on the seman- tic shift caused by critical causal modifications.
beach
better
Figure 1: Illustration of CF- CLM'ssss training frame- work. The model exposed both factual and counterfactual image-text pairs, where the latter introduce minimal but semantically decisive edits (e.g., "kick" - "not kick", "man" - "wommn")). In contrast to Triplet CLIP, which focuses on coarse-grained similarity (e.g., matching vs. non- matching)
2 Preliminary
Before detailing the CF-VLM framework, this section introduces foundational concepts, key no- tations, and the standard formulation of vision-language models (VLMss in cross-modal semantic understanding. Representation Learning in Vision-Languaa Models:The core objective of VL.Ms to learn joint multi-modal embeddings that align image and text representations in shared semantic space 20) Given an image I and its corresponding textual description T, typical VLM consists of two primary encoder components: Image Encoder fr(-): This module maps the input image / to a dd-dimensional visual embedding vector e₁ E RR = fr (I) Text Encoder (r)::: Analogously, the text encoder transforms the input textual sequence T into dd-dimensional text embedding vector er E Rdd /r(T). To ensure the comparability of these embeddings in the metric space and to stabilize similarity computations, both e, and er are typically T=z-normalized before being used in downstream contrastive learning objectives, resulting in unit-length representations.

---

2.1 Cross-Modal Similarity Measurement
The learned image embedding e₁ and text embedding er reside in shared representation space Within this space, semantically aligned image-text pairs are expected to exhibit high similarity, whereas unrelated pairs should demonstrate low similarity. This cross-modal semantic similarity, denoted as S(T, ?T), is typically measured via the cosine similarity between the normalized embedding vectors, which is mathematically equivalent to their dot product: S(1, T") = =ej er (T)) Il [T("))' In practice since both embedding vectors are 121nooralizee((1],, the above formulation simplifies to: (I1 e, e7. This similarity score serves as fundamental computational unit in the contrastive learning objective.
fr())-r((T)
2.2 Definition and Types of Counterfactual Samples
Our CF-VLMM leverages counterfactual samples for fine-grained supervision. While most existing vision-language learning methods rely on general data augmentation or negative sampling [33. such approaches often lack the capacity to systematically generate the minimal yet semantically critical interventions required for deep causal probing. In contrast, we define a counterfactual sample as new image-text pair, derived from anchor (T,) through precise, targeted intervention. These edits guided by prompt engineering for controlled and interpretable manipulation are specifically designed to enhance the model's understanding of key causal factors underlying image- text alignment. This focus distinguishes our CF-VLM from those methods that merely increase data diversity construct generic hard negatives. We concentrate two principal types of counterfactual interventions: (1) Single Object Attribute Modification: Targeting a core object in the image, this intervention modifies only one physical attribute time such color (e.g., red apple green apple), category (dog + cat), pose or state (standing + lying). spatial relation (left - right), or quantity. The objective to rigorously test and enhance the model's sensitivity to subtle yet semantically disruptive changes 34 (2) Key Causal Relationship Adjustment: This intervention modifies the key action, event outcome, or causal chain depicted in the scene (e.g.. changing "hits the ball" to "misses the ball," or "object falling" to "object floating"), directly challenging the model's s understanding of visual dynamics and physical plausibility. Based on these targeted intervention strategies, we construct two categories of counterfactual data structures for training:
Complete Counterfactual Scenario Pair (Lefa: Tefx)) Both the image and the text are jointly edited form logically coherent but factually distinct scenario from the anchor, facilitating the learning of plausible alternative realities.
Minimally Edited Counterfactual Image lof_edit; : A single critical intervention is applied to la. and the resulting Tef_ede _ff_edit, is paired with the original anchor text Ta as a hard negative sample, enabling fin-grained discrimination.
3 Methodology
CF-VLM Overview: Our CF-VLM is built upon a pretrained vision-language model and aims to refine its representational space by introducing carefully constructed counterfactual samples along with a set of targeted learning objectives (35ttivve[3333333 (35-39]. The central insight of CF-VLM is that merely learning to distinguish between matching and non-matching image-text pairs is insufficient. More crucially, the model must develop an understanding of why a given pair matches or not not-particllaryy when the differences are subtle and involve causally significant attributes 40 To this end, the overall training objective of CF-VLM, denoted as LCEVILM comprises three key loss components, each responsible for a distinct but complementary aspect of model enhancement: Maintaining the model's foundational cross-modal alignment capability via Traligni Strengthening the uniqueness and stability of anchor factual scene representations relative to plausible counterfactuals via Tesat Enhancing the model's 's causal sensitivity to semantic changes induced by minimal but critical edits via Traa- These components are integrated through a weighted LCF-LLF = a Lalign + 3 Lesd + 7 Lts where a, B, and y are hyperparameters that balance the contribution of each loss.
3.1 Counterfactual Data for Training
The organization of training data plays a pivotal role in fulfilling the learning objectives of CF-VLM. For each training iteration, our CF-VLM is provided with structured data unit comprising the
3

---

following elements: An anchor factual image-text pair (Iu, Ta); ; a set of K complete counterfactual scenario pairs (Tery: Teta )k = = where each (Icfk- Tess is derived from targeted modification of (Ia, Ta) and collectively forms a new, internally coherent counterfactual scene; and a set of J minimally edited counterfactual images lof.dit,j = 1", where each Iefadit, , is generated by applying single critical semantic edit-such as modifying an object attribute or altering a causal relation- the anchor image La- This counterfactual data structure enables CF-VLM to perform targeted supervision across multiple levels of semantic granularity [4144), ranging from basic alignment to causal sensitivity.
3.2 Core Learning Objectives of CF-VLM
Foundational Cross-Modal Alignment Loss. To ensure that CF-VLM preserves the foundational image-text alignment capability of the pretrained vision-language model during fine-grained tuning, we incorporate a standard contrastive loss. This loss is consistent with the pretraining objectives used in models such as CLIP and aims pull semantically relevant image-text pairs closer in the shared embedding space, while pushing apart unrelated pairs. Given a training batch of N factual image-text pairs {(1,, T; let es, and er; denote the normalized embeddings of image and text T, respectively. Their similarity score is computed as: S(T,, T2) = eT, eT, The foundational alignment loss aliggn is defined using symmetric InfoNCE objective:
1
(Xp(S(T,, Ti)TT) + log exp(S(T,, T:)/T) ΣE=1 exp()) T,)/T)
Lalign
log
2N
Σ (Sp(s((,,T.)/11
Here, T,) denotes the cosine similarity between the normalized image embedding el, and text exp(S(I..Tt/++)
embedding yTy. The first term inside the sum, - log Σ exp(5((,77))/v)' encourages each image I; to be most similar to its paired text T, compared to all other texts in the batch. The second exp(S(L.T))/r
term, - -log -;= exp(S((,.T.)/T))' encourages each text Ti to be most similar to its paired image 4, compared to all other images in the batch. The temperature parameter 7 controls the sharpness of the distribution. This symmetric formulation ensures that both image-to-tex and text-to-image alignments are jointly optimized.
Counterfactual Scenario Discrimination Loss. After establishing basic image-text alignment, CF-VLM introduces the loss term Les to reinforce the uniqueness and stability of the representation of the anchor factual scenario. The core idea is that the model should not only understand the current factual instance but also clearly distinguish it from logically coherent yet semantically different parallel counterfactual scenarios.
Given an anchor image-text pair Va,Tal. and a set of K complete counterfactual scenario pairs {(Lefe Tefs ) derived from it, the Lesd loss enforces that the similarity score S(1a, Ta) of the anchor pair is significantly higher than that of any counterfactual pair [[Lesk,Tefe The loss is implemented as hinge loss:
Lesd
max Tefe) S(la: Ta) m1)
Here, S(T,, Ta) denotes the similarity between the anchor image-text pair, while S(Tefe Tefe repre- sents the similarity of the k-th counterfactual pair derived from the anchor. For each counterfactual scenario, the loss encourages the similarity of the anchor pair exceed that of the counterfactual by at least a margin m. If the counterfactual pair is too close to or even exceeds the anchor in similarity, the hinge loss becomes positive and penalizes the model. When the anchor similarity already surpasses all counterfactuals by at least m1, the loss is zero. This mechanism enforces a clear separation between the factual anchor and its plausible but semantically different counterfactuals in the embedding space, thereby improving the model's discriminative power. Fine-Grained Causal Discrimination Loss. The loss term Lica represents the core innovation of the CF-VLM framework with respect to causal understanding. It is specifically designed to enhance the model's 's ability to detect and reason about changes in image-text matching relationships that arise from minimal but semantically critical edits, such as single attribute modifications or causal relationship reversals, as defined in Section 2.3. Given an anchor factual image-text pair (Ia, Ta) and a set of minimally edited counterfactual images (lef _ddit; 1/-1-ean each generated via an atomic semantic intervention on Ia- the objective of Lidd is to ensure that the similarity score S(La T.) is significantly higher

---

Image Encoding
Loss
Ori img img
Counterfactus Loss
Counterfactual Data Generation and Preprocessing
Base Model
ori img
Ori text
girl blue clothes runs in the park.
lo dertal
cf text
ori Text
Text Encoding
Counterfactual Loss
girl in blue clothes runs the park
Nest
Simiio
A boy blue clothes nuns park.
red clothes runs park.
Stable biffusion XL
Sm
A girl blue clothes walks park.
A girl in blue clothes runs the beach.
TiLi
TeL
S (Tori Tan)
Lagg
L
Test 1046
Lawe Test
Figure 2: CF-VLM training pipeline. Given a factual image-text anchor, the framework generates complete and minimally edited counterfactual images using fine-tuned SDXL model. The model optimized via three complementary objectives: cross-modal alignment (Lalign), counterfactual scenario diserimination (Lesd), and fine-grained causal discrimination (Lfea) enhancing semantic precision and causal sensitivity.
than that between T. and any Lef_edity- To strengthen training effectiveness, we focus on the most confusing counterfactual (i.e., the one with the highest similarity to T.), thereby forming hard negative sample. The loss adopts hinge formulation:
Liat max (0,
max S(Jef Ta) S(T,, Ta) + m2 jE(l..J)
Here, S(T,, Ta) is the similarity between the anchor image and its paired text, while S(Tefadit, Ta) denotes the similarity between the anchor text and the j-th minimally edited counterfactual image. Among all counterfactual images, only the one with the highest similarity to Ta (i.c., the hardest negative) is considered the loss calculation. The hinge loss encourages S(La, T,) to exceed this hardest negative similarity by at least a margin m2; otherwise, the loss penalizes the model. If all counterfactuals are sufficiently less similar than the anchor, the loss is zero. This mechanism ensures the model can robustly detect and distinguish subtle but semantically important changes introduced by minimal edits, thereby enhancing its fine- grained causal reasoning capabilities.
Experiment
We systematically evaluate our CF-VLM for vision-language models, i.e., Qwen-VL (7B) 44 and CLIP-ViT-B/22, with focus enhancing complex visual reasoning. The experimental setup covers (8.6M image-text pairs) and the CC3M [46] (120K pairs) optionally included for analy- sis. For each image-text pair, one counterfactual sample is generated by our dynamic counterfac- tual generation (DCF) strategy, effectively doubling the training data. Evaluation is conducted on compositional and generalization benchmarks. including the Conme 448 ARO [49], VL-
pretraining and fine-tuning on filtered CC12M (45)
subset (2.6M pairs), with MSCOCO Captions
47
Checklist (50), ImageNet-IIe 1k [5I] (zero-shot classification), and MSSCCO/Flickr30k [52 (zero-shot retrieval). Comparisons include Zero-shot Qwen- VL (7B), Standard Fine-tuning Text- Text-Negative Fine- tuning, leading CLIP-based models (TripletCLIP [2 CE-CLIP+ (33. COGT-CLIP (44, NegCLIP, Structure- PPucttre-CLIIP (55) L.LM-based VLMM (LLaVA-1. 1.5 (56), InstructBLIIP (57), MiniGPT-4 (58)),
and our CF-VLM (Qwen-V VI. fine-tuned on CC12M+DCF or CC3M+DCF). The counterfactual texts generated by Qwen2-72BB Inn-trut with 3-shot chain-of-thought prompting, and images

---

Table 1: Performance of CLiP-based SOTA vision-language models.
Mathed
Otp
Replace-Arr
Replax-lee Rel
Connz (Aag)
VG-Rdl
VG-Aar
(Aog)
Atniuz
Object
Reistion
VL-Checkäe (Arg)
NaCCIP (VIL-800, OC11M)
50.9
15.4
79.2
70.1
TipleCLIP (8T CC12M)
53.2
17.4
17.9
96.2
82.4
(FSCTIP- (MTF1/ OCTM+VG)
57.5
57.
53.8
56.4
13.6
76.2
75.9
76.3
85.2
75.3
78.9
Smactor- (ME-R0 CC12M)
S8.1
S8.1
53.8
35.87
55.2
$4.05
80.2
96.3
830
COGT-CIIP (NI--012, (1T176)
588
6017
54.3
57.9
18.2
67.75
66-2
81.4
35.4
510
Zemodax (MFB802)
50.9
51.7
45.5
49.73
59.7
51.9
60.8
6822
81.9
63.9
71.3
FC (NT CC12M)
35.7
5644
51.2
58.2
71.2
71.4
74
CR-VLM VT---2.. CC12M
60.4
00.4
56.6
59.33
90.3
88.4
87.6
844
Table 2: Performance of LLM-based VLMs, including Qwen- VL and LLaVA variants.
Methed
Replace Oij
Replace-Arr
Replaco Rel
(Aog)
VG
VG-AAr
ARD (hog)
Antiteso
Ofject
Relation
VI-Chedito
Inaasttl IP (Pxx Reponet)
19.0
$3.07
70.1
37.4
76.95
11.3
07.1
06.91
MicGPP- (TB Vioass Reyustal:
73.2
76.7
14.73
68.3
57.4
52.5
11.4
23.6
06.4
50.47
Zaro
78.3
Sandbrd IT (QNEn-VL OC12M)
K34
12.1
1223
82.6
16.3
8712
15:75
67.4
37.9
1953
N7.87
Tosthig (Qeee-11.
CFNLM (Ours, QL VL. 7B, OCIIM)
883
$7.5
86.9
87.57
91.8
946
93.3
89.6
98.7
91.4
90.57
LL Repusted)
65.4
73.2
71.
Sundbadd OLLAVALLA CESIN)
1664
774
12.4
723
Rishhng AVA-LA (CIIM)
623
610
09-
796
31.29
67.1
818
263
75.01
Refiner (40+15 step schedule). Models trained using Adam (81 = 0.9, B2 = 0.98, 6 = 1 x 10- peak learning rate of x 10-5 (cosine decay, 500 warmup steps), weight decay 0.1, and batch size 256 on single NVIDIA A100 (bf16). Training runs for 200K steps (CC12M) or 90K steps (CC3M), with each batch containing a 1:1 ratio of factual and counterfactual samples. The loss symmetric triplet contrastive loss C = G + CT-1 with a learnable temperature (init 0.07). We report main results, ablations, and analyses to show how counterfactual data aids hallucination mitigation and demonstrate CF-VLM's advantages over SOTA methods. For evaluation, compositional reasoning benchmarks use accuracy; ImageNet- Ik zero-shot classification is measured by top-1 and top-5 accuracy; image-text retrieval (MSCOCO, Flickr30k) is assessed via Recall @ 1/5/10. All results are averaged over three random seeds, with standard deviations reported.
use SDXL 1.0 Base 59
4.1 Main Results and Analyses
Compositional Reasoning Performance. We conduct a comprehensive evaluation of CF-VLM on key compositional reasoning benchmarks, including CONME, ARO, and VL-CHECKLIST, and com- pare its performance against both baseline models and relevant state-of-the-art (SOTA) approaches. The results are summarized in Table[[and Ta Performance Gains in CLIP--baed Models. As shown in Table [ CF-VLM (ViT-B/32, CC12M) outperforms all Cased-based baselines. On Conme, it achieves 59.1 193%% accuracy, exceeding Std FT by +4.7 points and surpassing TripletCLIP/CE-CIIII by +2.6/+2.7 points. On Visual Genome, it obtains $8.4% (VG-Rel) and 90.3% (VG-Attr), out- performing previous models. CF-VLM also achieves 88.4% on the VL-Checklist, showing strong robustness across benchmarks. Transferability and Advantage in LLM-based Architectures. As shown in Table 2 CF-VLM delivers strong results with achieves 87.57% % on Conme, improving over Std FT by +4.9 and over TextNeg FT by +3.4 points. On VG, it reaches 91.8% (Rel) and 94.6% (Attr). CF-VLM also transfers well to LLa LLa con- sistently outperforming Std FT and TextNeg FT, and surpasses strong instruction InstructBBLIP and MiniGPT-4 across most tasks, demonstrating broad generalization and adaptability.
models. For Qwen- VL 7B, it
models like
Generalization Performance on Standard Vision-Languaae Tasks. As shown in Figure 33] CF- VLM achieves strong generalization on standard tasks, including zero-shot ImageNet- lk classification (top-1: 8. 8.83%, top-5: 19.54%) and cross-modal retrieval (R @5: 5: 13.80% on MSCOCO(Avg), 26.58% on Flickr30((Avg)) outperforming or matching CLIP-based SOTA and relevant baselines.
4.2 Ablation Studies
Effectiveness of Counterfactual Sample Generation. To assess the impact of counterfactual batches on multi-modal causal reasoning, we design three ablation baselines: (1) Non-causal Baseline: training with original pairs and random negatives; (2) Text-only Counterfactual: counterfactual

---

MSCOCO
more
Figure Generalization results. Left columns): CF-VLM boosts baseline performance on classification and retrieval. Right (2 columns): LLM-based evaluation shows improved semantic alignment and Recall@5 over standard finetuning.
Table 3: Performance comparison of different ablation groups on causal reasoning benchmarks.
CF-VLM consistently outperforms all baselines across tasks.
Model
ConMe (Avg)
Replace-Ohj
Replace-Attr
Replace-Rel
ARO (Avg)
VG-Rel
VG-Attr
Non-causal Baseline
54.4
55.7
56.4
$1.2
71.2
74.2
68.2
Text only Counterfactuals
55.9
57.3
57.6
52.8
78.65
81.9
75.4
Image-only Counterfactuals
56.1
57.1
$7.9
53.4
78.25
82.2
74.3
CF-VLM
60.4
60.4
89.35
88.4
90.3
texts with original images; (3) Image-only Counterfactual. counterfactual images with original texts. All experiments use the CF-VLM framework with consistent settings. As shown in Table 3 the full CF-VLM model with both image and text counterfactuals achieves the best results all benchmarks (e.g., ConMe: 59.1, ARO: 89.35), outperforming all baselines.
Table 4: Ablation study of different loss components the ConMe dataset. The full model with all three loss terms achieves the best average performance.
Foundational Cross-Modal Alignment Loss
Counterfactual Sccnario Discriminution Loss
Fine-Grained Causal Discrimination Loss
Replace-Ohi
Replace-Att
Replace-Rel
Avg
59.4
59.2
55.1
$7.9
59.3
59.4
55.4
58.0
57.4
57.8
53.5
56.2
56.6
53.1
55.4
60.4
60.4
$6.6
59.1
Effectiveness Counterfactual Causal CLIP Losses
To evaluate the impact different loss components, we perform an ablation study on ConMe using: (1) Foundational Cross-Modal Alignment Loss, (2) Counterfactual Scenario Discrimination Loss, and (3) Fine-Grained Causal Discrimination Loss. We test various configu configurations by adding omitting each loss. The full model with all three losses achieves the best average accuracy (59.1), with 60.4 on Replace-Obj. 60.4 on Replace-Attr, and 56.6 on Replace-Rel Removing any loss reduces performance, e.g., excluding fine- fine-graing gree-graindd causal discrimination drops Replace-Rel by 1.5 points (average 57.9). These results demonstrate that combining alignment, counterfactual, and causal objectives is crucial for compositional reasoning in multi-modal models.
4.3 Effect of Counterfactusl Supervision Compositional Generalization
Impact of Counterfactual Types on Task-specific Accuracy. To further investigate the source of performance gains, we analyze the contribution of different types and quantities of counterfac- tual samples. The left of Figure reports the results of selectively removing specific types of

---

Peplace Onj
Peplico AIM
Raplo0o Rd
51.3
11.0
800
Figure 4: Removing any counterfactual type hurts task accuracy (left); more counterfactuals improve performance with diminishing returns (right).
counterfactual examples namely, attribute-level (Attr), object-level (Obj), and relation-level (Rel)
modifications -rrmm CF-VLM trained with ViT-B/32 on the ConMe benchmark.
The results indicate that omitting any single category of counterfactuals leads to a noticeable per- formance drop on the corresponding subtask. For instance, removing attribute-level counterfactuals reduces accuracy on the Replace-Attr task by 2.2 points (from 60.4 to 58.2), while removing object- and relation-level samples results in decreases of 2.9 and 3.3 points on Replace-Obi and Replace-Rei respectively. The overall average accuracy also declines accordingly.
Effect of Counterfactual Quantity: Tradeoff Between Performance and Efficiency. We further examine the effect of counterfactual data quantity on model performance. As shown in the right of Figure 4 the average accuracy on the ConMe benchmark increases steadily as the proportion of counterfactual samples (relative the number of original positive samples) grows from 0% (Standard Fine-tuning, 54.43%) to 100% (CF-VLM, 59.13%).
The performance curve begins plateau between approximately 60% and 80% counterfactual ratio, suggesting diminishing marginal gains as more counterfactual data is introduced. This observation highlights a practical tradeoff:: in real-world scenarios, one can balance performance improvement and computational cost by selecting an appropriate volume of counterfactual supervision.
4.4 Object-Level Hallucination Evaluation
Random Satting
Papular Satting
Advencarial Setting
Irccaion
Acamo
Ascutaa
Tscal
Mesicc
Natacc
Figure 5: Hallucination evaluation on the POPE benchmark. CF-VLM improves accuracy, precision, recall, and Fl on the object existence task over baselines. To further assess the practical applicability of CF-VLM, evaluate its potential in mitigating visual hallucinations-a long-standing challenge for vision-language models. Although CF-VLM is not explicitly designed optimize hallucination suppression, we hypothesize that its enhanced capability for fine-graing -reigga semantic modeling may indirectly improve factual alignment. To validate this hypothesis, we conduct evaluations on two widely adopted hallucination benchmarks: POPE (60 and MME [61]. As shown in Figure 5
POPE's object existence task, Standard FT (Qwen- VL, CC12M) signifi- cantly reduces the false positive rate compared to the zero-shot baseline, demonstrating the factual benefits of supervised fine-tuning. Building upon this, CF-VLM achieves marginal improvements across all four metrics accuray-accuuray, precision, recall, and Fl score-we average gains of 0.5-1.2 percentage points.
MME benchmark results (Figure 6 show CF-VLM outperforming Standard FT on hallucination- sensitive subtasks (Existence, Count, Position, Color). Notable gains in Color (+2.0) and Position
8

---

Metric Comparison (Excluding MiniGPPT-4)
Exislence
Counti
Position
Inetural
Zeo
Sundad
P1
VLM
CF VLM
CF MMM
140
160
180
170
182
170
133
100
150
115
120
Commonsenge Reasoning
LM0b-1
Uale-15
LoWk
HenunttiP IP
Instruck
Zare Ovenvt
Owen
Zer SwanV
Soodeed
St F7
Stundars FT
Hallucination evaluation on the MME benchmark. Relative performance gains on MME show CF-VLM's s advantage hallucination-sensiivve tasks, especially for color and position, indicating enhanced semantic grounding via counterfactual training.
Figure 6:
(+1.4) highlight counterfactual supervision's benefit for attribute-level reasoning, likely due to minimal perturbations in attribute representations reducing hallucination.
Related Work
Recent advances in dision-languee models (VLMs) 62 64), such as CLIP ( and BLIP [. have demonstrated remarkable performance cross-modal tasks through large-scale image-text pretraining. prerannng. However, these models often struggle with fine- tne-ggained discrimination and causal reasoning, relying on superficial statistical correlations rather than capturing deep causal relationships between visual and textual content [O]. Existing approaches employ contrastive learning to enhance embedding space discriminability, such as triplet loss but primarily focus on binary matching, overlooking the impact of causal attribute changes semantics
25
Counterfactual learning has emerged as a promising approach address the causal reasoning limitations of VLMs [24] 65] (66). introduced counterfactual visual explanations by modifying key visual elements to analyze model decisions, while CPL (25) leveraged counterfactual prompts to improve cross-modal understanding. However, these methods typically focus on single- mogality counterfactuals, lacking joint image-text counterfactual designs [67] and using singular objectives that fail to leverage counterfactual samples for multi-level semantic supervision.
Our CF-VLM framework advances VLM causal reasoning by integrating joint image-text counterfac- tual samples with three complementary training objectives: cross-modal alignment, counterfactual scenario discrimination, and fine- gre-grained causal discrimination. Unlike TripletCLIP optimizes coarse- -rasind similarity, CF-VLM targets causal decision points, achieving superior accuracy on benchmarks like ARO, ConMe, and VL-Checklist, while reducing visual hallucinations, demonstrating enhanced robustness and applicability in high-stakes scenarios.
22 which
Conclusion and Limitations
a counterfactual fine-tuning framework to enhance semantic granularity and causal reasoning in vision-languag models (VLMs). Addressing limitations in distinguishing subtle semantic differences, CF-VLM uses minimally edited image-text pairs from a fine-tuned SDXL pipeline, introducing counterfactual scenario discrimination and fine- ane-grained causal discrimination objectives. Evaluations on compositional reasoning (ARO, ConMe, VL-Checklist) and hallucina- tion benchmarks (POPE, MME) show CF-VLM outperforming CLIP-based and LLM- --auumeeeted VLMs, improving factual grounding and reducing hallucinations, even on untrained benchmarks, demonstrating the generality of counterfactual supervision. Future work includes integrating human- in-the-loop editing for richer counterfactuals, applying CF-VLM tasks like VQA image editing, and exploring interpretability via counterfactual sensitivity for more transparent VLMs.
We propose

---

Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62276283, in part by the China Metcorological Administration' 's Science and Technology Project under Grant CMAJBGS202517, in part by Guangdong Basic and Applied Basic Research Foundation under Grant 2023A151500 2985, in part by Guangdong- Hongn Kong-Macao
Macao Greater Bay Area Meteorological Technology Collaborative Research Project under Grant GHMA2024204, in part by Fundamental Research Funds for the Central Universities, Sun Yat-sen University under Grant 23hytd006, and in part by Guangdong Provincial High-LLevel Young Talent Program under Grant R1.2024-151--111
References
[1] J. Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation," , 2022. [Online]. Available: https:/aarxi.or///s/2220.... 1206
[2] Y. Du, Z. Liu, Li, and W. X. Zhao, "A survey of vision-language pre-trained models," " 2022. [Online]. Available: https//arxiiv.or//ss/2222..000
[3] J.-B. Alayrae, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, R. Ring, F. Rutherford, S. Cabi, T. Han, Z. Gong. S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisser- man, and K. Simonyan, "Flamingo: a visual language model few-shot learning," 2022.
[4] J. Ma. "Causal inference with large language model: A survey," 2025. [Online]. Available: https//awxv.orrgds//20....99
M. Chen, B. Peng, Y. Zhang, and C. Lu, "Cello: Causal evaluation of large vision-language models," 2024. [Online]. Available: https://arxvv.org/////466666 19131
[6] J. Zhang, K. Cai, J. Yang, and K. Wang, "Learring dynamics of ylm finetuning," 2025. [Online]. Available: https//arvv.org/ah225....1
Zhang, Y. Fan, K. Cai, Yang, J. Yao, Wang, G. Qu, Chen, and Wang, "Why keep your doubts to yourself? trading visual uncertainties in multi-agent bandit systems," 2026. [Online]. Available:
https://axiv..r//abb//b..LL..
[8] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," " 2021. [Online]. Available: httpss//artv.or//aa111.....00
[9] J. Zhang, K. Cai, Y. Fan, N. Liu, and K. Wang, "Mat-agent: Adaptive multi-agent training optimization," 2025. [Online]. Available: http:://arrw.on///a//555.......0 17845
[10] E. Bugliarello, Sartran, A. Agrawal, A. Hendricks, and A. Nematzadeh, "Measuring progress in fine-grained lision-and-languse understanding." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 1559-1582. [Online]. Available: https//addamttooooyyyoo/2...... 20tgg/222..clloog...11
[11] Peng, S. Xie, Z. You, Lan, and Z. Wu, "Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding." in 2024 IEEF/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2024. pp. 13279-17 288.
[12] J. Kim and H. Ji, "Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, Florida, USA: Association for Computational Linguisties. Nov. 2024, pp. 6187-6207. [Online]. Available: main.356/
https://ccanthhoooyyogg22222
[13] Z. Xu. Y. Zhu, S. Deng. A. Mittal, Y. Chen, M. Wang, P. Favaro. J. Tighe, and D. Modolo, "Benchmarking zero-shot recognition with vision-language models: Challenges on granularity and specificity." in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2024, pp. 1827-1836.
10

---

[14] W. Wang, Z. Yang. B. Xu, J. Li, and Y. Sun, "Vilta: Enhancing vision-language pre-training through textual augmentation." in 2023 IEEE/CVF International Conference Computer Vision (ICCV), 2023, pp. 3135-3146.
[15] J. Zhang, 7. Huang, Y. Fan, N. Liu, M. Li, 7. Yang. J. Yao, J. Wang, and K. Wang. "KABB: Knowledge-aware bayesian bandits for dynamic expert coordination multi-agent systems," in Forty-second International Conference on Machine Learning, 2025. [Online]. Available:
[16] J. Zhang, K. Cai, X. Guo, S. Liu, Q. Lv, R. Chen, J. Yang. Y. Fan, X. Sun, J. Wang, Z. Chen, L. Lin, and K. Wang. "Mm-cot: benchmark for probing visual chain-of-thought reasoning in multimodal models," 2025. [Online]. Available: http://aarkv.org//b225222222222
[17] V. Palit, R. Pandey, A. Arora, and P. P. Liang. "Towards vision-language mechanistic interpretabilityy A causal tracing tool for blip." " in 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023, pp. 2848 -853..
[18] Zhang, X. Guo, K. Cai. Q Lv, Y. Fan. W. Chai, J. Wang, K. Wang, "Hyyridtoken-VImm Hybrid token compression for Laiiu-language models," 2025. [Ouline]. Available: hutps://axxiv.rr//bb//Z2
[19] H. Laurençon, L. Tronchon, M. Cord, and V. Sanh, "What matters when building vision-language models?" 2024. [Online] Available:
[20] M. Wahed, X. Zhou. T. Yu, and I. Lourentzou, "Fine-grained alignment for cross-modal recipe retrieval," in 2024 IREF/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 5572-5581.
[21] A. D' D'Innocente, N. Garg, Y. Zhang, L. Bazzani, and M. Donoser, "Localized triplet loss for fine-grained fashion image retrieval," in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 9905-3910.
[22] M. Patel, A. Kusumba, S. Cheng, C. Kim, T. Gokhale, C. Baral, and Y. Yang. "Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives," " 2024. [Online]. Available: https:/aawi..rr//aas//4111..33.0
[23] F. Schroff, D. Kalenichenko, and Philbin, "Facenet: unified embedding for face recognition and clustering," in 2015 IEEF Conference on Computer Vision and Pattern Recognition (CVPR). TEFE, Jun. 2015, p. 815-823. [Online]. Available: http://dx..oo..org/ 10.
[24] Y. Rao, G. Chen, J. Lu, and J. Zhou, "Counterfactual attention learning for fine-grained visual categorization and re-identification"" in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 1005-1014 1014.
[25] X. He, D. Yang, W. Feng, T.-J. Fut, A. Akula, V. Jampani, P. Narayana, S. Basu, W. Y. Wang, and X. Wang, "CPL: Counterfactual prompt learning for vision and language models," in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Goldberg, Z. Kozareva, and Y. Zhang. Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, main. 224/
pp. 3407-3418. [Online]. Available: https:/aclaathooooy..g.//
[26] J. Zhang, K. Cai, Q. Zeng, N. Liu, S. Fan, Z. Chen, and K. Wang, "Failure-driven workflow refinement," 2025. [Online]. Available: http:://arxivoogg/bb///1....0.
[27] Y. Goyal, Z. Wu, J. Emst, D. Batra, D. Parikh, and S. Lee, "Counterfactual visual explanations," 2019. [Online], Available: httpss//arrxivoorg/abss 1904.04451
[28] A. Vaswani. N. Shazeer, N. Parmar. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and L. Polosukhin. "Attention is all you need," 2023. [Online]. Available:
[29] Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang. Y. Wang. W. Ye, Y. Zhang, Y. Chang, P. S. Yu, Q. Yang, and X. Xie, "A survey evaluation of large language models." " ACM Trans. Intell. Syst. Tecknol., vol. 15, no. 3, Mar. 2024. [Online]. Available: hups://doioorg 10. 145/3641289
[30] Y. Liu, Y. Zhang, Y. Wang, F. Hou, Yuan, J. Tian, Y. Zhang, Z. Shi, J. Fan, and Z. He, "A survey of visual transformers," IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 6, pp- 7478-7498, 2024.
[31] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations." in Proceedings of the 37th International Conference on Machine Learning, ser. ICML'20 20. JMLR.org, 2020.
11

---

[32] T. Wang and P. Isola, "Understanding contrastive representation learning through alignment and uniformiiy the hypersphere." " in Proceedings of the 37th International Conference Machine Learning, ser. ICML'20.
Jog.. 2020.
[33] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, 7. Li, and T. Duerig, "Scaling up visual and vision- laniuag representation learning with noisy text supervision." " 2021. [Online]. Available:
hmtps//aariiv.orgdbs22212....
[34] N. Kim and T. Linzen, "Cogs: A compositional generalization challenge based on semantic interpretation." " 2020. [Online]. Available:
0.06555
[35] D. Teney, E. Abbasnedjad, and A. van den Hengel, "Learning what makes a differrncce from counterfactual examples and gradient supervision," in Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X. Berlin, Heidelberg: 580-599. [Online]. Available: htps://ooi.org// 1wt.:///rg/0000007/-----00000000 (007/998-33033-58607-22
2020, p.
[36] R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel, "Teaching clip count ten," in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3147-3157.
[37] Z. Chen, Q. Gao, A. Bosselut, A. Sabharwal, and K. Richardson, "Disco: Distilling counterfactuals with large language models," 2023. [Online]. Available: https://ariivoorr/ab/22222..1.
[38] J. Zhang. Y. Fan, K. Cai. Z. Huang, X. Sun, J. Wang. Tang, and K. Wang. "Drdiff: Dynamic routing diffusion with hierarchical attention for breaking the efficiency-qualie trade-off." 2025. [Online]. Available: https://arxi..orr/a/////// http://o.rrrrba//2200000...
[39] J. Zhang, Y. Fan, K. Cai, X. Sun, and K. Wang, "Ose: Cognitive orchestration through dynamic knowledge alignment in multi-agent Ilm collaboration," 2025. [Online]. Available: hups://arxivv. org/abs/ wttpaab///////eeee// 2509904876
[40] Y. Zhang, J. Wang, and J. Sang. "Counterfactually measuring and eliminating social bias in lasiun-language pre-training models," in Proceedings of the 30th ACM International Conference on Multimedia, ser. MM '22. New York, NY, USA: Association for Computing Machinery, 2022, p. 4996-5004, [Online].
Available: https://doi.org/ 10. http///..rrrr....14453555....... 141114/65//0001133333366
[41] Y. Rao, G. Chen, J. Lu, and Zhou, "Counterfactual attention learning for fine- ane-grained visual categorization and re-identification." " 202/ IEFE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 1005-1014.
[42] L. Chen, Y. Zheng, Y. Niu, H. Zhang, and J. Xiao, "Counterfactual samples synthesizing and training for robust visual question answering," JEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 218 13 234, 2023.
[43] J. Zhang, Y. Fan, W. Lin, R. Chen, H. Jiang, W. Chai, J. Wang, and K. Wang, "Gam-agent: Gaae--thereete and incertainty-awan collaboration for complex visual reasoning." " 2025. [Online]. Available: hutps://ariiv.or/////25555..3332
[44] Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, Huang, H. Wei, H. Lin, Yang. J. Tu, J. Zhang, J. Yang, J. Yang. J. Zhou, J. Lin, K. Dang. K. Lu, K. Bao, K. Yang, L.. Yu, M. Li, M. Xue, P. Zhang, Q Zhu, R. Men, R. Lin, T. Li, T. Tang. T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang,
Y. Wan, Y. Liu, Z. Cui. Z. Zhang, and Z. Qiu, "Qwen2.5 technical report," 2025. [Online]. Available:
http://aariv.orr//b//422....111
[45] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, "Conceptual 12M: Pushing web-scale image-text pre-training recognize long-tail visual concepts," in CVPR, 2021.
[46] P. Sharma. N. Ding. S. Goodman, and R. Soricut, "Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning," " Proceedings of ACL, 2018
[47] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick, "Microsoft coco captions: Data collection and evaluation server," 2015. [Online]. Available: hntpss//arxiv.orr/ab////t/ http://aaa/a/////.....
[48] I. Huang, W. Lin, M. J. Mirza, J. A. Hansen, S. Doveh, V. I. Butoi, R. Herzig, Arbelle, H. Kuchne, T. Darrell, C. Gan, A. Oliva, R. Feris, and L. Karlinsky, "Conme: Rethinking evaluation of compositional reasoning for modern vims," 2024. [Online]. Available: hntpss//ariiv.rrg/ah///4..6 1864
[49] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, "When and why dision-language models behave like bags-oft of and what to do about it?" in International Conference on Learning Representations, 2023. [Online]. Available: http://pwweeee..e/////omoodddddd
12

---

[50] T. Zhao, T. Zhang, M. Zhu, H. Shen, K. Lee. X. Lu, and J. Yin, "VI-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations," 2023. [Online]. Available: https//ariv.oogdds2220..00
[51] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla, M. Bem- stein, A. C. Berg, and I.. Fei-Fei, "ImageNet Large Scale Visual Recognition Challenge," " International Journal of Computer Vision (IJC)), vol. 115, no. 3. pp. 211-252, 2015.
[52] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, "From image deseriptions to visual denotations: New similarity metries for semantic inference over event descriptions." Transactions of the Association for Computational Linguistics, vol. 2. pp. 67-78, 2014.
[53] L. 7hang, R. Awal, and A. Agrawal, "Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding." 2024. [Online]. Available:
httpss/aaxiv.orrg/hs//2.......
[54] F. Parascandolo, N. Moratelli, E. Sangineto. L. Baraldi, and R. Cucchiara, "Causal graphical models for vision-language compositional understanding." " 2025. [Online]. Available: hutps://arxi..rr//a///442
[55] Y. Huang. J. Tang, Z. Chen, R. Zhang, X. Zhang, Chen, Zhao, Z. Zhao, Lv, Z. Hu, and Zhang, "Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representations" " 2023.
[Online]. Available:
[56] H. Liu, C. Li, Ii, and Y. J. Lee, "Improved baselines with visual instruction tuning," 2024. [Online]. Available: httpss//arkiw.rr//////23/......0
[57] W. Dai. J. Li. D. Li, A. M. H. Tiong, J. Zhao. W. Wang, B. Li, P. Fung, and S. Hoi, "Instructblip: Towards general-purposs vision-language models with instruction tuning." 2023. [Online]. Available: https://ariv..rrgaa///2355.
[58] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing lision-l understanding with advanced large language models," 2023. [Online]. Available: hntpss//artiv.orr//////2...0...0
[59] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhom, Müller, Penna, and R. Rombach, "Sdxl: Improving latent diffusion models for high resoluccootton image synthesis," " 2023. [Ouline]. Available: wttp:/www...rgaaa////..00000
[60] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.RR. Wen, "Evaluating object hallucination in large vasion-language models," 2023. [Online]. Available: https://arxx..orr/ab//2233. 103..
[61] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang. X. Zheng. K. Li, X. Sun, Y. Wu, and R. Ji. "Mme: A comprehensive evaluation benchmark for multimodal large language models," 2024. [Online]. Available: https://arriv..orr//a//2....... 13394
[62] S. Long, F. Cao. S. C. Han, and H. Yang, aad-o---nn--lnnguge pretrained models: A survey," 2022. [Online]. Available:
https://arxiv.orr//////2...0...0
[63] A. Ghosh, A. Acharya, S. Saha, Jain, and A. Chadha, "Exploring the frontier of vision-language models: A survey of current methodologies and future directions," 2024. [Online]. Available: https://arii.orrglhh/2244400
[64] J. Zhang, J. Huang. S. Jin, and S. Lu, "Vision-languaae models for vision tasks: A survey," IEEE Transactions on Puttern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5625-5644, 2024.
[65] F. Abhasnejad, D. Teney, A. Parvaneh, J. Shi, and A. van den Hengel, "Counterfactual vision and language lcarning," in 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp- 100
[66] M.-K. Xie, J.-H. Xiao, P. Peng, G. Niu, M. Sugiyama, and S.-J. Huang. "Counterfactual reasoning for lubti-label image classification via patching- based training." in Proceedings of the 41st International Conference on Machine Learning, ser. ICML 24. Jrg.. 2024.
[67] F. Parascandolo, N. Moratelli, E. Sangincto, L. Baraldi, and R. Cucchiara, "Causal graphical models for vision-language compositional understanding." 2025. [Oaline]. Available: https//arxovorry/aab22222...333
[68] X. He, D. Yang. W. Feng, T.-J Fu, A. Akula, V. Jampani, P. Narayana, S. Basu, W. Y. Wang, and X. E. Wang, "Cpl: Counterfactual prompt learning for vision and language models," 2022. [Online]. Available: https:/aanivorrraas/22....3222
13

---

[69] Y. Goyal, Z. Wu, J. Erss. D. Batra, D. Parikh, and S. Lee, "Counterfactual visual explanations, 2019. [Online]. Available: https://arxix.orr/abb////0......
[70] Y. Rao, G. Chen, J. Lu, and J. Zhou, "Counterfactual attention learning for fine-grained visual categorization and re-identification, 2021. [Online]. Available: [ttpss//arxi.oog//b22211151533
14

---

A
Dataset Details
Our training and evaluation are conducted on series of standardized image-text datasets and structured evaluation benchmarks, detailed below:
Training Data: We utilize the cleaned version of CC12M (8.6M image-text pairs) and its subset CC3M (2.6M pairs) as our primary training corpora. All samples undergo standardized preprocessing, including image resizing to 224 224, text normalization, and language alignment to ensure stable input distribution.
Counterfactual Sample Generation: For each original image-text pair, we generate a semantically similar but minimally perturbed counterfactual pair using our Dynamic Counterfactual Generation (DCF) strategy. This effectively doubles the training data and enhances the model's capacity to learn fine- seee-graied semantic variations through contrastive supervision.
MSCOCO Captions (120K pairs): This dataset is not used for training, but optionally included for auxiliary analysis tasks (e.g., image-text retrieval or structural generalization), assess the transferability of our method beyond the training corpus.
Evaluation Benchmarks: We evaluate the model's compositional reasoning and cross- modal generalization on the following benchmarks:
This benchmark focuses on the model's ability to distinguish subtle semantic changes in compositional logic, such as negation, sequence, and conjunction. Each sample presents an image with two candidate captions-one semantically correct and one minimally perturbed. The task evaluates causal reasoning and structural understanding.
ConMe Suite:
ARO Bearibute-Rlltioon-Oheeee This benchmark introduces targeted perturba- tions to image-text pairs along three dimensions: attributes, relationships, and object identity. It assesses the model's consistency in recognizing multi-dimensional semantic alignment, particularly in fine- tne-grained visual-linguistic scenarios.
This benchmark contains three sub-tasks Lub-tasks-Replacc--000 Replace-Objj Replace- Attr, and Replace-Red Rellace-Rel-where only one semantic element in the text is systematically modified. It designed to evaluate the model's sensitivity attribute- and relation-level counterfactuals.
VL-Checklist:
ImageNet-1k 1k (Zero-shot Classification): This task transforms the standard ImageNet classification problem into image-text matching setup. Each image paired with multiple candidate textual descriptions, and the model must select the most semanti- cally aligned one. This evaluates the model's ability to generalize to natural image distributions in zero-shot setting.
MSCOCO Flickr30k (Image-Text Retrieval): These retricval tasks involve bi- directional matching (image - text and text - image). We report Recall @ I and Recall @5 metrics to evaluate whether the model can accurately locate the correct match from large candidate pool.
Preprocessing and Splits: All datasets follow their official train/validationntess splits. During training, each batch maintains fixed 1:4 ratio between factual and counterfactual samples, enhance the model's s sensitivity semantically critical perturbations.
15

---

Owen2.5-VL-77
Owen+CF-VLM
A woman in wheelchair sitting next to parking garage.
brown dog running the porch of house.
woman wheelchair sitting next to wall with the word park 1" it.
golden retriever running on the porch of a house.
A man is playing with his dog in the park.
Two people hiking the woods waterfall.
A manin playing with his dog the park, and the dog is jumping to catch frisbee.
Two people are standing on mossy path near waterfall,
A young boy is riding a bicycle while young girlin light blue dress watches.
Two women sit on a train and look the window.
A young girl in a green dress stands boy an bicycle.
Two women sit on a train, one reading and the other looking out the window.
A dog is standing the grass and looking up.
A woman pushing her daughter on swing a park.
A dog running on the grass with mouth open.
is helping young cross rope bridge in park.
white jeans is riding blue tricycle.
A group of people are playing video game room.
A man a white shirt and blue jeans riding blue bike.
A group of people are dancing in middle of with other people sitting around them.
A woman in white shirt blue shorts is playing tennis with man a blue shirt and red shorts.
A young girl in a red shirt running with her dog the grass
A woman in white shirt and blue shorts playing tennis while man in a blue shirt and red shorts
A young boy in a red shirt and gray pants running with his dog
watches.
Figure Qualitative comparisons showing VLM'S improvements in fine-grained attribute recognition, causal action modeling, and relational understanding across diverse visual scenes.
16

---

Counterfactual Rewriting Prompt
You are rewriting engine specialized in generating counterfactual or counter-attribute descriptions.
Your task: Rewrite the input sentence by changing exactly one element either:
physical attribute of an object, or
causal relationship.
You may change one the following object attributes (be creative and visually expressive):
Color (e.g., red + blue)
Material (e.g., wond metal)
Shape (c.g., round + square)
Size (e.g., large + small)
Position or orientation (c.g., on the left - on the right)
State pose (e.g., standing lying)
Object category (c.g., dog + cat)
Parts or structure (e.g., legs + four legs)
Movement status jc.g...mpping still)
Quantity (c.g., two one)
Facial expression emotion (e.g., smiling + angry)
You may use imaginative or creative combinations as long as they are visually descriptive and logically coherent.
Alternatively, you may flip causal relationship:
1. Identify cause-effect link in the sentence.
2. Invert it: If cause didn't happen, effect B would change.
3. Rewrite the sentence naturally under the new causal logic.
Rules:
Modify only thing (either attribute or causal link).
Keep all other elements in the sentence exactly the same.
Do not repeat any previously generated sentence.
The output must he single fluent, grammatically correct sentence.
Do not include any explanation, comment, tag, or prefix.
Only return the rewritten sentence. Nothing else.
Format: Please output the K counterfactual sentences in numbered list:
[sentence #1]
[sentence #2]
3. [sentence #3]
Example Input:
A young woman holding racket hit the ball, and the ball flew outward.
Output (attribute):
1. An old woman holding a racket hit the ball, and the ball flew outward.
2. transparent woman holding a racket the ball, and ball flew outward.
Output (causal):
3. A young woman holding racket missed the ball, and the ball dropped to the ground.
Now return K distinct counterfactual rewrites in the required format.
17

---

B Qualitative Analysis of Post-training Performance
B.1 Diversity and Quality Control of Counterfactual Samples
CF-VLM employs a structured counterfactual editing strategy that includes four types of controlled interventions: attribute substitution, object substitution, relationship substitution, and location sub- stitution. As illustrated in Figure 8 attribute edits (c.g., replacing "dirt" with "paved roads") and object edits (e.g., changing "two men" "two women") enable the model learn from variations in properties and subject identities. Meanwhile, relationship edits (e.g., switching from "throwing pitch" to "catching ball") and location edits (e.g., changing the spatial configuration between a "cat" and "television") enhance the model's 's understanding of action sequences and spatial semantics.
These controlled counterfactuals enable CF-VLM cover wider range of semantic types-from from color and texture to interaction and spatial reasoning- -hhile avoiding issues commonly observed in large language model (LLLM)-based generation, such redundancy and logical inconsistency. This structured supervision improves both the semantic diversity and logical coherence of the training samples.
Figure 7 provides qualitative evidence supporting this claim. the top-left example, the baseline model incorrectly identifies the scene "a parking garage" and ignores the textual cue "park" on the wall, while CF-VLM accurately describes the scene with attribute-awarr recognition. In the second-left row, CF-VLM enhances the original caption "a man is playing with his dog" by adding "and the dog is jumping to catch a frishee," " capturing the entire action-outcome chain. Similarly, in the top-right image, the baseline produces generic "brown dog," while CF-VLM correctly identifies it as a "golden retriever," offering greater specificity. In another example, CF-VLM replaces "hiking near a waterfall" with "standing on mossy path near waterfall," yielding more grounded spatial reasoning. These examples demonstrate CF-VLM's clear advantage in fine-grained attribute recognition, action completion, and relationship understanding. The generated counterfactuals are both more diverse and semantically consistent than those produced by conventional methods.
B.2 Causal Depth
CF-VLM also demonstrates superior causal reasoning by explicitly modeling the structural compo- nents visual events, including agents, objects, action stages, and outcome states. During training, counterfactual interventions are applied in structured manner, enabling the model to learn coherent "intervention-effccr" causal chains.
As shown in Figure 7 CF-VLM captures rich event semantics across range of complex scenes. In the "man-dog-frisbee" scenario, CF-VLM identifies three-step causal sequence: "a man is playing with his dog" - "the dog is jumping to catch frisbee," exhibiting strong temporal and causal coherence. In contrast, the baseline merely describes surface-level actions without modeling the outcome.
In the "tennis" example, CF-VLM accurately distinguishes between the active player and the ob- server, generating the caption "a woman is playing tennis while man watches," thus capturing role-based causal dynamics. In other scenes such as helping child cross bridge or shifting spatial Selationships-Cr---e CF-VLM successfully tracks pre- and post-action semantic states, encoding progression and continuity within the event timeline.
These results indicate that CF-VLM not only improves action recognition but also accurately models temporal structures and objntt interactions. Overall, CF-VLM outperforms conventional base- lines in understanding complex visual events, capturing semantically consistent changes triggered by causal interventions. This structured causal modeling provides stronger foundation for generaliza- tion and discrimination in multimodal semantic tasks, validating the effectiveness of counterfactual supervision for compositional reasoning.
C Detailed Comparison with Related Work and Highlighting Unique Contributions
In order to clarify CF-VLM's unique contributions in enhancing fine-grained discrimination and advancing deeper causal understanding in vision languale models (VLMs), and to directly address
18

---

concerns about novelty, this section presents thorough comparison of CF-VLM against several key related works the core mechanism level. We aim demonstrate that CF-VLM is not a simple application of counterfactual samples but carefully designed, multifaceted framework to improve sensitivity to specific interventions and identify causal visual elements.
C.1 Mechanistic-Level Comparison
We compare CF-VLM and representative works works-including including TriplerCLIP, CPL [68, Goyal et al. 69
and Rao et al. [70] along four dimensions: the type and generation of counterfactual samples, core learning objective/loss design, focus on causality or fine-grained discrimination, and simultaneous handling of image and text counterfactuals.
Type and Generation of Counterfactual Samples CF-VLM adopts comprehensive strategy for constructing counterfactuals: it combines complete counterfactual scenarios where both image and text are jointly edited, with minimally edited image perturbations targeting single object attributes or causal relations (e.g., color, class, state). We use fine-tuned SDXL 1.0 for image generation and Qwen2-72B 1nnstrrruct for text editing to ensure semantic control.
In contrast:
Triple(CLIP(22) synthesizes visual- languae negatives via rule- or template-basee text perturbations to improve compositional reasoning.
CPL (68 generates counterfactuals by modifying textual prompts using predefined templates to enhance cross-modal robustness.
Goyal et al. [69 focuses on image-based edits of key elements for interpretability, not directly optimizing prediction performance.
Rao et al. (70 learns counterfactual attention by masking or replacing salient regions, calibrating attention maps for visual tasks.
Core Learning Objectives and Loss Design plementary losses:
CF-VLM's training objective integrates three com-
1. Cross-modal Alignment Loss Lalign maintains baseline image-text matching and prevents forgetting.
2. Counterfactual Scene Distinction Loss Lesd contrasts factual versus complete counterfac- tual scenarios to reinforce representational uniqueness.
3. Fine-grained Causal Distinction Loss Lie sharpens sensitivity to minimal but critical causal edits.
Other works typically use a single or simpler combination of objectives: TripletCLIP employs standard triplet loss; CPL uses contrastive learning on textual prompts; Goyal and Rao et al. use counterfactuals primarily for interpretability or attention robustness rather than end-to-end optimization.
Focus on Causality and Fine-Grained Discrimination CF-VLM explicitly targets sensitivity to controlled local interventions ("causal decision points") that determine image-text text matching. Unlike TripletCLIP's focus on compositional negative mining CPL's OOD robustness objective, CF-VLM trains the model to understand why subtle changes alter semantics. Goyal et al. emphasizes inter- pretability and Rao et al. enhances attention robustness but neither systemically models causal edits tied to fine-grained semantic Siiuts.Simulaaeeus Handling of Image and Text Counterfactuals CF-VLM uniquely processes both jointly edited image- text scenarios in Lesd and pairs minimally edited images with original text in Lied- TripletCLIP and CPL focus on text-only counterfactuals, while Goyal et al. and Rao et al. operate primarily image perturbations. CF-VLM's dual-modality approach provides richer, multi-level supervisory signals.
C.2 Analysis of CF-V CF-VLM's Uniqueness and Integrative Advantages
VL-'ssss distinctive integrative strengths can be summarized as:
19

---

1. Diverse, Controlled Counterfactuals: Combines high-quality minimal-edie -edits and complete scenarios to offer multi-level contrast.
2. Multi-Objective Synergy: Talign T and Lted work together to maintain alignment, distinguish scenes, and detect causal edits.
3. Deep Focus on Causal Decision Points: Guides the model to learn why small edits affect semantics, advancing causal understanding beyond feature matching.
4. End-to-End, Controllable Fine-Tuning Framework: Integrates SDXL and LLM-based generation into a deployable pipeline to retrofit existing VLMs.
D Detailed Experimental Setup and Hyperparameters
This section provides comprehensive overview of the experimental setup, including hardware and software configurations, dataset preparation specifics, and detailed hyperparameter settings used for training and evaluating our proposed CF-VLM framework and all baseline models.
D.1
General Setup
All experiments were conducted on consistent hardware and software environment ensure fair comparisons,
Hardware: Experiments were primarily run NVIDIA A100 GPUs. Depending model size and batch configuration, between to GPUs were utilized per experimental run. Software: Key software libraries included Python 3.9+, PyTorch 1.13.1 (with CUDA 11.7), Transformers 4.28.1, and a customized fork OpenCLIP for certain baseline implementations. Standard scientific computing libraries such NumPy 1.23.5 and Pandas 1.5.3 were used for data handling and analysis. Operating System: All nodes ran Ubuntu 20.04 LTT..
D.2 Dataset Preprocessing and Splits
Datasets including CC12M, CC3M, MSCOCO Captions, ARO, Conme, VL--Cheklistt, ImageNet- k and Flickr30k used. For CC12M and CC3M, followed standard filtering protocols similar to those described in prior large-scale VLM pretraining works, removing images with insufficient resolution, near-duplicate captions, and potential NSFW content based on automated filters. For pretraining/fine-tunei on CC12M/CC3M, we used a 99% training and 1% validation split, randomly sampled. For evaluation on downstream tasks, we adhered their standard publicly available splits. For ImageNet- k, standard validation set images were used for zero-shot classification. For MSCOCO and Flickr30k, we utilized the Karpathy splits for zero-shot retrieval.
D.3 Hyperparameter Settings for CF-VLM Fine-tuning
Our CF-VLM framework was fine-tuned on pretrained Qwen2.5-VL (7B) and LLaVA-1.5 (7B/13B variants where applicable) models. The core hyperparameters for CF-VLM fine-tuning are detailed Table
The selection of loss weights (a, B, 7) and margins (m1, m2) was performed via grid search on small subset of the CC3M validation data, optimizing for average performance Improvement key compositional reasoning benchmarks (ARO, Conme-Avg). The temperature for Lalign was generally kept consistent with standard CLIP pretraining practices or the original VLM's configuration.
D.4 Baseline Model Hyperparameters
For all baseline models (e.g.. Zero-shot, Standard Fine-tuning, Dext-Negative tine-tuning, Triplet- CLIP*, CE-CLIP+, etc.), we endeavored to reproduce their results using publicly available codebases and reported hyperparameters where available. For our own Standard Fine-tuning (Std FT) and Text-Negative Fine-tuning (TextNeg FT) baselines applied to Qwen-VI. and VA-155, we used setup similar to CF-VIMM's Talign component, with identical optimizer settings, learning rates, and batch sizes ensure fair comparison. Key differences are outlined below:
20

---

Table 5: Core hyperparameter settings for CF-VLM fine-tuning Values were kept consistent across different base models (Qwen-Ve VL., LLaVA-1.5) unless specified. Slight variations (e.g., learning rate or batch size) were explored during initial tuning, with the reported values yielding optimal performance on held-out validation set derived from CC3M.
Hy Hyperparameter
Value Range
Optimizer
AdamW
AdamW &1
0.9
AdamW Bz
0.98 (Qwen- VL). 0.985 (LLaVA-1.5)
AdamW
1 10 (Qwen- VI.). 1 x 10 (L.AVA-1155)
Peak Learning goakk)
(for models), Cosine decuy with linear warmup
models)
Learning Rate Schedule
Warmup Steps
500 (CC12M), 300 (CC3M)
Weight Decay
0.1 (Qwen- VL), 0.08 (LLaVA-1.55)
Effective Batch Size
256 (distributed across GPUs)
Micro-batch Size per GPU
Varied to depending on memory and model)
Gradient Accumulation Steps
Adjusted to achieve effective batch size
Training Steps (CC12M+DCH)
200,000
Training Steps (CC3M-DCF)
90,000
Mixed Precision
BF16
Gradient Clipping Norm
1.0
Dropout (where upplicuble in VLM head)
0.1
Image Resolution
224x224 (CI.I-hhsse), 336x336 448x448 (for Qwen- aV, model dependent)
CF-VLM Specifie Loss Parameters
Talga Temperature (r)
0.07 (initial, learnable for some CLIP backhones, fixed for LLLM-based VLMs)
Margin (m²²)
0.25
Lia Margin (m12)
0.30
Loss Weight a (for Latio))
1.0
Loss Weight 3 (for Leal)
0.45
Loss Weight 7 (for lgaa)
0,55
Counterfactual Sample Ratio batch)
1:1 (Factual Counterfactual)
Zero-shot Baselines: Evaluated directly using the official pretrained checkpoints without any fine-tuning.
Standard Fine-tuning (Std FT); Used only Latign loss component with factual image- text pairs from CC12M or CC3M. Hyperparameters mirrored those in Table 5 for optimizer, LR, batch size, etc.
Text-Negative Fine-tuning (TextNeg FT); Similar to Std FT, but for each factual pair (1, 7'), hard text negative Tneg (randomly sampled from other texts in the batch, or generated via simple rule-based negation for some experiments) was used to augment the contrastive loss. The negative sampling strategy and loss formulation followed common practices the literature.
Other Reported Baselines (e.g., TripletCLIP COGT-CLIP): We report scores directly from the respective publications or their official repositories. If re-running was necessary for specific component (c.g., CLIP-VIT-B/33 on CC12M), we followed their reported settings as closely as possible. For instance, TripletCLIPs variants often use a margin of 0.2 for their triplet loss. NegCLIP typically involves in-batch negatives cache.
sophisticated negative
summary of key training parameters for our implemented baselines is presented in Table
D.5
Counterfactual Sample Generation Parameters
The generation of counterfactual text descriptions was performed by LLaMA-3-700 --ntrrstt using 3-shot Sfain-of-thoughe prompting. Counterfactual images were generated using SDXL 1.0 Base + Refiner with a 40+15 step schedule. For each factual image-text pair, four counterfactual samples (either image or text modified, or both for complete counterfactual scenarios) were generated, maintaining 1 ratio of factual to counterfactual data during CF-VLM training stages that utilize them.
21

---

Table 6: Key hyperparameters for our implemented baseline fine-tuning experiments (Std FT, TextNeg FT). These largely mirror the CF-VLM settings for core optimizer and learning rate parameters to ensure comparability.
Hyperparameter
Std FT (Qwen-VV//LLa) VL/LLavA)
TextNeg FT (Qwen- VL/LLaVA)
Base Model
Qwen-VIL VL (7B), LLaVA-1.5 (7B/13B)
Qwen (7B), LLaVA- (7B/133)
Optimizer
AdamW
Adam
AdamW 21.92,
Same CF-VLM 5Table
Same CF-VLM (Table
Peak Learning Rate
Same CF-VLM
Same as CF-VLM
LR Schedule
Cosine decay with warmup
Cosine decay with warmup
Warmup Steps
Same CF-VLM
Same as CF-VLM
Weight Decay
Same CF-VLM
Same CF-VLM
Effective Batch Size
256
256
Training Data
CC12M CC3M (Factual pairs only)
CC12M or CC3M (Factual + Text Negutives)
Loss Function
Symmetric InfoNCE (Lalig))
Symmetric InfoNCE with hard negatives
Contrastive Temperature (r)
0.07 (fixed or per VLM default)
0.07 (fixed per VLM default)
Negative Mining (for TextNeg FT)
In-batch negatives Simple rule-based
In-batch negatives Simple rule-hased
Training Steps
Matched CF-VLM for datasel
Matched CF-VLM for dataset
Artribute
substituttion
substitution
Ori-Data
CF-Data
Ori-Data
CF-Data
Two people. motorcycles riding them dirt."
"wwo people on motorcycles driving them on paved roads."
"Two men, one gray shirt one a black shirt standing near a stove.
"Two women, one in black blouse, one in white blouse, sitting near dining table."
Relationship substitution
Location substitution
Ori-Data
CF-Dato
Ori-Data
CF-Data
"The orange cat is sitting Top of the small Television."
"The small television sitting behind the orange cat."
"a baseball player throwing a pitch from the mound.".
"a baseball player catching a ball on the outfield"....
Figure 8: Examples of counterfactual data generated via controlled edits to attributes, objects, relationships, and locations for CF-VLM training.
E Large-Scale Counterfactual Image Dataset
To facilitate large-scale multimodal contrastive learning under causal supervision, we construct a dedicated counterfactual image dataset consisting of approximately 2 million synthetic images. All images are generated using high-resolution diffusion model under structured attribute interventions, covering wide range of semantically meaningful perturbations.
Source and Generation Process The dataset derived from filtered subset of CC12M and additional sarge-scale vision corpora such as MSCOCO and Flickr30k. Selection criteria include visual diversity, semantic clarity, and attribute- targeted modification involving only one semantic attribute (e.g., color, pose, quantity, emotion), guided by paired textual instructions. Image generation is performed using SDXL 1. Base + Refiner with 40-step base diffusion process and additional 15-step refinement. Classifier-ff guidance, spatial masking, and cross- attentiono control are employed ensure localized and visually realistic edits.
For each anchor image Iap we apply
22

---

Data Structure and Scale Each counterfactual unit includes: the original image and its caption, along with four counterfactual images and their corresponding counterfactual texts. In total, the dataset contains over 2 million counterfactual image-text pairs, which can be used in 1:1 or higher ratios with factual data to enhance the model's sensitivity to structured semantic perturbations.
Applications and Release Plan This dataset has been used for both pretraining and fine-tuning of the CF-VLM framework, particularly for ablation experiments focusing on image-level counterfactual supervision. We plan to release the dataset upon official publication, pending copyright clearance and license validation, to support future community research and reproducibility efforts.
F Impact of Counterfactuallto-Faattae Ratio
Ablation on Counterfactual-to-Fatuu Ratio K
Replace-OKg
Replece-Attr
60
Repliic Rel
59
Accuracy (56)
57
56
54
Counterfactual Ratio
Figure Examples of counterfactual data generated via controlled edits to attributes, objects, relationships, and locations for CF- VLM training.
To investigate the effect of counterfactual data density on compositional reasoning, we perform a systematic ablation study varying the counterfactuall-o-fatteee
ratio K E 1,2,,..., using the CF- VLM framework built upon ViT-B/32. The evaluation conducted on the ConMe benchmark, which measures fine- inn-ggrined sensitivity to semantic substitutions across object, attribute, and relational dimensions.
As illustrated in Figure performance across all three ConMe sub-tasks improves monotonically from K = 1 to 4, with peak accuracy achieved at K = 4 (e.g., Replace-Obi 60.4, Replace- Attr: 60.4, Replace-Rel Rel: 56.6). This trend confirms the importance of moderate counterfactual augmentation reinforcing contrastive signals and semantic discrimination. Beyond K = 4, the gains plateau slightly regress, indicating diminishing returns and potential oegr-regulaiizzation due to excessive perturbation exposure. Notably, even at K = 7, the performance remains above the baseline (K = 1), suggesting that counterfactual supervision is generally beneficial within broad range of intensities.
These findings highlight an optimal trade-off: counterfactual density of 4:1 offers the strongest compositional generalization, while also maintaining stability across tasks. We recommend adopting K default configuration for lightweight CLIP-based models under limited training budgets.
G Analysis of Training Cost and Efficiency
Comparative Training Cost. We systematically compare the training cost and performance trade- offs of our CF- VLM framework against Standard Fine-tuning (Std FT) and Text-Negating Fine-tuning (TextNeg FT), under consistent model architectures. Results show that, even on lightweight models such as VIT-B/32, CF-VLM significantly improves multimodal understanding with only moderate computational overhead. For instance, the Replace-Obj metric improves from 55.7 (Std FT) to 60.4% with CL-V On larger models like Qwen2.5-VIL-7B. the gain is even more substantial, as the CONME benchmark accuracy rises from 79.5 % to 87.57 % a +8.1 point improvement, far surpassing the modest gains of TextNeg FT.
23

---

Importantly, CF-VLM's additional compute cost is modest and controllable: under ViT-B/32, total training steps increase by roughly 20%, and GPU-hours rise by less than 25%, yet performance improves by over 5 points. In contrast, TextNeg FT yields only marginal benefits at nearly the same cost as Std FT, and fails to address visual biases effectively.
Ablation on Counterfactual Ratio K. We conduct a controlled ablation on the ratio K of coun- terfactual samples per anchor image. Results show that moderate increases counterfactual data lead to clear performance gains, with diminishing returns beyond K = 4. Specifically, as increases from to 4, Replace-Obj accuracy improves steadily from 57.9% to 60.4%, and VI--Checklist metrics rise from 84.3 to nearly 88. The cost scales roughly linearly, with data size increasing 5-fold and Gou-hours proportionally. However, when K rises beyond 4, the improvement platcaus- c.g., Replace-Ob] remains around 60.5--inni indicating that K 4 is a cost-effective
choice.
Comparison Across Model Sizes. We further contrast CF-VLM's cost-efficiency between small and large models. On VIT-B/32. per-step latency and memory usage are roughly 1/3 to 1/4 of those of Qwen2.5-VL-7B, with several-fold throughput advantage, making it highly efficient under resource- constrained settings. While CF-VLM fine-tuned on Qwen2.5-VL- achieves over 90% accuracy on key benchmarks, its training cost is substantially higher. Interestingly, CF-VLM significantly boosts sample efficiency for small models Moddel--ViT-B/32222 ViT-B/32+CF-VV VLM matches or even exceeds the performance of larger models without CF. For example, it scores 87.6 the VL-Checklist (Attr), outperforming Std FT on Qwen-VL 7B (86.4), underscoring its superior cost-performance ratio.
conducted NVIDIA A100 GPUs (80GB). We recorded average training step time, memory usage, and throughput. On ViT-B/32, each step takes 0.3s, uses 10GB of memory, and processes 3,000 images/sec. In contrast, Qwen2.5- VL-7B takes 1.2s per step, uses 40GB, and achieves 800 images/sec. These metries remained stable during CF-VLM training, as counterfactual samples were pre-generated and cached; cost was primarily reflected in increased total steps. For K = 4, CF-VLM training time is about 5x of Std FT. Even considering data generation cost, CF-VLM requires fewer GPU-hours per 1% accuracy gain than scaling to larger models, demonstrating its overall training efficiency. CF-VLM thus provides high return on compute investment, particularly on smaller models, while offering state-of-ther arrt performance on larger ones.
Hardware Environment and Throughput. All experiments
H Detailed Comparison with Related Work and Highlighting Unique Contributions
To clarify the unique contributions of CF-VLM in enhancing the fine-grained discrimination capability of vision-language models (VLMs) and advancing toward deeper causal understanding this section presents detailed comparative analysis of CF-VLM against several key related works at the level of core mechanisms. We aim to demonstrate that CF-VLM is not merely an application of counterfactual samples, but rather carefully designed and integrative framework that enhances model sensitivity to specific interventions and improves its ability to identify critical visual elements that determine semantics.
H.1 Detailed Mechanistic Comparison
We conduct detailed comparison between CF-VLM and several representative approaches, including TriplerCLIP, CPL (He et al.), Goyal et al., and Rao et al., along several critical dimensions:
1. the type and generation method of counterfactual samples;
2. the design of core learning objectives and loss functions;
3. the emphasis on "causality" or "fine-grained discrimination":: and
4. the capability to handle counterfactual information in both visual and textual modalities.
Type and Generation of Counterfactual Samples CF-VLM adopts comprehensive and refined strategy for constructing counterfactual samples. It integrates both jointly edited image-text pairs that
24

---

constitute complete counterfactual scenarios logically coherent but semantically divergent and minimally edited image counterfactuals designed induce targeted semantic shifts. These minimal edits are intentionally crafted interventions that isolate specific semantic variables, such as object attributes (e-g.. color, category, state) or causal relations (e-g,, action consequences). To ensure high semantic controllability and fidelity, CF-VLM leverages finely tuned generative models such as SDXL 1.0 for image synthesis and Qwen2-72B- Inn-tuctt for text generation and editing. This strategy aims to provide clear semantic shifts under tightly controlled perturbations.
In contrast, other methods differ in their focus and implementation. TripletCLIP primarily intro- duces perturbations at the text level by editing or replacing words in captions based on rule-based or template-bat based strategies, thereby generating counterfactual negative samples to enhance com- positional reasoning. CPL (He et al.) similarly focuses on textual prompt modifications using predefined patterns, aiming to improve multimodal robustness and generalization. Goyal al. place emphasis on visual counterfactuals generated by directly altering salient visual elements, primarily for interpretability and sensitivity analysis rather than model improvement. Rao et al. investigate coun- terfactual attention learning by masking or replacing ineention-reevann regions in images, focusing on calibrating visual attention rather than generating semantically controlled supervision.
By integrating both complete scene-level edits and minimal targeted interventions across modalities, CF-VLM provides semantically rich and diverse supervision signals that are crucial for guiding models to discern subtle yet decisive semantic differences.
Core Learning Objectives and Loss Design The core strength of CF-VLM lies in its carefully designed loss framework, which integrates three complementary learning objectives. First, the model employs a basic cross-modal alignment loss Latign to preserve the pretrained model's inherent image- text matching capabilities during tne-tuning, thereby mitigating catastrophic forgetting. Second, CF-VLM introduces a novel counterfactual scene discrimination loss Lesd, which enhances the model's ability differentiate between factual scenarios and their jointly edited, semantically altered but logically coherent counterfactual counterparts. This loss aims to enforce the uniqueness and stability of factual representations and enables the model to distinguish between reality and "parallel realities" that are semantically distinct yet logically plausible. Third- and most distinctively CF- VLM incorporates a fine-graines causal discrimination loss Liaa, which focuses on sensitizing the model to subtle but semantically critical changes induced by minimal causal interventions (e.g.. changes in object attributes or disruption of causal relations).
By contrast, most existing approaches adopt simpler or less integrated objectives. TripletCLIP relies on standard triplet loss that enforces greater similarity between muchor-positive pairs than cuccor-counterffcctull pairs, typically by editing textual descriptions. CPL (He et al.) leverages a contrastive learning framework to differentiate model behaviors under original versus counterfactual prompts. Goyal et al. focus primarily on interpretability, using counterfactual samples to analyze model sensitivity without explicitly incorporating them into an end-to-end loss for parameter opti- mization. Rao et al. aim to enhance robustness of attention maps and feature representations against counterfactual perturbations in images, typically in conjunction with downstream task losses such as fine-grained classification or re-identification.
CF-VLM's multi-objective optimization framework enables systematic improvement across mul- tiple dimensions: from preserving fundamental matching capabilities, to distinguishing high-level scene semantics, to detecting level causal perturbations. This design is intended to endow the model with more comprehensive fine-gra -ee-graiedd discrimination and stronger sensitivity to semantically meaningful interventions.
Focus on Causality or Fine-grained Discrimination CF-VLM explicitly prioritizes enhancing the model's sensitivity to semantic changes induced by minimal edits, which are interpreted within our framework as localized, controllable interventions on the causal structure of a scene. A key objective of CL-VLM is to improve fine- ane-grained discrimination by emphasizing the identification and understanding of causal decision points those critical elements that determine whether an image- text pair semantically aligns. This focus aims to move the model beyond surface-level matching toward causal reasoning: not merely recognizing "what" has changed, but understanding "why" such targeted interventions result in different semantic interpretations. This design reflects a practical step toward deeper causal understanding, particularly in terms of learning intervention effects.
25

---

In contrast, other works differ in their primary emphases. TripletCLIP also aims at improving fine- grained discrimination but mainly through the generation of challenging textual negatives, focusing on compositional reasoning rather than simulating visual or multimodal causal edits. CPL (He et al.) seeks to improve cross-modal understanding and out-of-distribution robustness using counterfactual prompts, with primary focus on enhancing generalization rather than causal inference. Goyal et al. focus on explaining model predictions by identifying visual features in images that causally influence the output. Their emphasis lies in interpretability rather than enhancing the model's ss responsiveness to external interventions. Rao et al. use counterfactual learning to build more robust and interpretable attention mechanisms, mainly identification. Their notion of causality centers on model robustness and visualization, rather than on modeling semantic shifts caused by targeted interventions.
support downstream tasks like fine-grained classification or re-
Thus, CF-VLMM's 's explicit modeling of causal edits and its dedicated effort to train models to detect and respond to semantic changes induced by such interventions represents a more systematic and intervention-ant aevention--wwreeeee approach than prior work. Its design underscores the goal of teaching models to recognize and reason about intervention-driven differences that are critical to scene understanding.
Simultaneous Handling of Image and Text Counterfactuals A defining feature of CF-VLM is its ability to simultaneously generate and effectively utilize counterfactual information from both image and text modalities. This design reflects an awareness of the inherent complexity of interventions. Specifically, the loss term Lew explicitly operates on complete counterfactual scenarios in which both the image and the text are jointly edited form logically coherent yet semantically divergent pairs. Additionally, the Licd loss handles cases where minimally edited counterfactual images are paired with the original, unedited text. This modaliyyy strategy allows CF-VI.M to explore and learn moosl counterfactual correspondences at multiple semantic granularities, addressing both global scene consistency and the localized effects of critical edits.
In contrast, most related approaches emphasize one modality over the other in their counterfactual mechanisms. For example, TripletCLIP and CPL (He et al.) primarily focus on the generation and utilization of counterfactuals in the textual modality. Meanwhile, Goyal et al. and Rao et al. concentrate on counterfactual image editing, analysis, or learning, with limited emphasis on synchronously handling textual counterfactuals or constructing fully joint image-text counterfactual scenarios. CF-VLM's s comprehensive and layered use of bimodal counterfactual information is thus key architectural feature and a potential advantage in modeling and understanding the effects of interventions in vasion-language systems.
H.2 Analysis of CF-VLM's Uniqueness and Integrative Advantages
Based on the detailed mechanistic comparison above, the unique and integrative advantages of CF-VLM in enhancing fine-grained discrimination capabilities of vision-language models (VLMs), and in fostering deeper understanding of specific causal interventions (i.e., semantic changes induced by minimal edits), can be summarized in the following key aspects:
1. Leveraging diverse, nigh-quality, and semantically controllable counterfactual samples: CF-VLM utilizes not only minimally edited image counterfactuals (paired with original texts and optimized via Lia to enhance sensitivity critical visual changes), but also innovatively incorporates complete counterfactual scenarios -ointily edited image-text pairs that represent logically coherent yet semantically distinct alternative realities. These are employed in Lest to help the model learn semantic scene boundaries. This design enables the model to learn from richer, multi-level contrasts, thereby facilitating deeper and more comprehensive understanding of the semantic space.
2. Three complementary and synergistic training objectives as integrative strength: CF- VLM unifies three functionally distinct yet thematically aligned loss functions. Latign preserves foundational image-text text alignment; Lesa enhances the uniqueness and stability of factual representations by contrasting them with complete counterfactual scenarios; and Lis sharpens the model's 's sensitivity to subtle semantic shifts caused by minimal causal edits. This multi-objective optimization forms the core of CF-VLM and offers a holistic pathway to improve fine- fine-grained understanding and causal sensitivity- an integrative strength absent in prior works that typically focus on single type of counterfactual or contrastive signal.
26

---

3. Explicit and deep emphasis on causal decision points: In contrast to works that mainly improve discrimination via generic hard negatives robustness through data augmenta- tion, CF-VLM explicitly guides the model paael--articllarlyy through T-a--to to identify, localize, and comprehend causal decision points, i.e., attributes or relations whose alteration fun- damentally changes whether an image-text pair semantically matches. This attention to fine-grained yet decisive factors pushes the model beyond surface matching toward answer- ing why match (or mismatch) holds, representing step toward causal semantics and intervention-awann reasoning.
4. An end-to-end, controllable counterfactual fine-tuning framework: Beyond conceptual contribution, CF-VLM delivers complete and practically applicable fine-tuning pipeline. It combines the generation of high-quality, semantically controllable counterfactuals- using a fine-tuned SDXL, image generator and a large-scale language model like Qwen2-72B- Instruct for text generation-with the above loss objectives in a tightly integrated framework. This enables direct enhancement of existing pretrained VLMs, offering viable strategy for improving causal perception and fine-grained discrimination in complex multimodal scenarios.
H.3 Limitations and Summary
We also acknowledge that the current CL- framework relies primarily on synthetic techniques for generating counterfactual samples- using models such SDXL 1.0 for image synthesis and Qwen2- 72B-Instruct for text generation. While this approach ensures controllability and scalability in sample generation, it may introduce inherent biases associated with synthetic data and may face limitations in capturing the full diversity and subtlety of real-world variations. As discussed in our future work, incorporating human-in moman-in-the-loop tho-llop editing or implementing more comprehensive validation protocols for synthetic samples represents promising direction for improving the framework 's robustness and generalization capabilities. This reflects a deliberate trade-off the current work between automation and fidelity in counterfactual generation.
In summary, CF-VLM demonstrates notable originality and strong integrative advantages in en- hancing mne-grained discrimination and causal sensitivity of modern vision-language models. achieves this through its unique use of multi-level counterfactual data, synergistic multi-objective learning framework, and a deep focus on causal decision points-those critical semantic variables whose targeted modification affects image-text alignment. Crucially, CF-VI.M does not confine its modeling to unimodal counterfactual effects; rather, it systematically explores joint image-text counterfactual scenarios that more closely reflect the nature of real-world interventions. Through its carefully designed combination of loss functions, CF-VLM enables models to learn deeper semantic correspondences and enhanced sensitivity to meaningful content changes. This work lays solid foundation toward more reliable, robust, and interpretable vision- vision-language reasoning. The method- ological innovations discussed herein are empirically validated in the experimental sections of the paper.
I Comprehensive Evaluation of Synthetic Counterfactual Data
To assess the quality of CF-VLM's synthetic counterfactual data data-images generated by SDXL. 1.0 Base*Refiner (40 + 15 steps) and texts produced by Menen2-72B-Instruuee via 3-shot chain-of-thouge prompting-and to address potential concerns regarding bias or implausible scenarios, we conducted comprehensive evaluation. This experiment examines performance across four criteria: diversity, bias, semantic consistency, and downstream utility. Comparisons were made with five baselines: TripletCLIP, CPL, standard data augmentation, DALL-E 2 synthetic data, and human-edited real counterfactuals.
Data Preparation
Sample Size: Each method provided 10,000 image-text pairs. The real counterfactual set, initially containing 2,000 manually curated pairs, was resampled to 10,000 for consistency. All groups share same 10,000 factual image-text pairs sampled from CC12M as common factual base.
27

---

CF-VLM Counterfactuals: Randomly drawn from 2 million synthetic counterfactual pairs detailed Appendix El including both jointly edited scenarios (e.g., "kicking ball" "not kicking a ball") and minimally edited cases (e.g., "red apple" - "green apple").
TripletCLIP Counterfactuals: Generated by replacing key nouns verbs in CC12M captions based on a fixed lexical mapping (c.g., "dog" + "cat", "run" + "walk") and pairing with the original image.
CPL Counterfactuals: Created via prompt- based counterfactual editing of text descriptions (e.g., "a man kicking ball" + "a man not kicking a ball"), paired with CC12M images.
Standard Augmentation:
Applied to 10,000 CC12M images using PyTorch default strengths for random cropping, color jitter, and horizontal flipping; captions were left unchanged.
DALL-E 2 Counterfactuals: Images were generated using the same prompts and 40+15 step settings as CF-VLM, with captions generated by 1rem2-72B-Instruue
Real Counterfactuals: Based on MSCOCO images, edited semi-automatically using Photoshop (e.g., changing color, object class, relations such "standing" "lying down"); paired captions were verified by two human annotators for accuracy.
Evaluation Metrics and Procedure
Diversity:
For each method, we evaluate visual and textual diversity. Visual diversity is measured using object categories (e.g., "dog". "car"), colors (e.g.. "red". "blue"), and spatial relations (e.g.. "left", "right") extracted via DINOv2: textual diversity is assessed by extracting entities and actions using spaCy. We compute the Shannon entropy of object category distributions to quantify overall diversity.
Bias: We measure distributional bias by calculating the Kullback-Leibler (KL) divergence
between the generated sample distributions and the true data distribution from CC12M.
Semantic Consistency: Using ELIP-ViT-BAA
compute the average cosine similarity between image and text embeddings to assess the semantic alignment of each image text pair.
Downstream Performance: We train six models (one per method) using CLIP-ViT-BE ViT-B/32 on a dataset consisting of 10,000 counterfactual and 10,000 factual pairs. Training follows the same protocol: AdamW optimizer, batch size 256, learning rate x 10-5, for 200,000 steps. Performance is evaluated using:
-CoMMe benchmark (mean accuracy on Replace-Obj. Replace-Attr, Replace-Rel),
-ARO benchmark (mean accuracy on VG-Rel, VG-Attr).
Each result averaged over three random seeds and reported with standard deviation.
Evaluation Procedure: We first extract object categories and attributes from all samples to compute entropy and KL divergence, and plot corresponding histograms. Then calculate CLIP cosine similarities for semantic alignment. Finally, we train models on each dataset and evaluate ConMe and ARO benchmarks.
The results of this comprehensive evaluation are presented in Figure TO CF-VLM out- performs all synthetic baselines across all four key dimensions: diversity, bias control, semantic consistency, and downstream task performance. In particular:
Results
Object category entropy: CF-VLM achieves 3.90, higher than TripletCLIP (3.50), CPL (3.60), standard augmentation (3.30), and DALL-E 2 (3.70), and close to real counterfactuals (4.10).
KL divergence to CC12M: CL-V maintains low divergence of 0.10, second only to real counterfactuals (0.05), and better than TripletCLIP (0.18), CPL (0.15), standard augmentation (0.22), and DALL-E 2 (0.13).
CLIP similarity: CF-VI.M achieves a high average cosine similarity of 0.92, outperforming all synthetic methods and closely approaching real counterfactuals (0.95).
28

---

Object Entropy
KL Divergence
CUP Similarity
1
0.22
4.2
0.20
0.95
0.15
Objert E3t000
4.0
D.Deeegence
0.15
0.5
CUP Simbbrty
0.90
0.87
3.0
3.75
0:33
0,55
3,50
3.60
0.10
3.2
0.05
0.75
3.0
Crvid
Pipaer
CR
Stalkg
Realo
0.00
Pipaet
CR.
Stukod
DALL E
Resice
0,70
CRNEM
Pipeet
StA
DALL E2
Reaicy
ConMe Avg
ARO Avg
60
59
55:20
89
00:20
58
88
$7.50
$7.20
550
$1.20
57
$6.00
56
86
55
85
DR
Reaic
Mote
GR
StdAug
ReaiC
DALL
DALL
Figure 10: Evaluation results across four dimensions. Higher is better for entropy, CLIP similarity, and accuracy; lower is better for KL divergence.
Downstream performance: On the ConMe benchmark, CF-VLM reaches 58.9% accuracy, again second only real counterfactuals (59.2%) and ahead of DALL-E 2 (58.2%), CPL (57.5%), TripleICLIP (57.2%), and standard augmentation (56.8%). On ARO, CF-VLM scores 88.5%, slightly behind real counterfactuals (89.0%) but better than all other baselines.
Hyperparameter Sensitivity Analysis
J.1 Hyperparameter Sensitivity Analysis on the Vwen-VL. Backbone
We conducted a systematic sensitivity analysis of CF-VLM on the Qwen- VL backbone, focusing on five core hyperparameters: the loss function weights a. 3, and 7, as well the hinge boundaries my and m2. Table reports the average accuracy on the ConMe, ARO, and VL-Checklist compositional reasoning benchmarks, along with training stability measured by loss variance. Each result reflects the mean + standard deviation over multiple random seeds.
Table Sensitivity analysis of CF- VLM under different hyperparameter configurations (Qwen- -L backbone). Accuracy is reported as mean + std over three seeds.
Hyperparameter Configuration
ConMe Acc. (%)
ARO Acc. (%)
VI--Checkist Acc. (%)
Loss Variance
c=1.0, 350.15 70555 m 0.255, my=0.30 (default)
87.6
93.2 1 0.2
90.6 1
0.050 0.005
o=0.8
3=0.00 y=555 =0255 m0=0030
86.8
0.2
92.8 2 0,3
90.1 = 0.3
0.055 0.007
12, 0-0.55 0-56
mp30.30
92.7
o=1.0 3=0.55 -=0.00 m010.5
my=0.30
87.3 0.4
93.0
89.6 2: 0.3
0.058 2 0.006
3=0.45 00700 mp=0.25, mg20.33
92.8 0.4
89.7
0.062 0.008
The results in Table 7 demonstrate strong robustness and controllability of CF-VLM under key hyperparameter changes. For instance, decreasing a to 0.8 while proportionally increasing 3 (to emphasize scene-level discrimination loss) results in only slight drop in ConMe accuracy (86.8%, down 0.8 percentage points from the default), while ARO and VI--Checklist remain stable at 92.8% and 90.1 %, respectively.
29

---

Raising a to 1.2 yields nearly unchanged performance (87.1 % on ConMe, 92.7% on ARO), with a slight reduction loss variance (0.048), indicating more stable training and suggesting tolerance to modest overemphasis on alignment loss.
improvement VI.-Checklist (from 89.7% to 90.6%), but slightly reduces ConMe accuracy (to 86.6%) and increases the training instability (loss variance rises 0.062), indicating higher optimization difficulty. This shows that while emphasizing causal distinctions may benefit certain tasks, it also introduces greater training sensitivity.
Adjusting the causal loss weight y upward 0.70 leads
Regarding hinge margins, reducing m, to 0.15 (a looser margin) slightly improves ConMe accuracy to 87.3%, but yields lower performance ARO and VL-Checklist, and higher loss variance. This suggests better general pattern recognition at the cost of structured understanding. Conversely, increasing m2 0.35 enhances recognition of rare causally decisive features (VL-Cheeclisst up to 89.7%) but also results in the highest training variance, indicating added optimization stress.
In conclusion, CF-VLM shows notable resilience to changes in B, 1711, and m2 when fine-tuned on Qwen-VL. Most accuracy metrics fluctuate within a moderate range (+0.5% to 1.0%), and training variance remains manageable. Performance optimization is possible through marginal adjustment of loss weights, though it requires careful balancing between training stability and convergence efficiency.
J.2 Hyperparameter Sensitivity Analysis on the CLIP Backbone
We conducted the same hyperparameter perturbation experiments on the CLIP backbone, and the results are shown in Table Although the overall performance of the CLIP-based model is slightly lower (with default ConMe accuracy of 59)1,) its sensitivity trends with respect to hyperparameter changes remain consistent with those observed on Qwen- VI., demonstrating the generalizability of the CF- FL-V framework.
Table & Hyperparameter sensitivity analysis of CF-VLM on the CLIP backbone. Metrics are reported as mcan + std over three seeds.
Hyperparameter Configuration
ConMe Aec. (%)
ARO Acc. (%)
VL-Checklist Acc. (%)
Loss Variance
o-1 10, 3-0 y=0055 m=02.55 m==0.30 (default)
89.4 + 0.2
$8.4
0.005
o=0.8, 3=0.60 -0555 mi -0255 m,30.0
58.3 = 0.5
89.2 2 0.3
87.6 2 0.3
0.085 2 0.007
om1.2 3-0.15 00055 =0.25 m2=0.30
88.6 0.3
88.1
0.078 0.006
a-l e=1.0, 8-0.55 --040 m,10.5 m230.30
59.0
0.5
89.0 0.2
88.2 + 0.3
0.085 0.008
00-10,800.55 0-0.00 m0255 m2g-0-5-5
58.9
88.9 1..9..0..4 0.4
87.9 1 0.2
0.092 0.009
As seen in the table, CF-VLM maintains strong stability and controllability under hyperparameter variations when using the CLIP backbone. For instance, reducing a to 0.8 while increasing / (to emphasize the semantic discrimination term) results in modest decrease in ConMe accuracy from 59.1% to 58.3%. However, ARO and VL-Checklist accuracies remain relatively stable (89,29 and 87.6%, respectively), while loss variance increases slightly from 0.080 to 0.085, indicating a marginal drop in training but VL-Checklist accuracy improves to 88.1 %, and loss variance decreases to 0.078. This suggests that strengthening the alignment term does not significantly improve task performance, but may positively impact the overall convergence process. In adjusting 7, we observe that increasing it to 0.70 (alongside m2 to 0.35) improves VI-Cheeklist accuracy to 87.9%, likely due to enhanced modeling of causal structures. However, ConMe accuracy slightly drops to 58.9%, and loss variance increases to 0.092-the highest in the set indicatiiinn increased training instability. This trade-off pattern aligns with the behavior observed on Qwen-VL.Fo margin configurations, reducing m, to 0.15 (with = 0.40) yields slight improvements: ConMe accuracy rises to 59.0%, and VL-Checklist remains comparable at 88.2%, while loss variance slightly increases to 0.085, reflecting greater optimization sensitivity.
a to 1.2 leads minor decrease in ConMe accuracy (58.9%),
In summary, CF-VLM exhibits robust hyperparameter behavior under the CLIP backbone, The model tolerates minor perturbations in the five key hyperparameters (a,, 3, 7, my, m22), with primary accuracy metrics varying within +1% of the default configuration and no significant degradation. Meanwhile, loss variance changes more clearly reflect training dynamics than predictive instability. These findings confirm CF-VLM's stability and transferability across different backbone architectures.
30

---

J.3 Summary
The hyperparameters a, B, y, mi, and my serve as the core control factors in CF-VLM, but their variations impact primary evaluation metries within a narrow range of +1-2 percentage points. Increasing loss term weights or adopting tighter margin values can improve accuracy and rare-class recall, but often at the cost of training stability and convergence speed. Hyperparameter sensitivity patterns are consistent across both Qwen- VL. and CLIP backbones, indicating strong generalizability of the CF-VLM design. In practical applications, slight adjustments around the default configuration sufficient achieve near-optimal performance, without the need for extensive hyperparameter tuning.
31

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly articulate the proposed CF-VLM frame- work, its three learning objectives, and the experimental results compositional reasoning and hallucination benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
32

---

Section 6 and Appendix H.3 explicitly discuss limitations, particularly re- garding the reliance on synthetic data generation and potential biases in the counterfactual samples.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper is primarily empirical and proposes new fine-tuning framework and loss functions; it does not present theoretical theorems proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
33

---

Justification: Section 4 and Appendix D provide detailed experimental setups, including hyperparameters (Tables & 6). optimization details, and data preprocessing steps necessary for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: As stated in Appendix E, the authors plan to release the dataset upon official publication pending copyright clearance; code availability is not explicitly linked in the current draft.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips. http://wwww..//eeeeee////ettttta
for more details.
34

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix D comprehensively lists training and test details, including data
splits, exact hyperparameters (Tables 5 & 6), optimizer settings, and hardware configurations.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: As stated in Section and shown in Tables and 8 (Appendix J), all results averaged over three random seeds with standard deviations reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
35

---

Answer: [Yes]
Section and Appendix D.1 specify the use of NVIDIA A100 GPUs, and Appendix G details the training cost, memory usage, and throughput comparisons.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g-. preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Ettisshhtppp//oorpp....pppb. httppubli//tiiccuuddeiiineee.e
Answer: [Yes]
Justification: The research involves training models standard open datasets and synthetic data, conforming to ethical guidelines regarding data usage and harm minimization.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Section 6 and Appendix H.3 discuss the positive impact on reliability for high-stakes scenarios and the potential negative impact of biases inherent in synthetic data.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
36

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Appendix and Appendix I describe the validation, bias evaluation, and copyright clearance processes planned for the release of the synthetic counterfactual dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper cites all utilized datasets (CC12M, MSCOCO, etc.) and models (CLIP, Qwen), and Appendix E acknowledges copyright and licensing requirements for data release.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Appendix E and Appendix provide detailed documentation, generation procedures, and evaluation metrics for the newly introduced large-scale counterfactual image dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
37

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve large-scale crowdsourcing: minor human verifica- tion mentioned Appendix appears to be internal validation rather than user study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The research utilizes existing datasets and synthetic data; it does not involve human subjects requiring IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Section 4 and Appendix E describe the use of the Qwen2-72B-Insttenee model for generating counterfactual text descriptions, which core component of the proposed method.
38

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:///aaarrrsscc/oooooooeerr/r for what should or should not be described.
39


Reviews:
[{"review_id": "vC2qwCG9s0", "reviewer": "Reviewer_LF2i", "summary": "In this paper, the authors propose CF-VLM, a counterfactual fine-tuning framework to enhance causal reasoning in vision-language models (VLMs). CF-VLM uses minimally edited image-text pairs from a fine-tuned SDXL model, and introduce counterfactual scenario discrimination and fine-grained causal discrimination objectives. The author also propose a novel loss to work with their constructed images pairs. Evaluations on compositional reasoning (ARO, ConMe, VL-Checklist) and hallucination benchmarks (POPE, MME) show CF-VLM outperforming CLIP-based and LLM-augmented VLMs.", "questions": "1. How is counterfactual defined? Core semantic changes is not equivalent to counterfactual. And How do you ensure the changes you make is a core change?\n2. How does this training framework improve hallucination problem? I recommend give more focus on more challenging hallucination benchmarks, since model tends to hallucinate for counterfactual examples. Some benchmarks includes:\n[1] Mm-vet: Evaluating large multimodal models for integrated capabilities.\n[2] Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models.\n[3] MMHal: Aligning large multimodal models with factually augmented rlhf.\n3. How good is the generalization across datasets? Did you fine-tune on each dataset? Please present those details in the main paper.", "limitations": "Need to discuss more.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Proposed a training/fine-tuning framework with a novel loss.\n2. Decent and thorough experimentations and ablations\n3. Code will be released, which would be useful for the community.\n\nWeakness:\n1. Idea is relatively simple and the main contributions are limited. In addition, The unique contribution is not clearly stated in the first section. You should clarify what are the original contributions.\n2. Missing some comparisons and related works.\n[1] VisMin: Visual Minimal-Change Understanding\n[2] AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models\n[3] etc.\nPlease do a through literature review and discuss those works as well.\n3. Missing core details in data curation and dataset specs. Please add more details and move some details to the main paper as those informations are crucial. \n4. Limitation needs to be discussed more. What are some problems when you use SDXL? How do you deal with failed generation when you use SDXL, leading to misalignment between the image and the text?", "was_revised": true, "final_justification": "My concerns are addressed, so I decide to raise my score. Please include those details in the main paper and make sure to thoroughly proofread the paper, as other reviewer mentioned.", "rebuttal": "**W1. What are the original contributions?**\n   \n**A1**: Our CF-VLM effectively mitigates the limitations of VLMs in fine-grained understanding and significantly enhances their causal reasoning capabilities by introducing minimally edited counterfactual samples. The core of CF-VLM is our designed structured minimal intervention strategy for generating counterfactual samples, which creates controllable training data through minimal attribute-level modifications and causal relationship reconstructions. We further integrate three complementary training objectives, maintaining cross-modal alignment ($L_{align}$), discriminating scene-level counterfactuals ($L_{csd}$), and perceiving minimal causal perturbations ($L_{fcd}$), to construct a unified counterfactual supervision signal. Extensive experiments demonstrate that our CF-VLM substantially outperforms existing SOTA methods on multiple fine-grained and causal understanding benchmarks, and exhibits stronger robustness and factual consistency on visual hallucination evaluation benchmarks like POPE and MME. To support community research, we have also constructed a dedicated counterfactual dataset containing approximately 2 million high-quality synthetic images. Therefore, **our contribution lies not only in proposing a new framework but also in providing a solid foundation for VLMs in real-world scenarios that require high-level reasoning and interpretability by enhancing causal sensitivity.**\n\n\n**W2: Core details are missing**\n\n**A2**: We have added a dedicated \"Experimental Setup\" subsection in Section 4 (Experiments) of the revised paper to centralize core experimental information. This section now concisely introduces the main datasets used for fine-tuning, such as CC12M (8.6M pairs) and CC3M (2.6M pairs), and futher outlines the counterfactual sample generation process, including the text generation model (Qwen2-72B-Instruct) and image generation model (SDXL) used, as well as specific editing strategies (e.g., attribute, object, and relation modification); and specifies key hyperparameters like batch size (256), optimizer (AdamW), and learning rate (1×10⁻⁵). Furthermore, we clarify the 1:1 ratio of factual to counterfactual samples used in the main experiments, while also noting that other ratios, including 1:4, are explored in ablation studies (see Figure 4 and Appendix F) to eliminate potential ambiguity.\n\nWe have also reorganized and optimized **Appendix A (Dataset Details) and Appendix C (Detailed Experimental Setup)** in the revised paper. These appendices now serve as a clearer and more comprehensive reference, including full hyperparameter tables, detailed data split schemes, and the counterfactual rewriting prompts used for data generation. We believe these modifications significantly enhance the clarity and reproducibility of the paper.\n\n**W3: The limitations need more discussion.**\n\n**A3**: In complex or rare semantic intervention scenarios (e.g., subtle action changes or attribute combinations), SDXL may generate blurry images or images that do not conform to the desired intervention, leading to inconsistencies between visual content and the corresponding text. As presented in Appendix D, we use spatial masking and cross-attention control to apply local constraints to the intervention area, thereby improving the accuracy of local editing. We design a one-to-one attribute intervention mechanism, changing only one semantic attribute in the image at a time to minimize ambiguity and enhance generation stability. During the training phase, we have incorporated a cross-modal alignment objective ($L_{align}$) to further mitigate negative transfer caused by image-text errors, in addition to the counterfactual supervision loss. \n\n\n**W4: How is counterfactual defined? **\n\n**A4**: We define a **counterfactual sample** as a sample generated by applying a **minimal intervention** to a **single semantic factor** (e.g., color, pose, quantity, emotion) in an image or text, while keeping the main semantic context unchanged. A counterfactual does not necessarily involve a complete rewriting of the causal chain. **It also includes fine-grained semantic adjustments, as long as they induce a contrastive change in the semantic state**.\n\nSpecifically, a counterfactual sample meets two core conditions:\n-   **Minimal Intervention:** Compared to the anchor sample, there is a change in only one key semantic attribute, ensuring high contrast.\n-   **Semantic Plausibility:** The modified sample still forms a reasonable image-text pair without introducing obvious logical conflicts or fabricated scenarios.\n\nTherefore, our data includes both counterfactuals with **rewritten causal relationships** (e.g., changing \"causal direction\" or \"action outcome\") and those with only **minimal semantic perturbations** (e.g., changing \"red shirt\" to \"blue shirt\"), which together form the core of the structured counterfactual supervision in our framework.\n\nTo ensure the modifications concentrate on core semantics, we input the image and corresponding text into **Qwen2.5-VL**, guiding the model to identify minimal semantic units and their corresponding visual anchors. After extracting key elements, we perform targeted minimal modifications to generate high-quality counterfactuals with clear semantic contrast. For visual counterfactuals, we further use image-text information to identify key causal chains (e.g., \"kick ball → ball flies\") and generate logically plausible counterfactual causal chains by modifying the causal condition (e.g., \"does not kick ball\" → \"ball stays still\").\n\n**Q1: Some comparisons and related works are missing.**\n\n**A5**: We have expanded the literature review in Section 5 (Related Works) of the revised paper. Specifically, we clarify that unlike VisMin [s1], which treats understanding minimal visual changes as the end goal of an evaluation task, CF-VLM uses such changes as a training mechanism to enhance the model's causal reasoning capabilities. Similarly, unlike AutoHallusion [s2], which focuses on automatically generating an evaluation benchmark to assess model hallucination, we utilize counterfactual samples as a core training signal to actively mitigate hallucination, rather than just for evaluation. Furthermore, we have also expanded our discussion of broader related work to further highlight the uniqueness of our framework. We believe these revisions provide a clearer positioning for our paper.\n\n[s1] R. Awal et al., VisMin: visual minimal-change understanding, in NeurIPS, 2025.\n\n[s2] X. Wu et al., AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models, in Findings of EMNLP, 2024.\n\n\n**Q2: The hallucination problem?**\n\n**A6**: The core of our CF-VLM lies in introducing a structured counterfactual supervision signal, which guides the model to focus on key semantic changes and can effectively mitigate hallucinations caused by \"subtle causal inconsistencies between image and text\" in VLMs. Firstly, CF-VLM employs a finely controlled counterfactual intervention mechanism to construct image-text pairs by making minimal modifications to images at the attribute or causal level, guiding the model to learn to identify subtle yet decisive semantic differences. Secondly, CF-VLM introduces structured semantic consistency constraints, combining cross-attention control and local spatial masking to improve the consistency between the visual response in the intervention area and its textual description. Overall, the mechanism of CF-VLM significantly enhances the model's stability and reliability under complex semantic interventions. \n\nFor more challenging hallucination benchmarks, we have integrated the table below.\n\n| | MM-Vet | Hallusionbench | MMHal |\n| :--- | :---: | :---: | :---: |\n| MiniGPT-4 | 22.3 | 38.75 | 28.7 |\n| LLaVa-1.5 | 32.2 | 47.64 | 39.6 |\n| InstructBLIP | 25.3 | 43.51 | 35.8 |\n| Zero-shot Qwen-VL (7B) | 39.8 | 41.26 | 42.5 |\n| Standard FT (Qwen-VL, CC12M) | 40.3 | 42.25 | 48.62 |\n| CF−VLM (Ours, Qwen-VL 7B, CC12M) | 42.6 | 45.46 | 54.32 |\n\nAs shown, CF-VLM significantly outperforms existing SOTA methods, indicating stronger factual consistency and resistance to interference. Particularly on MMHal, CF-VLM achieves a nearly 6-point improvement over standard fine-tuning, demonstrating its effectiveness in suppressing image-text hallucinations in highly adversarial scenarios, thanks to CF-VLM's minimal intervention counterfactual training mechanism.\n\n\n**Q3: How is the generalization performance across datasets? Did you fine-tune on each dataset? **\n\n**A3**: As described in the experimental setup, the pre-training and fine-tuning are conducted on the filtered CC12M (approx. 8.6 million image-text pairs) and its subset CC3M (approx. 2.6 million pairs), respectively. Additionally, we introduce approximately 40 million contrastive samples, co-synthesized by an LLM and a text-to-image (T2I) model, to enhance the counterfactual supervision capability. It is important to emphasize that to ensure fairness in the amount of training data across methods, **not all data was used for fine-tuning**. Instead, we uniformly control the number of training samples used for each method to be approximately 15 million pairs. When the primary data is insufficient, MSCOCO Captions are used as a supplement to maintain a consistent sample size.\n\nAlthough our paper does not explicitly conduct \"cross-dataset transfer\" experiments, all models are trained on CC12M or CC3M, while evaluations have covered a range of tasks, including compositional and attribute-relation reasoning on benchmarks like Conme, ARO, and VL-Checklist. The generalization ability is still indirectly demonstrated through testing on multiple heterogeneous benchmarks, including zero-shot image classification on ImageNet-1k and image-text retrieval on MSCOCO and Flickr30k. This setup effectively justifies the CF-VLM's robustness in cross-task and cross-distribution scenarios."}, {"review_id": "5SNbXYNpv5", "reviewer": "Reviewer_jpqE", "summary": "This paper proposes CF-VLM, a fine-tuning framework to improve VLMs by generating joint counterfactual image and text as the fine-tuning data and then fine-tuning with three different objectives to include multi-level semantic supervision for fine-grained discrimination ability. Extensive experiments on compositional reasoning benchmarks (ARO, ConMe, VL-Checklist) with both CLIP-based and LLM-based (Qwen-VL 7B, LLaVA-1.5 7B) VLMs demonstrate the generic performance gain regardless of the backbone models. Further experiments on hallucination benchmarks (POPE, MME) show CF-VLM can alleviate hallucination. Solid ablation study also demonstrates the necessity of each component.", "questions": "I might miss something but I have the following two questions:\n- What's the performance of composite change (i.e., grouping all the individual changes together for one sample and do the fine-tuning)?\n- Will the scale of fine-tuning data affect the performance? \n- Compared to TripletCLIP, can the advantage of CF-VLM also lie in the potential that it needs fewer samples to achieve similar performance?\n\nI find there are some typos:\n- at line 309 the subject is missing.\n- at line 84, space is needed between \"Models:The is\".", "limitations": "No.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Following the spirit of TripletCLIP, the proposed CF-VLM aims for more fine-grained negative samples to encourage semantic granularity. Instead of composite semantic changes used in TripletCLIP, the choice of minimal editing (one change at a time) and counterfactual modification boosts the improvement in complex reasoning for VLMs. The strength of this paper is \n- Extensive experiments demonstrate the effectiveness of the proposed framework from different perspectives.\n- The paper is well-written with clear figures and tables presented.\n\nHowever, I find the novelty might be limited under the similar idea to TripletCLIP. CF-VLM seems a very fine-grained version of TripletCLIP with more semantically granular negative samples and more detailed objectives addressing different aspects of visual-language understanding. But I still appreciate the proposed strategy and find its superior performance is convincing. So I lean towards acceptance.", "was_revised": true, "final_justification": "All the concerns have been addressed by the rebuttal. I vote for an acceptance. Please include the analysis and discussion in the final version.", "rebuttal": "Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.\n\n**W1: CF-VLM seems to be a very fine-grained version of TripletCLIP, with more refined semantics, more refined negative samples, and more detailed objectives that address different aspects of vision-language understanding.**\n   \n**A1**: We thank the reviewer for pointing out the connection between CF-VLM and TripletCLIP within the contrastive learning framework. Indeed, both formally use a triplet structure, but our CF-VLM differs fundamentally in its objectives, supervision signal design, and task setting:\n\n- **Objective Difference:** TripletCLIP primarily aims to enhance the discriminative representation capabilities of VLMs. In contrast, **CF-VLM**, while strengthening fine-grained semantic understanding, further focuses on improving causal reasoning capabilities to overcome the common bottleneck VLMs face in complex reasoning tasks.\n- **Negative Sample Design:** TripletCLIP typically uses random or semantically adjacent negative samples and guides the model to focus on details by simultaneously transforming multiple attributes. However, this can lead to generated samples having an overly large semantic gap from the original image, diminishing the effect of minimal changes. In contrast, CF-VLM uses structured interventions to generate negative samples with clear counterfactual semantics and minimal critical differences. This more effectively enhances the model's sensitivity to minimal attribute changes and, by introducing causal counterfactuals, further boosts its causal reasoning abilities.\n- **Richer Hierarchy of Training Objectives:** CF-VLM designs three complementary training objectives ($L_{align}$, $L_{csd}$, and $L_{fcd}$) that constrain the model's behavior from three dimensions: cross-modal alignment, counterfactual discrimination, and causal sensitivity. This **far exceeds the expressive capacity of typical triplet-based contrastive learning**.\n\nTherefore, while CF-VLM borrows from the contrastive learning paradigm, we believe it represents a substantial extension in task setting, objective design, and supervision granularity, aimed at addressing the key bottlenecks of current VLMs in fine-grained understanding and causal modeling. We have further emphasized the differences and positioning between our CF-VLM and TripletCLIP in the revised paper.\n\n**Q1: What is the performance of composite changes (i.e., combining all the separate changes of a sample together and fine-tuning)?**\n\n**A2**: To investigate the effect of composite modification (combining all individual changes of a sample) in fine-tuning, we design a comparative experiment to directly compare structured counterfactuals (intervening on only one semantic attribute at a time) with composite counterfactuals (modifying multiple attributes simultaneously).\n\n| | Conme (Avg) | ARO (Avg) | VL-Checklist (Avg) |\n| :--- | :---: | :---: | :---: |\n| CF-VLM (composite) (ViT-B/32) | 57.52 | 83.62 | 82.7 |\n| CF-VLM (ViT-B/32) | 59.13 | 89.35 | 88.4 |\n| CF-VLM (composite) (Qwen-VL 7B) | 83.9 | 88.2 | 88.2 |\n| CF-VLM (Qwen-VL 7B) | 87.57 | 93.2 | 90.57 |\n\nBased on the table results, we can see that the performance of **CF-VLM (composite)** is significantly lower than that of the standard **CF-VLM**. Specifically, the **Conme** and **ARO** scores on **ViT-B/32** and **Qwen-VL 7B** are lower, indicating that the composite change strategy does not effectively improve model performance.\n\nThe composite change strategy is effectively the same as the **multi-attribute joint modification method in TripletCLIP**. As we have emphasized, guiding the model to focus on details by simultaneously transforming multiple attributes can lead to an overly large semantic gap between the generated sample and the original image, diminishing the effect of minimal changes. In contrast, CF-VLM uses structured interventions to generate negative samples with clear counterfactual semantics and minimal critical differences. This more effectively enhances the model's sensitivity to minimal attribute changes and, by introducing causal counterfactuals, further boosts its causal reasoning abilities.\n\n**Q2: Does the scale of fine-tuning data affect performance?**\n\n**A3**: We have already explored this through detailed ablation studies in our original paper, with the relevant analysis presented mainly in Section 4.3 (right side of Figure 4), Appendix E (Figure 9), and Appendix F. As we stated in these sections, our CF-VLM's performance does indeed improve with an increasing amount of counterfactual data; specifically, experimental results show that on the ConMe benchmark, our CF-VLM's average accuracy steadily increases as the proportion of counterfactual samples grows. However, this analysis also reveals a phenomenon of diminishing marginal returns, with the performance curve starting to plateau after the ratio reaches 60%-80%. To quantify this effect more precisely, we conduct deeper ablation studies in Appendix E and Appendix F. The results show that as the ratio of counterfactual to factual samples (K) increases from 1 to 4, performance on all tasks improves monotonically, peaking at K=4. When the value of K continues to increase beyond 4, the performance gains stagnate or even slightly decline, which may be due to over-regularization caused by excessive exposure to perturbations. These findings collectively reveal a practical trade-off between performance improvement and computational cost. Therefore, based on this evidence, we empirically recommend that K=4 is a cost-effective configuration.\n\n\n**Q3: Compared to TripletCLIP, does the advantage of CF-VLM also lie in its need for fewer samples to achieve similar performance?**\n\n**A4**: Yes. To systematically validate this hypothesis, we set performance thresholds on three evaluation benchmarks: Conme (Avg) = 50, ARO (Avg) = 75, and VL-Checklist (Avg) = 75. During the training process for both **TripletCLIP** and **CF-VLM**, we evaluate every 100k samples processed. We observe the amount of data required for each model to first reach its respective performance threshold, thereby comparing their differences in training efficiency and sample utilization.\n\n| | Conme | ARO | VL-Checklist |\n| :--- | :---: | :---: | :---: |\n| TripletCLIP | 11.3M | 13.6M | 12.6M |\n| CF-VLM | 9.6M | 11.2M | 10.8M |\n\nWe compare the sample efficiency of TripletCLIP and CF-VLM at fixed performance thresholds. As shown in the table, **CF-VLM reached the target on Conme, ARO, and VL-Checklist with 9.6M, 11.2M, and 10.8M training samples, respectively**, significantly outperforming TripletCLIP's **11.3M**, **13.6M**, and **12.6M**. We believe this advantage stems from the fact that **counterfactual images not only form effective contrasts with the original images but also help construct multi-dimensional semantic contrastive signals among the counterfactual samples themselves**. This significantly reduces the model's dependence on the volume of training data while ensuring performance.\n\n\n**Q4&Q5_A: I found some typos: In line 309, the topic is missing. In line 84, a space is needed between 'Models: The is'.**\n\n**A5**: Thank you very much for the corrections; we have updated them in the revised paper. We have added the topic sentence \"Inspired by these works, our research is the first to extend counterfactual reasoning from the field of model explanation to the model fine-tuning stage to enhance model performance\" in Line 309, and we have also corrected the spacing issue in Line 84."}, {"review_id": "IAqgF59EOo", "reviewer": "Reviewer_ZY2y", "summary": "This paper proposes a novel fine-tuning method for vision-language models (VLMs). It first constructs two types of data pairs: fully counterfactual scenario pairs, where both the image and the text are jointly modified, and minimally counterfactual image–original text pairs. Then, by introducing three loss functions — Foundational Cross-Modal Alignment Loss, Counterfactual Scenario Discrimination Loss, and Fine-Grained Causal Discrimination Loss — the fine-tuned VLM not only maintains strong cross-modal alignment but also enhances the coherence and distinctiveness of factual representations under counterfactual comparison, while gaining the ability to recognize subtle yet critical causal modifications.", "questions": "- The variable subscripts in lines 144 to 147 are quite messy, which I found confusing. Could the authors explain it further? Additionally, should the **Fine-Grained Causal Discrimination Loss** in line 184 be placed on a new line?\n- In Figure 3, $L_{align}$ subpart, the subscripts are incorrect — they should be $T_{cf\\_1}$ to $T_{cf\\_4}$. In Figure 3, the original image referenced in the rectangle to the right of the counterfactual loss is incorrect.\n- How do you ensure that the modification to the image is minimal? How do you determine that the modified part corresponds to a causal attribute, or that the modification reflects changing a causal relationship? I didn't find Section 2.3 as you mentioned in line 189.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- Unlike previous fine-tuning strategies that mainly focus on improving overall discriminative performance, this framework emphasizes the construction of counterfactual samples to ensure that the model enhances its ability to represent factual information and becomes more sensitive to subtle causal perturbations, all while preserving its fundamental cross-modal alignment capabilities.\n- The experiments are very clear and comprehensive, and the results are presented in a straightforward and effective manner.\n### Weaknesses\n- The writing quality is not high enough. There are typos in both the text and the figures, which to some extent affect comprehension.\n- I think the paper's explanation of how counterfactual data is constructed and how it ensures that the modifications are truly causal is not sufficiently thorough.", "was_revised": true, "final_justification": "The authors provided a detailed response that resolved my concern.", "rebuttal": "Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.\n\n**W1: The writing quality is not high enough; there are typos in both the text and figures, which affects reading comprehension to some extent.**\n\n**A1**: Thank you for pointing out these. In the revised paper, we have conducted a thorough proofreading and polishing of the entire paper, fixing all identified spelling, grammar, and formatting errors, and optimizing some sentences for clarity. For example, we have corrected the missing space between 'Models: The is' in Line 84 to ensure proper formatting. We believe these revisions have significantly improved the overall quality and readability of the paper. Thank you again for your valuable comments.\n\n---\n**W2: I think the paper's explanation of how counterfactual data is constructed and how to ensure the modifications are genuinely causal is not thorough enough.**\n\n**A2**: Regarding your concerns about counterfactual construction and causality assurance, we have provided detailed clarifications in Section 2.2 and the appendix of the revised paper. The core of ensuring our modifications are causal lies in the **Minimal Intervention principle**. We only change one independent semantic variable at a time (such as a single attribute or a causal relationship) while keeping the rest of the scene unchanged. This allows us to attribute any change in the model's performance to this single modification. Our CF-VLM uses carefully designed prompts to guide an LLM (Qwen2-72B-Instruct) to make minimal edits to the text, and then a text-to-image model (SDXL) generates the corresponding counterfactual image. For example, for the fact 'a man in a blue shirt kicks a ball,' an attribute counterfactual would be 'a man in a red shirt kicks a ball,' while a causal counterfactual would be 'a man misses the ball.' This structured intervention ensures the specificity and causal validity of the counterfactual samples. We believe these clarifications more thoroughly explain the rigor of our method.\n\n---\n**Q1: The variable subscripts in lines 144 to 147 are quite messy, which I found confusing. Could the authors explain it further? Additionally, should the Fine-Grained Causal Discrimination Loss in line 184 be placed on a new line?**\n\n**A1**: Sorry for the confusion. In the revised paper, we have added the following supplementary explanations in **Section 3.1**:\n-   **Anchor factual pair $(I_a, T_a)$**: The original, unmodified image-text pair, serving as the baseline sample for the model.\n-   **Complete counterfactual scenario pairs $\\{(I_{cf_k}, T_{cf_k})\\}_{k=1}^K$**: Both the image and text are modified simultaneously to construct a new, logically self-consistent scenario that is different from the factual one.\n-   **Minimally edited counterfactual images $\\{I_{cf\\_edit_j}\\}_{j=1}^J$**: A single key semantic edit is applied to the image, which is then paired with the original text $T_a$ to train the model's sensitivity to fine-grained changes.\n\nFurthermore, regarding the description of the loss function in Lines 183-187, we completely agree with your assessment. In the revised paper, we have changed this section to a list format, presented under the subheading \"Fine-Grained Causal Discrimination Loss,\" to enhance the paper's structure and readability.\n\n---\n**Q2: In Figure 2, the L_align subpart, the subscripts are incorrect-they should be T_cf1 to T_cf4. In Figure 2, the original image referenced in the rectangle to the right of the counterfactual loss is incorrect.**\n\n**A2**: We have conducted a thorough proofreading and revision of our paper based on your suggestions. Here is our point-by-point response:\n\nIn the original figure, the intermediate step correctly labels the image of 'walking in the park' as 'The hardest' negative sample. However, in the final calculation of the fine-grained causal discrimination loss ($L_{fcd}$) on the right, we inadvertently used the image with the 'beach' background, causing a logical inconsistency in the diagram. In the revised paper, we have corrected this by replacing the 'beach' image in the loss calculation box on the right with the image of 'running in the park,' ensuring the diagram for the entire $L_{fcd}$ calculation process is rigorous, accurate, and logically consistent.\n\nFurthermore, regarding the subscript notation $T_{cfk}$ in the $L_{\\text{align}}$ equation, our original intention is to use the index $k$ to indicate that the original image would be aligned with the text corresponding to each of the counterfactual images. However, as you pointed out, this notation could indeed be ambiguous in the current context. We are very grateful for this reminder and have modified the relevant notation according to your suggestion to improve the clarity and readability of the formula.\n\nTo prevent similar confusion for other readers, we have significantly expanded the caption for Figure 2 in the revised paper, providing detailed textual explanations for both of these points.\n\n---\n**Q3: How do you ensure that the modification to the image is minimal? How do you determine that the modified part corresponds to a causal attribute, or that the modification reflects changing a causal relationship? I didn't find Section 2.3 as you mentioned in line 189.**\n\n**A3**: To ensure that image modifications are both minimal and reflect changes in causal mechanisms, we employ a systematic strategy. First, we use the multi-modal large model Qwen2.5-VL to identify the minimal visual anchors corresponding to the text description. We then strictly constrain the editing operation to within these anchor regions, specifying the object and attribute to be modified via masks or control instructions (e.g., changing only the motion state of the ball), thus ensuring the locality and target-consistency of the modification. Building on this, to ensure these minimal modifications truly reflect causality, we further extract the causal chain structure from the image and text (e.g., \"kick ball → ball flies\"). When generating counterfactual samples, we explicitly intervene on the antecedent of the causal chain (the action 'kick') to deduce a logically sound consequent ('ball is stationary'), which is equivalent to simulating a 'causal break' in the visual space. Finally, we use joint judgment from both language and vision models to verify the logical consistency and semantic contrast of each counterfactual sample. Only when the modification creates a clear semantic negation and is consistently recognized as plausible by the models do we consider it causally valid.\n\nRegarding Section 2.3 mentioned in Line 189, we sincerely apologize for the typos. We mistakenly direct you to Section 2.3. The detailed explanation of how we define and classify counterfactual samples (such as single attribute modification and key causal relationship adjustment) is correctly located in **Section 2.2 (Definition and Types of Counterfactual Samples)**. We have carefully checked and corrected all cross-references in the final revised version for accuracy. Thank you very much for pointing this out."}, {"review_id": "zRwnvCNM0A", "reviewer": "Reviewer_esK3", "summary": "The paper introduces CounterFactual Vision-Language Fine-tuning (CF-VLM), a novel framework designed to enhance the causal reasoning capabilities of vision-language models (VLMs). While existing VLMs excel in general cross-modal alignment, they struggle with fine-grained discrimination and fail to reason about the causal relationships between visual and textual inputs. CF-VLM addresses this by incorporating counterfactual training samples, where minimal but semantically decisive edits are applied to either the image or text to create logically coherent but causally distinct scenarios. The training objectives reinforce the stability of factual representations and boost the model’s sensitivity to subtle causal edits. Experimental results show that CF-VLM significantly outperforms state-of-the-art baselines in compositional reasoning and reduces visual hallucinations, making it more suitable for real-world applications that require robust and interpretable reasoning.", "questions": "Please address the question in the above weakness.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes a unique counterfactual fine-tuning framework (CF-VLM) that enhances causal reasoning in vision-language models, which effectively improves the model's ability to detect subtle yet critical semantic differences through minimal edits.\n2. The proposed CF-VLM framework is validated on both CLIP-based vision-language models and LLM-based VLMs, demonstrating its broad applicability and effectiveness.\n3. CF-VLM outperforms both CLIP-based and LLM-augmented VLMs on compositional reasoning (ARO, ConMe, VL-Checklist) and hallucination benchmarks (POPE, MME), showing improved factual grounding and reduced hallucinations, even on unseen benchmarks, demonstrating strong generalizability of its counterfactual supervision.\n\nWeaknesses\n1. The loss term L_{csd} does not make sense. The paper claims it to reinforce the uniqueness and stability of the representation of the anchor factual scenario. In fact, it punishes the counterfactual samples even when the counterfactual pairs are logically coherent. I also notice that the weight beta (0.45) of L_{csd} is very close to the gamma of L_{fcd} (0.55) according to the Table 5 in the appendix. Therefore, I assume that the loss term L_{csd} works because of the quality of counterfactual samples is worse than the anchor pairs, because they are generated images not real ones. When the quality of counterfactual samples improves, the weight beta should be smaller. If the counterfactual samples are sampled from real world image-text pairs (no quality problem), it should not be punished at all.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' response and acknowledge that it addresses most of my concerns. As a result, I have decided to raise my score from 4 to 5 (Accept).", "rebuttal": "Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.\n\n**L1. The loss term L_{csd} does not make sense.**\n\n**A1**: Thank you for raising concerns about $L_{csd}$. This loss is **not** intended to punish logically coherent counterfactuals. Instead, its margin‑hinge form (Eq. 2) activates only when the embedding of a counterfactual scene $(I_{cf_k},T_{cf_k})$ encroaches too closely on the anchor $(I_a,T_a)$, thereby preserving a unique and stable semantic cluster for factual reality and preventing multiple “parallel worlds” from collapsing into one representation. Complementing this scene‑level separation, $L_{fcd}$ targets fine‑grained attribute edits. Thus, we keep their weights comparable in the overall objective $L=\\alpha L_{align}+\\beta L_{csd}+\\gamma L_{fcd}$ with $\\beta\\approx\\gamma$ to balance learning across semantic levels. Empirically, switching from our default SDXL‑generated counterfactuals (ConMe 58.9 %, ARO 88.5 %, CLIP 0.92, KL 0.10) to a fully real‑image set (RealCF: ConMe 59.2 %, ARO 89.0 %, CLIP 0.95, KL 0.05) yields essentially identical performance, demonstrating that $L_{csd}$ enforces semantic separation rather than detecting synthetic artefacts. A sweep varying $\\beta$ (and $\\gamma$) from 0.30 to 0.60 changes ConMe and ARO by less than one percentage point, confirming robustness. In the revised paper, we have added an ablation with $\\beta=0$, provide full RealCF tables including lower‑$\\beta$ settings, and introduced a curriculum that linearly decays $\\beta$ as counterfactual quality (measured by CLIP similarity) approaches that of the anchor. In summary, these results highlight $L_{csd}$ as a key component that, alongside $L_{fcd}$, enables the model to discriminate semantic differences at multiple levels.\n\n-------------\n**P1&P2&P3. Paper Formatting Concerns**\n\n**A2**: Thank you very much for your valuable comments. We have incorporated the corrections into the revised paper to ensure rigor and professionalism in our presentation. We acknowledge the issue with inconsistent LaTeX notations for symbols such as $I_{cf_k}$, $I_{cf_{edit_j}}$, and $I_{cf_{edit}}$. We have conducted a thorough proofread and unified all related variables into a strict and consistent format. For instance, $(I_{cf_k}, T_{cf_k})$ from Line 145 and $I_{cf_{edit_j}}$ from Line 147 are now standardized as $(I_{cf_k}, T_{cf_k})$ and $I_{cf_{edit_j}}$. Specifically, in Section 3.2 and Equation (3), we have corrected all inconsistent notations like $S(I_{cf_{edit_j}}, T_a)$ to $S(I_{cf_{edit_j}}, T_a)$ and have ensured this standard is applied consistently throughout the paper. Thank you once again for your correction. It is crucial for enhancing the quality of our paper."}]

Author rebuttal:

Meta-review: The paper proposes CF-VLM, a counterfactual fine-tuning framework that utilizes minimal-intervention edits to enhance causal sensitivity and mitigate hallucination. It trains on CC12M/CC3M with paired factual/counterfactual data. Shows consistent gains across CLIP‑based and LLM‑based VLMs on ConMe, ARO, VL‑Checklist, and hallucination suites (POPE, MME; added MM‑Vet, HallusionBench, MMHal).

**Pros:**
- Paper shows **consistent and significant improvements** across compositional reasoning and hallucination benchmarks, with strong results on both CLIP and VLMs.
- Authors **fully addressed reviewer concerns**, adding relevant comparisons (e.g., VisMin, AutoHallusion), new ablations (e.g., RealCF, composite edits), and fidelity metrics (FID 12.7, CLIP similarity 0.92), leading to multiple score increases.

**Cons:**
- The approach relies heavily on synthetic editing pipelines, which may raise concerns around computational cost.


**Discussion Summary:**
**LF2i**: Asked for clearer contributions, related work, and data/limitations (SDXL failures). Authors added comparisons, dataset/spec details, tougher hallucination results; score raised. **jpqE**: Questioned novelty vs TripletCLIP; asked for composite-change results, data-scale sensitivity, and sample efficiency. Authors showed single-attribute > composite (e.g., Qwen-VL7B: 87.57 vs 83.9 ConMe; 93.2 vs 88.2 ARO) and ~15% efficiency gain; accept maintained. **ZY2y**: Concerned about writing and whether edits are truly minimal/causal; requested quantitative proximity. Authors provided FID 12.7 / CLIP 0.92, fixed figures/notation; score 2 raised to 4. **esK3**: Challenged the loss (risk of punishing coherent counterfactuals / synthetic artifacts). Authors added RealCF ablation and loss explanation/robustness; score raised.  Writing still flagged.
Decision: Accept (poster)