Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition
Authors: Pei Peng
Ming-Kun Xie
Hang Hao
Tong Jin
Sheng-Jun Huang
Venue: NeurIPS 2025 poster ()
Keywords: Zero-Shot Recognition
Counterfactual Reasoning
Causal Inference
CLIP
Primary area: general_machine_learning
TL;DR: We introduce a representation-level counterfactual framework to mitigate co-occurrence bias in vision–language models without retraining or prompt engineering.

Abstract: Object–context shortcuts remain a persistent challenge in vision‑language models, undermining zero‑shot reliability when test-time scenes diverge from familiar training co-occurrences. We recast this issue as a causal inference problem and ask: Would the prediction remain if the object appeared in a different environment? To answer it at inference time, we estimate object and background expectations within CLIP’s representation space, and synthesize counterfactual embeddings by recombining object features with diverse alternative contexts sampled from external datasets, batch neighbors, or text-derived descriptions. By estimating the Total Direct Effect and simulating intervention, we further subtract background‑only activation, preserving beneficial object–context interactions while mitigating hallucinated scores. Without retraining or prompt design, our method substantially improves both worst-group and average accuracy on context-sensitive benchmarks, establishing a new zero‑shot state of the art. Beyond performance, our framework provides a lightweight representation-level counterfactual approach, offering a practical causal avenue for debiased and reliable multimodal reasoning.

Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition
Pei Peng'
Tong Jin' Sheng-Jun Huang" Nanjing University of Aeronautics and Astronautics, Nanjing, China (pengpei, mkxie, haohang, tongj in, huangs j @nuaa. edu, en
Ming-Kun Xie
Hang Hao!
Abstract
Object-context shortcuts remain persistent challenge in vision-language mod- els, undermining zero-shot reliability when test-time scenes differ from familiar training co-occurrences. We recast this issue as a causal inference problem and ask: Would the prediction remain if the object appeared in a different environment? To answer this at inference time, we estimate object and background expectations within CLIP's representation space, and synthesize counterfactual embeddings by recombining object features with diverse alternative contexts sampled from external datasets. batch neighbors, text-derived descriptions. By estimating the Total Direct Effect and simulating intervention, we further subtract background-only activation, preserving beneficial object-context interactions while mitigating hal- lucinated scores. Without retraining or prompt design, our method substantially improves both worst-group and average accuracy on context-sensitive benchmarks, establishing new pero-shot state of the art. Beyond performance, our frame- work provides lightweight representation-levee counterfactual approach, offering practical causal avenue for debiased and reliable multimodal reasoning. The implementation is available https//gittub..oo/pipeengg
1
Introduction
Vision language models, such as CLIP [1], have demonstrated impressive success by embedding images and textual descriptions into shared semantic space, enabling strong zero-shot recognition across diverse datasets [2. 3, 4J. By training on large-scale natural image-text pairs, these models aim generalize beyond finite training distributions. However, recent studies [5] highlight a critical limitation: Real-world datasets inherently encode strong co oe-uurereence patterns between target classes and contextual cues, including background scenes and their complex interactions. These spurious correlations provide shortcut signals during training [6. 7], leading models to rely on context rather than the true causal semanties, and ultimately degrading their zero-shot recognition performance, particularly when the test-time context diverges from familiar training distribution.
The issue is especially pronounced CLIP, where training objectives that maximize mutual informa- tion across modalities unintentionally reinforce object-conted entanglement, During optimization, frequent co-occurrence patterns amplify semantic signals from contextual features, causing back- ground cues, co-occurring entities and scene-specific interactions to dominate the learned representa- tions. This focus shift skews decision boundaries away from intrinsic object feature. As shown in Fig. 1(a), although such biases may enhance performance on in-distribution data, they substantially degrade model performance when confronted with novel backgrounds [8, 9, 10. 11].
In response to the contextual entanglement, particularly the visual hallucinations induced by back- ground signals, recent strategies generally fall into two directions. A common and lightweight
"Corresoondence Sheng- Jun Huang: huangsj @ @uuaa.cdu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

C(a)-based
Averago
Best-group
Worstgrrop
100
Z sbot accurscy (%)
44.6
42.
15.0
60
CLIP
CLIP scure
Mune lier 2055
40
sooa 20U1
Iis BS-4
20-
caul 5-19
box frog 2366
jandan spader RAA2 Booster OUS
VIT-B/32
VIT-B/16
VIT-L/14
MT-H/14
(a) Accuracy drop under co-occurrence bias
(b) Visual bias and model hallucinations
Figure/ Bias and hallucinations caused by co-occurrence in CLIP. (a) Accuracy decreases significantly from the best to worst group across backbones in Waterbirds dataset, with groups defined by different class-contex co-occurrence patterns. (b) The Attention maps show image responses under the prompt "a photo of an albatross," comparing CLIP and our counterfactual embedding C. C(x), and context-only images trigger hallucinated score on ImageNet labels, highlighting vision-side bias.
indirect solution is to augment the textual modality, typically by enriching prompts with additional
contextual descriptors [12, 13, 14, 15]. However, two fundamental challenges remain: (i) generating comprehensive and accurate descriptions of the visual context is inherently challenging, and (ii) the text encoder itself inherits co-occurrence biases from pretraining, which may further entangle object and context semanties rather than disentangle them. On the other hand, directly operating the visual modality provides a more principled alternative that avoids the challenges in textual prompt- based methods. Yet current approach [16, 17, 18, 19] often rely generative models, complex augmentation pipelines, or fine-tuning of pretrained backbones, resulting in increased computational cost and limited scalability in practice. Detailed work is available in Appendix A.
Thus, motivated by causal inference [20, 21, 22]. we propose novel framework: instead of retraining models textual augmentation, inference time and let it autonomously answer the causal question: "Would this prediction remain consistent if the object appeared in different environment?" Specifically, separately estimate the conditional expectations object (target entity) and background (contextual cues), then construct their respective optimal estimation in the representation space. It will intervene objects by simulating in alternative contexts, derived from external scene datasets, internal scene from batch neighbors, or virtual scene descriptions constructed within the model's semantic space.
enable model imagine counterfactual scenarios directly at
Additionally, to suppress hallucinated background effects while preserving the beneficial interaction effects captured by the model, we introduce image-level Total Direct Effect (TDE) computation, which explicitly quantifics and removes the illusory influence arising from high-frequency object- context co-occurrences. Notably, our method requires additional training, external generative modules, and no handcrafted prompts, offering a lightweight, scalable, and causal solution for real-world deployment. In summary, contributions
follows:
We demonstrate a causal perspective to model co-occurrence biases in terms of beneficial interac- tions and potential hallucinations, and explain the sensitivity of prompt-based strategies.
We propose counterfactual estimation method to operate directly the representation level and develop lightweight, inference-only debias scheme. It synthesizes counterfactual embeddings by leveraging skills (e.g. scene descriptions), and employs targeted interventions and TDE estimation to eliminate hallucinated effects while preserving beneficial object-context interactions.
We present our method achieves state-of-the-art zero-shot recognition performance across multiple context-sensitiv. datasets, significantly improving model's 's reliability under distribution shifts.

---

Female
Male
Accuracy (%)
PMT
72
-44
-05
-63
dising Ledle homel
berip ploaee
di
whit
sm snffhurdd
book
aist Bitter
lipin spuits ball ratuee
rrack
spece howl
licyele
ord planpe
Eic
clais
coke
Lite frisher
csscl
sports tall rasinio troul
dining
Context Categories
Context Categories
(a) Gender-specific accuracy on contextual objects
(b) PMI between gender and contextual objects
Figure 2: Context-induced gender bias in CLIP. (a) Zero-shot accuracy gap on COCO-GB v1
across genders and contextual objects. (b) Corresponding PMI revealing co-occurrence bias pattern.
2 Revisiting Object-Context Bias in CLIP
Setup. Let E X' and Z E Z denote, respectively, the object category (labeled as Y E E) and contextual background." Each image instance i I is generated from p(i X= Z=z). CLIP transforms image via visual encoder f: I I-R and prompt T via textual encoder ft T TTRR RR. For brevity, we let fi X=z) abbreviate to x and similarly for Z = z.
Two-Factor Decompositions in CLIP Modalities. Following the Hoeffding perspective (a.k.a. ANOVA decompositions) [23, 24, 25J, with respect to object-context factors, the instance can be decomposed into the following structural causal function:
(ii)) (x) + (z) + ri( 2) 77i
(1)
where e; (a))= xx (1) (f(()) X-=) and e===)=Bapp((z=)))
-= are the first-order main effects of the object and background, (= z) is the interaction effects, and 1/s summarizes the higher-order residual. It proved in Appendix. B.1.1 that this is the only pairwise orthogonal
expansion of these terms of Eq. (1).
Similarly, the composite prompt T (Ta; Tz) is embedded
f(( e, (x) + ez(z) + r2( 7/5,
(2)
where (z)) and (z) are the textual modal main effects; rx( x) and n/ respectively interaction terms and residual. Detailed derivations of Eq. (2) appear in Appendix 1.1.2.
Co-occurrence Bias Emerges from Contrastive Learning. The InfoNCE objective used in CLIP maximizes the alignment between paired image and text embeddings while pushing apart mismatched ones. When representations are decomposed into object and context components, the InfoNCE gradient reveals a distinct structure: for any object-scene pair (æ, 2). high co-occurrence frequency leads to large gradient contributions on amplifying the interaction terms rg, r'z and the cross-modal similarity of (e,(),, (xx)), proved in Appendix B.2 and B.3. These components widen the margin between positive samples and their single- factor negatives (e.g., (x, x²) (x',z) z)).
Thus, the biases are enhanced during training and governed by the statistical co-occurrence of (æx, 2). which can be quantified by the Pointwise Mutual Information (PMI):
p(x,
PMT( z) = log
p(z) p(z)*
which measures the extent to which pair of variables co-occurs more frequently than expected under statistically independent. Since mutual information T(X; Z) is the expectation of PMI over (ax, 2), and the InfoNCE loss is lower bound on T(X, Z;T) T), minimizing InfoNCE inherently promotes high PMI pairs in cross-modal alignment. The detail are shown in Appendix B.4
Both label sets are finite, with Z enumerating all scenes present in the dataset.
3

---

As result, unless dataset samples objects and contexts uniformly, minimizing the InfoNCE loss will inherently entangle them, amplifying both interaction energy and cross- rross-modal similarity. This dependency is difficult to avoid in practice and is a key source of context-induced misclassifications in vision-language models. Fig. 2 provides a empirical evidence: on gender-balanced subset COCO- GB vI [10], the zero-shot accuracy gap between female and male correlates with their PMI to context, revealing similar gender-contex co-occurrence biases shared between COCO and LAION-2B [26]. These results confirm that CLIP models internalize context-driven associations, which persist even under zero-shot settings and impact fair prediction across different groups.
Causal Explanation for Ballucination-Indueer Misclassification. At inference, the relationship of causal effect can be constructed as shown in Fig. 3, where node I feeds two children: object X and scene Z: a solid arrow X + represents the ideal decision path, whereas the dashed links between Z and X indicate statistical co-occurrence learned during contrastive training. When conditioning on I, extra predictive paths emerge, including --Y and (X, 2) R- Y. Thus, the CLIP score' for any class c E y with an object-only prompt expands to
S(i,
f.), + (z), f((T.) + (r, (z, z), f((T)) + (n f((T)).
(4)
The inner product (e:(z), (T())) is the pure context score. It remains non-zero even for "background- only" images due to image encoder bias (see Fig. 1(b)), and may thus produce hallucination when the score favors a class in which the image does not exist. The term (a)), fe(T. is the ideal target, and the interaction term (1, S[ (Teg) )} acts as an object-specific prior when (x, x) really co-occur, but drops to low-level for totally novel pairs.
X: target object
I image instance
co-oceurring context
R
R interactive feature
V: model prediction
- causal path
X
Figure 3: The schematic of causal graph. Dashed line denotes X and Z co-occurring in the same image. R represents the interaction as observable features.
When (a, 2) follows the training distribution, the ideal target dominates the CLIP score, and the scene score and r; reinforce correct decision boundaries, just as contrastive training dose. However, for a novel pairing (æ, 2²) which rarely or never co-occurred during training, the interaction r; remains a low level. Meanwhile the frequent scene z' favors the class Cyr that often co-occurred with it during training stage. The decision margin between the correct class Cg and the confusable class oa is
8 == Sir.., Te,) Te..) ~ ((e.(()), (e((z), f((T.))) ((e.(=)), (e((2)))
(5)
If the contextual effect inherited from co-occurrence pairs (a',2')
22) dominates target evidence, then 5 < 0 and prediction incorrectly favors Ca', shown by zero-shot recognition failures cases in Fig. ((a).
With the object-only prompt, the prediction implicitly marginalizes over all possible contexts, as a priors in encoder, which are derived from the training set and influenced by co-occurrence patterns, that is p(Y I, T.,)) = Σz P(Y I, Te Z=z) p( Zz [Z) so the shortcut - Y remains active. Appending an explicit scene token T. amounts conditioning on (equivalently intervening in) Z:
Causal Explanation of Prompt-Basen Context Intervention.
p( I2, TT,, = (I 7=2),
(6)
thus clamping the latent context and cutting off the spurious branch I- Y.. The classifier then relies solely on the legitimate I- X + pathway. But these methods [12, 13, 14, 15] have to use an accurate T3 neutralizes the - Y shortcut. Imprecise or mismatched T2. only partially cancels the bias and may divert information to unrelated classes, even bringing more bias by Tz, which is accounting for the empirical sensitivity of prompt engineering. Details are shown in Appendix B.5.
Towards Counterfactual Solution. Causal inference provides principled approach to mitigating spurious correlations by identifying and suppressing causally irrelevant signals. However, existing methods often rely on generative models [27, 28], auxiliary networks [29, 30], or counterfactual losses terms [31], which increase model complexity and limit applicability in zero-shot setting. Con- sidering these limitations, as shown in Fig. 4, we propose lightweight, inference-only framework
In practice, the CLIP score is normalized, and we simplify the derivation by omitting the scaling factor 1/T.

---

that intervenes directly within the CLIP representation space, Inspired by the principle of randomized experiments, our method synthesizes counterfactual embeddings by recombining the object with di- verse, label-agnostic scene contexts, thus suppressing background-inducer illusions while preserving useful object-context interactions. This allows for causal debiasing without modifying the model architecture or relying handcrafted prompts.
Prequeat Scone
CLIP-VTT
Imagr repersottation
Infercnce
Takmn
TDEIL()
f. Y
Rure scone
Centea
Quers token
semsntics
Contest poel
effect
enbcdding
Busch
Eoomal
Minul
objects
TDEICLaa,,1
contsan?
Obsect
Counterfactaa
Clx) +
Intervention
Figure 4: Overall architecture. Our method consists of two components: the upper branch computes TDE by subtracting the predictive contribution of context; the lower branch constructs counterfac- tual embeddings by recombining object with diverse alternative contexts embeddings, simulating intervention. Aggregating those yields a robust, decoupled prediction. The gray dashed line, and red arrows denote blocked, preserved casual effect to prediction respectively. A detailed step-by-step description of our inference pipeline can be found in Algorithm in Appendix E.4.
3
Counterfactual Inference at Representation Level for Zero-shot Learning
3.1 Estimating Counterfactual Representational Level
A basic challenge in counterfactual reasoning lies in its numerical computation. From novel angle, we estimate the counterfactual embedding directly from the original image without any extra model.
Token-Level Direct-Effect Decomposition. We focus on ViT [32] as the backbone of CLIP, built from T. layers, each of which contains Multi-Head eelf-Attention (MSA) followed by MLP block. The representation fi is linear projection / Rd x of ViT output joint vision-and-languaee space. The matrix & Rdx(N+1), with tokens ( 19.................... 1% columns, constitutes the initial state of residual stream. Following the residual updates, f((1) split into
f((i) = Pro PPMMSA(LNN) (LN' (t (1))))) PLM)
where pre projection LayerNorm LN(-) can be absorbed into P and its bias is evenly spread across all summations. Thus, the image embedding are divided into three direct-effect terms of initial class token, MSAs and MIPs. Following previous work [33, 18), MSA output rewrites over H attention heads and tokens as
P[MSA (t)
ΣΣ h=1
LA "j
LA =dja WLLIN((G-")
(8)
where With E Rdxd are transition matrices and oj I.h E R are the attention weights from class token to the j-th token, which satisfy Σ/ 0 -11h1. More algebraic proofs are provided in Appendix B.6.
To investigate the effect of the component, we used average ablation [34], replacing the component with its average over batch, for measuring the decrease in accuracy after ablation. Experiments on
'The exact form i' MSA (SAA(LN) (LN(( -- MLP' (LN)(()(()) +

---

CLIP
C(x)-based
C1IIP
b(x))
CLIPP
C(x)-based
CLIP
(xx)-based
Figure 5: The attention map revealed by C(z)-based embeddings on NICO.
ImageNet with multiple backbones in Tab. found that the accuracy changed to less than 0.1% after MSA ablation, whereas after ablation for the effect of initial class token and MLP, it only decreased by about 1% compared the baseline. Therefore, it reasonable design the main effect of each token E on fi(i) as direct effect of the MSAs and bias terms on the initial class token and MLPs after ablation. Hence, we define semantic token effect:
My
N
1=1
where is constant for batch about class token and MLPs after mean ablation.
Optimal First-Order Estimates for Counterfactual Embeddings. Let an image i be partitioned into N patch-tokens (t1, f2..... }, }, and their semantic effect obtain from Eq. (9). We define the counterfactual embedding as 2'x,²'.. Specifically, when z' is set to base state, simplifies to ((x). which contains only the embedding of æ. Analogously, C(=) is defined in the same way.
We introduce a Bernoulli latent variable G indicate semantic property of image whether it is context target object æ. Its corresponding conditional distribution instance is defined as p(f((i) G=z), which means when we focus on background parts, how does embedding distribution look like. This exactly corresponds to C(z), representing the case there is only the background without any target objects. Formally, define C(z) as expectation vector under this condition:
C(z) (l) | G
TP(T G z)dr,
(10)
It is the center of gravity in the embedding space when "looking only at the background". and the optimal first-order estimate for the background counterfactual embedding in the image. Detailed proofs are provided in the Appendix B.7. And from visualization in Fig. 5, it confirms that C(x) can enable better attentional focus on targer objects.
In practice, we discretize Eq. (10). Assuming image is sampled uniformly and that all token priors are equal, Bayes' theorem gives p(ry (1) G = 2) p(G (i)). Therefore, the counterfactual embedding C(z) can be computed as:
ΣN w- (=(ts)e((i) E,W,((t)
C(=) R Σ (1(() G= E000GGGGG
w=(4))= - p(G 1
(11)
where normalizes the weights (x;t;) to ensure the results in the expectation form. And, to stay aligned with the rest of the CLIP embedding (which are all in the unit sphere), usually do another L2 normalization for ((z). C(x) can obtain in the same way by wy and u (i).
Then, as for each token ty and class we compute its sigmoid score:
wz (1,) = p(G = ea (1)) = (Sitt, Te,)),
(12)
'The base state sets the variable's effect fixed constant, defaulting in this paper.

---

where a is sigmoid function, i' the CLIP score mentioned before. We prefer sigmoid (over softmax) because it treats each class one-vs-rest, yielding independent probabilities and avoiding the need for any hand-crafted "background" prompt. We thus define the token's s background probability by
wx(t)) p = by (i)) = 1 max or (S(tz) (T))::
(13)
cEY
This mutual-exclusion rule treats background the complement of the most likely foreground class, which empirically provides a sharp, unambiguous estimate of wz-
3.2 TDE-Driven Context Preservation and Hallucination Suppression
Let ir.: contains object æ and context 2, as for any class E ) conventional reasoning is for computing the Total Effect: TE( c) =>=pp>)<) i) p(Y=c ir....)) where izo.:o means base state image. As shown in Fig. 3, TE- TE-based predictions are influenced by both benign interaction effects and malignant hallucination effects. Total Direct Effect (TDE) provides the most straightforward approach: preserving useful pathways (X, 2) + R + Y and suppressing hallucination channel Y, we obtain:
TDE( e) p(Y = c - p( = c izo.:) p(Y /e,,))) - -py=) c =)), (14)
Obviously, counterfactual embedding via Eq. (11) isolate malignant hallucinatory effect. In CLIP, the posterior takes the form p(Y=c. (S(i T.)). We thus update the TDE in logit space as
TDE c) = S(1,, TT) As(C
(15)
where Å is a suppression coefficient controlling the contribution of background-only signals. Because of the strictly increase of (-), the form of Eq. (15) satisfies order preservation and avoids the numerical saturation of sigmoid. Therefore, the logit-based TDE is used throughout our experiments to isolate beneficial object-scene cues and suppress hallucination.
3.3 Effortless Counterfactual Construction and Intervention on Inference
From the other side, we tent to simulate intervention that apply the do-operator on X for eradicating the path X I + Y, leaving only the genuine object-label causal edge X-+Y. Unlike the soft mitigation approach in the previous section, it presents a direct solution to yield a more purified ideal effect X - Y. To be exact, we select a set of optimal context embeddings synthesize counterfactual image embeddings, which serve as classifier inputs to simulate intervention. As the causal graph Fig. 3 satisfies the back-door criterion [20, 35]. we define:
M
p(Y = c do(x))
Σ p(==co = x = x, 2 = 2m zm)p(Z = 2m),
(16)
m=1
where M denotes counterfactual sample size, and do-operator [20, 35J formally models intervention.
Sources of Candidate Context Embedding. More comprehensive candidate context embeddings
yield enhanced intervention disentanglement. We consider three pools of context embeddings {ff((z))) corresponding to three experimental setups. (I) External scene datasets (e.g. Places-365
[36]): It samples dataset directly by CLIP image encoder to get fz(). (II) Internal scene from batch: It samples counterfactual context embedding C(2s) in a batch by Eq. (11). (III) Virtual scene by description: Leveraging CLIP's shared semantic space enables effortless generation of scene descriptions (Tz,)) to serve virtual counterfactual context embeddings.
For context embeddings obtained above, filter sampler is designed to improve its quality, It evaluates each candidate embedding for image in.y, using scoring function based on cosine similarities between fi(z)) and C(x) as well fi(z)) and C(2), preferentially selecting embeddings with lower combined scores to maximize scene diversity. Thus, define the synthesized counterfactual embeddings as
C(x, ) = aC(x) + (1 - f(mm), a E 10, Then, assuming a uniform sampling and considering the new bias from fi(z), Eq. (16) turns into
(17)
M
1 M
TDE(C( c) c)
Σ (S(C(x, z,,). T.) - -S(f Te)),
(18)
M m=1
M m=
Thee synthesized embedding will be L2 normalized by default.
7

---

This form can effectively prevent potential bias about Zm - Y in introduced scenes. Combining Eq. (15) and (18), we obtain
M Σ m=1
arg max (i),
=
+A M
TDE(C( zm):
(19)
where À controls for the extent to which interactions in the original image i affect prediction. From another angle, Eq. (19) is interpreted as the model's imagination beyond image i. By conceptualizing different scenarios mm combining the original scene x through weights, more robust and fair prediction result can be obtained.
4
Experiment
Settings. We evaluate our method on four widely adopted benchmarks that target context-sensitive dis- tribution shifts: Waterbirds [8], UrbanCars [9], COCO-GB [10], and NICO [11]. These datasets span various real-world correlations, including different of--iiitibuttion context. We report average and worst-group accuracy assess both overall performance and vulnerability specific context-label combinations under zero-shot classification. Detailed dataset information, implementation details, and variant configurations are provided in Appendix C.
Performance on Waterbirds. In Tab. 1, Our methods consistently achieve the SOTA in both average and worst-group accuracy across all backbones. Compared to vanilla CLIP, improves worst-group accuracy by an average of +32.3%, with remarkable gain of +44..6% on ViT-B/32. Meanwhile, average accuracy also sees significant improvement, with average increase of +14.3%, and maximum gain of +19.1% on ViT-B/32 In contrast, other methods such as PC introduces textual bias (see Fig. 8) when limited fine-graine grained prompt used, trading off average accuracy improve worst-group performance. Detailed results about Waterbirds are shown in Appendix D.1.
Table Average and worst-group accuracy (%) on Waterbirds on CLIP backbones. Best results are in bold, top-2 in gray. PC+ selects limited fine-grained prompts than coarse-graines PC.
VIT-B/32
VIT-B/16
VIT-L/14
VIT-H/14
Method
Avg.t
Worstt
Avg.t
Worst
Avg.t
Worstt
Avg.t
Worst
CLIP (CVPR'21 [I])
64.88
34.55
77.20
50.93
75.65
52.51
77.55
41.28
TBD (ICLR'24 [18))
66.98
38.63
83.88
64.17
85.04
75.25
84.98
48.91
PC (ICLR 24 [15])
71.73
57.25
81.79
55.45
87.83
70.72
82.05
45.02
PC (ICLR'24 [15])
65.27
46.12
79.06
66.67
81.65
75.08
76.35
66.82
B2T (CVPR'24 [14))
68.59
56.23
77.46
64.64
84.12
48.13
77.25
46.57
Ours (External)
79.96
79.16
86.40
70.87
91.08
82.09
88.23
63.40
Ours (Internal)
81.93
71.81
85.43
82.71
90.71
85.67
87.66
67.45
Ours (Virtual)
83.69
65.73
89.47
67.13
91.94
80.84
88.70
68.38
Table 2: Performance on COCO-GB on ViT-B/16. We report average, female. and male subgroup accuracies, as well as the gender per- formance gap. And smaller gap indicates bet- ter fairness across gender groups.
Table 3: Performance on UrbanCars on VIT- B/16. We report accuracy the original co- occurrence (I.D.), background shifted (BG), and Co-object shifted (Co-Obj) subsets, along with overall average accuracy.
Method
Avg.t
Female
Male
Gap!
Method
Avg.t
LD.
Co-Objt
CL.P
88.70
85.60
91.80
6.20
CLIP
63.07
82.00
37.20
70.00
TBD
90.80
89.60
92.00
TBD
53.87
58.00
50.80
52.80
PC
90.50
89.00
92.00
PC
64.67
79.60
46.40
68.00
PC+
90.40
88.20
92.60
4.40
PC*
65.33
83,60
43.20
B2T
90.10
89.40
90.30
B2T
63.40
78.40
44.40
67.40
Ours (external)
91.10
89.80
92.40
2.60
Outs(External)
68.53
86.40
45.60
73.60
Ours (internal)
91.25
90.50
92.00
1,50
Ours(Intemal)
68.27
88.00
45.60
71.20
Ours (virtual)
91.40
91.80
91.00
0.80
Ours( (irstVirtual)
71.87
89.60
48.00
78.00

---

CLIP
Exteral
CLIP Internal
CLEP
Virtal
100
190
Accurccy (%)
Accunicy (%)
Accurcy (%)
W
60
80
Categories
Categories
Cutegories
(a) Accuracy on External variant
(b) Accuracy on Internal variant
(c) Accuracy on Virtual variant
Figure 6: Per-Class worst-group accuracy with CLIP (VIT-B/16) NICO. Comparison of worst-group accuracy across 19 categories on NICO (VIT-B/16) using CLIP and our external, internal, and virtual variants, Our method consistently improves the lowest-performing context for each class.
Performance on COCO-GB and UrbanCars. Tab. 2 presents results of subset about gender on the MS-COCO, demonstrating that although CLIP is pre-trained on large-scale real-world data, it remains vulnerable to contextual co-occurrence bias, particularly evident in gender prediction (see Fig. 2). For example, CLIP exhibits a notable performance gap of 6.20% between male and female samples. While PC+ improves male accuracy, it further amplifies gender disparity. Tab. further evaluates performance on UrbanCars. By analyzing samples with background shifts and co-occurring object shifts, observe substantial accuracy drops for CLIP, revealing its reliance spurious, con-dssrimmiatiie features. Although TBD enhances robustness against background bias, prenttrvi samples (I.D.) due to architectural it significantly deteriorates performance high average accuracy while consistently improving disruption. In contrast, our method maintains model generalization and stability. Additional results can be found in Appendix D.2 and D.3.
Performance on NICO. Fig. shows performance improvements of our three variants over CLIP (ViT-B/16), demonstrating consistent gains across diverse class within context-shifted dataset. The detailed comparisons are shown in Appendix D.4. Besides, we compare attention maps [37] of the counterfactual C,, with CLIP in NICO, and the Fig. 5 proves that the counterfactual module captures more complete embeddings about target object. Details are shown Appendix D.4.
Table 4: Ablation study on Waterbirds. Average zero-shot accuracy (%) under different module configurations. X indicates the ablation of the component: the TDE module in Eq. (15), sampler and intervention module via counterfactual calibration in Eq. (18).
Method
Variant
Sampler
TDE
Intervention
VTT-B/32
VIT-B/16
VIT-L/14
VIT-H/14
CLIP
64.88
77.20
75.65
77.55
Random
65.19
59.94
77.36
69.55
X
75.70
79.57
87.26
84.21
76.10
83.85
89.96
85.38
External
X
79.92
84,97
90.89
87,50
79.96
86.40
91.08
88,23
X
75.46
79.38
84.09
84,2
Ours
x
77.94
82,29
87 .0
82.10
Internal
80.79
85.33
88
87.57
81.93
85.43
90.71
87.66
X
75.39
79.03
84.28
82.21
88.47
88.16
86.38
Virtual
x
83.66
88.18
89.51
87.69
83.69
89.47
91.94
88.70
Ablation. To assess the contribution of each module, we conduct ablations on three components. In Tab. 4, removing any component consistently degrades performance, and the full model achieves the highest gains, it confirms that these module are synergistic in mitigating contextual bias. Moreover, using sampler-filtered random vectors to ablate context pool confirms semantically specific contexts required for counterfactual synthesis, while still outperforming the baseline in certain backbones.

---

More parameter setting and analysis are shown in Appendix C.2 and E.3. In addition, Appendix E. provides an extended comparison with multiple augmentation based baselines (e.g., Mixup, CutMix, AugMix, Cutout, and ALIA [28]), showing our method surpasses these zero-shot variants in accuracy.
Efficiency. Our method achieves significant improvements with remarkably low computational cost. A key advantage lies in performing counterfactual reasoning entirely at the representation level, which incurs much lower cost compared to conventional data augmentation methods. This design makes our approach highly scalable for large-scale datasets and high-throughput inference. For instance, on a single GPU, we can generate 100,000 counterfactual embeddings for a batch of 1,000 samples less than 3 seconds, with effortless context embedding construction, particularly for the virtual variant.
To quantify efficiency further, Appendix E.2 provides detailed FLOPs analysis per image, assuming pre-computed text embeddings. Compared to diffusion- based ALIA and common augmentation methods, our representation-leye while consistently improving accuracy across all backbones. In contrast, pixel-level approaches require several orders of magnitude more computation (+100 10 G), demonstrating the scalability and efficiency of our approach.
variants incur only marginal additional cost (+0.002 0.004 G),
5 Conclusion
We propose a lightweight, inference-only framework that mitigates contextual hallucinations in vision-language models via representation-level causal reasoning. By estimating image embeddings into object and background components, we subtract spurious effects by the Total Direct Effect (TDE) measure, and simulate intervention by constructing counterfactual embeddings via recombining object features with diverse alternative contexts from external data, batch neighbors, and textual descriptions. Without requiring retraining, prompt tuning, or generative models, ours method achieves strong zero-shot performance and sets a new state-of-the oh-the-ar on context-sensitive benchmarks.
Acknowledgements
This work was supported by the NSFCCU22412885 62222605).
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8763- PmLR, 2021.
[2] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11115-111111 1115-1118,,,,,11 June 2023.
[3] Ségolène Martin, Yunshi Huang, Fereshteh Shakeri, Jean-Christophy Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot clip. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pages 288 18816628826, June 2024.
[4] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 146-754,70 2023.
[5] Zhuang Liu and Kaiming He. A decade's 's battle on dataset bias: Are we there yet? In The Thirteenth International Conference Learning Representations, 2025.
[6] Robert Geirhos. Jörn- Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 3:665-673,, 2020.
[7] Yining Wang. Junjie Sun, Chenyue Wang, Mi Zhang, and Min Yang. Navigate beyond shortcuts: Debiased learning through the lens of neural collapse. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 12322-12331, June 2024.
10

---

[8] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
[9] Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20071-20082, 2023.
[10] Ruixiang Tang, Mengnan Du, Yuening Li, Zirui Liu, Na Zou, and Xia Hu. Mitigating gender bias in captioning systems. In Proceedings of the Web Conference 2021, pages 633-645- 2021.
[11] Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, 110:107383, 2021.
[12] Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksan- dar Shtedritski, and Max Bain. A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning. arXiv preprint arXiv: 11933, 2022.
[13] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36:35544-35575. 2023.
[14] Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jacho Lee, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11082-11092, 2024.
[15] Bang An, Sicheng Zhu, Michael-Andrei Panaitescu- Liess, Chaithanya Kumar Mummadi, and Furong Huang. PerceptionCLIP. Visual classification by inferring and conditioning contexts. In The Twelfth International Conference on Learning Representations, 2024.
[16] Rwiddhi Chakraborty, Yinong O Wang. Jialu Gao, Runkai Zheng, Cheng Zhang, and Fernando D De la Torre. Visual data diagnosis and debiasing with concept graphs. Advances in Neural Information Processing Systems, 37: 17:100333-006410 2024.
[17] Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Re. Conrect-n-contrast: a contrastive approach for improving robustness to spurious correlations. In International Conference on Machine Learning, pages 26484-26511.. PMLR, 2022.
[18] Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP's image repre- sentation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024.
[19] Phillip Howard, Avinash Madasu, Tiep Le, Gustavo Lujan Moreno, Anahita Bhiwandiwalla, and Vasudev Lal. Socialcounterfactuals::: Probing and mitigating intersectional social biases in vision-language models with counterfactual examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11975-1 1985, 2024.
[20] Judea Pearl. An introduction to causal inference. The international journna journal of biostatistics, 6(2), 2010.
[21] Judea Pearl. Direct and indirect effects. Probabilistic and Causal Inference: The Works of Judea Pearl, page 373, 2001.
[22] Tyler J VanderWeele. three-way decomposition of total effect into direct, indirect, and interactive effects. Epidemiology. 24(2))224-233) 2013.
[23] Wassily Hoeffding. A class of statistics with asymptotically normal distribution. Breakthroughs in statistics: Foundations and basic theory, pages 308-334, 1992.
[24] Gaelle Chastaing. Fabrice Gamboa, and Clémentine Prieur. Generalized hoeffding-sobol decomposition for dependent variables-application to sensitivity analysis. Electronic Journal of Statistics, 6:2420-244 2248,, 2012.

---

[25] Omar El-Dakkak and Giovanni Peccati. Hoeffding decompositions and um sequences. The Annals of Probability, pages 2280-2310. 2008.
[26]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[27] Axel Sauer and Andreas Geiger. arXiv:2101.00046, 2021.
Counterfactual generative networks. ar preprint
[28] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion- based augmentation. Advances in neural information processing systems, 36:79024-79034, 2023.
[29] Zhao-Min Chen, Xin Jin, and Sixian Chan. In pursuit of causal label correlations for multi-label image recognition. Advances in Neural Information Processing Systems, 11:51634-516544 2024.
[30] Wei QIN, Hanwang ZHANG, Richang HONG, Ee-Peng LIM, and Qianru SUN. Causal interventional training for image recognition. IEEE Transactions on Multimedia, 25:1033, 2023.
[31] Ming-Kun Xie, Jia-Hao Xiao, Pei Peng, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Counterfactual reasoning for multi-label image classification via patching- -baned training. In Proceedings of the 41st International Conference on Machine Learning, pages 54576-54589, 2024.
[32] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[33] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
[34] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilityy arXiv preprint arXiv:2301.052/17 2023.
[35] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CumbridgeUnniversisyyress 19(2):3, 2000.
[36] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[37] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localization. In Proceedings of the IEEE international conference computer vision, pages 618-626, 2017.
[38] Simone Fabbrizzi, Symeon Papadopoulos, Eirini Ntoutsi, and Ioannis Kompatsiaris. A survey on bias in visual datasets. Computer Vision and Image Understanding, 223: 2022.
[39] Ming-Kun Xie, Jiahao Xiao, Hao-Zhe Liu, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. dlas-distribution-aaaal pseudo-labeling for semi-supervised multi-label learning. Advances in Neural Information Processing Systems, 36:25731
2023.
[40] Ali Abdollahi, Mahdi Ghaznavi, Mohammad Reza Karimi Nejad, Arash Mari Oriyad, Reza Abbasi, Ali Salesi, Melika Behjati, Mohammad Hossein Rohban, and Mahdieh Soley- mani Baghshah. Gabinsight: Exploring gender-activity binding bias in vasion-language models. In ECAI 2024, pages 729-736. IOS Press, 2024.
12

---

[41] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38642 2022.
[42] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12700- 2021.
[43]
David Williams. Probability with martingales. Cambridge university press, 1991.
[44] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLwrencee Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755, Springer, 2014.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly claim our contributions about counterfactual calibration in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include discussion of limitations in the Appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: We provide the references of our assumptions and proofs for our proposed propositions in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We disclose our implementation details in Section and Appendix D.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We will upload our code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n /nips. wwtppcccccc/cceeccccccccccccclct hataaigggddeeeedeeuniiieeeeeee
/Coo//eoomissssseeeee srriissssoppiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We disclose our experimental setting in Section 4 and Appendix C.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct experiments with fixed random seed and select the best round as the result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We disclose experimental setting in Appendix C.2.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httppp:////uiis./.eeee Sotpblic/thicshuideelllee
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics adequately and we make sure followed the code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We use the open-source code and dataset, and they are explicitly cited in the paper.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /ttpa//:///e.p../eeeee /c//////eeeeeeccc/////////1 for what should or should not be described.
20

---

A Extended Related Work
Co-occurrence Bias in Vision-Language Models. Vision-language models (VLMss) often learn spurious correlations from frequent concept co-occurrences in training data [7, 38], leading them to rely on context cues or stereotypes as shorteuts [5, 39]. For example, if men frequently appear repairing appliances, the model may incorrectly associate the activity with male gender [40]. Such biases degrade performance on unbiased cases and raise fairness concerns. Recent work shows that popular zero-shot VLMs exhibit notable gender disparities in retrieval and classification tasks [19], largely due to underrepresented attributes like gender, ethnicity, or background context. These issues highlight the need for effective debiasing strategies.
Prompt-Rased Bias Mitigation Techniques. Prompt engineering mitigates bias in vision-language models by injecting context-aware cues into textual inputs. Some methods learn prompt embeddings adversarially to suppress spurious features such as gender or stereotypes [12], while others rewrite biased captions with more balanced phrasing [13]. These techniques are lightweight and compatible with pre-trained models, requiring no additional image data or architectural changes. However, their effectiveness depends heavily on task-specific keywords or priors. Poorly designed prompts may reduce performance or introduce new biases. To address this, recent work explores generating high- quality prompts, such as estimating contextual probabilities from images [15] or deriving targeted keywords through misclassification analysis [14].
Data Augmentation and Model-level Strategies. Data-centric methods mitigate bias by enrich- ing the training distribution with additional image-text pairs that disrupt spurious co-occurrences. Concept graph-guided augmentation [16] targets rare object-contex. pairs but depends on generative models, raising concerns about output quality and scalability.
At the model level, contrastive approaches like CNC [17] align same-class samples across biases while separating spurious correlations, improving feature discrimination. Other methods prune biased classification heads [18] to reduce reliance on shortcut signals, though this may impair generalization. MEMO [41] adapt model predictions across multiple augmented views of each input by minimizing the entropy of their aggregated output distribution, thereby encouraging consistency under distribution shifts. While effective, these strategies typically require retraining or external modules, limiting their use in zero-shot settings.
Causal and Counterfactual Reasoning Approaches. Causal inference offers principled way to mitigate spurious correlations by modeling the data-generating process. Unlike prompt tuning or data augmentation, causal methods operate directly on variables to estimate and suppress undesired biases. In vangon-language tasks, counterfactual reasoning has been used to reweight or subtract biased modality signals, such as in counterfactual VQA [42]. which isolates language priors using Total Direct Effect (TDE).
Recent methods extend this idea by decomposing representations via generative models [27], training dual-branch architectures to decouple confounders [29), or applying TDE at the patch level to weaken local shortcuts [31]. However, these approaches often require segmentation, auxiliary branches, or retraining, limiting their practicality in zero-shot or inference-only settings.
In contrast, representation-level counterfactual methods are well-suitable to zero-shot VLMs due to their simplicity and compatibility. Combining causal reasoning with efficient inference-time interventions offers promising direction for scalable bias mitigation in open- open-world vasion-language applications.
B Theoretical Supplements
B.1 Decomposition for CLIP Embeddings
This section provides a compact derivation of the object-contex" factorization for both visual and textual encoders and explains, from an cotomationnthhoreee viewpoint, how interaction terms emerge under the InfoNCE objective which is the training loss for CLIP.
21

---

Notation Restatement. X E X' and Z E Z denote the object and context random variables re- spectively.For completeness we write p(i, xx 2) for the training distribution and assume the image embedding fi) IR and text embedding fz( IR belong to the Hilbert space L ( 2, z)).
Then, we use the shorthand E[- X==] for Ex-paxx=a]I1 and similar for Z or (X, Z). All
embeddings from dataset are assumed centered: [B[A(()) = B[A(T)) =.
B.1.1 Visual Two-Factor Hoeffding Decomposition
Let (z (i) |
X= Z==] denote the conditional mean embedding obtained from dataset information, which encapsulates the complete statistical memory of the training set for (z, x)
pairs. Projecting g onto the two one- mecto subspaces {g: (z. ))) and {g:(;, x)} of the Hilbert space L (p(x,, z)) yields the object and context main effect functions:
(xx) = Et)((t) | X = x]) e,(2) = (() Z = z].
(20)
Subtracting the two way projections from the joint expectations obtain the interaction residue r,( = ( 2) - (z) (z). By construction, for every fixed x we have
E (x,, 2) X æ E g (z, 2) x= x] ex( E(e
(21)
e ( (x(
0,
and similarly Bir; Z=z]= = 0. Orthogonality follows directly from this property: let Vx=(h(XX:
h (p(()))) } and Vz=(k(Z)= k E L [p(()) be the two one-factor subspaces of L (p(xx, x)).
For any ((X) E Vx, according to Law of Total Expectation:
(ra, (x)) = E(ra h(x) B(N(X) X7] 0,
(22)
and the same argument with k(Z) shows (ri, Vz) O. Hence r; is orthogonal to Vx UV..
The residual term T3 fi(i) g: (X, Z) satisfies E(n; X, -]
which for any function (x + (Z) + (X, Z) belonging to the direct sum Vx + Vz + Vxz (with centered in both arguments), the inner product with π7 vanishes after taking conditional expectations. It essentially captures higher-order variability orthogonal to all lower-order components (i.e., object, background, and interaction terms).
Consequently the four components ex(X). e; rx(X, Z), ') are pairwise orthogonal, giving rise to the unique expansion reported in Eq. (1) of the main text:
[i(1) = (x) + ez(z) + (x, + 1/--
(23)
When the data distribution is ideal distribution where the pairs of (æx, =) appears in dataset evenly, the joint mean embedding factorizes into the sum of margins and r, collapses to O, i.e. X Z which means object and background are then fully "decoupled" in this model. In practice, measures the extent which this ideal fails because of systematic co-occurrence.
What are fi) and (a, 2). The mapping f; : I -> Rd is a deterministic encoder produced by CLIP after contrastive training. For any concrete image it outputs an instance-level embedding (i). The quantity 9e (2 z) E [i) X==x, Z=z] is the conditional mean embedding: it averages the encoder output over all images in the training distribution that share the same abject-contes label pair (I, 2). Formally,
/ f.(1) p(i x= Z =) di,
9. 2)
p(i))ppi
(24)
which can be estimated in practice by empirical averaging across the mini-batch samples carrying the label (ax 2). Hence g may be viewed as the "prototype" or population center of all cebharross-inoceeer crane bird-in-forest, ... images that the encoder has seen during training. Because the encoder is fixed, the only randomness in (i) comes from the data distribution; conditioning on (X, Z) therefore yields well-defined mean element gi (x: 2) (z)())))
'The la normalization in CLIP is applied after the embedding is fed into the contrastive loss; the Hoetfding expansion applies to the pre- normalized vectors. For clarity, omit this distinction in symbols.
22

---

B.1.2 Textual Two-Factor Hoeffding Decomposition
Let (a 2) (T) x= --=] denote the conditional mean embedding of the text encoder
over all prompts (T T:) generated from label pair (x, 2). Projecting 9t onto the two one-factor
subspaces (g, and ()) of the Hilbert space L (2( 2)) yields the object and context
main effect functions:
ex(x) (T) X-x],
ex(2) = Bl)(((T Z=z].
(25)
Subtracting these from the joint expectations gives the text interaction remainder ra x) (xx =) (x (z). which, by construction, E[rr X=x E/r, Z= 0 and is
therefore orthogonal to both one-way subspaces.
Finally the residual terms V/2 [T(T) - gr ; Z) satisfies Ely X, Z 0. Collecting these pieces yields the unique orthogonal expansion:
fT T:) (x) e<(z) (x, 2) 7)
(26)
which analogous to Eq. (9) for the visual encoder but on the text side.
B.2 Interaction Amplification under InfoNCE
We now place the two-factor decompositions Eq. (23) and Eq. (26) into the CLIP training loss and give a self-contained argument that, whenever an object-context pair (x, 2) appears abnormally frequent than other pairs, gradient optimization enlarges the interaction terms (æ z) and (z z).
Score Decomposition. Insert the two- factor decomposed embeddings into the normalized inner product S(i, 7) = (fs, /T appearing in CLIP's InfoNCE loss. For a "positive" pair, where i in p(i X x, z) is an image whose true object label is æ and context label 2, we obtain:
S++ == Tz.,)
= (e ex())) + (e.(z), ex(x)) + (((())
+ (e((t)),
main effects
cmass alignment
(e,(x) e((2),)) + (x) rz(2),))) + (na.r)
(27)
interaction
+ (ms fi) + (me, fi)
residual
The two one-factor hard negatives (x, 2²) and (æ', 2) share exactly main effect vector with the positive sample, hence S4_ = Si Ta, and S_+ = S(i 2, Tw',)) differ from S4+ primarily in the interaction terms. Explicitly,
1 S++ S+ [(e(()) ex(z)) (e,(z'), e((2'))
+ (z) - e,()), ex(x)) + (e((2) - ex(2)) ex())) (e, (=)())))<(2xx) (e, -x,
(28)
+ (1)(x))) -z,, +e]
where is oigeer-order residual term about 7) and 7/-- Thus the margin separating the positive from its hardest negatives is governed by both the alignmat and interaction components. similar expression holds for S++ S_+-
InfoNCE Gradient on ræ. 2), (x, 2). For a mini-batch data DD, minimizing the InfoNCE loss: -[S_- logE(a (w' z'jen e((.T')] yields the gradient
Lncc = [T, 2) - -0t, re( 2²) o0,-+ (2(2
(29)
Lnce (2x, - -0H,+- ri(x 2') -0s-44 ri x)]]
TTo facilitate display, 1' consist of Th and T'r
23

---

where Ci and at are softmax weights for the two hardest negatives for visual and textual model. (z, = occurs more frequently than (x. x') (æ', ,2) in data distribution, then a a+-, a-+ are small and the gradient term -r, dominates, pushing ri in the +r, direction. By symmetry, an analogous expression holds for Vngen Iterated SGD steps therefore inflate the terms ra 2) and ry (a, 2) until the margin S++ max S_+} is large enough to satisfy the batch softmax.
B.3 Cross-modal Attraction on Co-occurring Context and Object under InfoNCE
In addition to the interaction gradients ri,'t, InfoNCE also induces cross-alignment gradients on the one-way main effects (=(=) and (x:) whenever (ær z) co-occurs more frequently than its one-factor negatives (ar,z') and (ar, z).
Recall that one of the positive score cross aliinmmnl term (e,(z), ex())) and the negative ones for (a', =): (z,(z), e, (xx)). )). All other negatives either involve x' (do not dependence on ex))) or only marginal effects. For a mini-batch data, minimizing the InfoNCE loss and its gradient projected onto (z) is
1 ex(a)) ez())
V Vei(s)Cnco
-
(30)
(e4-(e ex(zz)) -
Σ or es(2T))
( + -T
Since 0i is the softmax weight on the hardest negative (x, z') and (æ', 2). when (x, 2) is high-PMI (high -frequency co -occurrence) pair a4.F < 1. The dominant terms are (x). so the
net gradient points approximately toward +cx(2)).
Symmetrically, (an(Cuue contains -e,(z) + e,+),(z), also pulling ex(x) toward +=)(2)
0.9
Class from ImageNet
in case of high-PMI pair. Hence frequent co-occurrences induce a cross modal attrac- tion between background main effect (z) and object main effect (x),, explaining why the background vector can align with the object dictionary entry and thereby raise the spurious score (T.)) in Eq. (4).
bost
Similarit with "Sea" "Sea""(osii"" (cosine)
0.8
cake
line!
pir
sadda
brokwage
0.7
fletoa meuuin
cosh
0.6
diging
stampe Ce.
squirdl
pry
sitben
The onl- teme
0.4
These cross-alignment gradients demonstrate that InfoNCE training actively aligns the back- ground main effect (z) with the object prompt embedding (x) whenever (x, =) co-occurs frequently. Together with the interaction am- plification ri, rt, this mechanism explains why the spurious shorteut + Z - Y grows stronger under frequent co-occurrence and derpins the observed hallucination bias in novel scenes.
iulii
zilled gous
Carilin anee
0.3
ostele
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Similarity with "Forest" (cosine)
Figure The illustration of cross-modal attrac- tion. It compares the textual similarity of two scenes and a part of objects from ImageNet, shown by using f(')) as bridge.
B.4 Information Entropy Perspective for Co-occurrence Shortcuts.
The InfoNCE loss minimizes a tractable lower bound of the mutual information T(X;Z) between objects and scenes:
I(X; 2 log(K +1) - LInGONCF (0),
(31)
where K denotes the number of negative samples per positive pair (typically K = - for a batch size of N), therefore implies that training maximizes the average pointwise mutual information across observed pairs.
p(x,
I T(X; 2) Ep(x,s) (PMI z)] Ep( [log
(32)
p(rr) (<))p(z))
The CLIP score in Eq. (27) can be decomposed into three main parts: one-way terms, cross-modal terms, and interaction term involving residuals. These components are directly affected by the local PMI(#, x).
24

---

When (x. z) is frequently co-occurring object-scent pair (i.e., high (M 2)), the InfoNCE gradient includes attractive forces:
-v x rt. LV &T; + rı(2,, (x(
Meanwhile, the cross-modal alignment terms are also reinforced:
<V x e,(2), (x) e,())
(e,( 1
This "double boost" both magnifies the margin between positives and hard negatives, and strengthens the shortcut path which is a key mechanism underlying hallucination under co-occurrence bias.
In contrast, when (x, x) is attractive gradient to vanish or reverse. The optimizer keeps:
anti-correlated, the PMI becomes negative zero, causing the
ra 2), (x 2) - O, ea(z) 1. ex(2),
On the other hand, if pairs come with an almost fair and uniform data distribution, also allows the model learns decoupled, object-centric representation.
In conclusion, by maximizing the expectation of PMI, InfoNCE selectively increases interaction energy and modss alignment only for Ligh-PMII object-scene pairs. These components including rs, rt, and (=), e, (z)) serve as faithful proxies for dataset co-occurrence bias. When the joint distribution p(æ, =) is decomposed by independence, the learned representation is robust and object-
aligned; but when p(a, z) is skewed, InfoNCE implicitly injects shortcut dependencies exactly where the bias is strongest.
B.5 Prompt-Induced Mitigation of Bias
This section shows formally how appending explicit scene token T3 blocks the spurious causal path I- Z +Y, shown in Fig. 3, and also characterizes the effect of an imprecise or incorrect T".
Chusal-Intervention View for Prompt-Based Method. Let Z be the latent scene label inferred from image I, and the target class. In the object-only regime the model effectively computes the probability of class c:
p(Y I, To.)
ΣN I, T, p(Z = #E2
(33)
so the pathway I Z + Y remains open and can introduce spurious hallucination whenever p(Z Σ) favors classes co-occurring with certain scenes.
By appending Tr, intervene on Z, clamping it to the observed value:
P I, Te, T:) p(Y II, Te, Z 2) x p(Z Tc, where PZ z I.To, T.) T-Z-1 shorteut entirely, leaving only the legitimate I - Y path.
This removes the marginalization over and thus blocks the
However, if instead Z at the wrong value. The causal path is still blocked, but now the model conditions on 7 = Te which mismatches the true scene, shifting probability mass toward classes co-occurring with T and potentially introducing new nisclassifications.
imprecise incorrect scene token T + T., the intervention locks
As the experiment in Fig. 8 demonstrates, using simple object-only To fails on both object predic- tions. The model recovers the desired predictions when using the full scene descriptions Third (in a forest with dense vegetation and massy stones riverbank) and TX* the ocean with gentle waves and surrounded by green hills and clear sky), whereas the situation only improves to some extent when using the vague descriptions Trat trees) and Teat water). Alarmingly, the use of wrong descriptions can make the model predictions worse and introduce new misclassification targets, i.e., Third (in the zoo with lush green grass), Third (in foggy swamp, wading through murky water) and T (in the art gallery with a starry oil painting on the wall), Teat (in the zoo with lush green grass).
B.6 Proof of the Direct-F.ffect Decomposition
We provide the algebraic details omitted from the main text, showing how the residual stream unrolls into additive direct-effect terms and why all LayerNorm (LN) operations can absorbed into single projection matrix R up constant bias.
25

---

photo
Wile 9034
Sanao 2000
Siain oa 1835
13:0
13.6
11.64
mer 2442
Spcor
1agal
photo tabby
7g
7.
Figure 8: Impact of context-match level in T3 on prediction scores. Analysis performed on CLIP (ViT-B-16) for two images about albatross and tabby cat under novel contextual conditions. Five kinds of prompt variants were tested: To baseline (object-only. no context); Tj and T2 are ambiguous and precise scene descriptions; T's and Ta are incorrect or even adversarial scene descriptions
Residual Stream Induction Let to E7, f be the token matrix t E (N+)) fed the first Transformer block. Each layer I {I, consists of:
MSA (t' (t- ++++++++++ + = MLP (LN) (LN. (t)) +
(34)
Let the residual contributions applying the shared projection P. and substitute the first into the second, then the Eq. 34 become:
P Pitl P P Joio (LM (c
(35)
By simple induction), we obtain:
P P P
ols + P
MSA' Meels + P P (LL((
(36)
1=1
Noting that P Jat (i) recovers Eq. (7) and thus exhibits the desired additive decomposition into CLS, MSA, and ML. direct effects.
Absorbing the Pre-projection LayerNorm Most CLIP variants apply LN after t but before the projection layer,this term gives:
LN(I)
ny Val+c 8)
(37)
Vo24c
where e RV is token, μt, o'are the mean and standard deviation, and 7, 3 are learned vectors. We absorb the scale I into the projection P and distribute the constant bias 1I equally to every direct-effect term in Eq. (36). Since image-independent, this constant shift changes neither cosine similarities downstream zero-shot decisions.
Justification of the MSA Dominated Semantic Effect We follow mean-ablation way: each compo- nent (CLS, all MLPs, set of MSAs) is replaced by its batch mean, leaving the remainder of the network untouched. Formally, for batch D of size and we substitute a component vector:
5 x' R'DD
(38)
D
From Tab. 5, while the decomposition in Eq. (36) is algebraically exact, not all components contribute equally to semantic alignment with textual prompt zero-shot classification. The initial class token 10 acts as global visual prior and does not encode class-specific cues. Empirically, its mean-ablation results weakly decreases (around 0.5% on ImageNet)), suggesting that it plays stabilizing but non-discriminative role. The MLP blocks, which apply non-lincat transformations to
26

---

Table Mean-ablation results on CLIP-ViT components. We evaluate zero-shot top-1 accuracy on ImageNet by ablating different architectural components in four ViT structures. Each ablation [34] replaces the corresponding residual stream component (e.g. class token, MLPs, MSAs) with its mean over the batch, in Eq. (38).
Model
Base accuracy
Class token ablation
MLPs ablation
Class token + MLPs
MSA ablation
ViT-B/32
65.63
64.96
64.52
64.27
0.18
VIT-B/16
68.58
68. 16
67.73
67. 09.09
0.13
VIT-L/14
74.01
73.72
73.22
73.15
0.05
VIT-H/14
76.40
75.88
75.36
75.28
0.03
each token independently, serve primarily to normalize and rescale features. They do not aggregate new semantic content and result weakly decreases (around 1.0% on ImageNet)).
In contrast, the MSA blocks perform dynamic, content-dependeee token mixing. For each layer and attention head, patch tokens that are semantically aligned with the class description receive higher attention scores and are explicitly pooled into the CLS token. This operation introduces the main class-relevan signal into the residual stream. Quantitatively. when ablated for MSAs, it leads to the most significant drop in zero-shot classification accuracy. This clearly indicates that the MSA component is the principal driver of semantic alignment in CLIP-ViT.
Thus, in the way of Eq. (9), we isolate the semantically meaningful path through which each patch contributes to the final image embedding, and excludes the residuals (class token and MLPs) whose contributions are either generic or semantically neutral.
B.7 The Proof of Conditional Expectation as the Minimum-MSE Predictor
Without loss generality, this section focuses on C(z) as our object of study. The proof leads to the following conclusions. The vector C(z) is the unique minimizer of mean-squared error, the centroid of the conditional background manifold, making Eq. (11) the optimal first-order solution in both theoretical senses.
Notation. Let (2, F.P) P) be a probability space and
X X
X L (i.e. E| < oc).
Let G C. F be a sub-o-algebra that represents the information available to a predictor. A map a: Ω +B-RR R.D PD G-measurable when it depends only on g.
Orthogonality (Pythagoras) Identity. For every G-measurable a,
xX= = E||x - E[X 9 1 + ||a - E[X 9
(39)
The first term is the irreducible error; the second is the non- -engatittvve excess error that vanishes only E G). Hence
= X 9]
is the unique minimiser of MSE among all G-measurable predictors [43].
Instantiation for Counterfactual Embeddings. Let v(i) IR," be the image embedding of input i Define:
X = fi(1,, G= = oG 2) (token is background).
Applying (39) yields the centroid of the conditional background manifold
C(z) = = (1) G
J P(A,)))=2](T 2) dr.
(40)
which uniquely minimizes:
E (1) a all
DEI a is G-measurable.
Equivalently, after sampling for Eq. (40), each token can be written as (i); = 2 And, summation is composed of C(z). Thus, when only the background information G = 2 is known, C(=)
the MMSS--ptimal first-order mean approximation the true embedding vii). This provides principled baseline for constructing counterfactual representations.
27

---

C Details of the Experiment Setting
C.1 Dataset Description
We evaluate our methods on four widely adopted benchmark datasets specifically designed assess contextual robustness under spurious correlations and distribution shifts. These datasets span synthetic and real-world biases and provide controlled setups for evaluating zero-shot generalization under strong object-context entanglement:
Waterbirds [8]. constructed by compositing bird foregrounds from CUB-200-2011 with natural scenes from Places [36]. focuses on background correlation. It comprises total of 10,589 im- ages, including 4,795 training images and 5,794 test images, categorized into the classes: waterbirds and landbirds." In the training set reflecting reality, over 95% of waterbirds appear in water back- grounds and landbirds in land backgrounds, presenting strong spurious context-class associations. The test set balances background distributions to evaluate generalization beyond shortcuts
UrbanCars [9]. It derived from Stanford Cars, Places, and LVIS datasets, introduces multi-bias settings by synthetically composing each image with a car object, a background (urban or rural), and a co-occurring object (e.g., fire hydrants, cows). The training set contains 8,000 images, with strong three-way correlations between class, background, and co-occurring object. Both validation and test sets contain 1000 balanced samples where the spurious cues have extra tags which enables fine-grained analysis of shortcut reliance and interaction among multiple biases.
COCO-GB [10]. It reorganized from MS-COCO [44), targets gender context co-occurrence biases in image captioning. Based on gender annotation inferred from captions and manual verification, two versions are constructed: vI includes test set of images with equal male and female representation across balanced contexts for fairness analysis. These setups expose hallucinated gender attributions when models rely excessively on contextual priors rather than visual cues, shown in Fig. 2.
NICO [11]. is a real-world dataset contains over 25,000 images spanning 19 object classes (10 animals, 9 vehicles) across 188 distinct contexts (e.g.. "dog grass", "car in snow"). Each class appears in 10 different environmental or situational contexts. It serves as a benchmark built specifically to study context bias and causal learning, aiming at exploring causal relationships between objects and their contexts (e.g., background, scene, co-occurring objects) in object recognition tasks.
Together, these datasets enable rigorous and controlled evaluation of models' ability to disentangle semantic representations from contextually entangled signals, particularly under zero-shot and distribution-shifted scenarios. Fig. 9 displays a subset of images from those dataset.
C.2 Implementation Details
We four CLIP vision backbones: VIT-B/32, ViT-B/16, VIT-I/14, and ViT-H/14, all publicly avail- able via OpenCLIP and pretrained on the LAION-2B-en dataset [26], which contains over 2.3 billion image-text pairs. The pretrained model are: laion2b_ss s___3__b_____ b79k for ViT-B/32, laion2b s34b b88k for ViT-B/16, laion2b Lioon2__s2l________ b82k for both ViT-L/14 and laion2b_s322 2_69kbb79kk2 for ViT-H/14. Follow- ing CLIP's original setting [1], we adopt 80 hand-crafted text prompts per class to support standard zero-shot evaluation.
Then, implement three versions of CounterfactualCL.PP in this paper, distinguished by the source of counterfactual scene embeddings. Each follows a unified architecture and inference procedure, differing only in how background contexts are collected:
External Scene Variant. This variant uses scene images sampled from the Places dataset. For each target dataset (e.g.. Waterbirds), we manually select 16 relevant scene categories based on background diversity observed the data (see Tab. 6). For each category, 50 images are randomly drawn from Places, yielding fixed pool of 800 candidate backgrounds. At inference time, for each test image, We employed two constraints to uniformly select a total of M scenes from 16 relevant scene categories in the scene pool: (1) maximized scene dissimilarity with the original input; and (2) Maximized visual dissimilarity with the foreground object class. The selected scenes are embedded via the CLIP image encoder and used to synthesize counterfactual image representations.
Internal Scene Variant. Here, the candidate pool constructed online using other samples in the same test-time batch. We exclude the current image and select M candidates from remaining batch
28

---

Figure 9: Examples from four datasets: Waterbirds, UrbanCars, COCO-GB and NICO.
Table 6: Selected scene categories from the Places dataset for each target dataset the External Scene Variant.
Dataset
Scene Categories (16 total)
Waterbirds
marsh, pond, lake-natural, river, swamp, creek, canal-natural, fishpond, forest-broadleaf, bamboo forest, orchard, field-cultivated, field-wild, pasture, farm, forest_pathh
UrbanCars
street, downtown, parking Jot, parking garage-outdond highway, gas_ station, viaduct. crosswalk, field-cultivated. field road, forest road, village, farm, pasture, barn
COCO-GB
living room, bedroom, kitchen, bathroom, dining room, office, classroom, hotel room, restaurant, shopping lactiinggmlliinne mndl-rn bus settion--nnoon supermarket, park, street. playground
NICO
heach, bridge, desert_road, forest_patt river, snowfield, street, residential neighborhood, home_office, mountain snooii pasture, airplane_cabin, vrain_station-platfo garage-outdoor, amphitheater. corral
samples by Eq. (11), applying the same semantic dissimilarity filters as the external variant. This variant is more dynamic and requires no external data but depends on batch diversity.
For both external and internal variants contexts selected uniformly across categories (external) or available pool entries (internal) after dissimilarity filtering. This helps maintain scene diversity and avoid sampling dominant or overly similar backgrounds.
Virtual Scene Variant. This variant constructs purely text-based counterfactual pool. For each dataset, we generate 400 diverse scene descriptions using a large language model. The prompt is phrased as:
Generate [NUMBER] unique, single-sentence scene descriptions that together span the full range of environments found in [DATASET], without referring any [CLASS] specific content.
These descriptions, partly shown Tab. 7, are then encoded via CLIP text encoder to obtain virtual scene embeddings, which act as background surrogates during counterfactual synthesis. This variant lightweight, lanol-agnostic, and easily extensible. the experiment, we sample M background scenes from candidate pool of size B 400 for each image synthesize the virtual counterfactuals,
29

---

Table Examples of generated scene descriptions for the Virtual scene variant on Waterbirds and COCO-GB.
Waterbirds
COCO-GB
# Scene Description
# Scene Description
clear coast near tidal pool the early morn-
sidewalk café with metal chairs and small round tables facing the street.
ing.
2
A misty floodplain near gravel path rainy
2 A hospital room with medical equipment, mon-
day.
itor, and bed by the window.
3 shimmering marsh near a grassy bank at dawn.
3
forest clearing surrounded by tall trees and carpet fallen leaves.
4 A glassy wetland near a coastal bluff on rainy
4
A classroom with desks arranged in rows and a
day.
whiteboard covered in diagrams.
5 A foggy near gravel path on rainy day.
5
A theater stage with drawn curtains and lights focused on the center.
6 A shaded lagoon near a dune on a rainy day.
6
A public plaza with a fountain in the center and benches arranged around it.
7 A tranquil river near reed bed under midday 7
A mountain hiking trail winding between large boulders and dense pine
sun.
8 glassy lake near wooden pier in the early 8
shopping mall interior with decorative plants and store signs on every side.
morning.
for instance, with M (270, 270, 190, 903 for ViT-B/32, ViT-B/16, ViT-L/14, and VYT-H/14 on Waterbirds, respectively.
Additional Sarrae-praaeeter. During counterfactual synthesis we blend the object and background embeddings with fusion weight a. We find a selects 0.5-0.7 yields the best trade-off. The coefficient A in Eq. (19) regulates how much of the learned object-context interaction is retained in the final score; values 0.6-0.8 consistently work well. In Eq. (15), we introduce an additional factor X to control the malignant hallucination term, normally selecting unless stated otherwise.
When forming a counterfactual image embedding in Eq. (11) optionally discard token contributions whose probability is below a threshold, which can be tightened for unknown noisy. It sets around 0.3 usually provides good balance. Finally, to limit inference overhead, the operation of Eq. (18) are applied only to the top-5 classes returned by the initial softmax; all other classes still receive the TDE correction. The intuition stems from the observation that the model only needs to imagine the top-K most confusable categories in alternative contexts to achieve robust predictions. A sensitivity analysis of some of parameters above presented in Appendix E.3.
D Details of the Experiment Results
D.1 More Results on Waterbirds
Tab. 8 reports per-group accuracy Waterbirds for all VIT backbones, where each sample is defined by its object label (landbird: LB or waterbird: WB) and background (land: L or water: W). The diagonal cells (LB L. WB-W) WB) correspond familiar cells (LB-W) W. WB-L) represent the hardest OOD compositions that expose context bias.
pairs, whereas off-diagonal
Across backbones, vanilla CLIP shows strong degradation on LB-W and WB-L groups. For example, only 34.6% on LB-W with ViT-B/32 confirms its reliance on background cues. CounterfactualCL.PP substantially lifts these two worst groups while maintaining or improving the in-distribution pairs. These detailed results above further illustrate that our method suppresses spurious scene shortcuts and restores object-centric decision making.
D.2 More Results on COCO-GB
Tab. 9 presents the average and subgroup accuracy on COCO-GB v1 across ViT-B/32, VIT-L/14, and ViT-H/14 backbones. We observe that vanilla CLIP suffers from notable gender gaps (e.g.6.00% on ViT-B/32), indicating that model predictions are influenced by gender-context co-occurrence bias.
30

---

Table Performance on Waterbirds across different ViT backbones. LB: landbird, WB: water- bird; L: land background, W: water background.
Backbone
Method
LB-L
LB-W1
WB-L1
WB-W1
Avg.t
CLIP-base
88.47
34.55
59.03
94.39
64.88
Ours(Extemal)
79.16
80.67
79,91
80.37
79.96
ViT-B/32
Ours(Internal)
87.32
78.98
71.81
83.49
81.93
Ours( (Virtual)
90.64
82.93
65.73
79.91
83.69
CLIP--ase
95.57
63.81
50.93
85.98
77.20
(urs(Exteraa))
92.90
86.39
70.87
79.13
86.40
ViT-B/16
Ours(Internal)
88.16
82.88
82.71
87.54
85.43
Ours( Virtual)
95.30
94.41
67.13
73.99
89.47
CLIP-base
95.30
$2.51
71.03
92.52
75.65
(xts(Exteral))
92.82
93.13
86.76
82.09
91.08
ViT-L/14
Ours(Internll)
93.84
89.00
85.67
90,81
90.71
Ours( Virtual)
95.61
93.97
83.02
80.84
91.94
CLIP-base
96.27
71.65
41.28
68.85
77.55
Ours(Externa))
93.75
93.92
73.68
63,40
88.23
VIT-H/14
Ours(Internal)
92.24
92.68
74.14
67.45
87.66
Ours( Virtual)
97.21
91.18
68.38
70.40
88.70
Table 9: Average and gender-specific accuracy (%) across three CLIP backbones on COCO-GB v1. Gap denotes gender performance differences.
Backbone
Method
Avg.t
Femalet
Male
Gapl
CLIP
86.20
83.20
89.20
6.00
TBD
86.60
82.20
91.00
8,80
PC
86.30
84,80
87.80
3,00
PC+
86.80
83.40
90.20
6.80
B2T
86.60
81,40
91.80
10.40
VIT-B/32
Ours (Extemaa)
87.30
85.80
88.80
3.00
(Internal)
$7.00
85.40
3.20
Ours (Virtual)
87.20
84,80
89.60
4.80
CLIP
91.05
90,50
91.60
1.10
TBD
91.60
91.00
92.20
1.20
PC
91.80
92.60
91.00
1.60
PC+
91.50
90.40
92.60
2.20
B2T
91.50
90.00
93.00
3.00
VIT-L/14
Ours (External)
91.70
91.80
91.60
0.20
Ours (Internal)
91.75
92.10
91.40
0.70
Ours (Virtual)
92.15
92.60
91.70
0.90
CLIP
92.70
93.40
92.00
1.40
TBD PC PC+ B2T
92.30
93,40
91.20
2.20
92.80
92.60
93.00
0.40
93.10
92.60
94.00
1.40
91.70
92,20
91.20
1.00
VIT-H/114
Ours (Extemal)
92.90
92.60
93.20
0.60
Ours (Intemnal)
93.20
93.40
93.00
0.40
Ours (Virtual)
93.00
93.20
92.80
0.40
31

---

CounterfactualCLII maintains competitive or better average accuracy while significantly narrowing the gender gap across all backbones. On ViT-L/14, for example, our external variant achieves a gap of only 0.20%, and on ViT-H/14, the virtual variant reduces the gap to just 0.40%. These results confirm that our method effectively mitigates gender-associated spurious correlations without sacrificing overall performance. Results for ViT-B/16 are reported separately in Tab. 2).
Table 10: Average and factor-specific accuracy (%) on UrbanCars across three CLIP backbones. We report accuracy on the original co-occurrence (I.D.), background shifted (BG), and Co-object shifted (Co-Obj) subsets, along with overall average accuracy.
Backhone
Method
Avg.t
I.D.
BG
Co-Ohjf
CLIP
63.73
79.60
36.40
75.20
TBD
59.33
74.00
37.60
66.40
PC
69.73
83.20
52.40
73.60
PC
70,93
84.40
48.00
80.40
B2T
67.47
84.80
36.80
80.80
VIT-B/32
Ours (External)
71.60
84.40
50.00
80.40
Ours (Interna))
72.00
86.40
48.00
81.60
Ours (Virtual)
71.07
86.80
44.00
82.40
CLIP
62.27
80.40
32.80
73.60
TBD
55,60
66.00
44.80
56.00
PC
62.67
77.60
45.60
64.80
PC
64.40
84.80
36.00
72.40
B2T
61.60
79.60
32.00
73.20
VIT-L/14
Ours (External)
63.07
83.20
35.60
70.40
Ours (Internal)
64.40
84,00
34.80
74.40
Ours (Virtual)
66.00
88.40
36.00
73.60
CLIP
62.27
84,00
34.40
68.40
TBD
60.53
72.80
49.60
59.20
PC
63.20
82.00
39.60
68.00
PC
64.13
82.80
38.40
71.20
B2T
60.93
78.40
31.20
73.20
VIT-H/14
Ours (External)
66.00
85.20
44,00
68.80
Ours (Internal)
68.27
90.00
46.00
68.80
Ours (Virtual)
68.40
88.80
47.20
69.20
D.3 More Results on UrbanCars
Tab. 10 presents detailed breakdown on the UrbanCars dataset across VIT-B/32, VIT-L/14, and ViT-H/I4 backbones. The three columns (I.D., BG, Co-Obj) respectively represent accuracy on original co-occurring samples, background-shifted samples, and co-object-shifted samples. CLIP shows sharp performance drops under distribution shifts-fol example, on ViT-B/32, accuracy drops from 79.6% (I.D.) to only 36. (BG)). highlighting vulnerability background bias. In contrast, CounterfactualCL.PP maintains strong performance across all settings. On ViT-B/32, our internal variant achieves the highest average accuracy (72.00%). while virtual variant yields the best balance on ViT-H/14, reaching 68.40% average and 47. 47.20% BG accuracy-substantia substantially outperforming CLIP by over 12 points. For ViT-B/16 results, refer to Appendix Tab. 3.
D.4 More Results on NICO
NICO standard benchmark for evaluating contextual robustness, as it includes diverse object categories across varying backgrounds. Tab. 11 and 12 report both average and worst-group accuracy on ViT-B/32 and ViT-B/16. Across both backbones, outperform the baselines in worst-group accuracy, particularly for context-sensitive categories such as "cat" and "sheep." Notably, the virtual variant achieves the best balance between maintaining high average accuracy and significantly improving worst-group performance, highlighting its effectiveness
three variants ours methods consistently
32

---

in mitigating context-induce induced failures. This improvement stems from ours ability to simulate diverse scene interventions during inference without disrupting semantic alignment.
Table 11: Average and worst-group accuracy (%) per-category on NICO (VIT-B/32).
sirphon
CLPP
95.19
16.44
19.16
V1.25
18.46
V9.50
VALUE
95.17
MALTY
V6.25
V1.33
18.37
VIR
VALE
V9.15
VS.54
VIAS
97.79
95.83
58.53
17.47
99.14
96.40
93.74
95.12
98.25
95 5M
99.3
97.42
97.50
98.66
97.13
93.53
98.16
98
9400
Avrrage
PC-
05.54
48.00
$9.14
96.70
19.75
V7.57
07.88
91.96
94.54
29.15
95.41
VERS
96.73
94.54
$5.46
97.35
$6.44
96.74
66
97.57
96.35
97.15
93.94
Ounicaterms)
96.72
05.95
18.97
$6.57
96.79
92.39
94.60
98A77
95.11
99.66
97.31
98.66
99.06
95.05
98.54
Ounoniemal)
95.72
49.70
96.90
96.70
V6.16
97.80
96.97
92.96
94.76
95.41
90.0
Ounnirmua
98.18
96.62
99.11
58.15
56.70
07
94. 63
04
97.79
97.
96. 15
97.15
95.34
07:94
94.9
87.50
96.55
66.67
77.78
73.30
03.36
97.78
50.21
54.71
95,05
THD
79.37
141.00
15.03
12.24
62.22
75.001
99.05
12.59
89.51
92.24
94.10
$5.05
99.45
92,43
70.45
91.07
M8:75
97.61
07:20
61.41
191.0
77.15
94.05
97 T3
94.04
91.19
90.35
19.11
K3.55
01.04
95.34
79.55
PC1
58.73
97.75
94
92.20
54.71
91.34
BZT
$9.24
$4.51
$4.07
90.11
19.23
71.11
72.84
77.12
94.85
52.02
95.57
90.91
56.00
14.10
16.00
42.00
09.34
83.77
Ounmesttrss)
$9.24
90.14
05.90
91:84
97.96
71.11
77.75
19.51
96.27
91 905
97.78
96.55
89.15
95.97
91.00
64.00
03:94
95.24
75/00
Oundintenna
89:29
88.73
95.85
91.43
99.79
71.01
77.78
14.51
96.27
91.51
97.78
96.62
99.36
90.97
92.18
55.88
93.94
95.34
77.27
Ounniittull
Table 12: Average and worst-group accuracy (%) per-category on NICO (ViT-B/16).
8
Metri
Methool
CLIP
95.83
97.11
99.47
95.09
99.14
96.39
93.45
94.82
98.17
96.62
19.53
98.38
99.21
91.11
97.71
95.57
9292
99.00
95.63
TED
89.00
55.70
98.93
97.84
94.25
96.39
94.33
94.36
95.42
96.31
56.92
97.57
98.66
98.65
90.59
95 10
48
98.08
92.06
94.76
Avarge
PO
97.00
96.50
99.17
98.33
99.26
96.79
94.33
94.89
96.17
96.13
99.92
97.94
99
95.12
59.20
9545
99.02
9M
94
B27
94.15
98.05
97.15
96.79
91.52
95.25
96.31
98.84
57.77
(um/esternal)
97.34
57.11
99:47
08.31
90:33
18.64
$7.85
87.71
95.34
45.13
90.48
99.17
97.5
07.52
91.5
9947
94
Ourolisterma)
91.25
98.04
99.59
95.33
99:41
98.33
90.05
91:36
95.67
97.82
99.21
99.91
98.22
99.24
Dunistrtedd
91.78
96.40
96.70
$6.64
96.41
98.64
90.45
96.67
1000
99.21
96.08
99.60
CLIP
$7.50
$1.50
58.22
93.85
96.54
73.33
69.41
74.51
55.52
94.25
98.55
16.30
95.74
96.55
50.07
56.05
95.45
99.41
72.73
TED
59.89
$5.64
$1.12
71.78
71:23
75.05
$2.36
44.25
56.55
99.17
91.02
91.09
87.17
16.05
05.85
79.55
90.70
96.45
97.56
72.03
56.27
95.00
95.45
Wond
POX")
91:47
$9-47
05.45
91.50
95:94
71:76
77:78
74.51
$1.64
04.25
55.55
W6.10
15-55
15.55
84.00
05.05
96.65
79.55
Ourslesteraad)
$1.50
91.90
98.65
95.00
97.45
91.01
09:44
$4.37
97.44
95.20
1000
98.25
16.45
99.41
00:07
90.70
96.00
98.41
81.82
Oussisttcnal
87.50
98.23
97.45
75.00
29.41
59.37
79.55
Oussisirtma)
91.07
95.55
96.83
95.92
97.45
91.11
72.23
81.35
91.01
94.46
100.0
06.55
I5:45
98.73
$8.57
96.00
98.41
79,55
D.5 Visualization on NICO
We include extended visualization experiment in Fig. 10.
E
More Discussion at CounterfactualCLIPP
E.1
Comparison with Image augmentation Variants under Zero-shot Classification
We further benchmark method against four classical image augmentation techniques including Mixup, CutMix, AugMix, and Cutout and adapted into counterfactual variants compatible with zero-shot settings. Specifically, instead of retraining, test image during inference, and aggregate their prediction logits, obtaining the revisions to prediction. Low-quality augmented samples are filtered [28] ensure the relevance of counterfactual variations.
apply each augmentation multiple times to
Among these, Mixup performs linear interpolation image pairs; CutMix pastes patches from another image; AugMix applies diverse augmentation chains followed by probabilistic blending; and Cutout randomly masks out rectangular regions. All operate at the pixel level, without using pretrained semantic priors. Results in Tab. 13, 14 and 15 show some augmentation variants offer partial
33

---

CLIP
C(x))-based
CLIP
-(x)-based
CLIP
C(x))bbsed
CLIP
C(x)-baaed
CLIIP
U(x))-based
CLIP
C(xx -based
C1.IP
O(xs-
CLIP
((x))-based
Figure 10: More Attention Map Revealed by C(x:)-basedd Embeddings NICO.
improvements. For example, AugMix reduces the gender gap to 1.6% on COCO-GB and pushes Co-Object accuracy to 78.0% on UrbanCars. However, these improvements are inconsistent and highly dataset-dependent. In contrast, our method consistently outperforms all baselines: the virtual variant improves Waterbirds average accuracy by +14.1% and narrows COCO gender gap to just 0.8%.
Table 13: Performance comparison with image enhancement variants on Waterbirds (ViT-B/16). LB: landbird, WB: waterbird; L: land background, W: water background.
Method
LB-L1
LB-Wt
WB-L*
WB-W+
Avg.f
CI.IP
95.57
63.81
59.03
85.98
77.20
Mixup
88.60
57.12
64.33
90.50
73.87
CutMix
95.21
57.87
47.04
85.98
74.32
AugMix
91.53
52.28
59.50
90.03
72.54
Cutout
97.12
55.65
38.16
86.45
73.27
Ours (External)
92.90
86.39
70.87
79.13
86.40
Ours (Intemma)
88.16
82.88
82,71
87.54
85.43
Ours (Virtual)
95.30
94.41
67.13
73.99
89.47
E.2 Analysis of Module efficiency
As shown in Table 16, the computational cost introduced by our method is minimal, especially compared diffusion based methods. While our method introduces additional inference steps, emphasize that all operations are performed in the embedding space, with no image re- re-encoding or pixel-level augmentation required. CLIP performs single forward pass through a frozen encoder, and our method preserves this efficiency: it constructs counterfactual embeddings using closed-form computations Eq. (10), (18) over precomputed token features, followed by lightweight TDE fusion in Eq. (19). All operations involve only matrix multiplications and element- wise computations, without any backpropagation or additional network training. Moreover, since the context candidate pool can be prepared offline, method remains highly efficient in large-scale data deployment.
34

---

Table 14: Performance comparison with image enhancement variants on COCO-GB v1 (VIT-
B/16), Gap denotes gender performance differences.
Method
Avg.t
Femalet
Malet
Gap4
CLIP
88,70
85,60
91.80
6.20
Mixup
87.30
91.00
83,60
7.40
CutMix
88.00
86.40
3.20
AugMix
90.00
89.20
90.80
1.60
Cutout
89.50
88.90
90.10
1.20
Ours (External)
91.10
89.80
92.40
2.60
Ours (Internal)
91.25
90.50
92.00
1.50
Ours (Virtual)
91.40
91.80
91.00
0.80
Table 15: Performance comparison with image enhancement variants UrbanCars (ViT-B/16).
We report accuracy on the original co-occurrence (I.D.), background shifted (BG), and Co-object shifted (Co-Obj) subsets, along with overall average accuracy.
Method
Avg-t
L.D.1
BGt
Co-Objt
CLIP
63.07
82.00
37.20
70.00
Mixup
61.73
78.40
36.80
70.00
CutMix
62.00
77.20
38.40
70.40
AugMix
64.13
84.00
30.40
78.00
Cutout
62,80
80.00
36.40
72.00
Ours (External)
68.53
86.40
45.60
73.60
Ours (Internal)
68.27
88.00
45.60
71.20
Ours (Virtual)
71.87
89.60
48.00
78.00
Table 16 and 17 summarize the computational cost and efficiency analysis. Table 16 shows the FLOPs image required for different methods, with the increments relative to the CLIP baseline in Waterbirds dataset. Our method's overhead negligible compared ALIA [28] and other sugmentation-bass methods, highlighting the advantages of representation-les reasoning.
Table 16: Efficiency Comparison on Waterbirds across Different VIT Backbones. GFLOPs per image is reported increments relative to the CLIP baseline ("+" indicates additional GFLOPs). M represents the number of counterfactual samples per image.
Method
M
ViT-B/32
VIT-B/16
ViT-L/14
ViT-H/14
CLIP (base)
8.8214
35.1417
162.0866
334.6752
ALIA (diffusion-based)
7
+1448068
+1452343
+1478801
+1512448
AugMix
+390.5
+3995.7
+28467
+67880
Ours (External)
100
+0.0027
+0.0033
+0.0053
+0.0071
Ours (Internal)
100
+0.0015
+0.0023
+0.0035
+0.0043
Ours (Virtual)
100
+0.0019
+0.0025
+0.0041
+0.0055
Table 17 further evaluates the efficiency of our method as a function of the number of counterfactual samples (M) per image. shown, while increasing M generally improves performance, the additional computation cost remains very low, especially compared to conventional data augmentation methods. This demonstrates the scalability and efficiency of our approach, even for large datasets and high-throughput inference scenarios.
E.3 Analysis of Module Parameters
We conduct ablation studies to analyze the sensitivity of our method to key hyperparameters under the virtual variant setting. Experiments are performed across four CLIP backbones, and results are reported in Fig. 11. We examine the following factors: the number of sampled counterfactual scenes M used in the intervention module via Eq. (18); the hallucination suppression coefficient Á from
35

---

Table 17: Efficiency with Varying Counterfactual Samples M on Waterbirds (ViT-B/16), FLOPs
per image is reported as increments relative to the CLIP baseline.
Method
M = 50
M = 100
M = 200
M = 300
Ours (Extermal)
+0.0028
+0.0033
+0.0042
+0.0051
Ours (Internal)
+0.0018
+0.0023
+0.0032
+0.0041
Ours (Virtual)
+0.0020
+0.0025
+0.0034
+0.0043
Eq. (15); and the predictive balance coefficient controlling the weight of interaction terms in final classification in Eq. (19).
Impact of the Number of Counterfactual Scenes M. As shown in the left panels of each figure in Fig. 11, increasing M initially improves performance across all backbones by introducing more diverse context embeddings, which enhances the effectiveness of the intervention. However, beyond certain point, accuracy saturates and slightly declines. We attribute it to two factors: first, excessive samples may introduce redundant or noisy descriptions, especially the virtual variant; second, the filtering mechanism becomes less effective the number of samples increase, allowing lower-quality scenes to influence the final embedding. Notably, larger backbones such as ViT-L/14 and ViT-H/14 reach optimal performance with fewer samples, indicating that stronger models can generalize with smaller but nigher-qualing interventions.
Impact of Predictive Balancing Coefficient X. The rightmost plots in Fig. 11 illustrate the influence of X, which balances the predictive weight between the interaction in original image and its intervention terms (see Eq. (19)). As À increases, performance improves initially, benefiting from richer imagined scenarios. However, overly high À may dilute the grounded signal in slightly hurting prediction reliability due to excessive dependence on synthetic embeddings.
Impact of Hallucination Suppression Coefficient 7. As reflected in the center plots of Fig. 11, increasing leads to improved performance up to moderate range (typically E [0.5, 1.5)). This coefficient controls the suppression of Eelgrroudd-ndducee hallucination during the TDE computation (see Eq. (15)), by down-weighting contributions from background-oni hallucination effects. Larger values of Î more aggressively remove spurious context effects, improving model reliability. However, if Å is set too high, it may overly penalize useful signals, causing slight drops in accuracy.
E.4 Counterfactual Framework Pseudo-code
Algorithm reflects our inference method combining representation-lvvd counterfactual construction and total direct effect computation. Candidate object scores token p(y e 02) computed via sigmoid activation, while background likelihood is defined as Pog (j) = - max; ply = c vy). Tokens with Pb)(j) are used to construct the background embedding C(2). while class-conditional object tokens satisfying p(y = Ck U3)) > T yield C(r..). We adopt thresholds for background and object embeddings.
The base prediction score is computed as Eq. (15), where the hallucinated contribution of background- only signals is controlled by suppression coefficient Å. For each top-K class on original prediction, select M counterfactual contexts } that jointly dissimilar to C(2) and C(xx). ensuring semantic orthogonality
intervention. Counterfactual embeddings C(I., Zm.) are generated by linearly combining C(x,) and fz())) with mixing ratio and then used in Eq. (18) to simulate interventions. We note that although it aggregates over M counterfactuals, practical implementation applies a filtering mechanism prior to sampling to avoid noisy entangled contexts.
In Eq. (19), only top-K predicted classes undergo counterfactual imagination and fusion, while others retain their base TDE scores, achieving robust calibration with minimal overhead. The entire inference free of additional training, leveraging only frozen CLIP representations and batch-accessible or externally precomputed scene embeddings.
36

---

The impact of number
The A TDE module
impact of prediction
85
acouaay
Vinui catias
Vinaal vara2
Vitaal valar
3
05
65
(8) Virtual variant on ViT-B/32
impuct semple
impact of module
impoat
50
80
Vatea nunur
Virtat
Vatall
(b) Virtual variant on ViT-B/16
impact sample
impact
impact
Vitual naturz
Virial
Vataal
82
76
(c) Virtual variant on ViT-L/14
impuct number sample soenes
impact
impoat of prodiction
Dasiii
Saurko
Basda:
Zero--ton
50
150
200
250
300
350
0.3
035
1.0
1.5
230
61
832
03
64
035
66
43
03
(d) Virtual variant on ViT-H/14
Figure 11: Parameter analysis of the virtual variant on Waterbirds across different backbones. Each panel shows the impact of (left) number of sampled counterfactual scenes M, (middle) halluci- nation suppression coefficient Á and (right) predictive balance coefficient Á zero-shot accuracy.
37

---

Algorithm 1: INFERENCE WITH TDE AND REPRESENTAAION-LVVEL COUNTERFAC-
TUAL CALIBRATION
Input Token embeddings {1,(i)}}= text embeddings (fc (f((To))ex)
parameters a, A, A, T, K; context pool 2in=)i(2h))b)
Output: : Predicted label y
1. Estimate Main Effects from Image Tokens
Compute per-token class probabilities: ply = 0j) = (S(Ug,
Background score: Pbg (j)
max, ply =c Uj);
Background weights: w2 (j) = P6 (i) > T)
Candidate object weights (per class): w ( (j) = I(p(y = c 0,) > r);
2. Compute First-Order Embeddings:
Background embedding:C =) Normalize
E (z( . /// E, ws (i)::
foreach top-K class C₁: do
Object embedding: C( ) Normalize
E, WS w( (j) -0.00 /// ES us dow (i)
3. Compute Base TDE score by Eq. (15):
Siny = (((a(i), Tc); Sng = S(C(2), Tz): TDE (c) = Sing Å Sog.
Construct Representation-Leww
Counterfactuals by Eq. (11):
foreach top-K K class C₁ do
Sample M scenes {zm} C Zare
Filter by dissimilarity to C(z) and C(xzy))
Construct: C(Ery 2m) = a C(xre)) + (1 (1-a) a) fi(zm)
5. Compute the intervention on object by Counterfactuals via Eq. (18):
(DEx = 1 M
M S(C(xee zm). Ta.) - 2. ((f((zm)TTa))
6. Final Prediction Score by Eq. (19):
foreach class c E y do if c top-K then
ci; (1 - A) TDEnne (c) + - TDEa(()
else
v(;;c) TDEneee (c) return 0 arg m c(
Limitation and Future Work
Our method operates entirely at the embedding level and relies estimating object and background prototypes through linear aggregation. While effective, it may overlook certain spatial interactions to some extent. Additionally, performance gains depend partially the diversity and representaaiveness of the available alternative contexts (external scenes, batch samples, or textual descriptions); thus, effectiveness may vary if suitable context alternatives limited. Future work could extend our causal approach to multi-label and dynamic visual scenarios, refine nonlinear estimation methods for representation, and develop adaptive context-selecins strategies to further enhance reliabilty and generalization.
38


Reviews:
[{"review_id": "ZVAs6BCbI7", "reviewer": "Reviewer_XSqm", "summary": "Deep learning models, including vision-language models, often tend to overfit on their training sets, which may include background, texture, and shape information. This paper approaches the issue by framing it as a causal inference problem, raising an important question: If an object appears in a different environment, will the prediction still hold true?\nThe authors use a CLIP model to examine its representation space. They estimate the total direct effect and subtract the activation related to the background only, which allows them to preserve object-content interactions. Notably, they propose a method to enhance performance in a zero-shot manner, without the need for prompt design or retraining. They also assert that their framework represents a lightweight, counterfactual-based approach. Furthermore, the additional theory and experiments, which include analyses of module parameters and comparisons to classical augmentation techniques, strengthen their proposal.", "questions": "- Regarding the implementation details, how many samples are considered at test time to obtain the counterfactual embeddings?\n- As an additional relevant baseline, I suggest that the authors include simple data augmentations as described in [2] for an input image and report the accuracy. This can be considered a first step in addressing the generalization problem at inference.  \n- This recommendation is in addition to the baseline comparisons outlined in Appendix E1, which include mixup, cutmix, augmix, and cutout.  \n- The performance on the CoCo-GB dataset appears to be incremental and not the best. Can the authors provide a justification for this performance drop?\n- The authors currently use multiple predictions (ensemble of predictions) over multiple (generated) inputs. I am curious to see the performance of 'multiple models' (ensemble of models) on single input as a comparison. Needless to say, these needs multiple models compared to their approach", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- It is interesting to note that counterfactual embeddings are obtained without relying on any external models.  \n- Overall, the paper is well-written, and the authors' contribution is clear.  \n- The inference costs are noteworthy and minimal, despite multiple counterfactual embeddings\n- The experiments and theoretical aspects are designed and articulated clearly, effectively complementing the paper's approach.  \n\nWeaknesses\n- Similar work by Sauer et al. (2021) [27] utilizes multiple GANs, where the object is decomposed into texture, shape, and background using external networks. This raises the question of why the authors are solely decomposing the object into its background. To what extent does overfitting or ambiguity in inference occur solely because of the background?  \n- The proposed method enforces consistency in predictions across diverse scenarios, which is similar to the concept presented in Test-Time Augmentation papers, such as Memo. I recommend that the authors cite this work.  \n\nReferences:\n[1] Zhang, Marvin, Sergey Levine, and Chelsea Finn. \"Memo: Test time robustness via adaptation and augmentation.\" NeurIPS 2022", "was_revised": true, "final_justification": "The authors have responded well to my questions during the rebuttal. I am in favor of acceptance, hence I will keep my original score. \nThe original submission lacked discussions about focusing on contextual bias, additional classical computer vision augmentation comparisons, ALIA [4], and its implementation, adjusting for better zero-shot performance. The reviewers have now answered these questions in a satisfactory manner.", "rebuttal": "Thanks for your appreciation of our paper. We are glad that you considered our work “well-written, effective, interesting”. We are glad to answer all your questions.\n\n**Q1:** **Comparison to [1]  and why focus mainly on contextual bias**\n\n**A1:** Thanks for the insightful comment. While Sauer et al. [1] utilize generative models to disentangle different components, our method focuses specifically on addressing contextual bias considering in three aspects:\n\n**- *Empirical evidence that context drives most failures*:** Well-trained models easily capture low-order, “easy-to-learn” semantics, while high-order cues are harder to generalize [2]. In our error analysis on Waterbirds and COCO-GB, we found that most misclassifications arise from small, ambiguous, or obstructed object, which leads the model **tend to over-rely on background**. After all, we wouldn’t want a system that sees a kitchen and always predicts woman, or sees shrubbery and always predicts landbirds.\n\n**- *No external generative networks or fine-tuning*:** Disentangling texture and shape requires training or fine-tuning separate encoder-decoder networks. In contrast, we operate directly at inference time on a frozen CLIP model, using a closed-form solution (Eq. (9), Appendix B.7) to estimate object and background direct effects. This **avoids** the need for **additional parameters and training**.\n\n**- *Better zero-shot generalization*:** Introducing dedicated texture and shape models may **anchors the model to its specific data domains** (e.g., bird textures in Waterbirds), which **limits transfer to novel** categories or instances (e.g., humans appearances in COCO-GB). Since context is more readily decoupled from object identity than low-level appearance, our strategy better supports zero-shot recognition performance.\n\nWe will include these discussions in the revised version.\n\n$ $\n\n**Q2: Supplemental citation [3]**\n\n**A2:** Thank you for suggesting the Memo [3]. We have added this citation in Section A and clarified that, differing from pixel-level TTA methods (which adjusts model more robust under augmentations), our framework **operates in the semantic space of frozen CLIP**. It decomposes and subtracts background effects and synthesizes counterfactual embeddings. This causal, representation-level approach directly addresses the issue of object-context confusion and controls hallucination effects caused by context.\n\n$ $\n\n**Q3: Experimental details on sample size for constructing counterfactual embeddings**\n\n**A3:** Thank you for your interest in experimental details. The full parameter settings, including sample size selection, are provided in the README.md of our original supplementary materials. We also promise to release the code in the future.\n\nRegarding the number of counterfactual samples generated, we conducted a parameter analysis in Fig. 11. For instance, the ViT backbones with increasing parameter sizes in Waterbirds , the virtual variant method sampled 270, 270, 190, and 90 from pool, as shown in Tab. 1 (sampling rules can be found in lines 220-223). **The scale of the candidate pool (400) can be adjusted by the experiment designer offline. Larger and more diverse candidate pools may result in fewer size.**\n\n$ $\n\n**Q4: Additional baseline information and clear baseline comparisons**\n\n**A4:** Thank you for the helpful suggestion. While the information citation \"[2]\" may have been unintentionally omitted by reviewer, we agree with the intention of introducing more augmentation-based baselines. Beyond Mixup, CutMix, AugMix, and Cutout **shown in Appendix E.1 (Tab. 13, 14)**, we further include **ALIA** [4], a diffusion-based method that performs language-guided image editing. We adapt ALIA into a counterfactual variant (see lines 773–777) and report its results. However, it incurs much higher computation cost compared to our lightweight representation-level approach.\n\n| Method         | Avg. ↑ | Worst. ↑ | FLOPs per image ↓ |\n| -------------- | ------- | ---------- | --------------- |\n| CLIP           | 77.20   | 50.93      | 8.8214 G        |\n| **ALIA** [4]   | 78.61   | 55.18      | +1452343 G      |\n| Ours(External) | 86.40   | 70.87      | +0.0033 G       |\n| Ours(Internal) | 85.43   | 82.71      | +0.0023 G       |\n| Ours(Virtual)  | 89.47   | 67.13      | +0.0025 G       |\n\nTable Note: Performance on Waterbirds on ViT-B/16 backbones. Efficiency are reported as FLOPs *increments* relative to CLIP baseline. Avg. and Worst. indicates average accuracy and the worst group accuracy.\n\nWe acknowledge that Appendix E.1 did not clearly reference and we will update the draft to clarify table 13-14 indices and improve formatting for better readability.\n\n$ $\n\n**Q5: Justification for COCO-GB performance**\n\n**A5:** Thank you for raising this point. On COCO-GB v1 we report two key metrics: **average accuracy** which measures overall recognition performance and utility; **gender gap**, which indicates fairness. In Tab.8, our virtual variant achieves **the lowest gap of 0.8%** (compared to CLIP’s 6.2%) while still reaching 91.6 % average accuracy, **outperforming all other methods on both fairness and utility simultaneously.**\n\nA further advantage of our framework is its **tunable suppression factor $\\hat{\\lambda}$** . Users who prefer higher accuracy over strict parity (or vice-versa) can adjust $\\hat{\\lambda}$ to smoothly trade off between average accuracy and gap, making the method adaptable to different deployment requirements. We will clarify these points in the revised draft.\n\n$ $\n\n**Q6: Discussion on multiple models approach**\n\n**A6:** Thank you for thoughtful suggestion. Ensemble methods that aggregate predictions from multiple models or strategies are indeed effective in bias mitigation. For instance, Li .et al [5] combines classifiers trained on various augmentation models (e.g., watermark, texture, background) and aggregates their outputs to improve robustness. However, this approach requires retraining multiple specialized models, making it unsuitable for zero-shot settings. To provide a fairer comparison, we included the counterfactual variant of AugMix, which uses various image transformation strategies similar to [5].\n\n Additionally, we implemented the **zero-shot ensemble strategy** proposed in [6]. This approach aggregates multiple pre-trained VLMs (e.g., CLIP variants) without fine-tuning, improving over any single model by leveraging architectural diversity. We evaluate these on benchmarks and present results below.\n\n| Method         | Waterbirds  |            | COCO-GB     |                  |      |\n| -------------- | ----------- | ---------- | ----------- | ---------------- | ---- |\n|                | **Avg. ↑** | **Worst. ↑** | **Avg. ↑** | **Gender Gap ↓** |      |\n| AugMix         | 72.54       | 52.28      | 90.00       | 1.60             |      |\n| $ZN_{E_n}$ [6]  | 77.32       | 54.86      | 90.80       | 2.80             |      |\n| Ours(External) | 86.40       | 70.87      | 91.10       | 2.60             |      |\n| Ours(Internal) | 85.43       | 82.71      | 91.25       | 1.50             |      |\n| Ours(Virtual)  | 89.47       | 67.13      | 91.40       | 0.80             |      |\n\nTable Note: Performance on ViT-B/16 backbones.\n\n**Reference:**\n\n[1] Sauer A, Geiger A. \"Counterfactual generative networks\". arXiv 2021.\n\n[2] Nam J, Cha H, Ahn S. \"Learning from failure: De-biasing classifier from biased classifier\". NIPS 2020.\n\n[3] Zhang, Marvin, Sergey Levine, and Chelsea Finn. \"Memo: Test time robustness via adaptation and augmentation.\" NeurIPS 2022.\n\n[4] Dunlap et al., “Diversify your vision datasets with automatic diffusion-based augmentation,” NeurIPS 2023.\n\n[5] Li, Zhiheng, et al. \"A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others.\" CVPR2023.\n\n[6] Lu, Zhihe, et al. \"Beyond sole strength: Customized ensembles for generalized vision-language models.\" ICML2024."}, {"review_id": "fuwial7pDd", "reviewer": "Reviewer_7obT", "summary": "This paper handles the zero-shot recognition problem from the causal view. It tackles the out-of-distribution problem during the inference time and formulates it as a causal inference question: Would the prediction remain if the object appeared in a different environment? The paper also proposes a counterfactual estimation method to operate directly at the representation level and achieves state-of-the-art zero-shot recognition performance across multiple context-sensitive datasets.", "questions": "The authors discuss the inference speed in the cost section, that is good. But I am also curious about what is the efficiency of the proposed method compared with standard CLIP in terms of flops?\n\nI wonder if for stronger models with larger compacity, whether the proposed approach could work more and whether it can be generalized and whether if there is scaling law applies here. From Figure 1 (a), it looks like ViT-B/32 and ViT-H/14 drop more performance from the best to the worst group.\n\nIs the method trainable (applied to the training paradigms)? Normally, when more data gets involved in training, the OOD problem can be more greatly mitigated.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The problem formulation in this paper is interesting and novel. It gives a proper explanation and solution by formulating the problem from the causal perspective and model co-occurrence biases in terms of beneficial interactions and potential hallucinations, and explain the sensitivity of prompt-based strategies.\n\n- There is solid theoretical explanations and proofs in this paper.\n\n\nWeaknesses:\n- The motivation of this paper seems to be weak, the author claims \" leading models to rely on context rather than the true causal semantics, and ultimately degrading their zero-shot recognition performance, particularly when the test-time context diverges from familiar training distribution. The issue is especially pronounced in CLIP\". But recent models including encoder like Dino, PE, etc, can do zero shot recognition very well even on unseen context.\n- The experiments are only based on the CLIP model, which is kind of toy and outdated. There are many more advanced models such as PE, BLIP, SigLIP, etc, while the paper does not try.\n- The datasets covered in this paper only include Waterbirds, UrbanCars, COCO-GB, and NICO. While there are a lot of interesting datasets outside there for zero-shot recognition such as Winoground etc, which are not tested.\n- Although the paper did a detour to introduce the method from the causal perspective and give many theorems and proofs, but the actual implementation of the method is unclear, even from the Appendix it is still hard to reproduce the method and not sure if the method can be generalized.\n- From Figure 4 and the method introduction, it looks like the method can add a lot of computational complexity. The authors should also compare the efficiency of the proposed method with other baselines like flops, memory metrics.", "was_revised": true, "final_justification": "Thank the authors for their detailed response. The clarifications regarding the choice of CLIP as the base model, the motivation from a causal perspective, and the explanation of computational efficiency address some of my concerns. The additional quantitative results and implementation details also address many of the concerns raised in the initial review. I will raise my rating.", "rebuttal": "Thanks for your constructive comments. We are glad that you considered our work “novel, well-structured theoretical, interesting” and we will answer all your questions.\n\n**Q1: Motivation and reason for selecting CLIP**\n\n**A1:** Thank you for raising these valuable concerns. We are glad to clarify our motivation in the revised version.\n\n**-*Stronger encoders do not eliminate co-occurrence bias*:** While recent backbones (e.g., DINO, PE) improve average, they still rely on object-context shortcuts **in challenging or occluded scenarios** [1]. Dataset classification studies [2] interestingly show that **larger pre-trained models amplifies such easy inter‑domain cues and passes them to downstream tasks** [1]. PMI in our paper confirms that LAION‑2B (CLIP’s pretraining dataset) shares the same gender‑scene patterns as COCO, meanwhile Li et al. [3] reveal watermark–carton co‑occurrences that skew predictions. Thus, simply switching to larger encoders or datasets does not remove co‑occurrence bias and it continues to threaten both safety and fairness.\n\n**-*Lightweight way to address shortcut reliance, not encoder competition*:** Our goal is to provide a lightweight, inference-time plugin to mitigate hallucinations, without the high cost of generative editing. We adopt CLIP-style VLMs as our base models because CLIP serves as a strong and widely used baseline, and its contrastive pretraining reveals the object–context shortcuts analyzed in Section 2.\n\nMoreover, our method is architecture‑agnostic: built on attention‑based direct‑effect decomposition and TDE, it can be transferred to stronger encoders. We provide initial evidence across four CLIP backbones with increasing capacity, demonstrating its compatibility (Tab.1, 8-12). **Starting with the CLIP allows a clean theoretical exposition and avoids interference from extra modules in newer models**.\n\n$ $\n\n**Q2: Additional benchmark**\n\n**A2:** Thank you for this suggestion. We deliberately chose Waterbirds, UrbanCars, COCO‑GB, and NICO because each dataset **explicitly encodes object‑context shortcuts** (e.g. backgrounds, gender cues, co‑occurring objects, etc.) that **let us quantify worst‑group accuracy**, the primary metric for our debiasing method. As shown in Fig. 1 and Tab. 8, worst‑group accuracy reveals a model’s true bias sensitivity: even strong zero‑shot CLIP variants score only ~40 % on this metric across all four architectures.\n\nDatasets such as Winoground are designed for fine‑grained text–image reasoning. They focus on text-generated hallucinations or caption mismatches and also do not report group‑wise accuracy. Hence they are better suited to evaluating LVLMs hallucinations when CLIP is used merely as an image encoder  component [1]. While that setting is not our current focus, our method can be applied to MiniGPT‑4, LLaVA, etc. methods on these datasets. In future work we plan a broader evaluation on benchmarks like Winoground to complement the object‑context bias tests.\n\n$ $\n\n**Q3: Algorithm implementation clarity**\n\n**A3:** We are glad to explain any ambiguity. The complete implementation can already be reconstructed from three parts of the paper: \n\nThe **end-to-end inference pipeline is detailed in *Algorithm 1*** (Appendix C.3, p. 29): \n\n(i) extracting main-effect embeddings (Eq. 6 and 8) → (ii) constructing counterfactual samples(Eq. 15 and 16)→ (iii) fusing scores (Eq. 14, 17, and 18).\n\nAdditionally, we provide ready-to-use YAML configs for each dataset (Waterbirds, UrbanCars, COCO-GB, NICO) in **the original supplementary README.md**, including the sampling script and full hyperparameter settings. In the revised version, we will reference Algorithm 1 more clearly in the main paper, expand more hyperparameter details, and release a public repo for reproducibility.\n\n$ $\n\n**Q4: Quantitative Analysis of Efficiency**\n\n**A4:** We appreciate the reviewer’s concern. While our method introduces additional inference steps, we emphasize that all operations are performed in the embedding space, with **no image re-encoding or pixel-level augmentation** required.\n\nWe agree that quantitative metrics are essential for evaluating overhead. In the revised version, we will add some experiments, including comparing to ALIA [4], a language-guided generative baseline. By default, FLOPs measures the total number of floating-point operations, assuming pre-computed text embeddings. **Values in table are reported as FLOPs *increments* relative to CLIP baseline.**\n\n| Method(GFLOPs per image) |      | ViT-B/32 | ViT-B/16 | ViT-L/14  | ViT-H/14  |\n| ------------------------ | ---- | -------- | -------- | --------- | --------- |\n| CLIP                     |      | =8.8214  | =35.1417 | =162.0866 | =334.6752 |\n| ALIA  ($M=7$)            |      | +1448068 | +1452343 | +1478801  | +1512448  |\n| AugMix ($M=7$)           |      | +390.5   | +3995.7  | +28467    | +67880    |\n| External ($M=100$)       |      | +0.0027  | +0.0033  | +0.0053   | +0.0071   |\n| Internal ($M=100$)       |      | +0.0015  | +0.0023  | +0.0035   | +0.0043   |\n| Virtual ($M=100$)        |      | +0.0019  | +0.0025  | +0.0041   | +0.0055   |\n\nTable Note: Efficiency on Waterbirds across four different backbones during inference-time. $M$ is the number of counterfactual samples per image in Eq. 15. \n\n| Method(GFLOPs per image) | $M=50$  | $M=100$ | $M=200$ | $M=300$ |\n| ------------------------ | ------- | ------- | ------- | ------- |\n| Ours(External)           | +0.0028 | +0.0033 | +0.0042 | +0.0051 |\n| Ours(Internal)           | +0.0018 | +0.0023 | +0.0032 | +0.0041 |\n| Ours(Virtual)            | +0.0020 | +0.0025 | +0.0034 | +0.0043 |\n\nTable Note: Efficiency on ViT-B/16 varies from the number of counterfactual samples $M$ on Waterbirds.\n\nAs shown above, **the computational introduced by our method is minimal, especially compared with generative approaches.** CLIP performs a single forward pass through a frozen encoder, and our method preserves this efficiency: it constructs counterfactual embeddings using closed-form computations (Eq. 9, 17) over precomputed token features, followed by lightweight TDE fusion (Eq. 18). **All operations involve only matrix multiplications and element-wise computations, without any backpropagation or additional network training.** Moreover, since the context candidate pool can be prepared offline, our method remains highly efficient in large-scale data deployment..\n\n$ $\n\n**Q5: Discussion on scaling law in our work**\n\n**A5:** Thank you for this insightful question. Our results on Waterbirds (Fig. 1(a) and Appendix Tab. 8) closely mirror those reported by Kaiming He et al. [2]. Across four CLIP backbones' results, we draw two key conclusions:\n\n-***Small models (B/32, B/16)*:**  Their limited ability under-represents the object itself, so worst-group errors stem primarily from **insufficient object representation**. Increasing capacity or resolution directly boosts both average and worst-group accuracy.\n\n-***Large models (L/14, H/14)*:** For “easy” high-resolution images, object cues quickly saturate. Extra capacity then tends to **amplify memorized object–context shortcuts** in ambiguous or occluded cases, leading the model to “guess” from background more aggressively, precisely the hallucination our method mentions.\n\nThis pattern mirrors tests in [2]: even a tiny 7K-parameter ConvNeXt can achieve 72.4% on dataset-ID classification, rising to 85% when scaled to 87M parameters. Intuitively, classifying dataset splits should be randomly, but larger models systematically learn dataset-specific shortcuts, producing stronger hallucinations downstream [1].\n\nHence, **increasing backbone scale without bias control actually reinforce shortcut learning**. In contrast, our representation-level counterfactual plugin pairs integrates with any model size to preserve object recognition while suppressing context-driven hallucinations,enabling safer and more reliable deployment. We plan to clarify these scaling effects in the revised version.\n\n$ $\n\n**Q6: Discussion on training paradigms**\n\n**A6:** This is a very insightful question. Although current design is inference-only, the counterfactual mechanism is **compatible with training paradigms**. The key idea which modifies object–context combinations via counterfactual synthesis, can be naturally integrated as a **data augmentation or regularization strategy** during training.\n\nMore broadly, we agree that incorporating more data often enhances generalization. However, **data scaling alone is insufficient**—large-scale datasets tend to contain **spurious correlations** that models exploit as shortcuts. Our counterfactual approach provides a **targeted and controllable intervention** to address it directly.\n\nIn particular, the **virtual variant** constructs semantically grounded counterfactual embeddings using text prompts in the vision–language space. This enables the generation of **lightweight synthetic supervision signals** without retraining or external annotation cost. It also makes our approach suitable for scalable training-time integration in the future, offering a promising path to debias models.\n\n**Reference:**\n\n[1] Chen, Junzhe, et al. \"Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models.\" CVPR 2025.\n\n[2] Zhuang Liu and Kaiming He. \"A decade’s battle on dataset bias: Are we there yet?\" ICLR2025.\n\n[3] Li, Zhiheng, et al. \"A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others.\" CVPR2023.\n\n[4] Dunlap et al., “Diversify your vision datasets with automatic diffusion-based augmentation,” NeurIPS2023.\n\n[5] Yossi Gandelsman, et al. \"Interpreting CLIP’s image representation via text-based decomposition.\" ICML2024."}, {"review_id": "Wtfacy0yZF", "reviewer": "Reviewer_pF3x", "summary": "This work introduces a framework for improving the zero-shot classification performance of Vision-Language Models (VLMs) by reducing the impact of background features. \n\nThe CLS token used as image embedding is decomposed into multiple additive components which are then separated into background and object components by their CLIP score with respect to the class prompts. \nThe object component is then combined with new background components that can be obtained from external data, in-batch samples or “virtual” backgrounds from text embeddings, producing counterfactual embeddings which simulate the object on various other backgrounds. \n\nThe final prediction is based on a weighted sum of the Total Direct Effect (TDE, Eq. 14) of the original image and the average TDE of the simulated samples.", "questions": "Questions\n1. The paragraph starting on line 117 seems a bit unclear to me. The authors supposedly want to compare a sample $(x, z)$ with another sample $(x’, z)$, but the inequality on line 123 rather resembles the difference in CLIP score of the sample $(x,z’)$ with respect to the classes $c_x$ and $c_{x’}$. My intuition is that the authors were referring to a $(x,z’)$ sample instead of  $(x’,z)$ on line 121 and that $z’$ is a background which favors the predictions of a class $c_{x’}$. Could the authors clarify whether there is simply a typo in this paragraph or I missed something else?\n\n2. From Appendix B, lines 456-457: “All embeddings from dataset are assumed centered”. Do the authors themselves perform a centering of the embeddings in this regard?\n\nSuggestions:\n- I think that Algorithm 1 in the Appendix clearly illustrates the steps of the framework and that it could help a reader to more easily understand the proposed method. While there may be no room to add it in the main paper, I think that it would be a good idea to mention it somewhere in Section 3.\n- The authors could consider also including the $\\hat{\\lambda}$ parameter in Eq. 14. It’s not necessarily a problem that it is only mentioned in the Appendix, but adding it shouldn’t overload the paper.\n\nTypos (not an exhaustive list, I would recommend the authors to proofread the entire paper):\n- Line 29 - o-occuring \n- Line 88 - the bias(es?) are enhanced\n- Line 95 - will inherently entangle**s** \n- Line 123 - I think that $f_t$ should be used for the embeddings of texts instead of $f_i$\n- Line 159 - shouldn’t the attention weights sum up to 1 instead of 0?\n- Line 184 - missing blank space between the dot and the first word of the next sentence\n- Line 201 - sentence starting in lower case - “we thus update…”\n- Line 453 - I think the authors meant to use $X \\in \\mathcal{X}$ instead of $X \\in \\mathcal {Y}$, as in Line 67\n- Line 457 - Shouldn't the second expectancy be of $f_t(T)$?", "limitations": "The authors discuss the limitations of their framework in Appendix F.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed methodology is sound and it surpasses prior works in the considered setups\n- The framework is validated on multiple datasets and ablations of its components (Table 4) or hyper-parameters (Appendix E.2) are provided\n- The method does not have heavy hardware requirements\n- Implementation details are properly provided, the code itself is also shared\n\nWeaknesses:\n- As noted by the authors in the Limitations and Future work section, the end performance depends upon the available backgrounds for synthesizing the counterfactual embeddings. While an ablation is performed on the number of backgrounds (scenes) used (Appendix E.2, Fig. 11), it is unclear what the results would look like if the pool of background scenes lacks in diversity (for any setting among external, internal or virtual). For the external setting of the Waterbirds dataset the authors use scenes from the Places-365 dataset, which is the very source of background pictures used in creating the Waterbirds dataset. To some extent, this could explain why the highest performance gains are obtained on this dataset. A simple test in this regard would be to use either only water-related backgrounds or land-related backgrounds in the virtual setting for the Waterbirds datasets and see how that affects the group-wise accuracies. This might end up similarly to having a low value for M (as in Fig. 11), so implementing it is not a must.\n- Some results are harder to read out as the dataset on which they are obtained is not explicitly mentioned in the caption of the table or figure, so the reader must first find the corresponding paragraph where they are referenced - Tables 1, 4, 11, 12, Figure 11.", "was_revised": true, "final_justification": "This paper introduces a method for synthesizing counterfactuals directly in the embedding space of CLIP models. As shown in the rebuttal, this approach is much more computationally efficient than using input-level editing with diffusion models.\n\nThe counterfactuals are successfully used to reduce the dependency of CLIP-based zero-shot classifiers on background features over four datasets. \n\nThe authors have addressed the concerns of the reviewers and clarified some parts of their work in the rebuttal.", "rebuttal": "Thanks for your appreciation of our paper. We are glad that you considered our work “sound, effective, easy to follow”. We are glad to answer all your questions.\n\n**Q1: Discussion on the impact of context diversity**\n\n**A1:** Thank you for the insightful comment on the diversity of the background pool for synthesizing counterfactual embeddings. We fully agree that **the quality, especially the diversity, of counterfactual embeddings is more critical than their quantity**. As discussed in lines 94–97 and analyzed in Fig. 11, a diverse candidate pool better approximates ideal interventions and more effectively mitigates biases embedded during training.\n\nOur sampling strategy (lines 220–223) is designed to achieve scene diversity and semantic plausibility. The external and internal variants sample from Places365 and in-batch sources, respectively, helping align the synthesized embeddings with real-world object–context co-occurrences. Additionally, we introduce a virtual variant, where counterfactual embeddings are constructed directly in the shared vision–language space via text. Leveraging recent advances in LLMs such as GPT-4, **we can efficiently generate rich and diverse scene prompts at scale, enabling low-cost counterfactual construction.**\n\nThis approach also offers a practical benefit: it enables **task-agnostic deployment**. Once a diverse virtual pool is built, the same intervention strategy can be reused across tasks without additional manual effort.\n\nWe will clarify these points in the revised draft, and we thank the reviewer again for emphasizing this important aspect.\n\n$ $\n\n**Q2: Clarifying table and figure representation**\n\n**A2:** Thank you for pointing this out. We will revise the layout and captions to make the results easier to read. Specifically, we will explicitly indicate the corresponding dataset in the captions of Tables 1, 4, 11, 12, and Fig. 11, and **place each table or figure closer to its first mention in the text**. We will also include brief descriptions of the metrics and datasets in each caption to avoid confusion and reduce back-and-forth reading. These changes will be reflected in the revised version to enhance readability and accessibility.\n\n$ $\n\n**Q3: Explanation of typos on lines 121-124**\n\n**A3:** Thank you for pointing out this confusion. You are right, and there is indeed a typo in lines 121–124. Our intention was to **illustrate how a novel background $z'$ can reverse the prediction for the same object $x$,** especially considering the co-occurrence pairs $(x,z)$ and $(x',z')$ observed during training. Concretely, consider a novel pair $(x,z')$ that rarely (or never) co-occurred during training, and thus has a low interaction term $r_i$. Define the decision margin as:\n$$\n\\delta = S(\\mathbf i_{x,z'},T_{c_x})-S(\\mathbf i_{x,z'},T_{c_{x'}})\n$$\nwhere $c_x$ is the true class of $x$ and $c_{x'}$ is a confusable alternative.\n\nThen, using the decomposition in Eq. 4, this margin can be rewritten as:\n$$\n\\delta =(\\langle e_i(x), f_i(T_{c_x}) \\rangle - \\langle e_i(x), f_i(T_{c_{x'}}) \\rangle) \n\\;-\\;\n(\\langle e_i(z'), f_i(T_{c_{x'}}) \\rangle - \\langle e_i(z'), f_i(T_{c_x}) \\rangle)\n$$\nThis expression shows that if the contextual interaction effect (inherited from the co-occurrence pairs $(x',z')$) dominates the object evidence (the first term), then $\\delta<0$ and the model incorrectly favors $c_{x'}$. This explains the failure case shown in Fig. 1(a), and is the underlying mechanism for the hallucinated scores visualized in Fig. 1(b). \n\nWe will correct the notation and clarify the intent in the final version.\n\n$ $\n\n**Q4: Clarification on whether the embeddings are centered**\n\n**A4:** Thank you for raising this point. Yes, **we explicitly apply mean-centering to the embeddings.** Before downstream operations, such as CLIP inner product scoring or TDE estimation, we subtract the mean embedding (computed over the full dataset or mini-batch) from each vector. This ensures that the theoretical assumption in Appendix B (lines 456–457) is faithfully reflected in our actual implementation. We will publish the algorithm code in the future.\n\n$ $\n\n**Q5: Suggestions on clarity, presentation, and typos**\n\n**A5:** Thank you for the detailed and constructive suggestions. We agree that Algorithm 1 provides a clear summary of our method, and we will explicitly reference it in Section 3 to help readers navigate to the full procedure in Appendix C.3. We also appreciate the recommendation to introduce the hyperparameter $\\hat{\\lambda}$ earlier, and will include a brief explanation alongside Eq. 14 in the main text. \n\nAdditionally, we are grateful for the careful review of typos, inconsistent notations, and formatting issues (e.g., notation for $f_t$, attention weights, expectation symbols). We will thoroughly proofread the paper and correct all potential issues to improve overall clarity and precision in the final version."}, {"review_id": "G7LtYatUcO", "reviewer": "Reviewer_d8Ai", "summary": "In this paper, the authors propose a theoretical causal approach to enhance zero-shot reliability of VL models (CLIP). Utilizing TDE, the authors emphasize the effective removal of context bias, and highlight the efficacy of the proposed method in terms of no need of retraining and heuristic prompt design, as well as practicality of the light-weight design.", "questions": "Q1. The reviewer thinks that the term “Hallucination” is wrongly defined in this paper. Assigning high scores for the albatross for the given sea image is misclassified results rather than the hallucinated results (many researchers may agree the term hallucination is normally used for generative models that generated persuasive responses not grounding on the factual information). The reviewer thinks that context bias is more proper terminology for the shortcuts.\n\nQ2. The definition of semantic token effect seems too much simplified. (tbh sec 3.1 should be improved a lot - e.g., through line 160-167 the readers cannot have insights of the semantic designs for the implementation of the TDE formulation) The reviewer also agree of the importance of MSA for the feature extraction. But considering the complicated non-linearity in the transformer architecture, the reviewer concerns the role of MSA in the semantic effects (beyond the ablation results). Can the authors generalize this assumption on more general datasets or any other reference to support this claim?", "limitations": "The limitation or discussion section is not described in the currnet version.\n\nDespite the impressive research topic and their theoretical revisit of CLIP models, the current method of counterfactual intervention is limited to certain benchmarks, which cannot support the efficacy in the more generalized cases (e.g., in-the-wild, ood, etc,.). The reviewer leans to negative for the current version and will discuss during the discussion period.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The reviewer enjoyed to read this paper. Especially, the theoretical part of the feature decomposition and the origin of object-context bias in CLIP (sec 2) are quite impressive to follow. The authors theoretically present that the current SSL training (w/ infoNCE loss) inherently provoke entanglement of object-context in terms of information theory and empirical observations on toy examples support such claims.\n\nDespite of the intriguing research direction, there are several points arguing about. \n\n1. The experimental benchmarks are ideal playground to show the effectiveness of the proposed method due to the explicit definition of object-context bias in the datasets. The reviewer believes the essential strength of CLIP model is its powerful generalizability for the any given combination of the object-context, where the current method is limited show the effectiveness in such environments.\n2. The proposed method requires context pool to alter the object-context pair, which is extracted from other datasets, other images within the batches or the generated features from text side. That is, the current approach relies on the predefined or closed range of context, which is limited to show the effectiveness under the open-ended and zero-shot cases.\n3. In short, the formulation of TDE (eq. 14) is the score difference between the original image and (estimated) background-only image. Here, C(z) is the solely weighted summation of token embeddings that is mostly like background. But the improvement is more than 44%p in WB dataset, which is quite impressive and also doubtful. As aforementioned, the reviewer thinks that this may the results of the overfitting in the benchmarks (that includes only simple instances of single object and bg context).\n4. The analyses of the proposed method is limited in this paper. The reviewer thinks that simply listing benchmark performance cannot deliver enough insights to potential readers (line 261-272).", "was_revised": true, "final_justification": "Most of my concerns have been addressed during the rebuttal. I appreciate the clarifications and hope that the discussed points will be properly reflected in the potential final version.", "rebuttal": "Thank you for your thoughtful review and encouraging feedback. We are glad that you considered our work “enjoyed reading, impressive theory, effective”. We are glad to address your questions.\n\n**Q1: Why do CLIP models still suffer from co-occurrence bias despite their strong zero-shot capabilities?**\n\n**A1:**  Thank you for raising this important point. While CLIP performs well on familiar object–context pairs, it also encodes object–context **shortcuts from its pretraining data** (e.g., LAION-2B), which limits its generalization to novel combinations. Our benchmark are designed to expose such biases, with worst-group accuracy dropping as **low as 34.5%** (Tab. 8). We believe addressing this issue is critical for the following reasons:\n\n-***Context bias is inherent in pre-trained VLMs.*** As shown in our PMI analysis (Fig. 2), LAION-2B shares gender–scene co-occurrence patterns with COCO and contains low-level artifacts like watermark–carton correlations [1]. These biases stem from the data itself and cannot be removed by simply scaling it, but can be addressed through targeted counterfactual intervention.\n\n-***Larger models amplify shortcut learning.*** Scaling to ViT-L/14 or ViT-H/14 does not resolve the issue. On the contrary, larger models tend to memorize and exploit spurious correlations more aggressively in ambiguous or occluded cases. This matches findings from Kaiming He et al. [2], which show stronger models are more sensitive to dataset-specific shortcuts, making bias mitigation even more crucial as models scale.\n\n-***Biases affect downstream systems.*** Many LVLMs, such as MiniGPT-4, LLaVA, use CLIP as their image encoder and inherit these hallucination patterns [3], leading to potential failures in real-world applications. Therefore, we believe CLIP remains a relevant and important testbed for studying and mitigating object–context bias.\n\nWe will include these discussions in the revised version.\n\n$ $\n\n**Q2: Whether predefined context pools are limited in open-ended or zero-shot settings?**\n\n**A2:** Thank you for raising this insightful concern. We agree that **context pool diversity** is essential to our method. Our approach simulates interventions by imagining an object in alternative contexts at representation-level, which is conceptually similar to randomized trials. For these to be effective, we only require a diverse set of dissimilar contexts, which discourages reliance on pretraining shortcuts (see Fig. 11).\n\nIn practice, the candidate pool is **not fixed or dataset-specific**. For structured datasets like Waterbirds, we can design targeted pools to match domain-specific scenes via a **priori**. For open-domain datasets like COCO, it makes sense to sample from Places365. Our method also **supports more larger open-world datasets** (e.g., Open Images, SUN397, ADE20K), which cover diverse real-world scenes without extra labeling costs. With our sampling strategy (lines 220–223), selecting from such pools remains efficient and lightweight.\n\nMoreover, the virtual variant enables text-guided counterfactual construction using LLMs like GPT-4, allowing for rich and diverse prompts beyond any limited scenes dataset. This makes **our method task-agnostic and extensible: once a diverse virtual context pool is built, it can be reused across tasks or deployed in zero-shot settings.**\n\nWe will clarify this in the revision and emphasize that our method does not rely on a fixed context pool, but rather supports open-ended generalization through flexible sampling strategies.\n\n$ $\n\n**Q3: Is the high performance of the WB dataset a coincidence that TDE solves overfitting?**\n\n**A3:** Thank you for raising this question. We clarify that the large improvement on the Waterbirds dataset (over 44%) is not due to overfitting, but rather from effectively **reducing the strong object–context bias** that CLIP inherits from pretraining on LAION-2B. As shown in Tab. 8, CLIP performs well on common object–background pairs but fails on rare ones, which is precisely the issue our method targets. \n\nWhile some gain may come from using Places365 as an external pool which **aligns well with WB backgrounds**, we also observe strong performance with internal and virtual variants, confirming the method’s generalization across context sources.  Moreover, Our method **generalizes well across different CLIP backbones** (ViT-B/32 to H/14) **and datasets** (UrbanCars, COCO-GB, NICO), consistently improving worst-group accuracy. This suggests that it reliably mitigates hallucinations caused by contextual bias.\n\nBesides, the effectiveness is not due to a simple subtraction. It is derived from a **counterfactual optimal estimation framework** (Sec. 3.1), where we enable to compute the Total Direct Effect (TDE), using $\\hat \\lambda$ to suppress the confounding context-only pathway. This formulation, along with **multiple $C(z)$ variants simulating interventions** (Eq. 17), helps prevent context-driven misclassification, such as predicting “albatross” based only on the ocean in Fig. 1(b).\n\nWe will revise the paper to clarify these distinctions and reinforce the broad applicability of our method beyond any single dataset.\n\n$ $\n\n**Q4: Clarification of the analytical experiments**\n\n**A4:** Thank you for pointing this out. We fully agree that in-depth analysis is crucial to help readers better understand our method. Due to space limitations in the main paper, many of our extended analyses were **placed in the appendix**.\n\nIn particular, we provide additional experiments across four datasets in Appendix D, and introduce detailed pseudocode in Appendix C.3 to facilitate reproducibility. Moreover, Appendix E.2 offers a comprehensive  analysis of hyperparameters, and Appendix E.1 compares our method with 4–5 classical augmentation techniques by adapting them into counterfactual variants.\n\nAdditionally, we plan to include **more comparison baselines and provide FLOPs analysis** (access to relevant tables in the responses to reviewers XSqm and 7obT), to further strengthen the evaluation. Finally, we will improve the main paper by explicitly pointing to relevant appendix sections, so readers can more easily access the extended analysis.\n\n$ $\n\n**Q5: Discussion on the use of term \"hallucination\"**\n\n**A5:** Thank you for raising this point regarding term. We clarify that, at the mechanism level, *object–context bias* refers to the learned co-occurrence shortcuts during training, while *hallucination* describes the inference-time outcome where the model makes overconfident predictions based on misleading contextual cues [4, 5].\n\nIn our paper, “hallucination” refers to **cases where the model predicts a label even when the target object is absent**. For example, in Fig. 1(b), the model predicts “albatross” based solely on the ocean background. Alternatively, one can view it as CLIP **mistakenly retrieving or generating a caption** that strongly matches the background, but does not reflect the true content of the image.\n\nTo avoid ambiguity, we will clarify it in the revised draft, avoiding the use of “hallucination” or replacing it with “context-induced misprediction”.\n\n$ $\n\n**Q6: Discussion on semantic-token direct effect**\n\n**A6:** Thank you for the insightful suggestion. We agree that Section 3.1 can be improved, and we will revise it to better explain how the semantic-token is captured. Specifically, we will clarify how the direct-effect decomposition helps assign distinct semantic roles to different tokens.\n\n**MSA dominates semantics.** Unlike MLPs which process each token independently, multi-head self-attention (MSA) allows tokens to exchange information, **enabling the model to gather semantic cues across spatial regions**. Prior works support this: Caron et al. [6] show that attention heads can recover segmentation structures even without supervision, and Naseer et al. [7] find that ViTs use long-range attention to model global semantics. Recent study [8] also shows that attention-guided token grouping transfers well across datasets and tasks. Our mean-ablation study on ImageNet (Tab. 5) further confirms that removing MSA significantly weakens semantic attribution.\n\n**Direct effect makes sense**. Although Transformers contain nonlinear components, our decomposition remains interpretable in CLIP, as the final prediction is computed by a **linear inner product** between image and text embeddings. This allows us to estimate each token’s semantic contribution to the final representation (see Eq. 6). Furthermore, ViTs include **residual connections** after each MSA and MLP block, allowing us to compare representations with and without specific modules (see Eq. 33), making direct-effect estimation possible even in nonlinear models.\n\nWe will clarify these points in the revision and cite relevant work to support our method.\n\n**Reference:**\n\n[1] Li, Zhiheng, et al. \"A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others.\" CVPR2023.\n\n[2] Zhuang Liu and Kaiming He. \"A decade’s battle on dataset bias: Are we there yet?\" ICLR2025.\n\n[3] Chen, Junzhe, et al. \"Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models.\" CVPR 2025.\n\n[4] Liu, Yufang, et al. \"Investigating and mitigating object hallucinations in pretrained vision-language (clip) models.\" EMNLP 2024.\n\n[5] Chen, Xuweiyi, et al. \"Multi-object hallucination in vision language models.\" NIPS2024.\n\n[6] Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” ICCV 2021.\n\n[7] Naseer et al., “Intriguing Properties of Vision Transformers,” NeurIPS 2021.\n\n[8] Yossi Gandelsman, et al. \"Interpreting CLIP’s image representation via text-based decomposition.\" ICML2024."}]

Author rebuttal:

Meta-review: The final scores for this submission are 5/5/4/4. After the rebuttal, the reviewers consistently agreed that the authors had addressed their concerns and gave a positive attitude toward the work, highlighting its satisfactory novelty, moderate resource requirements, and solid theoretical explanations. After carefully reviewing both the paper and the rebuttal, I recommend accepting this submission.
Decision: Accept (poster)