Title: Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zero-Shot Recognition

Counterfactual Reasoning

Causal Inference

CLIP
Abstract: Object–context shortcuts remain a persistent challenge in vision‑language models, undermining zero‑shot reliability when test-time scenes diverge from familiar training co-occurrences. We recast this issue as a causal inference problem and ask: Would the prediction remain if the object appeared in a different environment? To answer it at inference time, we estimate object and background expectations within CLIP’s representation space, and synthesize counterfactual embeddings by recombining object features with diverse alternative contexts sampled from external datasets, batch neighbors, or text-derived descriptions. By estimating the Total Direct Effect and simulating intervention, we further subtract background‑only activation, preserving beneficial object–context interactions while mitigating hallucinated scores. Without retraining or prompt design, our method substantially improves both worst-group and average accuracy on context-sensitive benchmarks, establishing a new zero‑shot state of the art. Beyond performance, our framework provides a lightweight representation-level counterfactual approach, offering a practical causal avenue for debiased and reliable multimodal reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a theoretical causal approach to enhance zero-shot reliability of VL models (CLIP). Utilizing TDE, the authors emphasize the effective removal of context bias, and highlight the efficacy of the proposed method in terms of no need of retraining and heuristic prompt design, as well as practicality of the light-weight design.\n\nSTRENGTHS AND WEAKNESSES:\nThe reviewer enjoyed to read this paper. Especially, the theoretical part of the feature decomposition and the origin of object-context bias in CLIP (sec 2) are quite impressive to follow. The authors theoretically present that the current SSL training (w/ infoNCE loss) inherently provoke entanglement of object-context in terms of information theory and empirical observations on toy examples support such claims.\n\nDespite of the intriguing research direction, there are several points arguing about. \n\n1. The experimental benchmarks are ideal playground to show the effectiveness of the proposed method due to the explicit definition of object-context bias in the datasets. The reviewer believes the essential strength of CLIP model is its powerful generalizability for the any given combination of the object-context, where the current method is limited show the effectiveness in such environments.\n2. The proposed method requires context pool to alter the object-context pair, which is extracted from other datasets, other images within the batches or the generated features from text side. That is, the current approach relies on the predefined or closed range of context, which is limited to show the effectiveness under the open-ended and zero-shot cases.\n3. In short, the formulation of TDE (eq. 14) is the score difference between the original image and (estimated) background-only image. Here, C(z) is the solely weighted summation of token embeddings that is mostly like background. But the improvement is more than 44%p in WB dataset, which is quite impressive and also doubtful. As aforementioned, the reviewer thinks that this may the results of the overfitting in the benchmarks (that includes only simple instances of single object and bg context).\n4. The analyses of the proposed method is limited in this paper. The reviewer thinks that simply listing benchmark performance cannot deliver enough insights to potential readers (line 261-272).\n\nQUESTIONS:\nQ1. The reviewer thinks that the term “Hallucination” is wrongly defined in this paper. Assigning high scores for the albatross for the given sea image is misclassified results rather than the hallucinated results (many researchers may agree the term hallucination is normally used for generative models that generated persuasive responses not grounding on the factual information). The reviewer thinks that context bias is more proper terminology for the shortcuts.\n\nQ2. The definition of semantic token effect seems too much simplified. (tbh sec 3.1 should be improved a lot - e.g., through line 160-167 the readers cannot have insights of the semantic designs for the implementation of the TDE formulation) The reviewer also agree of the importance of MSA for the feature extraction. But considering the complicated non-linearity in the transformer architecture, the reviewer concerns the role of MSA in the semantic effects (beyond the ablation results). Can the authors generalize this assumption on more general datasets or any other reference to support this claim?\n\nLIMITATIONS:\nThe limitation or discussion section is not described in the currnet version.\n\nDespite the impressive research topic and their theoretical revisit of CLIP models, the current method of counterfactual intervention is limited to certain benchmarks, which cannot support the efficacy in the more generalized cases (e.g., in-the-wild, ood, etc,.). The reviewer leans to negative for the current version and will discuss during the discussion period.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a framework for improving the zero-shot classification performance of Vision-Language Models (VLMs) by reducing the impact of background features. \n\nThe CLS token used as image embedding is decomposed into multiple additive components which are then separated into background and object components by their CLIP score with respect to the class prompts. \nThe object component is then combined with new background components that can be obtained from external data, in-batch samples or “virtual” backgrounds from text embeddings, producing counterfactual embeddings which simulate the object on various other backgrounds. \n\nThe final prediction is based on a weighted sum of the Total Direct Effect (TDE, Eq. 14) of the original image and the average TDE of the simulated samples.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed methodology is sound and it surpasses prior works in the considered setups\n- The framework is validated on multiple datasets and ablations of its components (Table 4) or hyper-parameters (Appendix E.2) are provided\n- The method does not have heavy hardware requirements\n- Implementation details are properly provided, the code itself is also shared\n\nWeaknesses:\n- As noted by the authors in the Limitations and Future work section, the end performance depends upon the available backgrounds for synthesizing the counterfactual embeddings. While an ablation is performed on the number of backgrounds (scenes) used (Appendix E.2, Fig. 11), it is unclear what the results would look like if the pool of background scenes lacks in diversity (for any setting among external, internal or virtual). For the external setting of the Waterbirds dataset the authors use scenes from the Places-365 dataset, which is the very source of background pictures used in creating the Waterbirds dataset. To some extent, this could explain why the highest performance gains are obtained on this dataset. A simple test in this regard would be to use either only water-related backgrounds or land-related backgrounds in the virtual setting for the Waterbirds datasets and see how that affects the group-wise accuracies. This might end up similarly to having a low value for M (as in Fig. 11), so implementing it is not a must.\n- Some results are harder to read out as the dataset on which they are obtained is not explicitly mentioned in the caption of the table or figure, so the reader must first find the corresponding paragraph where they are referenced - Tables 1, 4, 11, 12, Figure 11.\n\nQUESTIONS:\nQuestions\n1. The paragraph starting on line 117 seems a bit unclear to me. The authors supposedly want to compare a sample $(x, z)$ with another sample $(x’, z)$, but the inequality on line 123 rather resembles the difference in CLIP score of the sample $(x,z’)$ with respect to the classes $c_x$ and $c_{x’}$. My intuition is that the authors were referring to a $(x,z’)$ sample instead of  $(x’,z)$ on line 121 and that $z’$ is a background which favors the predictions of a class $c_{x’}$. Could the authors clarify whether there is simply a typo in this paragraph or I missed something else?\n\n2. From Appendix B, lines 456-457: “All embeddings from dataset are assumed centered”. Do the authors themselves perform a centering of the embeddings in this regard?\n\nSuggestions:\n- I think that Algorithm 1 in the Appendix clearly illustrates the steps of the framework and that it could help a reader to more easily understand the proposed method. While there may be no room to add it in the main paper, I think that it would be a good idea to mention it somewhere in Section 3.\n- The authors could consider also including the $\\hat{\\lambda}$ parameter in Eq. 14. It’s not necessarily a problem that it is only mentioned in the Appendix, but adding it shouldn’t overload the paper.\n\nTypos (not an exhaustive list, I would recommend the authors to proofread the entire paper):\n- Line 29 - o-occuring \n- Line 88 - the bias(es?) are enhanced\n- Line 95 - will inherently entangle**s** \n- Line 123 - I think that $f_t$ should be used for the embeddings of texts instead of $f_i$\n- Line 159 - shouldn’t the attention weights sum up to 1 instead of 0?\n- Line 184 - missing blank space between the dot and the first word of the next sentence\n- Line 201 - sentence starting in lower case - “we thus update…”\n- Line 453 - I think the authors meant to use $X \\in \\mathcal{X}$ instead of $X \\in \\mathcal {Y}$, as in Line 67\n- Line 457 - Shouldn't the second expectancy be of $f_t(T)$?\n\nLIMITATIONS:\nThe authors discuss the limitations of their framework in Appendix F.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper handles the zero-shot recognition problem from the causal view. It tackles the out-of-distribution problem during the inference time and formulates it as a causal inference question: Would the prediction remain if the object appeared in a different environment? The paper also proposes a counterfactual estimation method to operate directly at the representation level and achieves state-of-the-art zero-shot recognition performance across multiple context-sensitive datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The problem formulation in this paper is interesting and novel. It gives a proper explanation and solution by formulating the problem from the causal perspective and model co-occurrence biases in terms of beneficial interactions and potential hallucinations, and explain the sensitivity of prompt-based strategies.\n\n- There is solid theoretical explanations and proofs in this paper.\n\n\nWeaknesses:\n- The motivation of this paper seems to be weak, the author claims \" leading models to rely on context rather than the true causal semantics, and ultimately degrading their zero-shot recognition performance, particularly when the test-time context diverges from familiar training distribution. The issue is especially pronounced in CLIP\". But recent models including encoder like Dino, PE, etc, can do zero shot recognition very well even on unseen context.\n- The experiments are only based on the CLIP model, which is kind of toy and outdated. There are many more advanced models such as PE, BLIP, SigLIP, etc, while the paper does not try.\n- The datasets covered in this paper only include Waterbirds, UrbanCars, COCO-GB, and NICO. While there are a lot of interesting datasets outside there for zero-shot recognition such as Winoground etc, which are not tested.\n- Although the paper did a detour to introduce the method from the causal perspective and give many theorems and proofs, but the actual implementation of the method is unclear, even from the Appendix it is still hard to reproduce the method and not sure if the method can be generalized.\n- From Figure 4 and the method introduction, it looks like the method can add a lot of computational complexity. The authors should also compare the efficiency of the proposed method with other baselines like flops, memory metrics.\n\nQUESTIONS:\nThe authors discuss the inference speed in the cost section, that is good. But I am also curious about what is the efficiency of the proposed method compared with standard CLIP in terms of flops?\n\nI wonder if for stronger models with larger compacity, whether the proposed approach could work more and whether it can be generalized and whether if there is scaling law applies here. From Figure 1 (a), it looks like ViT-B/32 and ViT-H/14 drop more performance from the best to the worst group.\n\nIs the method trainable (applied to the training paradigms)? Normally, when more data gets involved in training, the OOD problem can be more greatly mitigated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDeep learning models, including vision-language models, often tend to overfit on their training sets, which may include background, texture, and shape information. This paper approaches the issue by framing it as a causal inference problem, raising an important question: If an object appears in a different environment, will the prediction still hold true?\nThe authors use a CLIP model to examine its representation space. They estimate the total direct effect and subtract the activation related to the background only, which allows them to preserve object-content interactions. Notably, they propose a method to enhance performance in a zero-shot manner, without the need for prompt design or retraining. They also assert that their framework represents a lightweight, counterfactual-based approach. Furthermore, the additional theory and experiments, which include analyses of module parameters and comparisons to classical augmentation techniques, strengthen their proposal.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- It is interesting to note that counterfactual embeddings are obtained without relying on any external models.  \n- Overall, the paper is well-written, and the authors' contribution is clear.  \n- The inference costs are noteworthy and minimal, despite multiple counterfactual embeddings\n- The experiments and theoretical aspects are designed and articulated clearly, effectively complementing the paper's approach.  \n\nWeaknesses\n- Similar work by Sauer et al. (2021) [27] utilizes multiple GANs, where the object is decomposed into texture, shape, and background using external networks. This raises the question of why the authors are solely decomposing the object into its background. To what extent does overfitting or ambiguity in inference occur solely because of the background?  \n- The proposed method enforces consistency in predictions across diverse scenarios, which is similar to the concept presented in Test-Time Augmentation papers, such as Memo. I recommend that the authors cite this work.  \n\nReferences:\n[1] Zhang, Marvin, Sergey Levine, and Chelsea Finn. \"Memo: Test time robustness via adaptation and augmentation.\" NeurIPS 2022\n\nQUESTIONS:\n- Regarding the implementation details, how many samples are considered at test time to obtain the counterfactual embeddings?\n- As an additional relevant baseline, I suggest that the authors include simple data augmentations as described in [2] for an input image and report the accuracy. This can be considered a first step in addressing the generalization problem at inference.  \n- This recommendation is in addition to the baseline comparisons outlined in Appendix E1, which include mixup, cutmix, augmix, and cutout.  \n- The performance on the CoCo-GB dataset appears to be incremental and not the best. Can the authors provide a justification for this performance drop?\n- The authors currently use multiple predictions (ensemble of predictions) over multiple (generated) inputs. I am curious to see the performance of 'multiple models' (ensemble of models) on single input as a comparison. Needless to say, these needs multiple models compared to their approach\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}