Learning Invariant Causal Mechanism from Vision-Language Models
Authors: Zeen Song
Siyu Zhao
Xingyu Zhang
Jiangmeng Li
Changwen Zheng
Wenwen Qiang
Venue: ICML 2025 poster (2025)
Keywords: Vision-Language Models
Causal representation Learning
Out-of-distribution generalization
Representation Learning
Primary area: general_machine_learning->representation_learning

Abstract: Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success, but its performance can degrade when fine-tuned in out-of-distribution (OOD) scenarios. We model the prediction process using a Structural Causal Model (SCM) and show that the causal mechanism involving both invariant and variant factors in training environments differs from that in test environments. In contrast, the causal mechanism with solely invariant factors remains consistent across environments. We theoretically prove the existence of a linear mapping from CLIP embeddings to invariant factors, which can be estimated using interventional data. Additionally, we provide a condition to guarantee low OOD risk of the invariant predictor. Based on these insights, we propose the Invariant Causal Mechanism of CLIP (CLIP-ICM) framework. CLIP-ICM involves collecting interventional data, estimating a linear projection matrix, and making predictions within the invariant subspace. Experiments on several OOD datasets show that CLIP-ICM significantly improves the performance of CLIP. Our method offers a simple but powerful enhancement, boosting the reliability of CLIP in real-world applications.

Learning Invariant Causal Mechanism from Vision-Language Models
Zeen Song"
12
Siyu Zhao
Xingyu Zhang
Jiangmeng Li Changwen Zheng' Wenwen Qiang'
Abstract
tasks by measuring semantic similarity between images and textual category descriptions (Radford et al., 2021). Notably, CLIP performs well in out-of-distribtioo (OOD) scenarios, effectively adapting to domain variations and recognizing previously unseen categories.
Contrastive Language-Iman Pretraining (CLIP) has achieved remarkable success, but its perfor- mance can degrade when fine-tuned in out-of- distribution (OOD) scenarios. We model the pre- diction process using a Structural Causal Model (SCM) and show that the causal mechanism in- volving both invariant and variant factors in train- ing environments differs from that test envi- ronments. In contrast, the causal mechanism with solely invariant factors remains consistent across environments. We theoretically prove the existence of linear mapping from CLIP em- beddings to invariant factors, which can be es- timated using interventional data. Additionally, we provide condition to guarantee low OOD risk of the invariant predictor. Based on these insights, we propose the Invariant Causal Mech- anism of CLIP (CLIP-ICM) framework. CLIP- ICM involves collecting interventional data, es- timating a linear projection matrix, and making predictions within the invariant subspace. Experi- ments on several OOD datasets show that CLIP- ICM significantly improves the performance of CL.P. Our method offers simple but powerful enhancement, boosting the reliability of CLIP in applications. The source code is avail- able at https//giihabb.oomm//enoooCCCC.
However, this ability of CLIP task-independent. many real-world applications, CLIP requires fine-tuning adapt to specific tasks (Gao et al., 2023; Zhou et al., 2022a; Shu et al., 2023; Zhang al., 2023b). When fine-tuning CLIP for downstream tasks, a critical question arises: Can CLIP maintain its strong capabilities when faced with test data that differs from the training distribution? More importantly, can still perform well when encountering new classes not seen during training? To address this, we conduct an experiment the Terra Incognita dataset (Beery al., 2018), which contains species data collected from diverse real-world environments. The results in Table show that fine-tuning CLIP in the training domain does not necessarily lead to good performance in the test domain. Furthermore, when encountering new classes not observed in the training domain, fine-tuning on the training domain can harm the zero-shot ability of CLIP.
To address the above problem, we find that inconsistent pre- diction issues can be well explained from a causal perspec- tive. Inspired by (Pearl, 2009; Pearl & Bareinboim, 2014; Schölkopf et al., 2021). we first propose a Structural Causal Model (SCM) to capture the causal processes underlying OOD predictions of CLIP. this causal model, predictions rely two types of latent factors, which are assumed to generate the image data: (I) environment intronment-invariint factors that remain unchanged across different environments and (2) environment-vatte
1. Introduction
As one of the most successful large-scale pre-trained vision- language models, Contrastive Language-Image Pretraining (CLIP) (Radford al., 2021) has garnered significant atten- tion. Trained on 400 million diverse image-text pairs, CLIP achieves robust cross modal alignment without task-specific supervision. This enables direct generalization to unseen
factors that vary with the environ- ment. By analyzing the causal mechanisms in the SCM, we observe that if predictions rely on both invariant and variant factors, the causal mechanism in the training environment differs from that in the test environment. However, when based solely invariant factors, the prediction mechanism remains consistent across environments. This raises an in- triguing question: could we design a predictor that relies only on the invariant factors?
'Equal contribution 'Institute of Software Chinese Academy of Sciences, Beijing, China University of the Chinese Academy of Sciences. Correspondence to: Wenwen Qiang <qiangwen- wen acc.s.
A key challenge in obtaining such an invariant predictor that it is unclear which parts of the extracted representations of CLIP correspond to the invariant factors (Hyvärinen &
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Invariant Causal Mechanism from Vision-Language Models
Pajunen, 1999; Locatello et al., 2019). By analyzing the identifiability of CLIP, we first demonstrate that the rep- resentations learned by CLIP can be viewed as linear combination of invariant and variant factors. Under this condition, we prove that, if interventional data (Refer to Appendix H.1 and Section for details) are available, it is possible to derive a linear projection matrix that maps the representations of CLIP to the invariant factors. The learned projection matrix maps the image and text representations of CLIP into the same invariant space. In this space, it be- comes possible to perform invariant prediction. We further analyze the OOD risk of this invariant predictor compared to a regular predictor. We provide conditions where the invariant predictor indeed achieve lower OOD risk.
2. Related Work
Vision-Language Pre-training. Incorporating language with the visual modality has been a long-standing issue, as extensively researched in previous works (Frome et al.. 2013; Socher et al., 2013; Norouzi et al., 2014; Elhoseiny et al., 2013). Recent years have seen significant success in multi-modal pre-training models, spearheaded by CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), with the aim of enabling cross- moddll learning by mapping lan- guage and visual modalities into the same embedding space, Notably, researchers have turned their focus towards fine- tuning CLIP in downstream tasks (Gao et al., 2023; Zhou et al., 2022a;b; Shu et al., 2023), In our work, we delve into the reasons behind the poor OOD generalization ability of CLIP from a causal perspective. We propose CLIP-ICM as a solution to address this issue.
Guided by the above theoretical analysis, we propose a novel modeling framework called the Invariant Causal Mecha- nism of CLJP (CIIP-ICM). This framework consists of three stages: (1) Collect the interventional data. (2) Estimate a linear projection to invariant subspace. (3) Perform invariant prediction in the invariant subspace. Under the CLIP-ICM framework, we propose specific methods to obtain interven- tional data using either image or text data. Additionally, we propose learning objective for estimating the projection matrix. Notably, our method does not require retraining the backbone network of CLIP. We evaluate the proposed CLIP- ICM on OOD generalization datasets, including Domainbed (Gulrajani & Lopez-Pazt and variants of ImageNet (Recht et al., 2019; Hendrycks et al., 20210:a Wang et al., 2019). Experimental results demonstrate the outstanding effective- ness of our approach. Considering its low computational cost and significant performance improvement, our method is both simple and effective.
Causal Representation Learning. The latent factor hy- pothesis posits that observational data is generated by un- derlying latent factors, with causal representation learning aiming to identify these factors- challenge akin to Inde- pendent Component Analysis (ICA) (Bishop, 1998; Bengio et al., 2013). While linear ICA is solvable (Hyvärinen & Oja, 2000), nonlinear ICA requires inductive biases such as auxiliary labels, sparsity constraints, or restricted function classes (Hyvärinen & Pajunen, 1999; Locatello et al., 2019; Schölkopf et al., 2021). Recent work has shifted from purely observational settings (Roeder et al., 2021) to leveraging interventional data (Ahuja et al., 2023; Squires et al., 2023), Aligning with this trend, we propose a method using inter- vention data to identify invariant features in CLIP, offering simple yet effective solution.
-uitrr-distribution Generaliztion. It is evident in many cases that the performance of deep learning methods is weakened when applied to different distributions (Beery et al., 2018; Taori et al., 2020). Studies have been conducted to address OOD generalization issues (Arjovsky et al., 2020; Ahuja et al., 2020; Gulrajani & Lopez-Paz; Miller et al., 2021; Abbe et al., 2023; Chen et al.). One important branch of work aims to find invariant causal mechanisms across domains, inspired by the invariance principle from causality (Arjovsky al., 2020: Peters et al., 2016; Suter et al., 2019; Ahuja et al., 2020; 2021; Chen et al.). Additionally, some studies incorporate SCM when analyzing the OOD prob- lem (Robey et al., 2021; Li et al., 2024). Vision languae pre-trained models, such as CLIP (Radford et al., 2021), ex- hibit impressive performance across various domains. How- ever, recent works emphasize that adapting CLIP with task- specific data often comes at the cost of OOD generalization ability (Shu et al., 2023; Gao et al., 2023; Pham et al., 2023). We also provide a detailed analysis with some prior works in Section L..
Our contributions can be summarized as follows: 1) We identify and demonstrate that CLIP exhibits inconsistent performance in the train and test domain when fine-tuned in OOD generalization scenarios. Through an in-depth causal analysis, we demonstrate that it is possible achieve consis- tent predictions using only invariant factors. 2) Through an analysis of the identifiability of CLIP, we theoretically prove the existence of a linear mapping from CLIP embeddings to the invariant factors. This mapping can be estimated using interventional data. Additionally, the OOD risk analysis provides a condition to guarantee lower OOD risk for the invariant predictor. 3) We propose the CLIP-ICM frame- work, which includes collecting interventional data using either image or text data, estimating the linear projection, and performing invariant prediction. Extensive experiments across various OOD scenarios demonstrate that CLIP-ICM significantly enhances the performance of CLIP. Addition- ally, a comprehensive set of ablation studies validates the effectiveness of our approach.
2

---

Learning Invariant Causal Mechanism from Vision-Language Models
3. Problem Setting
fers from the distribution of æ in the training environment while open-class scenarios involve previously unseen classes appearing at test time. Although the zero-shot capability of CLIP excels in handling open-class scenarios, maintaining this ability under domain shift presents a challenge.
3.1. Contrastive Language-Imag Pre-training
We begin by revisiting the framework for CLIP (Radford et al., 2021) !. Let æ E X' represent an arbitrary image, and & E T denote text sequence describing this image, forming an image-text pair (ær, t). Here, X' and TT refer to the image and text spaces, respectively. CLIP processes these inputs through two pre-trained encoders: an image encoder ft : X' + 2 and a text encoder fr T + 2. Both encoders map their respective inputs into a shared embedding space 2 C RD, where the embeddings of the image-text pair (xx t) are aligned. Here, D denotes the dimensionality of the embedding space.
Several established methods exist for fine-tuning CLIP to acquire such a predictor h for tasks in OOD scenarios: (1) Linear probe: A linear classifier is trained on the fixed output of f1- (2) Learnable prompt (Zhou et al., 2022b;a; Zhang et al., 2024): The fixed text descriptions for each class are replaced with learnable parameters. (3) Adapter (Gao et al., 2023): Both fr and fr are frozen, and a multi- layer perceptron (MLP) is trained on their outputs. (4) Fine- tuning image encoder (Shu et al., 2023): All parameters of fr are optimized, while fr are fixed. Despite these differences, all methods share common procedure when fine-tuned in OOD scenarios. A pre-trained CLIIP model is taken as the backbone, and some task-specific parameters O, such as classifier weights or prompt vectors, are introduced. Data from the training environments E,, C Eall is then collected. During training, model parameters are optimized to reduce classification errors across the aggregated training distribution. After this phase, the model is evaluated on test environments Eaul Eir using the updated parameters.
A specific characteristic of CLIP is the zero-shot prediction. The goal of zero-shot prediction is to use a predictor h X' - y' to map an input image x E X' to a predicted category label ğ E V. where y denotes the set of possible class labels. For each class e E V. a corresponding text description &e is defined. For instance, for class c, the description may be something like "a photo of [CLASS]" where [CLASS] is replaced by the name of class c. CLIP encodes both the image x and the text description t. for each class into the shared embedding space 2 C P. Then, it computes the similarity between the image embedding far) and the embedding for the text. description of each class (te). The predictor h is defined as:
4. Motivation Experiment
In this section, we evaluate the performance of CLIP under OOD generalization using the Terra Incognita dataset (Beery et al., 2018), which contains identical object categories of wildlife species across diverse, in-the- environmental conditions. We first examine only the domain shift scenario, where we adopt a leave-one-out protocol and train linear probe with the frozen CLIP image encoder. Specifically, for a given target domain, a linear classifier is trained on frozen CLIP image embeddings from all other domains and tested on the held-out domain to assess how well the model handles shifts in distribution. Next, we examine case combining domain shift and open classes to assess whether the strong zero-shot capabilities of CLIIP remain after fine-tuning in OOD settings. We divide categories into base and new classes and apply leave-one-ous out protocol. The image encoder fine-tuned on base-class data from training domains, while the text encoder remains frozen.
h(æ)) = arg max P(c(a), eEY'
s.t. P(c[x) = exp (S(((())),
Edey exp (S(f
Here, P(c]x) is the probability of class c given X, ((F((a) (t.)) denotes the cosine similarity between fz()) and (tz).. The predicted label á is the one associ- ated with the text description that has the highest similarity to the image embedding.
3.2. Of Generalization
Next, we present a unified framework for adapting CLIP to OOD generalization scenarios. In this framework, each distinct scenario (e.g., varying lighting, styles, or camera gles) is treated as an environment resents the set of all possible environments, and each envi- ronment corresponds to a unique data distribution P (æx y). The goal of OOD generalization is to learn a predictor h from training environments E., C Eau to perform well in unseen environments. Achieving this involves addressing two challenges that often occur together in practice: domain shift and open- plan-class scenarios. Here, domain shift refers to when the distribution of æ in the test environment dif-
Ealt where Eaul rep-
The model is then evaluated on both base and new classes in the target domain to assess whether the zero-shot ability of CLIP remains after fine-tuning.
From the results in Table 1, we can draw two important conclusions. First, in the domain shift scenario, the perfor- mance of fine-tuning is unsatisfactory. For instance, in the
We primarily refer to the case in standard domain generaliza- tion tasks.
'The definition of all notations is provided Appendix A.
3

---

Learning Invariant Causal Mechanism from Vision-Language Models
Table The performance of CLIP the Terra Incognita dataset in accuracy (%). L.00, 1.38, 143, and 1.46 represent different environments. In the inear-Probe case, the values in (-) represent the accuracy achieved by directly fine-tuning with the linear probe on the target domain.
DOMAIN SHIFT
METHOD
L100
L38
L43
L46
AvG
LINEAR-PROBE
73.6 (90.9)
58.3 (86.3)
61.0 (76.0)
47.8 (78.9)
60.2
(83.0)
OPEN CLASS UNDER DOMAIN SHIFT
SPLIT
METHOD
L100
L38
L43
L46
AvG
ZERO-SHOT
56.4
23.2
30.4
25.8
33.9
BASE
FINE-TUNE
75.6
58.5
65.2
55.1
63.6
ZERO-SHOT
47.6
17.6
35.2
37.4
34.5
NEW
FINE-TUNE
36.7
11.2
25.1
25.5
24.6
ZERO-SHOT
52.0
20.4
32.8
31.6
34.2
TOTAL
FINE-TUNE
56.2
34.9
45.2
40.3
44.2
L46 domain, accuracy achieved using the leave-one-out protocol (47.8%) is significantly lower by 31.1% 1% compared directly fine-tuning this domain (78.9%). This indi- cates that training on other domains does not ensure good performance on the L46 domain. Second, in the open-class scenario, the zero-shot performance remains nearly con- sistent across base and new classes. While the fine-tuned model outperforms zero-shot predictions on base classes, it performs worse on new classes. These results suggest that naively fine-tuning the CLIP model in OOD scenarios harm its strong zero-shot capabilities. This raises critical question: What causes the discrepancy between training and testing domains in the OOD generalization of CLIP, and how can we preserve the strong zero-shot capacity of CLIP for unseen classes while adapting to OOD scenarios?
E
7
Z
X
X
Y
Zino
Zino
(a) Data generation
Prediction
Figure 1. The SCMs for: (a) the data- aitta-generatnng process (b) the prediction process. The square denotes selection variable, see Appendix C for details.
variable Z P(z). defined on Z = Zinv Zvars consists of two components: the invariant factor Zinv P(Zine), and the variant factor Zear ~ P(Zwar).
5. Theoretical Analysis
The SCM Figure I(a) illustrates the data data-generating pro- cess. In this SCM, the edges Y - Zine and Y - Zear indicate that the class label Y influences both Zenz and Zyar- The edge E Zeur shows that Z is affected by the vironment E. Finally, the edges Zinr X and Zuar + X represent that the image X is generated from Zine and Zear. For example, consider the class label "bird" (Y). The wing shape and beak shape (Ziwv) do not change across different environments. However, feather color (Zwar) appears differ- ently day or night (E). Resulting varying images of birds (X) across environments.
5.1. Causal Analysis
To investigate the reasons behind the unsatisfactory OOD generalization of CLIP further, we propose to analyze it from causal perspective. During analysis, we adopt the latent factor hypothesis, which assumes that a set of latent factors generates the observed data. Based on this, we model both the gette-geneaatigg and prediction processes using Structural Causal Model (SCM) in Figure 3
Let X ~ P(x) and Y ~ P(y) be the random variables for images and class labels, defined on A' and V respectively. selection variable E ~ P(e), defined on Eaur represents the environment (Pearl & Bareinboim, 2014). The latent
The SCM in Figure 1(b) depicts the prediction process, which can be considered as the inverse of the data- generating process (Bengio et al., 2013; Schölkopf et al., 2021; Zimmermann et al., 2021). 1). this SCM, the edges X Zear and X - 7ine denote the representation learn-
Pleasse refer to Definition C.I for the definition of the SCM and the causal mechanism.

---

Learning Invariant Causal Mechanism from Vision-Language Models
y
Zin inv
Shift
2 inv
E = e
Projection
Z Zinv
E = e
0:
=
y
y
::2
00
00
0:1
O: :22
:2
fr((1)
f (t1
fr (t1)
(T((2)
Zvar
fT((2)
Z vvar
fT((2)
(a)
(b)
(c)
Figure 2. An example of a classifier defined on Z = Zine X Zear when applied to different environments, where the dashed lines represent the decision boundary and solid arrows depict the normal vectors of these boundaries. y is the set of labels. (a) illustrates an example of a classifier that is trained and tested in the same environment. (b) demonstrates applying the classifier from (a) to another enviromment E- e". The change in E only affects the distribution of Zear (c) demonstrate the case when samples and classifiers are projected to the Zi space.
ing process. The edges Zver Y and Zine Y repre- sent the prediction process. In OOD prediction scenarios of CLIP, the process X + Zino and X + Zvar corre- spond to the image being passed through the image en- coder to obtain its feature representation, i.e., fr(2)= 2, The process Zine + Y and Zvar Y involve comparing the image embedding with the embeddings of text descriptions (t.) for each class E V, and the predicted label is derived based on the simi- larity, as shown in Equation (1).
environments, ensuring predictor trained in one domain can generalize to others.
To illustrate this process, Figure 2 provides an example. In the figure, we plot the image samples in the latent space 2 Zine X Zear Samples from different classes are plotted in different colors. The arrows in varying colors represent the embedding of text descriptions for each class. Figure 2(a) shows the sample distribution in train environ- ment e, while Figure 2(b) depicts the distribution in test environment e". In Figure 2(a), the samples of a given color are similar to the corresponding text description features, ensuring correct predictions in the training environment." However, in Figure 2(b), the distribution of samples shifts along the Zvar dimension due environmental changes. This shift causes samples to move away from their corre- sponding text embeddings, leading to misclassification. In Figure 2(c), both the samples and text embeddings are pro- jected into the Zine subspace. Here, the environmental shift no longer affects the prediction. This ensures a consistent performance in both train and test environments.
where 2- Z
In an SCM, each edge represents a specific causal mech- anism. Our focus is the causal mechanism in the OOD prediction scenario. Specifically, we investigate whether predictor learned from training environment Err C Faul can still perform well in an unknown test environment ex E Eau- The following proposition outlines how the causal mecha- nisms change under environmental shifts:
Proposition 5.1. Let (*( P(-le") denote the distribu- tion in test environment E = e". The causal mechanisms are formalized through interventional distributions. We focus on the predictions P(y (d) where do(.) represents an in- tervention. The causal mechanism satisfies P* ()) + ()y)do(=)). When using only Zinn the causal mechanism remains unchanged: pa (yy (uldo(zsn))
5.2. Identifiability Analysis
In the previous section, we propose that if the prediction performed in the space Z the results in test environment remain consistent with training environments. However, in practice, it is unclear which part of the image representation obtained by CLIP corresponds to Z,nv- The task of finding invariant factors mirrors the well-established problem of latent factor identification (Schölkopf et al., 2021; Hyväri- nen & Oja, 2000). The observational data are assumed to be generated by the latent factor through an unknown,
((y)dd((zne)))
The detailed proof is provided in Appendix D. Proposi- tion 5.1 shows that the causal mechanism P dz)))) re- lying on both invariant and variant factors, varies across environments, making a predictor trained in one domain unusable in another. In contrast, the causal mechanism P(y]do(z 2),))), based only on Zinu> remains stable across
FFor simplicity, this process assumes ideal scenario, which we discuss further in Section 5.2.
*This remains true regardless the fine-tuning strategy, even in open-class scenarios. For further discussion, refer to Appendix B.
5

---

Learning Invariant Causal Mechanism from Vision-Language Models
non-linear process g. Identifying the latent factors meaning finding function f : X' ++ 2 such that f is the inverse of data generation process 9 Z + X. i.e., f = g . In this section, we theoretically prove that it is possible to find a linear mapping from the CLIP embedding space to Zine-
tion remains unchanged. Fortunately, Proposition 5.3 in- dicates there exists linear mapping from 2 to Zine- Consider an observed representation 2 = =f((2), Proposi- tion 5.3 suggests & is linear transformation of the true latent vector, i.e., 2 = Az, where A is an invertible matrix, x = [Zino: =var]" is the true latent vector. The true latent vector can be recovered by applying the inverse of A, so we have = = A We can decompose A into two blocks A = [Ain Avor T And Zinv can be obtained with Zine = Ain Since A is a fixed, invertible matrix, the above derivative shows that there always exists a linear mapping Aie that maps & to the true invariant factor Zinv-
We aim to explore the ability of CLIP to identify latent variables. To begin with, we assume the data generation process g Z + X is injective. This assumption implies that g is invertible, and different latent factors generate dif- ferent images. Based on this generation process, we define the ideal image encoder fJ- : X' + Z as the inverse of g. i.e. f)- = g (z)) Furthermore, we define fT- as the corresponding text encoder for fi.. The outputs of pairs (ar, through fr- and f₁- are aligned in the shared embed- ding space. For the actual encoders fr and fr in CLIP, we prove in Proposition 5.3 that if Condition 5.2 is satisfied, fr identifies E up to an invertible linear transformation.
Since both A and the true latent factors z unobservable. It is unlikely to directly obtain the matrix Aine through only the observed representation 2. However, by using interventional data, where certain latent factors are fixed, as outlined in Condition 5.4, we can prove that it becomes possible to estimate the Aino-
Condition 5.2. For an image encoder fi and a text en- coder fo with output dimensions D, for any x sampled from P(x), there exist D + 1 distinct text description pairs (ta, 1s) satisfying:
Condition 5.4. Let pdo(zune) denote an interventional distribution of X. The invariant factors of the images from this distribution are required to take a fixed value, i.e. Zinv = 21. Here, do(.) denotes an intervention flag. This condition require sampling from pdo(esww)) (x) is available.
exp(fi(e)) fT((a)) exp(fr. (x)(fr(()) fT-(ta)) (((T((a)TT((tt))) exp(Jr- (x) JT. (to))
ST-
(2)
Proposition 5.5. For the encoder fi satisfies Propo- sition 5.3, if Conndiion 5.4 is satisfied and sample from pdo(zino) (x)) is possible. Then for any pair (z) dollzioo) , ,22 dolsine)) sampledfrom pp)lzsn))xx)) The map-
Proposition 5.3. For an image encoder fl: X' + 2 and its corresponding lext encoder TT - 2, where 2 C RDD is the embedding space. If Condition 5.2 is satisfied, then fr(x) identifies Z up to an invertible linear transformation A, i.e. fı(x) = Ag (x) Az
ping Asno satisfies:
Amn Amm(fr (fr(x) (ooottiio))) - (r (202 dolkino)))
The proof is provided in Appendix ES, Condition 5.2 ensures both consistency and diversity. Consistency is achieved as the similarity between image embeddings and text embed- dings produced by f1 and fr aligns with those produced by the ideal encoders fi- and fr-. Diversity is ensured through the existence of sufficient pairs (tg. to). which guarantees the invertibility of the matrix A. Consequently, satisfying Proposition 5.3 ensures that predictions based on similarity remain consistent with the ideal setting, allowing CLIP to achieve accurate predictions. The remarkable performance of CLIP, supported by its large-scale training data, suggests that it can satisfy this condition in practice.
The proof is detailed in Appendix F. Proposition 5.5 pro- vides the theoretical foundation for extracting Zine and per- forming prediction the Ziee space. Once Condition 5.4 satisfied, interventional data can be obtained. By repeatedly sampling from this interventional distribution, it becomes possible to estimate a matrix that satisfies Equation (3). This matrix then maps the observable data into the Zinv space, allowing an invariant prediction across environments.
5.3. OOD Generalization Analysis
In the previous sections, we prove the feasibility of project- ing representations of CLIP into an invariant subspace and performing predictions within that subspace. In this section, we provide theoretical guarantees for the OOD generaliza- tion performance of this approach.
Proposition 5.3 suggests that the output representation of CLIP is linear combination of the true latent space, compassing both invariant and variant factors. As a re- sult, changes in the environment lead to shifts in all dimen- sions of the output of CLIP. As we discussed in Section 5.1, this can lead to incorrect prediction when the text descrip-
One way to assess OOD generalization is to measure per- formance in the worst-case environment. Formally, this is captured by the OOD risk:
'This conclusion aligns with results established in prior works (Roeder et al., 2021; Ahuja et al., 2022; Hyvarinen & Morioka, 2016; Zimmermann et al., 2021), and Proposition 5.3 demonstrates this result within our specific scenario.
ROOD
max R (h),,
(4)
ec e
6

---

Learning Invariant Causal Mechanism from Vision-Language Models
Collect Interventional Data
Estimate Ainu
3
Perform Invariant Prediction
Frozen Learnable
photo cat A photo of dog
Image-Based
A photo of person
Reconstruct
Assar
A lin
A
Ainn
Consistent
Ann
Ann
Text Based
High Similarity Low Similarity
Figure The pipeline the proposed CLIP-ICM framework. CLIP-ICM consists of three stages: (1) collect interventional data. (2)
estimate Aine. (3) perform invariant prediction.
where R (h) denotes the risk of h in environment e, and can be calculated with:
The proof is detailed in Appendix G. Theorem 5.6 state that, to ensure smaller OOD risk, needs to ensure high T(Zino: Z). This means Zine should contain enough information related to Z, thereby ensuring that the invariant predictor also performs well in the training environment.
Re (l)) = (xs)) [T(((>)> A y)],
(5)
where [(()(z)) + y) is the indicator function. It equals if the prediction of is incorrect and 0 otherwise.
6. Method
We consider two types of predictors: the conventional pre- dictor h and the invariant predictor hinn- The definition of the conventional predictor is provided in Equation (I), while the invariant predictor is defined as follows:
Building the theoretical results in Section 5, pro- pose Invariant Causal Mechanism of CLIP (CLIP-ICM), CLIP-ICM comprises three stages: (1) Collect interven- tional data. (2) Estimating Aino using the interventional data. (3) Perform prediction in the invariant subspace. The key challenges lie in two aspects: acquiring interventional data and estimating Aine satisfying the condition both Theorem 5.6 and Proposition 5.5.
Dine (x: = arg max P(c|2r, Ainr), cEy
(6)
where P(clær, A is the probability of class computed in the invariant subspace. It can be obtained by projecting with Aino and applying the following softmax function:
Collect Interventional data. According to Condition 5.4, interventional data are those generated by fixing the in- variant factors while allowing the variant factors to vary.
Pinok (c/x) exp (S(Amm (xx), A fT(.))) Σ exp (n( (x) fr e'EY
There are numerous ways to obtain such data (Nikolenko, 2021), including the use of rendering engines (Zhang et al., 2025), generative models (Zhang et al., 2023a; Azizi et al.), or large language models (Li et al., 2023; OpenAI, 2023). Leveraging the property of CLIP to align image and text features within the same embedding space, we propose two approaches for collecting interventional data: one based on images and the other based on text.
By investigating the OOD risk of these predictors, we present the following theorem:
Theorem 5.6. Let H be a hypothesis class over X x V, and let Eau denote a set of enviromments with distributions {P*) DeEEa over X x y. Let hine E H. be a predictor relying solely on Zine, and h E a predictor utilizing both Zinv and Zear- If the mutual information between Zino for some constant c > O. Th Then, the worst-case OOD risk strictly satisfies:
We first collect image data from subset of training environ- ments E., Ean- The interventional data is generated by applying data augmentation techniques. Data augmentation preserves Zine while altering Z This works because
and Z satisfies I IZZini Z)
ROOD (hame) < ROOD (h))(h))

---

Learning Invariant Causal Mechanism from Vision-Language Models
invariant factors, like class-relevant features, remain un- changed, while variant factors, such as color, orientation, or background, are modified. We use a combination of meth- ods, including color jittering, grayscale, and Gaussian blur, to create diverse transformations. Details on these tech- niques are provided in Appendix H, and an ablation study of specific augmentations is discussed in Appendix M. For each image æ, we apply random augmentation a<.)) to generate an augmented sample (ax). Embeddings of the pair Jr (a()))) (xx) are then used as interventional data.
invariant subspace using Ain. Predictions are made using the invariant predictor defined in Equation (6).
7. Experiment
To evaluate the CLIP-ICM framework in OOD scenarios, we conduct experiments on the DomainBed benchmark (Gulra- jani & Lopez-Paz). We use five datasets from DomainBed: PACS (Li et al., 2017), VLCS (Fang al., 2013), Office- Home (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018), and DomainNet (Peng et al., 2019).8
Due to the ability of CLIP to align the image and text embed- ded in the same latent space, interventional data can also be generated by modifying the text descriptions associated with a given image. We use the same training images from Exr- Pre-intervention text descriptions are generated for each image using an image captioning model. Post-intervention text descriptions are created by modifying these captions using a large language model. Details of the prompts used for generating and modifying text descriptions are provided in Appendix H. Finally, the embeddings of the pre- and post-intervention text descriptions, (t) and fr(((t)), are used interventional data for further processing.
We evaluate CLIP-ICM for OOD generalization in two set- tings: (1) domain shift and (2) domain shift with open-class scenarios. For domain shift, we use a leave-one-out proto- col, training on all domains except the target and testing on the target domain (Table 2). For the combined setting, we split data into base and new classes, train basc classes in training domains, and evaluate both base and new classes in the target domain (Table 3). Each value in Table 2 and Table represents the mean and standard deviation over 5 runs with different random seeds.
We compare three interventional data collection methods: models marked with use image-based data, use text- based data, and unmarked use both. We also compare two prediction strategies: CLIP-ICM, which maps embeddings through Aian, and CLIP-ICM Linear Probe, which trains a classifier on image embeddings projected by A
Estimate Amn. Once the interventional data is obtained, the next step is to learn the Alvo matrix. The Aive matrix must satisfy two key conditions: (1) According to Propo- sition 5.5, Ainn should satisfy Equation (3), ensuring that the embeddings obtained from interventional data pairs are equal after applying Aine- (2) Based on Theorem 5.6, the projected embedding should retain as much information about the original representation fr (ar) as possible. To meet these conditions, we formulate the learning objective as the following optimization problem:
Results. From Table 2, CLIP-ICM consistently outper- forms previous fine-tuning methods for CLIP. On the Tera Incognita dataset, CLIP-ICM Linear Probe surpasses the best-performing method, CLIPOOD, by 6%. From Table 3, while CLIP-ICM does not achieve the best performance on base classes, it significantly outperforms other methods on new classes. The average performance on new classes exceeds the best-performing method, CoOp, by 5.5%. This shows that the CLIP-ICM learning method preserves the zero-shot capability of CLIP even after fine-tuning. Com- paring the three approaches for interventional data, models using only image-based data outperform those using only text-based data. Combining both further improves perfor- mance, highlighting their complementary benefits.
min| 2 Aino- inv 212 + Arn
++A11A7 Al1A7m (2- - 2d0(zs >>
s.t.
Ainu ID...'
Here, & represents the matrix of observable embeddings ob- tained from either the image encoder fr or the text encoder fr. 7 corresponds to the embeddings obtained from interventional data, where the invariant factors Zinv are fixed while the variant factors Zvar vary. I.... denotes the identity matrix of dimension Dine. and À is a regularization hyperparameter. The first term of Equation (9) along with the constraint Aine = Dine ensures that 2 is effec- tively reconstructed within the invariant subspace defined by Amne. The second term enforces consistency between the embeddings of interventional data in the invariant subspace.
8. Conclusion
In our work, we observe the limitations of CLIP in OOD scenarios. Causal analysis reveals that predictors relying
Alternatively, predictions can be made by projecting only the image embeddings through Ais and training a classifier on the projected space. This approach is more suitable when the classes in the training environments are identical to those in the test environments (i.e.. open-class scenarios).
Perform invariant prediction. After obtaining Aino through Equation (9), we can perform invariant prediction. For a test image x and a fixed set of text prompts (te (oel the embeddings are first computed using fi and fr, re- spectively. These embeddings are then projected into the
*See Appendices J and for additional results and Ap- pendix M for ablation studies.
8

---

Learning Invariant Causal Mechanism from Vision-Language Models
Table Accuracy on the DomainBed benchmark with domain shift. Methods with indicate training with only image- interventional data. Methods with indicate training with only text-based interventional data.
METHOD
BaCKBoNE
PACS
VLCS
OFFiCEHome
TErRaInc
DOMAINNET
AvG.
ZERO-SHOT
CLIP
96.1
82.4
71.5
34.2
56.8
68.2
LINEAR-PRODE
CLIP
96.4
78.7
81.9
60.220a
55.020.
74.4
MIRO (CHA ET AI.,, 2022)
CLIP
95.61n.3
82.2101
82.510.3
54.310a
54.01os
73.710.5
CoOp (ZHOU ET AL.,, 2022b)
CLIP
97.0+0.2
83.0+0.1
81.1+0.4
54.6+02
59.5+02
75.0+m.2
CoCoOp (ZHOо
2022A)
CLIP
96.7+0.4
83.6=0.1
80.7+0.1
56.2+0.a
59.7+0.4
75.4+0.4
CLIP-ADAPTER (GAO ET AL., 2023)
CLIP
96.410.3
84,3
82.210.2
57.529.4
59.910.
76.110.1
DPL (ZHaNG ET AL., 20238)
CLIP
97.3
84.3
84.2
52.610.4
56.7
75.0
CLIPOOD
CLIP
97.3+0.1
85.0=0.4
87.0+0.2
60.4+0.7
63.5+0.1
78.6+0.1
CLIP-ICM"
CLIP
97.3
84.1
82, 22..3
49.9.0.a
60.520.
74.9
CLIP-ICM" LINEAR- PROBE
CLIP
97.510.5
86.5
84.610.4
64.3193
64.0102
79.010.
CLIP-ICM
CLIP
96.810.4
83.4goa
82.1
45.2203
$7.410
73.010.
CLIP-ICM' LINEAR-PROBE
CLIP
97.2
85.2
82.4+0.3
61.2+a.
59 59.65044
77.1
CLIP-ICM
CLII
97.7+0.2
86.2+0.3
84.6+0.2
52.5+0.4
61.1
76.4
CLIP-ICM LINEAR-PRURE
CLIP
97,820.3
86.620.1
87.110
66.5
65.0
80.610.2
Table Accuracy OfficeHome and DomainNet with both domain shift and open classes. Methods with indicate training with only image-based interventional data. Methods with indicate training with only text-based interventional data.
OFfcELlOme
DOMAINNET
SPLIT
METHOD
A
C
P
R
C
1
P
Q
R
S
CLIP
86.8
75.5
89.5
92.6
72.8
51.7
66.0
13,5
83.4
66.9
CoOp
87.0ga.4
78.321.2
92.4
91.440.6
75.7
58.8
68.5
13.1
84.0g0.5
70.0
CLIPOOD
90.1
79.720.2
93.1
94.8
79.0
62.220.1
73.0
20.2
86.220.1
73.8
BASE
CLIP-ICM'
88. 38.6
78.0+a.2
90.2+0.3
93. +3.2
74.4+0.3
53.2+0.a
67.2+0.3
14.6+0.2
85.2+0.3
67.8+0.2
CLIP-ICM'
87.1
77.2+0.2
89.3
92.0+a.2
73.1
52.0+0.1
+6.1
14.1
83.8+0.4
66.8+0.4
CLIP-ICM
89.2
78.620.4
90.6
93.7
75.040.3
53.920.a
67.610.4
14.9
85.9
68.420.4
CLIP
76.6
59.4
88.1
86.2
70.2
44.1
66.4
14.1
83.5
61.0
CoOp
26.5
56.642.4
88.0
86.8
71.5+0.2
47.2+0.3
67.3
14.8
83.7
63.1
CLIPOOD
77.8
60.020.2
88.3
86.7.0.1
71.220.1
48.1
68.2+0.2
18.0+0.4
83.420.1
62.940.1
NEW
CLIP-ICM"
81.7
66.5
90.2+..4
90.640.4
76.740.2
50.9
69.1
17.240.5
83.640.a
67.7.0.5
CLIP-ICM
81.2+0.2
65.2+0.2
89.4+0.4
89.9+0.2
76.0+6.2
49.8+0.1
67.9+0.2
15.7+0.1
82.5
67.0+0.4
CLIP-ICM
82.6
67.5
20.9
91.5
77.8
51.610.6
70.2
18.010.4
84.5
68.610.6
CLIP
82.6
67.3
88.8
89.5
71.4
47.1
66.2
13.8
83.4
63.4
CoOp
82.7+0.5
67.2+a.7
90.2+1.0
89.2+a.6
73.4
51.840.a
67.9+1.
13.7+0.8
83.9
66.0+0.2
CLIPOOD
85.1
69.6.0.2
90.8+a.1
91.0+a.1
74.8
53.620.1
70.6:0.1
19.1
84.8
67.420.1
TOTAL
CLIP-ICM"
85.2g0.4
72.320.3
90.2
91.920.3
75.620.2
52.1
68.220.4
15.940.3
84.4g0.3
67.820.3
CLIP-ICM'
84.2+a.1
71.2+0.2
89.4
91.0+a.2
74.640.2
50.9+6.1
67.040.1
14.9+0.1
83.240.2
66.9+0.4
CLIP-ICM
85.920.2
73.1
90.8
92.620.2
76.4
52.810.4
68.9
16.5
85.2g0
68.5
solely on invariant factors achieve consistent performance across training and test environments. Identifiability anal- ysis further guarantees the existence of linear mapping from CLIP embeddings to invariant subspace, which can be estimated using interventional data. If certain con- ditions are met. the invariant predictor achieves a lower OOD risk. Building on these insights, we propose CLIP- ICM, a framework for collecting interventional data and estimating the required linear mapping. Experiments on benchmark datasets demonstrate the superior performance of CLIP-ICM compared to existing methods.
consequences of our work, none of which we feel must be specifically highlighted here.
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work supported in part by the China Postdoctoral Science Foundation, Grant No.2024MM533566 and in part by the National Natural Sci- ence Foundation of China, No. 62406313.
References
Impact Statement
Abbe, E., Bengio, S., Lotfi, A., and Rizk, K. Generalization the Unseen, Logic Reasoning and Degree Curriculum. In Proceedings of the 40th International Conference on
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal

---

Learning Invariant Causal Mechanism from Vision-Language Models
Machine Learning, pp. 31-60. PMLR, July 2023. ISSN: 2640-3498.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th Interna- tional Conference on Machine Learning. pp. 1597-1607. PMLR, November 2020. ISSN: 2640-3498.
Ahuja, K., Shanmugam, K., Varshney, K., and Dhurandhar, A. Invariant Risk Minimization Games. In Proceedings of the 37th International Conference on Machine Learning,
pp. 155, PMLR, November 2020. ISSN: 2640-3498.
Chen, Y., Zhou, K., Bian, Y., Xie, B., Wu. B., Zhang. Y., KAILI, M., Yang, H., Zhao, P., Han, B., et al. Pareto invariant risk minimization: Towards mitigating the op- timization dilemma in ofist--diitriuution generalization. The Eleventh International Conference on I Learning Representations.
Ahuja, K., Caballero, E., Zhang. D., Gagnon-Audet, J.-C., Bengio, Y., Mitliagkas, I., and Rish, I. Invariance princi- ple meets information bottleneck for uut-of-diitribution generalization. Advances in Neural Information Process- ing Systems, 34:3438-34555 2021.
Chuang, C.-Y,, Torralba, A., and Jegelka, S. Estimating gen- eralization under distribution shifts via Nomain-invarian representations. In International Conference on Machine Learning, pp. 1984-1994, PMLR, 2020.
Ahuja, K., Mahajan, D., Syrgkanis, V., and Mitliagkas, I. Towards efficient representation identification in super- vised learning, In Conference on Causal Learning and Reasoning, pp. 19 43. PMLR, 2022.
Elhoseiny, M., Saleh, B., and Elgammal, A. Write a classi- fier: Zero-shot learning using purely textual descriptions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2584 2591, 2013.
Ahuja, K., Mahajan, D., Wang, Y., and Bengio, Y. Interven- tional Causal Representation Learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 372-407. PMLR, July 2023. ISSN: 2640-3498.
Fang, C., Xu, Y., and Rockmore, D. N. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference Computer Vision, pp. 1657- 1664, 2013.
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez- Paz, D. Invariant Risk Minimization, March 2020. MiXiv:1907.02899 [es, stat].
Azizi, S., Kornblith, S., Saharia, C., Norouzi, M., and Fleet, D. Synthetic data from diffusion models improves ima- genet classification. Transactions on Machine Learning Research.
Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. Devise: A deep visual- semantic embedding model. Advances in neural informa- tion processing systems, 26, 2013.
Beery, S., Van Horn, G., and Perona, Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pp. 4564473, 2018
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., March, M., and Lempitsky, V. Domain- adversarial training of neural networks. Journal of ma- chine learning research, 17(59) 2016.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R.,, Zhang, Y., Li, H., and Qiao, Y. CLIP-Adapter: Better Vision-Language Models with Feature Adapters. International Journal of Computer Vision, September 2023. ISSN 0920-5691, 1573-1405. doi:
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transac- tions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
Glymour, M.. Pearl, J., and Jewell, N. P. Causal Infer- ence in Statistics: A Primer. John Wiley & Sons, Jan- uary 2016. ISBN 978-1-119-111 1-119-18686--.. Google-Books-. IOV2CWAAQBAJ.
Bishop, C. M. Latent Variable Models. In Jordan, M. I. (ed.),
Learning in Graphical Models, pp- 371 371-403. Springer Netherlands, Dordrecht, 1998. ISBN 978-94-010-6104-9 978-94-011-501499 doi: 0011//9778-9--115014--- 13.
Guirajani, I. and Lopez-Paz, D. In search of lost domain generalization. International Conference on Learning Representations.
Cha, J., Lee, K., Park, S., and Chun, S. Domain gener- alization by mutual-information regularization with pre- trained models. In European conference on computer vision, pp. 440-457. Springer, 2022.
Hendrycks et al. The many faces of robustness: A critical analysis of diitti--iitiiittonnnn generalization. Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 9340-8349, 2021a.
10

---

Learning Invariant Causal Mechanism from Vision-Language Models
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15262-15271, ,, 2021b.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124, PMLR, 2019.
Hyvarinen, A. and Morioka, H. Unsupervised feature ex- traction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Miller, J. P., Taori, R., Raghumathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. Accuracy on the line: on the strong correlation between ditt---distribution and in-distribution generalization. In International Conference on Machine Learning, pp. 7721- 7735. PMLR, 2021.
Hyvärinen, A. and Oja, E. Independent component analysis: algorithms and applications. Neural Networks, 13(4):
411-430, June 2000. ISSN 0893-6080. doi: 10.1016/ 508993-608000000000---5
Nikolenko, S. I. Synthetic datafor deep learning, volume 174. Springer, 2021.
Hyvärinen, A. and Pajunen, P. Nonlinear independent com- ponent analysis: Existence and uniqueness results. Neural Networks, 12(3):429 439, April 1999. ISSN 0893-6080. doi: 10. 10.10016/50089--6000999900010010
Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G. S., and Dean, J. Zero-shot learning by convex combination of semantic embeddings. In 2nd International Conference on Learning Representa- tions, ICLR 2014, 2014.
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q.. Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904 4916. PMLR, 2021.
OpenAL. GPT-4 Technical Report, December 2023. EXXiv:300308874 [cs].
Pearl, J. Causality. Cambridge university press, 2009.
Pearl, J. and Bareinboim, E. External Validity: From Do-
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B. A., Haque, I. S., Beery, S., Leskovec, J. Kundaje, A., Pierson, E., Levine, S., Finn, C., and Liang, P. WILDS: A benchmark of in-the-wild distri- bution shifts. In International Conference on Machine Learning (ICML), 2021.
Calculus to Transportability Across Populations. Statis- tical Science, 29(4):579-595 November 2014. ISSN 0883-4237, 2168-8745 doi: 10. 10.114444-SSS488 Pub- lisher: Institute of Mathematical Statistics.
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Peters, J., Bühlmann, P., and Meinshausen. N. Causal In-
ference by using Invariant Prediction: Identification and Confidence Intervals. Journal of the Royal Statistical So- ciety Series B: Statistical Methodology, 7865):947-1012, November 2016. ISSN 1369-7412, 1467-9868. doi: 10.1 101111/rssb.1111 12167.
Li, H., Pan, S. J., Wang, S., and Kot, A. C. Domain general- ization with adversarial feature learning, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018.
Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A. W., Yu, J.. Chen, Y.-T., Luong, M.-T., Wu, Y., Tan, M., and Le, Q. V. Combined scaling for zero-shot transfer learning. Neurocomputing. 555:126658, October 2023, ISSN 09252312, doi: 10. 00.10166j.eeucom.2023. 202.2...00..2....6666... 126658.
Li, Z., Zhu, H., Lu, Z., and Yin, M. Synthetic data gener- ation with large language models for text classification: Potential and limitations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10461, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Ma- chine Learning, pp. 1748-8765 PMLR, July 2021. ISSN: 2640-3498.
Li, Z., Cai, R., Chen, G., Sun, B., Hao, Z., and Zhang, K. Subspace identification for multi-source domain adapta- tion. Advances in Neural Information Processing Systems, 36, 2024.

---

Learning Invariant Causal Mechanism from Vision-Language Models
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize imagenet? In Interna- tional conference on machine learning, pp. 5389-5400. PMLR. 2019.
Von Kügelgen, J., Sharma, Y., Gresele, L.., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, R. Self- supervised learning with data augmentations provably isolates content from style. Advances in neural informa- tion processing systems, 34: 64511-6667,, 2021.
Robey, A., Pappas, G. J., and Hassani, H. Model-based domain generalization. Advances in Neural Information Processing Systems, 14:20210-20222 2021.
Wang, H., Ge, S., Lipton, Z. and Xing, E. P. Learning robust global representations by penalizing local predic- tive power. Advances in Neural Information Processing Systems, 32, 2019.
Roeder, G., Metz, L.,, and Kingma, D. On linear identifia- bility of learned representations. International Con- ference on Machine Learning, pp. 9030-9039. PMLR, 2021.
Yan, S., Song, H., Li, N., Zou, L.,, and Ren, L. Improve un- supervised domain adaptation with mixup training. arXiv preprint arXiv:2001 2020.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations.
Yang, M., Fang. Z., Zhang, Y., Du, Y., Liu, F., Ton, J.-F. Wang, J., and Wang. J. Invariant learning via probability of sufficient and necessary causes. Advances in Neural Information Processing Systems, 36.79832-79857, 2023.
Schölkopf. B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward Causal Representation Learning. Proceedings of the IEEE, 109(5):612-634 May 2021. ISSN 1558-2256. doi: 10.11000/PR0C2220203355555 Conference Name: Pro- ceedings of the IEEE.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 1836-3847, 2023a.
Zhang, L., Rao, A., and Agrawala, M. Scaling in-the-wild training for biffus illumination harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference on Learning Representations, 2025.
Shu, Y., Guo, X., Wu, J., Wang, X., Wang, J., and Long, M. Clipood: Generalizing clip to outri--distributions. In International Conference on Machine Learning, pp- 31716-31731. PMLR, 2023.
Socher, R., Ganjoo, M., Manning, C. D., and Ng, A. Zero- shot learning through cross-modal transfer. Advances in neural information processing systems, 26, 2013.
Zhang, M., Marklund, H., Dhawan, N., Gupta, A., Levine, S., and Finn, C. Adaptive risk minimization: Learning adapt to domain shift. Advances in Neural Information Processing Systems, 01.23664-236777 2021.
Squires, C., Seigal, A., Bhate, S. S., and Uhler, C. Lin- ear causal disentanglement via interventions. In Inter- national Conference on Machine Learning, pp. 32540- 32560, PMLR, 2023.
Zhang, X., Gu, S. S., Matsuo, Y., and Iwasawa, Y. Domain prompt learning for efficiently adapting clip to unseen do- mains. Transactions of the Japanese Society for Artificial Intelligence, 38(6)B-MM22. 2023b.
Suter, R., Miladinovic, D., Schölkopf, B., and Bauer, S. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pp. 6056- 6065. PMLR, 2019.
Zhang, Y., Li, J., Liu, L., and Qiang, W. Rethinking mis- alignment vision-language model adaptation from a causal perspective. arXiv preprint arXiv:2410.12116 2024.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribu- tion shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599, 2020.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16816-16888 168255, 2022a.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, S. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learn- ing to Prompt for Vision-Language Models. Interna- tional Journal of Computer Vision, 13009))2337-2348 September 2022b. ISSN 0920-5691, 1573-1405. doi: 10. 10.1007/s1 1 063-022-01653 arXiv:2109.0113 [es].
12

---

Learning Invariant Causal Mechanism from Vision-Language Models
Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive learning inverts the data generating process. In International conference machine learning, pp. 12979-12990. PMLR, 2021.
13

---

Learning Invariant Causal Mechanism from Vision-Language Models
Appendix
This supplementary material provides detailed proofs for the theorem and proposition mentioned in the main text. Further- more, additional experimental results and implementation details are provided.
Appendix provides the definition of all notations of the main text.
Appendix B presents detailed discussion of various fine-tuning strategies within the context established in Section 5.1.
Appendix provides some background of the causality.
Appendix provides the proof of Proposition 5.1 in the main text.
Appendix E provides the proof of Proposition 5.3 in the main text.
Appendix F provides the proof of Proposition 5.5 in the main text.
Appendix provides the proof of Theorem 5.6 in the main text.
Appendix H provides the implementation details for CLIP-ICM.
Appendix provides the performance of CLIP-ICM in the setting of Section 4.
Appendix J provides the experimental results in ImageNet and variants ImageNet.
Appendix K provides detailed description of the used datasets.
Appendix L provides detailed comparison with the existing works.
Appendix M provides the ablation study.
Appendix N provides an analysis where the domain prompt is used in zero-shot case.
Appendix O provides full results in each domain in DomainNet Benchmark.
A. List of Notations
We list the definitions of all notations from the main text as follows:
Variables
x: An image sample.
t: A text description.
An environment.
e": The test enviromment.
A class.
A ground-truth label.
A predicted label.
2: The image embedding.
A latent factor.
Zing: An invariant latent factor.
Zear: A variant latent factor.
ww: The normal vector of the decision boundary.
Random Variables
X: The random variable of the image.
Y: The random variable of the class label.
Zin:: The random variable of the invariant factors.
14

---

Learning Invariant Causal Mechanism from Vision-Language Models
zerr: The random variable of the variant factors.
zar The random variable of variant factors in environment e*.
E: The selection variable of environments.
Sample Spaces Supports
- T: The support text description.
X: The support of image samples.
:doolsioo); The support of image samples when the latent factor Zine takes fixed value.
-
Y: The support of the class label.
2: The support the embedding.
Z: The support of the latent factors.
Ziv: The invariant subspace.
-
Zvar:
The variant subspace.
Functions
fr: The image encoder.
fr: The text encoder.
fr= : The ideal image encoder.
fr+: The ideal text encoder.
g: The ideal data- generatnng process.
f: The ideal representation learning process.
h: The regular predictor.
hive: The invariant predictor.
I: The indicator function.
W: The coefficient matrix of a linear classifier.
P: The probability distribution.
P": The probability distribution in environment e".
pdo(x)mo); The distribution of other latent factors when Zino takes fixed value.
Ain:: The mapping from 1 to Zine.
Avar: The mapping from 2 Zvar.
a: The data augmentation function.
3: The function to alter text description.
Sets
E The set all training environments.
Fau: The of all environments.
D: The datasets from training environments.
Dc: The dataset in environment e.
- H: The hypothesis class of predictor.
2: The collection of embedding vectors.
do((suu)); The collection of embeddings obtained from interventional data, where the invariant factors Zino are fixed while the variant factors Zpar vary.
Functionals
ROOD. The OOD risk.
R The generalization risk in environment e.
Rir: The generalization risk on E,r.
R": The generalization risk in environment e*.
15

---

Learning Invariant Causal Mechanism from Vision-Language Models
Constants
D: The dimension of embedding space / latent factors.
Dine: The dimension of the invariant factors.
Dvar: The dimension of the variant factors,
No; The number of instances in environment
C: The total number of classes.
K: The total number of data augmentations.
B. Discussion on Fine-tune Strategies
2
Before fine-tune
Zine
Linear Probe
Zino
Fine-tune
1 n :22 New
O Bass
Image Encoder
(ar,)
[i(2))
F ::2 New
fr-mm)
oo
80
(x(t))
W1
(6x)
Zear
Zuar
Zuar
frtta)
fr fr. (a2)
(1
Pou n D fi(2)))
w.
Weights of daasmer WF, bj
Optimized fr=:
Image Enceder
(a) Before Fine-tune
(b) Linear Probe
(c) Fine-tune fz
Zinv
Adapter
y
Zino
Learnable Prompt
New
Wafeiar
New
O
Wafs(ti
Dr
Zear
Wa
fe( Ma....
Myy image odapner
Wifr(ms)
Learnable tokens
Wyr wdapter
[V5, VS....
(d) Adapter
(c) Learnable Prompt
Figure An illustration the fine-tuning process under the setting Section 5.1. The red circle represents samples from the hase class while the green square represents the samples from the new class.
In this section, we discuss why almost all fine-tuning methods can be understood as leading to the scenario illustrated in Figure 2. This is detailed in Figure 4.
Before fine-tuning the training domains, the image embeddings and text embeddings not well aligned, shown in Figure 4(a). However, there exists class-specific text embeddings (+1)), fr((2) for both base and new classes.
When fine-tuning with linear probe, the method learns a classifier specifically for the base classes, with parameters w1, b1. Within the base classes, the classifier parameters W1 align with the image embeddings, as illustrated Figure 4(b). Thus, satisfies the conditions depicted in Figure 2(a). However, linear probe fine-tuning does not utilize information from CLIP's text encoder, making it unable predict new classes.
When only the image encoder is fine-tuned, the distribution of image embeddings changes, while the text embeddings remain fixed. This adjustment aligns the image embeddings for base classes in the training domains with their corresponding text embeddings. For new classes, the image embedding distribution also shifts, and ideally, they should align with the fixed text embeddings, as shown in Figure 4(c). This satisfies the conditions described in Figure 2(a).
When using an adapter for fine-tuning on the training domains, both image and text embeddings are altered. This ensures that the image and text embeddings for base classes are aligned. Ideally, by applying the same adapter to new classes, the
16

---

Learning Invariant Causal Mechanism from Vision-Language Models
text and image embeddings of new classes should also align in the training environment, as depicted in Figure 4(d).
When using learnable prompts for fine-tuning, the image embeddings remain unchanged, but the introduction of learnable tokens modifies the text embeddings. This allows the text embeddings of base classes to align with the image embeddings in the training domains. Ideally, by applying the same learnable tokens to new classes, the text embeddings of new classes adjust accordingly, aligning with the image embeddings in the training environment, as shown in Figure 4(e).
is important to note that these fine-tuning methods utilize the same text description embeddings or linear classifiers during testing across environments. However, in the test environments, the distribution of image data shifts, potentially leading to the situation illustrated in Figure 2(b), where incorrect predictions occur for both base and new classes
Worse still, when fine-tuning is performed only on the base classes, the learnable parameters may overfit to the base classes. This overfitting can result in the misalignment of embeddings in Figure 4(c), Figure 4(d), and Figure 4(c) even within the training environment for new classes. Consequently, the misalignment is further exacerbated in the test environments, making it even more challenging to correctly classify new classes.
C. Background in Causality
Structural Causal Model of SCM is:
The SCMs can be used describe the causal variables and their relationships. The definition
Definition C.1. (Structural Causal Model (Pearl, 2009))
1. A set U of background or exogenous variables, representing factors outside the model, which nevertheless affect relationships within the model.
2. set = .V. Vn of endogenous variables, assumed to be observable. Each of these variables is functionally
dependent on some subset PA, of U V. Here, PA; means the set of parent nodes of the endogenous variable V, 3. A set F of functions (/,....: : [n] such that each determines the value v, of V; V. 0z Se(pas; u). These functions also known as causal mechanisms.
4. A joint probability distribution P(u) over U.
The SCMs can be represented by Directed Acyelic Graphs (DAG) (Glymour et al., 2016). In the DAG, the arrow X + Y denotes that changes in the value of X directly cause changes in Y. However, this relationship does not hold in the reverse order.
The Selection Diagram and Transportability Pearl et al. (Pearl & Barcinboim, 2014) propose the selection diagram, which is used to model the causal relation between different environments with an external selection variable. The definition of the selection diagram is:
Definition C.2. (Selection diagram): Let < M, M > be pair of SCMs relative to domain < II, II >, sharing a causal diagram G. M, M >> said induce selection diagram D if D is constructed follows:
1. Every edge in G' is also an edge in D 2. D contains an extra edge S + V; whenever there might exist a discrepancy fs + f or P(uu) + P*(u:) between M and M
The set are denoted as selection variables.
Transportability aims to study whether the causal relation in a specific environment can be applied to any other environment, which is defined as:
Definition C.3. (Transportability):: Let be a selection diagram relative to domain < II, >. Let < P, > be the pair of observational and interventional distributions of II, and p be the observational distribution of II. The causal relation R(IT") P* (y do(x:)) is said to be transportable from II to II in D if R(IT*) is uniquely computable from P, P* in any model that induces D.
D. Proof for Proposition 5.1
Before we proceed to the proof of Proposition 5.1, we first provide some useful definitions and lemma.
Definition D.1. (d-separation): A set of nodes is said to block a path p if either p contains at least one arrow-emiting node that is in S.
17

---

Learning Invariant Causal Mechanism from Vision-Language Models
p contains at least one collision node that is outside S and has no descendant in S. If S blocks all paths from X Y, it is said to d-separate X and Y. X and Y are independent given S, written X T Y|S.
The d-separation reflects the conditional independencies that hold the distribution that is compatible with the DAG.
Definition D.2. (Rules of do-calculus) Let X, Y, Z, W be arbitrary disjoint sets of nodes in a causal DAG G. We denote by Gx the graph obtained by deleting from G all arrows pointing nodes in X. Likewise, we denote by Gy the graph obtained by deleting from G' all arrows emerging from nodes in X. The notation Gxz represents the deletion of both incoming and outgoing arrows.
1. Insertion/deletion of observations.
P(y]do(z), P(vldo(x), w) if и Z|X, W)
2. Action/observation exchange.
P(y)do(z), (oldo()),d((),.u)))) P(yldo(z),w)) if 1 Z|X, W) GTzZ'
3. Insertion/delction of actions.
P(yldo(z), do(2). w) P(y]do(x). w) if (Y 1 Z|X, W
X
where Z(W) is the set of Znooees that are not ancestors of any W W-node in Gx Definition D.3. (Trivial transportability). causal relation It is said to trivially transportable from II to II", if R(II) is identifiable from (G*,P*)
R is trivially transportable if can directly estimate R(II") from observational data of II", unaided by the causal information from II. The following state the sufficient and necessary conditions of transportability of average causal effect P* (y)dd(x))
Definition D.4. (S-admissibility). set Z of variables satisfying (Y 5) X) ) in Dx will be called S-admissible with respect the causal effect of X on Y. Dx denote deleting all arrows pointing node X in D.
Lemma D.5. (sufficient and necessary conditions of transportability (Pearl & Bareinboim, 2014)) The average causal effect Pa (1))d(()) is transportable from II to II if either one of the following conditions holds: 1. P (y do(x)) is trivially transportable.
There exists a set of covariates Z (possibly affected by X) such that is S-admissible and for which P" (=(2do((z) is transportable.
There exists set of covariates, W that satisfy (X YIW) X(W) and for which P* (w]do(<)) is transportable.
Proof. 1. According to Definition D.3, the causal relationship can be directly estimated from observational data of II"
If condition holds, it implies:
(*)l(o((x) = P(yldo(x), 8)
(10)
Σ (()))l))))
Σ ((y)lo(s))
The transportability of P(z]do(z)) reduces (* do(x)) to star-free expression and therefore (y)dd(x)) is transportable.
3. If condition 3 holds, it implies:
P*(y)o((z) Plyldo(z).s)
(11)
Σ P(yldo(x), w. w. P(w)do((z)) s)
Σ Plylw, 8) (wdllo(()
According to Rule 3 of Definition D.2, the transportability of Pa (w)((())) would render P(w]do(x), s) to a star-free expression and render P* P* (y]do(x)) transportable. This ends the proof.
18

---

Learning Invariant Causal Mechanism from Vision-Language Models
We now prepared present the proof for Proposition 5.1 in the main text:
Proposition D.6. Let (*) := P(-le") denote the distribution in test environment E = E. The causal mechanisms are formalized through interventional distributions. We focus on the predictions P(y]do(-)), where do(-) represents an intervention. The causal mechanism satisfies P* (yldo(x)) + P(y)do(x)). When using only Zine, the causal mechanism ((y))o(Zinoo))
remains unchanged: P (y])d((zmmo)
Proof. The causal mechanism (o())) can be decomposed as:
P(yldo(x)) = ΣΣ P(yldo(a:) Zvar; Zine)
P
Ido(z))
Similarly, the causal mechanism PM (y]lo((=)) can be decomposed
Pa (*]y]do(z)) P(y)do(x))
(12)
Σ P(y]do(<z), wear, Zinu, E²)
ZARO Zear
P(Zvar|do(2)) e²) P(zinn (d((z)) E*)
From the Rule of Definition D.2, we have
P(y]do(a)), zwar, Zinn; P(y]do(()) Zvar: Zinu);
(13)
because Zear satisfies (Y IL E var, X) in Dx, and according to Definition D.4, the Zvar E-admissible with respect to the causal effect X on Y. Then, since (Y X Zwar, Zine). the following holds:
Plyldo(x) zvar: kine) = P(yl Zears Zinv)
(14)
According to Definition D.1, Zine - Y Zear + E is collision node:
Plz ())o e²) P(z (do(z)).
(15)
And since there is other path from X - Zino and X Zvar, the do- operator is trivial. Therefore, the Equation (12) can be rewrite as:
P*(y)do(xx) = ΣΣ Σ
Zine) P(zinol (z (x)
(16)
Zino
Since (* (x: # P(Zearla)) Pa (y) (o())) # P(y)))(x)))
If the prediction is solely based on Zinei Y is independent of Zvar. The prediction process is now P (yyldo(zzin P(yldo(Zine)) e*). Since the do(-) cut-off all edges entering Zino, Zino is independent of E. Therefore, P (3) (=ddd(((n)))))))))))))))(()))( P(yldo(Zinn)) This ends the proof.
E. Proof for Proposition 5.3
Building upon prior works (Roeder et al., 2021; Ahuja al., 2022; Hyvarinen & Morioka, 2016; Zimmermann et al., 2021). we offer proof demonstrating that CLIP identifies latent factors up to an invertible linear transformation, and we also reuse some of their proof techniques.
Proposition E.1. For an image encoder fi : X 2 and its corresponding text encoder T 2, where 2 C R the embedding space. Condition 5.2 is satisfied, then fi(x) identifies Z up to an invertible linear transformation A, i.e. fı(x) Ag (xx) Az.
Proof. Consider a training dataset D ((r, 4) sampled from the joint distribution p(æ, l). Let T denote the set of all possible values of t. Let & denote the parameters of f, and fr, and let 6* denote the parameters of fr- and fr- (to which we have no access). The ground-truth conditional probability can be regarded as produced by fi- and fr-:
exp(Sr- (x) (1)
po- (t x1, xx = Evercxp(fr- (x) f (t'))
(17)
19

---

Learning Invariant Causal Mechanism from Vision-Language Models
Similarly, the CLIP model functions /1 and Jr produce the distribution:
exp(Jr(x)" (r(I))
po(t T) Everexp(fi(n) fr(())
(18)
The training objective for CLIP is to minimize the KI. divergence KL we have pe z, T) Pa- (t x, T), that is:
KI,(Do(() ppll 2, T) pe (1 a, T)). Ideally, after training,
exp(fi(x) fT(t)) ΣVET exp(fi(x) fT(()) Everexpi (x) fr- (t))
exp(fi- (x) fr. (t))
(19)
The above equality illustrates the consistency aspect of Condition 5.2. Building on this, for any pair and the following ratio should hold:
(t x, T) (t6 2,, T) Pa- (t, 2, T)
Pg- (ta x, T)
(20)
This implies that under Condition 5.2 there exist D + 1 distinct text description pairs (ta, tú) satisfying:
fr(tt) fr (ta)) (xp(f(())( exp(fr- (2)(f)-(x)'1 fr.
(21)
Taking the logarithm both sides, this simplifies to:
(fr(ts) - fr(t))
(fr- (r-(t))) fi-(x).
(22)
On the left-hand side of Equation (22), (Sr(ta) (r((())))))))))) can treated as basis vector. Since there exist D + pairs of (ta, to), least D linearly independent vectors (fr(ta) fr())) can form an invertible matrix L E DDDD The same holds for the right hand side, where (fz-(ta) fi (t:)) forms another matrix L'. Substituting these into the equation gives the following system of D linear equations:
( fi.
(23)
Defining A= L'L=', which is invertible, arrive at:
f+(2)) = ASr- (x) Az
(24)
This completes the proof.
F. Proof for Proposition 5.5
Proposition F.1. For the encoder Ji satisfies Proposition 5.3, if Condition 5.4 is satisfied and sample from pdo(ssno) (a)) is The proof technique of Proposition 5.5 follows the approach of Ahuja et al. (2023), which we adapt to our setting.
possible. Then for any pair (æ) do( dol do(sna))) from pdo(zsno) (xt). The mapping Aino satisfies:
22
Aino (fr (ar (ar dolzsmu)) fr do( =0
(25)
Proof. Condition 5.4 states that sampling from pdo(zian) (x) is available. The invariant factors of the images from pdo(sins) (x) are fixed at specific value, i.c., Zino zt. For sample ø1 do(zmar) = [2] ævar. "", Similarly, for another sample x
do(zmno) its ground-truth latent representation,
do((snn) = =g" (x dolssne)), can be written as 2 #1
do(s.n>) we have
do(kino) [2 2
22
From Proposition 5.3, the following relationships hold:
fr (x (oloonn))))
Az do(zen))
A fz do(suou))
do Bo(2enz)
= =
(26)
Annn Avar
fr do(s)ne))
zt
Z Zear:
20

---

Learning Invariant Causal Mechanism from Vision-Language Models
The same holds for 22 do(sinv), By focusing only on Ainos we have:
Aino doleino))
(2) =21,
(27)
and
Ainv (102 (222
dolsino))
=z,,
(28)
Subtracting these two equations yields:
Ainv (favlfle (fr(cz
fi(2z (oltmno))) (xoolamn))
(29)
This ends the proof.
G. Proof for Theorem 5.6
Before proceed the proof of Theorem 5.6, give the following definition. Let etr E Eall denote the train- ing environment. Define the distribution on some test environments (e((())))() be the expected disagreement between two hypotheses h, h E H. where l is some loss function (e.g. cross-yn loss). This represents a measure of how much two hypotheses disagree with each other on the training distribution. We use HAH divverrgeee (Ben-David et al., 2010; Chuang al., 2020) measure whether there is any pair of hypotheses whose risk differs significantly between P., and Pte.
E Eatt, Ctr * Cte as Pte. Let R (h,
Definition G.1. (Hvv--divergeece (Ben-David al., 2010)) Given two distribution Per and Pte, and hypothesis class H. the HiA-diieegencee between Ptr and Pte is:
Pie) == sup h.EER
[R* Rete (hte(h,
(30)
Next, we are going to show how the risk in etr can be related to test environment ete Eat with the H HAH--ivergence. Lemma G.2. (Ben-Davier al., 2010 (Ben-David al., 2010)) For all hypothesis h E H. the risk Pte is bounded as:
t (h) < - (h) + ddaH Pie) Ans
(31)
where Au is the best joint risk:
dy = inf (R=*(M))+ R (1'))())))) (11))/2.
(32)
NEH
Proof. By the definition of duau (Ptr, Pte).
dHAH (Pir, Pte)
sup A.
1R (h, 10 (h,l²))
(33)
Also, with the triangle inequality for classification error:
Re (h) < Re (h') + R (h, h!)
I Re (h') + R (h, h!' + R (h, h/) R (h, h()l
Rew (h') + Re (h, h') + dHAH (Prr, Pte)
(34)
Ne (h') + Ret (h) R (h') + dHAH (Por; Pre)
< u (h) + dHHH Pie) A2
This ends the proof.
Next, we formalize some basic assumptions:
Assumption G.3. 1. let (hinv) is constant for E Eait-
2. The Bayes-optimal predictor ht E H and hime E H exist.
3. The loss e y x y R T--Lipschitz and bounded by M
21

---

Learning Invariant Causal Mechanism from Vision-Language Models
Now can proof Theorem 5.6 from the main text.
Theorem G.A. Let H. be hypothesis class over X y, and let Eall denote set of environments with distributions (ee EeE Ean over X y. Let hine E H be predictor relying solety on Zino, and h E H predictor utilizing both Zino and Zvar. If the mutual information between Zimu and Z satisfies:
I(Zin: 2) > for some constant e >> 0.
(35)
Then, the worst-case OOD risk strictly satisfies:
ROOD (himo)
ROOD (h),
(36)
Proof. Step I: Controlling In-Environment Risk via (Zinvi Z)
The mutual information condition / (Zznn; 2) guaranttes that Zino captures most predictive information in Z. By the data processing inequality,
I(Y; Zinu) 2 2) - e(c)
(37)
where e(c) + 0 as c - oc. This implies the conditional entropy gap between hi and is bounded:
H(Y Zino) H(Y %) = T(Y; Zvar 7snr) S c(c).
(1)
For the T--Lipschitz loss, the risk gap becomes:
Ret (hine) S Retr (h²) + T. c(c).
(38)
By realizability, hive approximates hine SO:
Ret (hino) S Reer (hāno) + S (h") + L. E(c) + n.
(39)
or small 1) > 0.
Step 2: Bounding OOD Risk via HAH-Divergencee
By Definition G.1, the HAA--divergence between Pe and Pe- is:
Po. = sup R (1) 1 (h, h')l
(40)
h.MEH
For hine, invariance ensures minimal divergence:
dHAM Par; hini) S danes
(41)
while for h, the divergence scales with -haftshift:
shift:
(e:sPP)) deor >> Sinv.
(42)
Step 3: Worst-Case Risk Comparison
By definition, ROOD (h) = max{ R (h), R (h)). From (2) and (3):
R (h) < R (h) Swar + Ax S (RoDD((sm))) L e(c) - -n) + 8par + Nz.
(43)
Since the risk of hine is stable (ROOD (hinv) Re (hinu)), have:
(OOD 2 Re (h)( 2 (pOD((smu) L [.E(cc) - dear + Au-
(44)
Under I(Zinvi Z) > c, e(c) becomes negligible. With Svar > 0 and dy >> 0, it follows that: ROOD (hmm)) < ROOD(
(45)
This ends the proof.
22

---

Learning Invariant Causal Mechanism from Vision-Language Models
H. Implementation of CLIP-ICM
H.1. Interventional Data Collection
In Section 6, we propose two approaches for collecting interventional data. In the following sections, we provide a detailed explanation of each method.
Image-based Interventional Data Collection To generate interventional data, we employ the following data augmentation techniques: ColorJitter, GrayScale, GaussianBlur, RandomInvert, RandomRotation, RandomPosteriza RandomSolarize. and RandomEqualize. After applying these methods, the changes in the latent factors of the corresponding images summarized in Table 4. In practice, for each image, randomly select one augmentation technique from the set and generate an augmented sample, which is then paired with the original sample.
Table 4. Impact of data augmentation specific latent factors (Zinvi v, Zeari
Augmentation Technique
Shape
Structure
Color
Texture
ColorJitter
GrayScale
GaussianBlur
V
RandomInvert
V
X
V
RandomPosterize
V
X
X
RandomSolarize
V
X
X
RandomEqualize
V
X
RandomRotation
Text-based Interventional Data Collection To collect text-based interventional data, two components required: image description model and text intervention model. We utilize GPT-4o (OpenAI, 2023) for both purposes. The text description model takes an image as input, processes it using a carefully crafted prompt, and outputs a descriptive caption of the image. The text intervention model then takes this caption as input, applies a specified intervention using a modified prompt, and outputs an altered version of the original description.
Prompt for image description Describe this image in detail.
Prompt for text intervention:
You are tasked with creating interventional descriptions of images for a classification task. Given the following JSON input:
...
"image_ description": "A detailed textual description of the image, including all features (relevant and irrelevant the classification task).", ["Class1", "Class2", "Class3"]
} ""
Your goal is to:
Identify features related to the classification task that are invariant to environmental shifts. These features should remain unchanged in the output description.
2. Intervene on all other features (e.g., texture, background, lighting) by replacing their description with new, plausible alternatives.
3. Maintain the original structure of the description while altering only the specified features.
23

---

Learning Invariant Causal Mechanism from Vision-Language Models
Output a single JSON object containing only the "intervened_ deserrpptio "" which reflects these modifications.
We provide examples of both descriptions for samples from the PACS dataset in Figure
PACS
This image depict white dog with black or dark grey spots, The dog shown in profile, facing to the right, with focus head and neck. The background is painted in neutral, earthy tone, possibly brown, which contrasts gently with the light coloration of dog's texture
painting suggests traditional medium such oil acrylic on canvas, that add depth and dimension to the fur and background. The lighting accentuates the contours of the dog's head, highlighting its structure and features, such as the ears, muzzle, and eyes, in a subtle and naturalistic manner.
visible brushstrokes
Photo
This image depicts white with black dark gray spots. The dogis shown profile, facingto the right, with focus heed and neck. The background ls painted in a vibrant, sky-blue tone, which contraste gently with the light colorstion of the dog's painting suggests a modern medium such as digital art, smooth transitions dimension the fur and background. The lighting accentuates the contours the dog's head, highlighting its structure and features, such as the ears, muzzle, and eyes, in bright and slightly stylized manner.
depth
This image is painting of dog. focusing closely on the dog's face and part of its neck. The artwork employs a vibrant and stylized color palette, with prominent shedes blus, teal, and purple blending together to create an elmost surreal effect. The dog's left (on right
and contrasting with the cooler the face. The background ls minimel and light-colored, which helps emphaeize the vivid colors of the dog. The use of bold and expressive brushstrokes contributes to the artistic and abstract quality the piece, highlighting the painter's creative interpratation subjact.
image)
tones
the of
Art
focusing closely dog's face and its neck. The artwork employs rich and imaginative color palette, with prominent shades of green, gold, and pink blending together to create an almost ethereal effect. The dog's left ear (on right side of the image) painted cool tones of gray and silver, contresting with the we mer golden hues rest of the face. The background dark and textured, which helps emphasize the vibrent colors the The of smooth and dynemic bruahstrokes contributes to the artistic and modern quality of the piece, showcasing the painter's innovative interpretation subject.
This image painting
Figure The example images with image description and the interventional description from the PACS dataset.
H.2. Details in Estimating Aino
The process begins with collecting and _dd((zno)) For any given image sample æ, we first generate an augmented version a(x) using data augmentation. We then obtain the corresponding text description t using a text description model and create an intervened description (t)) using text intervention model. This allows us to construct four embedding pairs: (T( (+(t))) (fr(t), (t) - -y (x)))) (f)()), fr(z) fr(()))),, and (fr(#), (xx) fr (B())). These pairs stacked across all samples to form Z and Ž 2dd()ano))
Once the data is collected, the next step is to fit Aine- One approach is to use principal component analysis (PCA) to compute Aire that satisfies Equation (9). Another approach is to treat Aine as learnable matrix and optimize it directly using gradient descent.
H.3. Pseudo Code
The pseodo code of CLIP-ICM is illustrated in Algorithm 1.
H.4. Computation Cost and Data Collection Cost
All experiments are conducted on single NVIDIA-RTX A6000 GPU, and the average training duration for each result is less than one hour.
Computation Complexity Analysis We provide detailed analysis of the computational complexity of CLIP-ICM.
24

---

Learning Invariant Causal Mechanism from Vision-Language Models
Algorithm 1 CLIP-ICM
Require: CLIP image encoder fI, text encoder fr images from training domain E D, (Drr types data
augmentation A = {ax I Hyperparameter À and Dine. Text description of all images æ E D. The text intervention model B.
1: Initialize the container Z = for original representation
2: Initialize the container 2' = for intervened representation
3: for æ in D do
4: Sample random augmentation as from K
5: Get the text description of æz
6: Get the original image representation = (x)
7: Get the original text representation & fr(t)
8:
Get x da(#iww) = (ar)
9: Get gdo()one) ((())
Get intervened representation 3' fr((n(z))
11: Get the intervened representation & (r((()))
Append into 7 // image pair
13: Append 3' into 2'
14:
Append & into 2 // text pair
15:
Append & into ž'
16:
Append into 7 // image-text pair
17: Append E into Z'
18: Append & into // text-image pair
Append z' into 2''
end for
21: Get the correlation matrix Corr = 2 X. (2 2²) 2). Exd ERR Corr E Rddd
22: Perform Eigenvalue Decomposition for Corr to get V.V.E
23: Get the highest Dine singular vector U.D. E R DiauDD
Output: The target transform Acnu U U.D.ne
According to Algorithm 1, the training process of CLIP-ICM can be divided into four key steps: data preprocessing, obtaining the representation of the training data, calculating Corr, and obtaining the projection matrix through SVD. Let N denote the size of the training dataset and D the dimension the representation:
Data Preprocessing: Since data augmentation applied to individual samples, the computational complexity for this step O(N).
2. Obtaining Representations: The pre-trained network is applied individual samples. Like the previous step, this operation also has complexity of O(N).
3. Calculating the Corr: Let Z E RD be the matrix containing all training data representations. The complexity of computing Z is O). Since the dimension of representation and is independent of the dataset size, can be considered constant, the complexity also O(N).
The computational complexity of SVD for the correlation matrix is O(D*). However, since the matrix Corr E PxD is independent of the dataset size, this step's 's complexity can be considered O(1).
SVD of the Correlation Matrix:
In summary, CLIP-ICM has computational complexity of O(N), growing linearly with dataset size. For larger datasets, Ainn is treated as a learnable matrix and optimized via gradient descent, requiring only linear matrix in Dx In both approaches, CLIP-ICM remains highly efficient.
Data Collection Cost We use GPT-4o as the generator for text-based interventional data, which involves three steps: processing image inputs, generating textual descriptions, and producing intervention-modifee descriptions. For smaller datasets such as PACS, VLCS, Terra Incognita, and Office-Home, we generate interventional text data for all image samples. For large datasets such as DomainNet and ImageNet, we sample 20% of the dataset for text-based interventions. Data is collected using the GPT-4o batch API at a cost of $1.25 per million input tokens and $5 per million output tokens. The total cost for data collection is $162.79. 79.
25

---

Learning Invariant Causal Mechanism from Vision-Language Models
I. OOD Performance in Terra Incognita
evaluate CLIP-ICM using the same experimental setup as described in Section 4, with results presented in Table 5. The results show that CLIP-ICM Ainer-Proobe improves performance by an average of 6.3% compared to direct Linear-Probe, Furthermore, CLIP-ICM significantly outperforms naive fine-Tune new classes under both open-class and domain shift conditions.
Table 5. The performance of CLIP on the Terra Incognita dataset in accuracy (%). L100, L38, L43, and L46 represent different environments. In the Linear probe case, the values (-) represent the accuracy achieved by directly fine-tuning with the linear probe on the target domain.
DOMAIN SHIFT
METHOD
L100
L38
L43
L46
AVG
LINEAR PROEe
73.6 (90.9)
58.3 (86.3)
61.0 (76.0)
47.8 (78.9)
60.2 (83.0)
CLIP-ICM LINEAR-PRORH
79.8 6.2)
61.5 3.2)
66.9 5.9)
57.9 10.1)
66.5 (1 6.3)
OPEN CLASS UNDER DOMAIN SHIFT
SPLIT
METHOD
L100
L38
L43
L46
AvG
ZERO-SHOP
56.4
23.2
30.4
25.8
33.9
BASE
FINE-TUNE
75.6
58.5
65.2
55.1
63.6
CLIP-ICM
63.7
52.6
58.8
51.2
56.6
ZERO-SHOT
47.6
17.6
35.2
37.4
34.5
NEW
FINE-TUNE
36.7
11.2
25.1
25.5
24.6
CLIP-ICM
59.4
43.2
51.6
46.8
50.2
ZERO-SHOV
52.0
20.4
32.8
31.6
34.2
TOTAL
FINE-TUNE
56.2
34.9
45.2
40.3
44.2
CLIP-ICM
61.6
47.9
55.2
49.0
53.4
J. More Results
We also utilize ImageNet as the training data and validated the OOD generalization ability of CLIP-ICM on four datasets: ImageNet V2 (Recht al., 2019), ImageNet-A (Hendrycks al., 2021b), and ImageNet-Sketch (Wang et al., 2019), ImageNet R (Hendrycks et al., 2021a). Specifically:
ImageNet V2 (Recht al., 2019) contains new test data for the ImageNet benchmark, providing a fresh evaluation of model performance.
ImageNet-A (Hendrycks et al., 2021b) consists of real-world, unmodified, and naturally occurring examples that are often misclassified by ResNet models, presenting challenging cases for OOD generalization.
ImageNet-Skee Skeeeth (Wang et al., 2019) includes sketch representations of ImageNet classes, offering a test of the model's ability to generalize different visual styles.
ImageNet-R (Hendrycks et al., 2021a) contains art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes.
We additionally conducted experiments on the WWldCam--WIL 2020 dataset (Koh al., 2021). iWildCAM comprises 203,029 images from 182 different animal species, which were collected from 323 camera traps distributed across various locations. The images obtained from different locations exhibit variations in lighting, color, camera angle, background, vegetation, and relative animal frequencies. We follow the setting in (Koh et al., 2021), use images from 243 locations as the training domain, and those from 48 other locations as the test domain. We report the average macro Fl score of CLIP, CLIP-ICM', CLIP-ICMÍ, and CLIP-ICM under both ID and OOD conditions, as shown in the Table 7: From the results in Table 7, we can observe that regardless of type of interventional data, CLIP-ICM consistently outperforms CLIP both ID and OOD settings.
26

---

CLIP-ICM Learning Invariant Causal Mechanism from Vision-Language Models
Table Accuracy on ImageNet with various domain shifts.
IN-DISTRIUUTION
DUT-OR-DISTRIUUTIOSS
METHOD
IMAGENET
IMAGENET-V2
IMAGENET-S
IMAGENET-A
IMAGENET-R
ZERO-SHOT
66.7
60.8
46.1
47.8
74.0
57.2
FINE-TUNE
68.220.1
61.910
46.820.1
46.4
75.1
57.610.1
COOP (ZHOU ET AL., 2022B)
71.5
64.2
48.0
49.7
75.2
CoCoOr (ZHOU
2022A)
71.0
64.2
48.8
50.6
76.2
59.9
CLIPOOD (SHU ET AL., 2023)
71.600.1
64.920.1
49.310.1
50.4
77.220.1
60.4
CLIP-ICM'
70.5
64.710
49. 49.910.4
50.320.
76.220.
60.210.
71.2+0.4
63.7
48. 48.8
46.8
74.8
58.5
CLIP-ICM
71.600.1
65.8+0.3
+0.9
51.4+0.8
77.6.0.2
61.4+0.3
Table 7. Accuracy on the iWildCam WILDS 2020 dataset.
Method
ID (48 Locations) OOD (243 Locations)
CLIP
14.2
10.6
CLIP-ICM"
15.6
13.3
CLIP-ICM
15.2
12.2
CLIP-ICM
15.8
14.1
CLIP Linear-Probe
54.6
41.4
CLIP-ICM' Linear-Probe
56.2
42.2
CLIP-ICM Linear-Probe
55.6
44.3
CLIP-ICM Linear-Probe
57.1
46.1
K. Dataset Details from DomainBed Benchmark
DOMAINBED includes downloaders and loaders for seven multi-domale image classification tasks:
PACS (Li al., 2017) comprises four domains e E {art, cartoons, photos, sketches). This dataset contains 9,991 examples of dimension (3, 224, 224) and 7 classes.
VLCS (Fang al., 2013) comprises photographic domains e [Caltech101, LabelMe, SUN09, VOC2007]. This dataset contains 10,729 examples of dimension (3, 224, 224) and 5 classes.
Office-Home (Venkateswara et al., 2017) includes domains E (art, clipart, product, real). This dataset contains 15,588 examples of dimension (3, 224, 224) and 65 classes.
Terra Incognita (Beery et al., 2018) contains photographs of wild animals taken by camera traps at locations (L1000, L38, L43, L46). Our version of this dataset contains 24,788 examples of dimension (3, 224, 224) and 10 classes.
DomainNet (Peng et al., 2019) has six domains e E {clipart, infograph, painting, quickdraw, real, sketch). This dataset contains 586,575 examples of size (3, 224, 224) and 345 classes.
For datasets, we first pool the raw training. validation, and testing images together. For each random seed, we then instantiate random training, validation, and testing splits.
L. More comparison with related works
L.1. Invariant Risk Minimization (Arjovsky et al., 2020)
Invariant Risk Minimization (IRM) and IRM-based methods (Ahuja al., 2021; Yang et al., 2023) also aim to minimize the OOD risks with an invariant predictor. However, note that these methods obtain invariant predictors without identifying the invariant factors. Moreover, as suggested in Figure 2 of Arjovsky et al. 2020 (Arjovsky et al., 2020), there are multiple solutions to the IRM objective. Therefore finding an IRM solution doesn't mean the identification Zinv- Compared to these methods, our method ensures the identification of Zine-
27

---

Learning Invariant Causal Mechanism from Vision-Language Models
L.2. Model-based Domain Generalization (Robey et al., 2021)
Model-based Domain Generalization (Robey al., 2021) also also proposes an SCM, their analysis method is different from ours. They believe that in domain shift, instance X is generated by an underlying random variable X and jointly passing through domain-transfer model ((X,e), e). In our causal diagram analysis, we believe that the sample is generated by Zvar and Zine jointly.
MBDG obtains the invariant features by utilizing pre-trained domain-transfer model to constrain the distance between the features extracted by the feature extractor and the representations generated by the domain-transfer model.
In contrast, our approach learn a linear transformation of the pre-trained representation obtain features that are invariant to data augmentation. We provide the comparison results between our method and MBDG in Table 15 and Table 12.
L.3. Subspace Identification Guarantee (Li et al., 2024)
From the perspective of SCM analysis, SIG views the data generation process through 4 kinds of latent factors. In our SCM, Zine can be regarded as a domain-invariant variable, and Znar can be regarded domain-variant variable. However, in causal diagram, do not distinguish between label-irrelevant and label-relevant variables in modeling.
From a methodological perspective, SIG uses an end-to-end, reconstruction-basee network, while we use pre-trained CLIP backbone network. SIG uses a variation-inference method to identify latent factors, while we use an intervention method to identify latent factors. We present the performance of our method alongside the SIG to the Office-Home dataset in Table 13.
L.4. Domain Prompt Learning (Zhang et al., 2023b)
(Zhang et al., 2023b) introduces Domain Prompt Learning (DPL), a novel approach that improves the domain generalization performance of the CLIP model on various datasets by generating conditional prompts, achieving higher accuracy without fine-tuning the entire model. DPI. can be regarded as Test-Time Adaptation (TTA) approach, while our method can be regarded as domain generalization approach. Nevertheless, we provide the comparison results in Table 2.
L.5. Sele-Supervised Learning with Data Augmentations Provably Isolates Content from Style (Von Kügglgenn et al., 2021)
(Von Kügelgen al., 2021) also interpret data augmentations as counterfactuals, and use them obtain the invariant factors. Our work differs from theirs in three key aspects: the background setting, the research problem, and the implementation method.
Firstly, the problem studied by (Von Kügelgen et al., 2021) is set within the framework of self-supervised contrastive learning. In their paper, the term "invariant factor" refers to the invariant parts derived from two different augmented perspectives within a positive sample pair. In contrast, the "invariant factor" our paper refers to the latent factors that remain unchanged across different domains. Due to the diversity of augmentation methods employed self-supervision (Chen et al., 2020), the invariant factors they investigate do not align with the invariant factors we examine.
Second, our theoretical approach diverges fundamentally from (Von Kügelgen et al., 2021) in both methodology and objectives. While their work establishes that non-linear encoders can achieve block identification of content factors through embedding space alignment of view- augmentedd samples, our analysis proceeds through two critical theoretical advancements: (1) We first demonstrate that CLIP learns representations that identify all latent factors up to an invertible linear transformation; (2) We subsequently prove that interventional data enables estimation a linear mapping from the embedding space to invariant subspaces.
Finally, the research objectives of the two works are fundamentally distinct. While (Von Kügelgen et al., 2021) focuses on analyzing SimCLR's factor identification capabilities, our work specifically addresses CLIP's out-of-distribution generalization limitations by developing linear mappings to invariant subspaces. This difference in both theoretical focus and practical application underscores the novel contributions of our approach.
28

---

Learning Invariant Causal Mechanism from Vision-Language Models
Jiter
72.93
73.04
73.37
73.37
72.92
74.09
73.88
100
Caltech101
Drey
LabelMe
SUN09
GBu
73.55
71.82
73.02
72.74
72.15
72.15
73.02
73.5
linear-probe Accuracy(%)
VOC2007
Augmentatonn
73.0
85
Invert
73.63
72.05
72.81
73.15
72.33
72.51
72.97
80-
72:15
72.5
74.40
71.54
72.41
72.76
72.18
72.25
72.74
72.0
Equalize
72.41
73.02
72:45
72.43
71.5
65
Jiter
Gray
Gaussian
Invert Rosterize Solarie Equalize
Augmentation
(a)
(b)
100
interventional data per sample
0
70.1
73.6
73.7
73.9
75
linear-probe Accuracy(S)
74.7
75.6
75.6
75.6
Image
70
2-
74.7
75.6
75.7
75.7
72
Caltech101
LabelMe
SUN09
74.8
75.6
75.7
75.8
VOC2007
300
Text
(C)
(d)
Figure 6. The experimental results for ahlation studies. (a) The zero-shot accuracy of CLIP-ICM the LabelMe domain of the VI.CS dataset with different combinations of augmentations, and brighter colors indicate a higher accuracy. (h) The lincar-probe accuracies CLIP-ICM on the VLCS dataset with different X. (c) The accuracy CLIP-ICM on VLCS with different choice of Dinv. (d) The accuracy of CLIP-ICM on the LabelMe domain of the VLCS dataset with different numbers of interventional data pairs.
M. Ablation Study
M.I. The impact of different data augmentation
We investigate how CLIP-ICM performs with varying data augmentation strategies the VLCS dataset. In this experiment, only image-based interventional data utilized. The experiments include different combinations of augmentation techniques, and the prediction accuracy the LabelMe domain is illustrated in Figure 6(a). The diagonal of Figure 6(a) indicates scenarios where only one augmentation is applied, while other cells represent combinations of data augmentation techniques. noteworthy that the use of data augmentation significantly improves performance, with certain combinations showing even more promising results.
M.2. The impact of hyperparameter X.
We conduct experiments on the VLCS dataset and visualize the results in Figure 6(b). In this experiment, only image-based interventional data are utilized. From the results, we can observe that has a significant impact on the prediction results. This is because that higher value of indicates the second term of Equation (9) is more important and has a higher impact.
29

---

Learning Invariant Causal Mechanism from Vision-Language Models
These results illustrate the effectiveness of our proposed method.
M.3. Ablation study on dimension of Zine
As we mentioned at Section 6, the dimension Dino of Zine is a hyperparameter. To investigate the influence of Dines we conduct an ablation study regarding the choice of Dinv. All results are the performance CLIP-ICM with only image-based interventional data utilized, conducted on the VLCS dataset with ViT-B/16 as the backbone, where Diav < 512. The results are shown in Figure 6(c).
From Figure 6(c) we can see that as the Dinu increases, the accuracy first increases, stabilizes, slowly decreases, and finally drops. Therefore, the optimum dimension of Dino should be around 300 to 350 for all domains in VLCSS.
M.4. Number of Interventional data pairs.
By default, each sample from the training environments is paired with an image-based interventional sample via data augmentation and a text-based interventional sample generated by an LLM. In this section, we examine whether increasing interventional data enhances CLIP-ICM's performance, as shown in Figure 6(d). The results indicate that while adding more data pairs slightly improves performance, the gain is marginal.
M.5. The role of Aie
To validate the role of Aine, we conduct an ablation study to illustrate its contribution. Specifically, we use our generated image-based interventional data train a linear probe on the DomainBed benchmark. The experimental results are presented in Table 8. From the results, we observe that incorporating image-based interventional data into the linear probe leads to
Table 8. Ablation results on the DomainBed benchmark under domain shift. We compare the standard linear probe, linear probe trained with image-based interventional data, and CLIP-ICM' linear probe trained with the proposed Abs module. Methods with indicate training with only image- masee interventional data.
Method
PACS
VLCS
OfficeHome
TerraInc
DomainNet
AVG.
Linear-Probe
96.4
78.7
81.9
60.2
55.0
74.4
Linear-Probe + Interventional data
96.8
79.3
82.3
60.5
55.8
74.9
CLIP-ICM" Linear-Probe
97.5
86.5
84.6
64.3
64.0
79.0
only marginal improvements. In contrast, although both settings utilize the same interventional data, the performance of CLIP + Linear-Probe is significantly higher. These findings demonstrate that our proposed Aie module substantially enhances the OOD generalization ability of CLIP.
N. An analysis of the zero-shot prediction with domain prompt
In experiments from Section 7, we use the standard text prompt template when performing zero-shot prediction, i.e. "A photo of [cls]" where [cls] is the name of the corresponding class.
To explore the influence of text prompts on zero-shot prediction, we conducted experiments using the PACS, VLCS, and -ifice-Home datasets. In these experiments, we employed text prompts that incorporate domain information. For datasets where the domain names hold particular significance, such as PACS and -ffice-Home, we utilized the prompt 'A [domain] of [cls]'. In contrast, for the VLCS dataset, where the domain names do not have specific meanings, we adopted the prompt "A photo of [cls] [domain]'. Here, [domain] represents the name of the test domain. The results are illustrated Table 9, Table 10 and Table 11.
It is important that the objective of domain generalization is to evaluate the performance of model unseen domains. Therefore, incorporating domain information, such as (a sketch of [cls]) would not align with the requirements of the domain generalization task. We only conduct this experiment for exploration.
From the results in Table 9, we find that only in some domains, incorporating domain information in the prompt template can improve the zero-shot performance, such as Sketch in PACS, and Clipart in Office-Hoiee Home. However, in most domains, where the domain name doesn't 't provide any information, or can't 't describe the domain properly, the performance lower
30

---

Learning Invariant Causal Mechanism from Vision-Language Models
than the original template.
Another important observation is that CLIP-ICM consistently improves the performance of CLIP in most domains, without any domainn-rlater information or other kinds of template design.
Table 9. Comparison results of zero-shot performance of CLIP with domain prompt in PACS dataset. f denotes that the results of CLIP with domain prompt are higher than standard prompt, while denotes the results CLIP with domain prompt are lower than standard prompt.
Algorithm
Photo
Art
Cartoon
Sketch
Avg
CLIP
100.0
97.4
99.2
88.1
96.1
CLIP Domain Prompt
100.0
97.4
99.0
90.2 f
96.6
CLIP-ICM
99.1
99.5
99.7
92.2
97.7
Table 10. Comparison results of zero-shot performance of CLIP with domain prompt in VLCS dataset. denotes that the results of CLIPP with domain prompt are higher than standard prompt, while J. denotes that the results of CLIP with domain prompt are lower than standard prompt.
Algorithm
Caltech101
LabelMe
SUN09
VOC2007
Avg
CLIP
99.9
70.1
73.5
86.1
82.4
CLIP Domain Prompt
99.2 4
69.7
65.1 4
80.3 4
78.5
CLIP-ICM
100.0
75.6
79.2
90.0
86.2
Table 11. Comparison results of zero-shot performance of CLIP with domain prompt in Office- Home dataset. 1 denotes that the results of CLIP with domain prompt higher than standard prompt, while denotes that the results of CLIP with domain prompt are lower than standard prompt.
Algorithm
Art
Clipart
Product
Real
Avg
CLIP
83.3
65.3
89.0
89.3
81.7
CLIP Domain Prompt
82.4 +
67.01 f
87.8 +
88.7 +
81.4 1
CLIP-ICM
84.6
70.7
91.7
91.5
84.6
O. Full Results on Domainbed
In this section, we provide the full results on datasets from the Domainbed benchmark.
31

---

Learning Invariant Causal Mechanism from Vision-Language Models
Table 12. Accuracy on PACS dataset. A.C.P and represents different domains.
Algorithm
P
A
C
S
Avg
Backbone ResNet-50
ERM
97.2++03
84.7++0.4
80.8+0.6
79.3+
85.5
IRM (Arjovsky et al., 2020)
96.710.6
8.,8#113
76.411.1
76.1
83.5
Go (Sagawa et al.)
96.7+0..3
83.5:1.9
79.140.6
78.3+2.0
84.4
Mixup (Yan et al., 2020)
97.620.1
86.1=0.55
78.940.8
75.8
84.6
MMD (Li et al., 2018)
96.6+0022
86.1=11.4
79.4+0.9
76.5++0.5
84.6
DANN (Ganin al., 2016)
97.3+0.4
86.4
77.4
73.5:2.3
83.6
ARM (Zhang et al., 2021)
97.4+0.3
86.8=0.6
76.8=0.55
79.3
85.1
MBDG (Robey et al., 2021)
97.0
80.6
79.3
85.2
85.6
CLIP Linear-probe
99.2+0.7
91.7=#0.8
92.5
82.3+0.2
91.4
CLIP Zero-shot
99.3+0.0
91.0+10.0
93.1+0.0
80.5
91.0
CLIP-ICM'
99.420.4
93.6-1.2
99.3+0.55
83.1#0.6
93.8
CLIP-ICM' Linear-probe
99.410.5
92.8=0.7
93.9+0.4
83.8+0.5
92.5
CLIP-ICM
99.020.2
93.320.4
98.9
82.8+0.4
93.8
CLIP-ICM* Linear-probe
99.1+0.55
92.5+0.4
93.4+0.3
83.5+0.4
92.5
CLIP-ICM
99.810.44
94.2:1.22
99.710.44
83.7
93.8
CLIP-ICM Linear-probe
100.0
93.4
94.5+10.4
84.2.10.2
92.5
Backbone
ViT-B/16
CLIP Zero-shot
100.0
97.4++0.0
99.2+0.0
88.1+0.0
96.1
CLIP Linear-probe
97.3+0.77
98.4=1011
99.5
90.410.3
96.4
CLIP-ICM'
98.7
99.1=0.44
99.9
91.7.1..4
97.3
CLIP-ICM' Linear-probe
98.810.2
99.410.3
99.8+0.88
92.1+0.55
97.5
CLIP-ICM
98.3+0.4
98.6=10.4
99.5#0.3
90.7+0.3
96.8
CLIP-ICM Linear-probe
98.410.11
98.9.1..3
99.3.0.11
92.140.1
97.2
CLIP-ICM
99.1+0..
99.5+0.2
99.7+0033
92.2+0.44
97.7
CLIP-ICM Linear-probe
99.110.4
99.910.1
99.911.11
92.411.11
97.8
32

---

Learning Invariant Causal Mechanism from Vision-Language Models
Table 13. Accuracy on Office-Home dataset. A.C.P and represents different domains.
Algorithm
A
C
P
R
Avg
Backbone ResNet-50
ERM
61.3+0.7
52,4+0.3
75.8+0.1
76.6++0.3
66.5
IRM (Arjovsky et al., 2020)
58.942.3
52.211.6
72.1=2.9
74.0=2.5
64.3
GroupDRO (Sagawa et al.)
60.4+0.7
52.7#1.0
75.0=0.7
76.0+0.7
66.0
Mixup (Yan ct al., 2020)
62.420.8
54.820.6
76.9
78.3+0.2
68.1
MMD (L. et al., 2018)
60.4+0.2
53.3+0.3
74.3+0.1
77.4:10.6
66.3
DANN (Ganin et al., 2016)
59.911.3
53.0.1..3
73.61:0.7
76.910.55
65.9
ARM (Zhang al., 2021)
58.910.8
51.020.5
74.1=0.1
75.220.3
64.8
SIG (Li et al., 2024)
76.4
63.9
85.4
85.8
77.8
CLIP Zero-shot
71.3+0.0
50.4+0.0
81.7=000
82.6#0.0
71.5
CLJP Linear-probe
68.0+0.8
46.3+0.3
80.410.9
81.910.7
69.1
CLIP-ICM"
72.6+0.4
55.010.9
83.2+0.3
83.720.8
73.6
CLIP-ICM" Linear-probe
78.3+...3
56.410.8
88.610.8
87.710.7
77.8
CLIP-ICM
72.2=00.4
54.6210.5
82.710.2
83.310.3
73.2
CLIP-ICM Linear-probe
77.9+0.1
55.9+0.4
88.2+005
87.3+0.2
77.3
CLIP-ICM
72.910.2
55,410.2
83.610.11
84.210.3
74.0
CLIP-ICM Linear-probe
78.8
56.7:10.3
8..1
88.1
78.2
Backbone
ViT-B/16
CLIP Zero-shot
83.3+0.0
65.3++0.0
89.0+0.0
89.3+0.0
81.7
CLIP Linear-probe
78.9+0.9
69.3+00.9
90.
89.010.2
81.9
CLIP-ICM"
83.740.7
67.410.6
90.8=0.9
90.1
82.6
CLIP-ICM" Linear-probe
84.3+0.5
71.410.2
92.510.2
90.210.8
84.6
CLIP-ICM
82.8+0.4
66.810.3
89.8#0.2
89.010.2
82.1
CLIP-ICM Linear-probe
83.3+0.4
70.5+10.4
91.4110.4
84.411..2
82.4
CLIP-ICM
8..6
70.7++0.3
91.7++0.3
91.5
84.6
CLIP-ICM Linear-probe
84.3+00.5
7..4
92.511.2
90.2:10.8
87.1
33

---

Learning Invariant Causal Mechanism from Vision-Language Models
Table 14, Accuracy Terralncognita datasct.
Algorithm
L100
L38
L43
L,6
Avg
Backbone ResNet-50
ERM
49.8
12.1
56,9
35.7+3.9
46.1
IRM (Arjovsky et al., 2020)
54.6#1.3
39.8
56.2
39.6=0.8
47.6
GroupDRO (Sagawa al.)
41.2
38.6+2..1
56.7+0.9
36.412.1
43.2
Mixup (Yan et al., 2020)
59.612.00
42.241.44
55.910.8
33.9+1.4
47.9
MMD (Li al., 2018)
41.9#3.0
34.8#1.0
57.0=1.9
35.2=1.8
42.2
DANN (Ganin et al., 2016)
51.14335
40.6+0.6
57.400.5
37.7=1.8
46.7
ARM (Zhang et al., 2021)
49.3+0.7
38.3+2.4
55.810.8
38.7=113
45.5
CLIP Zero-shot
7.740.0
14.8+0..0
32.4+0.0
20.9+0.0
19.0
CLIP Lincar-probe
52.011.0
34.420.6
56.1
32.8=022
44.1
CLIP-ICM"
38.8+0.4
33.6+0.66
33.6+0.11
30.1
34.0
CLIP-ICM" Linear-probe
64.140.44
46.5+0.7
61.1-0.6
42.8+0.4
53.6
CLIP-ICM
38.3+0.11
32.9+0.4
33.0+0.2
29.4+0..
33.4
CLIP-ICM Linear-probe
63.5+0.2
45.940.5
60.3+0.2
42.0+0.2
52.9
CLIP-ICM
39.6-10.3
34.5+00.5
34.8+0.4
31.2++0.3
35.0
CLIP-ICM Linear-probe
65.310.5
47.010.2
62.010.3
43.4-0.3
54.4
Backbone is VIT-B/16
CLIPOOD (Shu et al., 2023)
74.
57.3
59.1
47.7
59.7
CLIP Zero-shot
52.0+0.0
20.4
328810.0
31.620.0
34.2
CLIP Linear-probe
73.6+0.5
58.3+0.77
61.0+0.1
47.8+110
60.2
CLIP-ICM"
59.3+0.44
46.2+0.4
52.4
41.620.8
49.9
CLIP-ICM" Linear-probe
7..7+00.
60.2+0.9
66.0+0.8
52.4+0.1
64.3
CLIP-ICM
58.5+0.2
45.240.2
51.610.2
37.410.2
48.2
CLIP-ICM Linear-probe
77.940.3
59.1+0.2
65.410.1
50.5+0.2
63.2
CLIP-ICM
60.8+0.11
47.340.4
53.5+0.1
48.410.1
52.5
CLIP-ICM Linear-probe
7..
61,5++0.3
66.9-0.3
57.9+-03
66.5
34

---

Learning Invariant Causal Mechanism from Vision-Language Models
Table 15. Accuracy VLCS dataset.
Algorithm
S
Avg
Backbone ResNet-50
ERM
97.7+0..
6..3
73.4
1..6
77.5
IRM (Arjovsky et al., 2020)
98.6#0.1
64.9=0.9
73.420.6
77.3+0.9
78.5
GroupDRO (Sagawa al.)
97.3+0.3
63.4+.0.
69.5+
76.7+00.7
76.7
Mixup (Yan al., 2020)
98.340.6
64.821.0
72.1:0.5
74.340.8
77.4
MMD (Li al., 2018)
97.7+0.1
64.0#11.1
72.8+0.22
75.3433.3
77.5
DANN (Ganin et al., 2016)
99.020.3
65.1=1.4
73.1=0.3
77.240.6
78.6
ARM (Zhang et al., 2021)
98.7+0.2
63.6=0.7
71.3+112
76.7+0.6
77.6
MBDG (Robey et al., 2021)
98.3
68.1
68.8
76.3
77.9
CLIP Zero-shot
99.2+0.00
69.5+0.0
69.8=0.0
84.9+0.0
80.9
CLIP Linear-probe
98.1
63.810.8
79.8=0.7
84.5#1.
81.5
CLIP-ICM*
99.420.4
71.8=0.8
71.7=0.8
85.3#1.0
82.1
CLIP-ICM" Linear-probe
99.2+0.11
69.9+0.3
80.5+0.3
89.5+0.9
84.8
CLIP-ICM
98.7+0.5
71.1=0.2
71.0=10.4
84.4+0.4
81.3
CLIP-ICM Linear-probe
98.410..3
69.3+
79.8+10.2
88.6+1.22
84.0
CLIP-ICM
100.0
72.4=0.2
72.5=0.4
86.0+0.2
82.7
CLIP-ICM Linear-probe
99.7+
70.7+0.2
81.0=0.4
90.0++0.3
85.4
Backbone is VIT-B/16
CLIPOOD (Shu et al., 2023)
97.5+
68.3
83.9
88.7+110
84.6
CLIP Zero-shot
99,9
70.1
73.5+0.0
86.140.0
82.4
CLIP Linear-probe
93.7+0.22
65.4+0.2
76.4+0.2
79.1+0.3
78.7
CLIP-ICM*
100.0+0.0
74.7+10.4
74.5+10.4
87.1
84.1
CLIP-ICM' Lincar-probe
98.8+0.3
72.7=0.8
86.4-0.9
87.940.9
86.5
CLIP-ICM
98.810..2
73.6=10.2
73.9+0.5
87.3+0.4
83.4
CLIP-ICM Linear-probe
98.2+0.4
71.7=0.4
85.8-0.1
85.1+0.2
85.2
CLIP-ICM
100.010.0
75.6-00.
79.210.4
90.010.1
86.2
CLIP-ICM Linear-probs
99.8+
73.9+0.3
87.3+0.5
85.5+0.3
86.6
35

---

Learning Invariant Causal Mechanism from Vision-Language Models
Table 16. Accuracy DomainNet dataset.
Algorithm
CLIPART INFOGRAPH
PAINTING QUICKDRAW
REAL
SKETCH
Avg
Backbone ResNet-50
ERM
58.1
18.8=0.3
46.7110.3
12.240.4
59.6#0.1
49.8+0.4
40.9
IRM (Arjovsky al., 2020)
48.512.8
15.0.1.55
38.3114.3
10.910.5
48.215.2
42.313.1
33,9
GroupDRO (Sagawa et al.)
47.2+0.55
17.5+0.4
33.8+00.5
9.3+0.3
51.6+0.4
40.1
33.3
Mixup (Yan al., 2020)
55.710.3
18.5
44.310.5
12.510.4
53.810.33
48.2+0.5
39.2
MMD (Li et al., 2018)
32.1+13.3
11.0+4.6
26.8+11.3
8.742.1
32.7++3.8
28.9+11.9
23.4
DANN (Ganin et al., 2016)
53.1
18.320.1
44.2+00.7
11,840.1
55.520.4
46.8
38.3
ARM (Zhang al., 2021)
49.710.3
16.3.0.5
40.911.1
9.410.11
53.410.4
43.5110.4
35.5
CLIP Zero-shot
53.1
39.2=0.0
52.9+0.0
5.7+0.0
76.7=0.0
48.0+0.0
45.9
CLIP Lincar-probe
59.2+0.1
38.6+0.9
54.1+0.7
8.240.7
56.420.7
39.6+0.7
42.6
CLIP-ICM"
58.010.1
42.310.5
54.410.2
12.810.7
78.210.1
49.610.2
49.2
CLIP-ICM" Linear-probe
63.1+0.1
50.1+0.2
66.2+1.0
19.7+0.3
83.8+0.4
63.2+0.3
57.8
CLIP-ICM
57.4+0.2
41.7+0.3
53.910.1
12.1+0.5
77.620.3
49.010.3
48.6
CLIP-ICM Linear-prod probe
62,4+0.2
49.4=:02
65.7+0.1
18.9#0.4
83.2=0.55
62.610.2
57.0
CLIP-ICM
59.1
43.4=:.2
55.5+0.3
13.8#0.2
79.1
50.8+0.3
50.3
CLIP-ICM Linear-probe
6..1
10,5
6..2.10.5
20.6.10.5
84.9
64.1
58.7
Backbone is ViT-B/16
CLIPOOD (Shu al., 2023)
77.6
54.7
72.5
20.7
85.7
69.9
63.5
CLIP Zero-shot
70.2+0.0
46.6++..0
65.0++0.0
13.7+0.0
82.9+0.0
62.7+000
56.8
CLIP Linear-probe
73.9:0.5
40.2:10.5
62.110.2
15.110.3
76.910.1
62.0+0.7
55.0
CLIP-ICM"
77.1+0.5
51.8+0055
67.9+1.0
16.7++0.3
82.940.8
66.9+0.2
60.5
CLIP-ICM" Lincar-probe
78.6+0.2
55.6+0.2
72.910.6
22.1+:0.7
86.120.3
68.5+0.3
64.0
CLIP-ICM
75.910..5
50.110.3
66.2.10.4
15.0.10.5
81.410.1
55.810.3
57.4
CLIP-ICM Lincar-probe
76.7+0.3
53.5+002
70.6+0.4
20.8+0.4
84.0=002
52.1+0.5
59.6
CLIP-ICM
78.110.4
52.810.3
69.010.2
17.6.10.4
84.210.2
64.810.2
61.1
CLIP-ICM Linear-probe
79.6+0.2
56.7+-0.2
74.1+00.3
23.5++0.2
87.3+0.2
68.9++0.3
65.0
36


Reviews:
[{"review_id": "jZnSLFjpfa", "reviewer": "Reviewer_Pp7f", "summary": "This paper introduces CLIP-ICM, a framework that improves CLIP’s OOD robustness by leveraging a causal perspective to separate invariant and variant factors. By learning a linear mapping to the invariant subspace using interventional data, CLIP-ICM enhances performance across multiple OOD datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and valuable suggestions. We sincerely appreciate the reviewer for their positive feedback, especially for finding our claims **well-supported**, recognizing the **clarity** and **comprehensiveness** of our paper's presentation, affirming that our evaluation methodology **aligns with community standards**, and highlighting our contribution in **bridging** causal inference and OOD generalization. Additionally, we provide detailed responses to address the two specific concerns raised by the reviewer as follows.\n\n## Response to Other Strengths And Weaknesses\n\n### W1:\n\nWe appreciate the reviewer's interest in understanding the contribution of the linear projection matrix $A_{inv}$ and the role of interventional data. Regarding the role of interventional data, we would like to first emphasize a few points:\n\n1. Interventional data and the linear projection matrix $A_{inv}$ are  mutually dependent components of our method. According to Equation (9), without interventional data, it is unlikely to estimate $A_{inv}$ in our framework. \n2. As shown in Tables 2, 3, 5, 6, and 10–14, we have conducted extensive comparisons between three variants of our method utilizing different type of interventional data. Specifically:\n   - CLIP-ICM$^*$: using only image-based interventional data,  \n   - CLIP-ICM$^\\dagger$: using only text-based interventional data, \n   - CLIP-ICM: using both types of interventional data.  \n3. We have provided an ablation study on the effect of different numbers of interventional data pairs in Appendix M, Figure 6 (d).\n\nRegarding the role of $ A_{inv} $, we agree with your suggestion and thus we include an additional ablation experiment to further illustrate its importance. Specifically, we use our generated image-based interventional data to train a linear-probe on the DomainBed dataset.  The experimental results are summarized as follows. \n\n| Method                              |   PACS   |   VLCS   | OfficeHome | TerraInc | DomainNet |   AVG.   |\n| :---------------------------------- | :------: | :------: | :--------: | :------: | :-------: | :------: |\n| Linear Probe                        |   96.4   |   78.7   |    81.9    |   60.2   |   55.0    |   74.4   |\n| Linear Probe + Interventional  data |   96.8   |   79.3   |    82.3    |   60.5   |   55.8    |   74.9   |\n| CLIP-ICM$^*$ + Linear Probe         | **97.5** | **86.5** |  **84.6**  | **64.3** | **64.0**  | **79.0** |\n\nFrom the results, we can observe that:\n\n1. Incorporate image-based interventional data with linear probe only slightly (0.5%) improve the performance of linear probe.\n2. Despite all incorporate image-based interventional data for training, the performance of CLIP-ICM$^*$ + Linear Probe is significantly better than that of Linear Probe + Interventional data. \n\nThese findings demonstrate that our proposed $A_{inv}$ module (i.e., the projection to the invariant subspace) indeed improve the performance of CLIP in OOD scenarios.\n\n### W2:\n\nRegarding the source of variance in Tables 2 and 3, each value in Tables 2 and 3 represents the mean and standard deviation over 5 runs with different random seeds. According to the standard evaluation protocol of the DomainBed Benchmark, we believe that the primary source of variance originates from the various splits of the training, validation, and test datasets across multiple runs.", "claims_and_evidence": "To the best of my knowledge, the evidence supports the claims well.", "methods_and_evaluation_criteria": "To the best of my knowledge, the evaluation follows the community convention.", "theoretical_claims": "I have checked section 5.1 and do not find any issue", "experimental_designs_or_analyses": "I have checked section 7 and believe it follows the community standard", "relation_to_broader_scientific_literature": "The paper studies the OOD from a causal inference perspective, which bridges the gap between the two fields", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "1. I am curious on the role of A_inv and the interventional data. After I check the ablation study, seems like there is no ablation study on this level. would be beneficial to present the ablation of the three steps in figure 3 to help reader to understand the importance of each component.\n\n2. Where does the variance come from in the data reported in Table 2 and 3? Is it from the difference of interventional data or different initialization? It would be beneficial to have clarity don't that\n\n2. Overall the paper's presentation is very clear and comprehensive, and studies an important problem that lies in the interest of the community.", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mm3mOUpbdl", "reviewer": "Reviewer_q6oc", "summary": "This work is motivated from the OOD generalization issue in CLIP, it addresses this problem via learning an invariant causal mechanism and proposes CLIP-ICM framework, which includes collecting interventional data, estimating a linear projection matrix, and predicting in the invariant subspace. The proposed CLIP-ICM shows improvement in OOD datasets.\n\n## update after rebuttal\nI appreciate the authors for their response, which addresses most of my concerns. I intend to maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and positive feedback. We are pleased that the reviewer recognizes our work as **well-supported**, highlighting our **clear** pipeline and **sound** theoretical analysis. Below, we provide detailed responses addressing the specific concerns raised by the reviewer.\n\n## Response to Methods and Evaluation Criteria\n\nWe appreciate the reviewer’s comments regarding the interventional data generation process. We would like to clarify the following points:\n\n1. In the original manuscript, lines 369–371 (left column) briefly outline the steps for collecting image-based interventional data, while lines 381–384 (left column) and line 330 (right column) briefly describe the process for collecting text-based interventional data. And we mention that the detailed collection process is provided in Appendix H.1.\n3. In Appendix H.1, we present detailed description of the collection procedures for both image-based and text-based interventional data:\n   1. For the image-based interventional data, we explain that it is generated using eight data augmentation techniques, including: ColorJitter, GrayScale, GaussianBlur, RandomInvert, RandomRotation, RandomPosterize, RandomSolarize, and RandomEqualiz. These data augmentation techniques directly implemented from the *torchvision.transform* package.\n   2. The text-based interventional data comprises two components: the text description model and the text intervention model. Both models are generated by invoking GPT-4o. The prompts used for these models are provided on page 21, lines 1118–1142, and Figure 5 presents an example of the text-based interventional data.\n\nPlease let us know if you have any additional suggestions for further improvements regarding this aspect.\n\n## Response to Experimental Designs Or Analyses\n\nTo address your concerns, in addition to the datasets used in the original manuscript (PACS, VLCS, OfficeHome, Terra Incognita, DomainNets, ImageNet, ImageNet-V2, ImageNet-S, ImageNet-A, and ImageNet-R), we additionally conducted experiments on the iWildCam-WILDS 2020 dataset. \n\niWildCAM comprises 203,029 images from 182 different animal species, which were collected from 323 camera traps distributed across various locations. The images obtained from different locations exhibit variations in lighting, color, camera angle, background, vegetation, and relative animal frequencies.\n\nWe follow the setting of Koh et al. (2021)[1], use images from 243 locations as the training domain and those from 48 other locations as the test domain. We report the average macro F1 score of CLIP, CLIP-ICM$^*$, CLIP-ICM$^\\dagger$, and CLIP-ICM under both ID and OOD conditions, as shown in the table below:\n\n| Method| ID (48 Locations) | OOD (243 Locations) |\n| :- | :-: | :-: |\n| CLIP|14.2|10.6|\n| CLIP Linear-Probe|54.6|41.4|\n| CLIP-ICM$^*$|15.6|13.3|\n| CLIP-ICM$^*$ Linear-Probe|56.2|42.1|\n| CLIP-ICM$^\\dagger$|15.2|12.2|\n| CLIP-ICM$^\\dagger$ Linear-Probe|55.6|44.3|\n| CLIP-ICM|15.8|14.1|\n| CLIP-ICM Linear-Probe|57.1|46.1|\n\n[1] Koh et al. Wilds: A benchmark of in-the-wild distribution shifts. ICML 2021.\n\n## Response to Other Strengths And Weaknesses\n\nThank you for suggesting a more comprehensive evaluation. We have extensively validated our method across multiple datasets, including PACS, VLCS, OfficeHome, Terra Incognita, DomainNets, ImageNet, ImageNet-V2, ImageNet-S, ImageNet-A and ImageNet-R along with detailed ablation studies. \n\nAdditionally, we have included experiments on the iWildCam-WILDS 2020 dataset in our previous response. Moreover, in our reply to Reviewer Pp7f, we added ablation studies concerning the role of $A_{inv}$. Please let us know if you have any further suggestions regarding other experiments.\n\n## Response to Other Comments Or Suggestions\n\nWe thank the reviewer for pointing out this issue. In the final version, we will check all formatting issues and make the necessary revisions.\n\n## Response to Questions For Authors\n\nWe thank the reviewer for the question and are happy to elaborate. \n\n1. The environmental diversity across our evaluation datasets comes in different ways. For datasets like PACS, Office-Home, DomainNet, ImageNet-Sketch, and ImageNet-R, diversity primarily stems from variations in visual styles. In VLCS and Terra Incognita, it is reflected in background complexity, lighting conditions, and camera viewpoints. For ImageNet V2 and ImageNet-A, diversity arises from changes in image sources and the inclusion of hard-to-classify samples, respectively.\n2. Our method is generally applicable to real-world scenarios where environment-induced distribution shifts occur. Potential applications include monitoring in wildlife habitats, perception systems in autonomous driving, and cross-domain image-text retrieval. In particular, tasks that require stable semantic understanding across diverse environments can benefit from the CLIP-ICM framework’s ability to isolate invariant semantic factors from CLIP representations.", "claims_and_evidence": "In general, the claims are well supported. The paper originates from a well-studied principle in invariant learning, the high-level idea is not new, but it would still contribute to the CLIP model generalization.", "methods_and_evaluation_criteria": "The pipeline of the method is generally clear, but some of the technical details may need further clarification. For example, it would be better to include a more detailed description of the interventional data generation process.", "theoretical_claims": "The theoretical analysis looks good to me.", "experimental_designs_or_analyses": "The experiment would be improved if further diverse environments and contexts are included.", "relation_to_broader_scientific_literature": "The paper is related to CLIP applications in different domains.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "In general, the paper would be further improved with a more comprehensive evaluation.", "other_comments_or_suggestions": "There are a few informal writing styles, e.g., the footnote in page 4 takes a single sentence in the main text.", "questions_for_authors": "Could the authors share more details of the environment diversity and applicable scenarios of the proposed method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ktJmjrP1B", "reviewer": "Reviewer_HsFQ", "summary": "The paper analyzes the OOD generalization of CLIP via the lens of causal/invariant predictor learning, where the goal is to make predictions via the invariant (causal) features for the downstream task. Motivated by the failure cases of naive funetuning of CLIP, the authors propose CLIP-ICM as a principled approach. The proposed approach relies on the linear identifiability guarantees in CLIP's representation space, which is further disentangled into invariant and environment specific features by leveraging interventional data. With the identified invariant features, CLIP-ICM train a linear probe for making predictions in the downstream task. CLIP-ICM is benchmarked on widely used OOD generalization datasets, where it outperforms baselines (existing strategies for finetuning CLIP) especially for the case of open class domain shifts.\n\n\n## Update after rebuttal \n\nI have read the author rebuttal and other reviews as well. I think the paper is very interesting, technically sound, and make a good use for proposing methodology inspired from the latent identification literature. Hence, I retrain my rating and vouch for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful evaluation and positive feedback. We appreciate the acknowledgment that our approach offers a **solid** theoretical foundation and demonstrates **clear** empirical benefits for OOD generalization. We also value the reviewer’s recognition that our **claims are well-supported** by both theoretical analysis and practical experiments, as well as the confirmation that our references to existing literature provide **sufficient** context. Moreover, we are pleased that the reviewer finds our writing to be **clear** and our explanation of the proposed method to be **thorough**.\n\nBelow, we address the reviewer’s additional questions and suggestions in detail.\n\n## Response to Weaknesses\n\nWe appreciate the reviewer’s positive comments and would like to clarify our position. We acknowledge that Theorem 5.3 and Theorem 5.5 indeed build upon previous work in the literature. However, our primary interest lies in extending these interesting theoretical insights to practical applications.\n\nIn our manuscript, we carefully discuss the conditions under which Theorem 5.5 and Theorem 5.6 hold, and we leverage these conditions to propose our CLIP-ICM method, which is designed to guarantee lower OOD generalization error. One particularly surprising and encouraging observation is that by mapping both image and text embeddings into a shared invariant subspace, CLIP is able to maintain its original zero-shot performance even when confronted with domain shift—thus, ensuring that it continues to perform well on new classes after task-specific fine-tuning.\n\nWe are grateful for the reviewer’s recognition of our work and believe that integrating theoretical results with real-world application strategies represents a significant contribution to the field.\n\n## Response to Other Comments or Suggestion\n\n1. > Given that the theoretical results (Theorem 5.3, 5.4) are mostly an application of existing theoretical results, I suggest the authors should rename the theorems to propositions.\n\nThank you for your suggestion. In the final version, we will change these two theorems into propositions.\n\n2. > the same should be done for Theorem 5.4 with the prior work by Ahuja et al. 2023 on interventional causal representation learning.\n\nWe thank the reviewer for this suggestion. In the final version of the manuscript, we will explicitly highlight both the connections and distinctions between our Theorem 5.4 and the interventional causal representation learning work by Ahuja et al. (2023).\n\n## Response to Questions For Authors\n\n1. >  Were the CLIP representations finetuned with linear probe or only a linear probe was trained with frozen representations?\n\nWe apologize for the confusion caused by our description, and thank the reviewer for highlighting this issue.\n\nTo clarify, in the domain-shift scenario (line 153), the CLIP embeddings remain frozen, and only the linear probe is trainable. At line 168, when we mentioned \"fine-tuning,\" we were referring specifically to the linear probe training process, rather than updating the original CLIP image encoder or text encoder.\n\nWe will carefully revise this description in the final manuscript to clearly differentiate these two settings and avoid further confusion.\n\n2. > Table 1, open class domain shift scenario, why does finetuning improve the performance for base classes but it deteriorates the performance for novel classes?\n\nWe thank the reviewer for raising this insightful question. This phenomenon—where fine-tuning improves performance on the base classes but degrades performance on novel classes—has been widely acknowledged in studies on adapting CLIP, including CoOP, CoCoOp, CLIP-Adapter, and CLIPood. As noted by Wortsman et al. [1] and Shu et al. [2], naively fine-tuning CLIP often results in a loss of its inherent strong generalization ability, manifesting as improved performance on the specifically fine-tuned downstream task but significantly weakened robustness under distribution shift (including both covariate shift and label shift).\n\nA likely explanation for this deterioration on novel classes is tied to **catastrophic forgetting**, a phenomenon wherein a model “forgets” previously learned information when trained on new data. In the context of Table 1, when adapting CLIP to  a set of base classes, the fine-tuning procedure heavily optimizes for accurate classification of those base classes. Consequently, the original parameters—particularly those responsible for generalizing to unseen classes—are overwritten. As a result, the previously robust zero-shot capability of CLIP (which was central to its strong open-class performance) is compromised. \n\n\n\n[1] Wortsman, Mitchell, et al. Robust fine-tuning of zero-shot models. CVPR 2022.\n\n[2] Shu, Yang, et al. Clipood: Generalizing clip to out-of-distributions. ICML 2023.", "claims_and_evidence": "Yes, the claims made in the submission are well supported with clear and convincing evidence.\n\n**Strong empirical evidence for the claims**\n\n- The failure cases with naive finetuning strategies of CLIP are highlighted clearly with experiments on the Terra Incognita dataset (Table 1).\n\n- The main experiments in Table 2 test CLIP-ICM with a variety of baselines on multiple benchmarks, with CLIP-ICM providing improved  performance in nearly all the cases. Further, experiments in Table 3 with results for the open classes domain shift strengthen the author's claim of superior OOD generalization.\n\n- Given the requirement of interventional data in CLIP-ICM, the authors generate interventional data to test by manipulating both the base images and captions. This helps to analyze CLIP-ICM's performance with access to diverse interventional data, and ablations CLIP-ICM$^{\\star}$ and CLIP-ICM$^{\\dagger}$ provide further details.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem at hand. All the benchmarks used in this paper are widely used for out-of-distribution generalization. Regarding baselines for finetuning CLIP,  I am not the best judge if all the relevant baselines have been used, since I am not familiar with recent works on CLIP finetuning.", "theoretical_claims": "Yes, I checked the correctness of the proof for all the theorems and I did not find any major issues.", "experimental_designs_or_analyses": "Yes, I checked the soundness/validity of all the experiments in the paper, and the experiment design doesn't have any flaws. Further, the authors have done a good job at analyzing their findings, its coherent with the experiment results.", "relation_to_broader_scientific_literature": "The papers utilizes the methodology of invariant predictor learning, a fairly common approach for tackling out-of-distribution generalization.  \nSpecifically, identifiable invariant predictor learning approaches have been proposed in prior works [1, 2]. The key contribution of the paper is to apply these ideas in the framework of CLIP.\n\nReferences\n\n- [1] Lu, Chaochao, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. \"Invariant causal representation learning for out-of-distribution generalization.\" In International Conference on Learning Representations. 2021.\n\n- [2] Yao, Dingling, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. \"Unifying Causal Representation Learning with the Invariance Principle.\" arXiv preprint arXiv:2409.02772 (2024).", "essential_references_not_discussed": "No, I believe all essential references have been discussed to the best of my knowledge. The authors have written have a very detailed related works section.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper is well written, with details about the proposed method easy to follow and the empirical findings are clear and easy to follow.\n\n\n**Weaknesses**\n\n- The core ideas behind CLIP-ICM are not very original, the theoretical results in the paper mostly build upon existing proof techniques in the literature. Even the methodology of extract invariant features from representation with linear identification guarantees is not very novel. However, I don't think this is a major concern, as the application of identifiable invariant feature learning specifically to CLIP framework is novel to the best of my knowledge.", "other_comments_or_suggestions": "- Given that the theoretical results (Theorem 5.3, 5.4) are mostly an application of existing theoretical results, I suggest the authors should rename the theorems to propositions.\n\n- Just like the authors mention that Theorem 5.3 aligns with results in prior works, the same should should be done for Theorem 5.4 with the prior work by Ahuja et al. 2023 on interventional causal representation learning.", "questions_for_authors": "- The description in section 4 about the domain shift is a bit confusing. The authors mention with linear probe the CLIP embeddings are kept frozen (line 153), but when analyzing the results they mention finetuning CLIP (line 168). Were the CLIP representations finetuned with linear probe or only a linear probe was trained with frozen representations?\n\n- Table 1, open class domain shift scenario, why does finetuning improve the performance for base classes but it deteriorates the performance for novel classes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PYQ4pOcA4j", "reviewer": "Reviewer_5jTK", "summary": "This work aims to leverage Invariant Causal Mechanisms in causality to improve prediction under distribution shifts. However, a detailed summary is challenging for me due to several fundamental issues, including an unclear problem formulation, misconceptions of key concepts, and unrealistic theoretical assumptions.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Re: Claims And Evidence & Q1\n\n**Regarding the definition of domain shift and open-class.**\n\n1. In our context, domain shift primarily refers to covariate shift, where the $p(x)$ differs between the training and testing phases while  $p(y|x)$ remains unchanged. This scenario is widely adopted in standard domain generalization tasks [1].\n\n2. The open-class prediction problem is well defined [2–5]. The problem is addressable,  and one of the solution is CLIP [3]. Several studies [4,5] also provide theoretical support.\n\nWe will include detailed explanations regarding these scenarios in the final version.\n\n**Regarding the causal mechanism in Section 5.1.** \n\nWe clarify as follows:\n\n1. The construction of SCM depends on the **type of task** [10]. For example, whether the chicken causes the egg or the egg causes the chicken depends on the objective: if we study how chickens produce eggs, the chicken is the cause and the egg is the effect; if we study how eggs develop into chickens, then the egg is the cause. Therefore, when we study the task of prediction, the input image is the cause, and the predicted label is the effect.\n2. We study **two** SCMs in our paper: Figure 1(a) is the generation process, while Figure 1(b) is the prediction process. Since our work primarily focuses on prediction, the causal mechanisms $p(y|do(x))$ and $p(y|do(z_{inv}))$ in Proposition 5.1 are both defined based on the SCM **in Figure 1(b)** rather than Figure 1(a). In Figure 1(b), the model construct a causal chain $X \\to Z \\to Y$, where $X$ is the cause, $Y$ is the effect. Therefore, it **does not means inferring the cause from the effect**. In this SCM, the causal effect is: \"how change in image $X$ affects the prediction of output $Y$\".\n3. As stated in lines 194–197, the prediction process can be viewed as the inverse of the data generation process. We emphasize that the term “inverse process” here is **solely a mathematical construct** to derive the structural equations. These equations correspond to edges in Figure 1(b), which we further provide a detailed explanation in lines 197–208. Therefore, Figure 1(b) is a valid SCM.\n\nIn summary, our proposed SCM doesn't contain \"cause given the effect\" scenarios. And the causal mechanism in Figure 1(b) is valid and well-defined.\n\n## Re: Theoretical Claims & Q2\n\n1. The reviewer may have misunderstood the logical connection between Condition 5.2 and Theorem 5.3. Theorem 5.3 aims to prove identifiability under certain conditions, and our work focuses on **formulating such a condition**—namely, Condition 5.2. Therefore, although the proof of Theorem 5.3 is relatively straightforward, its validity relies on the formulation of Condition 5.2.\n2. Literature [6] proves that without additional constraint, latent factors are unidentifiable. Motivated by literatures [7,8,9], we **identify and formalize** this condition in this paper, thereby facilitating a clear understanding and straightforward implementation of the proof for Theorem 5.3. \n3. The **advantage** of this condition is that it **does not require** additional assumptions on the latent factors (prior distribution) or on the generative process. Instead, it relies solely on observable label data.\n4. The condition consists of two parts: consistency and diversity. The consistency (Equation 12) only requires the output distribution $\\hat{p}(y|x)$ to match the **observable distribution** $p(y|x)$ , rather than an unknown prior. \n5. We demonstrate in lines 247~255 why CLIP can be considered satisfied this condition.\n\nIn summary, the Condition 5.2 and Theorem 5.3 are formulated within a standard theoretical framework based on extensive prior work and have practical implications. \n\n## Re: Experimental Designs Or Analyses & Q3\n\nWe understand the reviewer's concerns, but:\n\n1. To date, OpenAI has not released the training data used for CLIP, which makes it extremely challenging to verify whether there is any overlap between the experimental data and the data used to train CLIP.\n2. Our experimental design strictly adheres to the established community standards for fine-tuning CLIP in domain generalization tasks. (including CLIP-Adapter, CLIPood, CoOp, CoCoOp, MIRO, and DPL)\n\nTherefore, we believe that our experimental setup is both reasonable and widely accepted.\n\n\n[1] Domain generalization: A survey.\n\n[2] A survey of zero-shot learning: Settings, methods, and applications.\n\n[3] Learning transferable visual models from natural language supervision.\n\n[4] Zero-shot learning with semantic output codes. \n\n[5] Attribute-based classification for zero-shot visual object categorization.\n\n[6] Nonlinear independent component analysis: Existence and uniqueness results.\n\n[7] Nonlinear ICA using auxiliary variables and generalized contrastive learning.\n\n[8] On linear identifiability of learned representations.\n\n[9] Contrastive learning inverts the data generating process.\n\n[10] Toward causal representation learning.", "claims_and_evidence": "There are several unclear or even mistaken claims in the paper. For example:\n\n1) Problem Setting: The claim \"The goal of OOD generalization is to learn a predictor from training environments... domain shift and open-class scenarios. Domain shift arises when the data distribution in the test environment differs from that in the training environment, while open-class scenarios involve previously unseen classes appearing at test time.\" is confusing.\n\nDomain shift is a broad category encompassing various settings, such as covariate shift, conditional shift, and label distribution shift. The authors should specify which type of domain shift their work addresses to avoid ambiguity.\nOpen-class scenarios, where previously unseen classes appear at test time, present a significant challenge. The authors should clarify whether this setting is realistically addressable and, if so, whether there exists a theoretical solution for it.\n\n2) Conceptual Misuse: You have assumed a causal generative model, as shown on the left in Figure 1, where there is a clear causal relationship, e.g., y causes z, and z causes x. Causal mechanisms should be defined from cause to effect, rather than as p(y∣do(x)) or p(y∣do(z)), as claimed. From my understanding, a causal mechanism refers to the underlying process or system that explains how one variable influences another in a causal relationship. It describes how causes bring about effects, and is typically assumed to be invariant. Therefore, one cannot claim that the relationship from effect to cause constitutes a causal mechanism, as this relationship is generally variant and does not align with the principles of causal inference. Further, in Proposition 5.1, p(y|do(z)) or p(y|do (x)) (e.g., \"cause given the effect\") typically does not have a well-defined meaning in the standard framework. Please let me know if I have misunderstood this.", "methods_and_evaluation_criteria": "Since the identifiability theory is problematic (see below for concerns regarding the theory), I am not confident that the method's effectiveness is due to causality.", "theoretical_claims": "Theorem 5.3 is central to supporting this work. However, the assumption in Condition 5.2 is quite peculiar. It broadly states, \"There exist some samples such that the inference model can be equal to the generative model on these samples.\" This is strange, because the generative model is completely unknown. How can one enforce the inference model to match an unknown prior from the generative model? If this assumption holds, one could simply assume that the inference model equals the generative model, which would make the proof trivial. In fact, after reviewing the proof, I found that there is almost no technical challenge to the identifiability proof under the assumption in Condition 5.2.", "experimental_designs_or_analyses": "For the experiment results based on CLIP, there is a significant concern regarding whether the training process of CLIP truly does not use the data in the experiments. Since CLIP is trained on a large number of image-text pairs, it’s important to question whether there is any potential data leakage. Specifically, it should be clarified whether the data used in the experiments overlaps with or has any connection to the data CLIP was trained on, as this could lead to biased or invalid results. Ensuring that no data leakage occurs is critical to maintaining the integrity of the experiment's findings.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Causality is a challenging concept to understand. I believe it is particularly effective in handling distribution shift tasks, as it not only provides a theoretical framework but also offers practical tools in certain cases. However, we must be cautious in how we apply it, and at the very least, it requires a deep understanding of causality.", "questions_for_authors": "Overall, \n1) the problem setting is unclear, and some fundamental concepts in causality are misused (see Claims And Evidence). \n\n2) The identifiability analysis is unrealistic and nearly flawed (Theoretical Claims), which undermines confidence in the proposed methods. \n\n3) Additionally, using the CLIP model to claim OOD distribution experiments should be approached with caution and carefully considered (see Experimental Designs Or Analyses).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper proposes a causal mechanism for addressing CLIP's out-of-distribution (OOD) problem. The paper is well written, and reviewers noted the strong empirical evidence demonstrating the effectiveness of the proposed method compared to naive finetuning strategies and other baseline models. However, the theoretical claims were not unanimously accepted by reviewers. In particular, one reviewer raised significant criticisms regarding the theoretical foundation of the work, its assumptions, and the proposed theorems. During discussions with reviewers, it became clear that while these theoretical uncertainties are important, they may not undermine the strength of the empirical results. Moreover, reviewers disagreed among themselves about the theoretical merits of the work. Given these considerations, I believe the paper represents an interesting contribution to the field and therefore recommend acceptance. However, the authors should carefully address the theoretical concerns raised by reviewers in their revised manuscript.
Decision: Accept (poster)