When and How Does CLIP Enable Domain and Compositional Generalization?
Authors: Elias Kempf
Simon Schrodi
Max Argus
Thomas Brox
Venue: ICML 2025 spotlightposter (2025)
Keywords: CLIP
Compositional Generalization
Domain Generalization
Out-of-Distribution Robustness
OOD generalization
Primary area: deep_learning->robustness
TL;DR: We studied CLIP's domain and compositional generalization via systematic data-centric experiments and mechanistic analyses, revealing that domain diversity, sufficiently shared intermediate features and circuitry are crucial for generalization.

Abstract: The remarkable generalization performance of contrastive vision-language models like CLIP is often attributed to the diversity of their training distributions. However,  key questions remain unanswered: Can CLIP generalize to an entirely unseen domain when trained on a diverse mixture of domains (domain generalization)? Can it generalize to unseen classes within partially seen domains (compositional generalization)? What factors affect such generalization? To answer these questions, we trained CLIP models on systematically constructed training distributions with controlled domain diversity and object class exposure. Our experiments show that domain diversity is essential for both domain and compositional generalization, yet compositional generalization can be surprisingly weaker than domain generalization when the training distribution contains a suboptimal subset of the test domain. Through data-centric *and* mechanistic analyses, we find that successful generalization requires the learning of sufficiently shared representations in intermediate layers and circuits.

When and How Does CLIP Enable Domain and Compositional Gemeralization?
Elias Kempf" Simon Schrodi
Max Argus'
Thomas Brox
Abstract
Recent work has converged on the conclusion that CLIP's diverse training distribution is the primary factor driving its unprecedented generalization performance. For example, Fang et al. (2022) found that other factors such as language supervision, training data size, or the contrastive loss play only a minor role, while Nguyen et al. (2022) showed that data quality is more important than quantity. More recently, Mayilvahanan et al. (2025) demonstrated that CLIP's "gen- eralization performance [-..] drops to levels similar to what has been observed for ImageNet-trained models" (Mayil- vahanan et al., 2025, p. 10) by limiting the diversity of (visual) domains' to a minimum, i.e., by removing all non- natural samples. This shows that the mixture of various (non-natural) domains plays an important role for CLIP's generalization, yet the underlying mechanisms remain un- explored. This brings to core research question:
The remarkable generalization performance of contrastive vision-language models like CLIP is often attributed to the diversity of their training distributions. However, key questions remain unanswered: Can CLIP generalize to an entirely unseen domain when trained on diverse mix- ture of domains (domain generalization)? Can it generalize to unseen classes within partially seen domains (compositional generalization)? What factors affect such generalization? To answer these questions, we trained CLIP models on sys- tematically constructed training distributions with controlled domain diversity and object class expo- sure. Our experiments show that domain diversity is essential for both domain and compositional generalization, yet compositional generalization can be surprisingly weaker than domain general- ization when the training distribution contains a suboptimal subset of the test domain. Through data-centric and mechanistic analyses, we find that successful generalization requires the leamn ing of sufficiently shared representations in inter- mediate layers and circuits.
How does the mixture of diverse (visual) domains in the training data affect CLIP's generalization performance?
In particular, investigate under which circumstances CLIP can learn the object class invariances across the train- ing domains with the aim to generalize to entirely unseen domains-a fundamental question about its domain gener- alization capability (Blanchard et al., 2011; Muandet et al., 2013; Gulrajani Lopez-Paz, 2021). We also study ques- tions about CLIP's compositional generalization (Hupkes et al., 2020; Wiedemer et al., 2023), which is believed be important factor of its generalization performance (May- ilvahanan et al., 2024; Udandarao et al., 2024) and long- standing challenge of machine learning research. Adapting Szabó's (2012) classical example, we ask pictorially: Can CLIP, trained on natural images of cats and dogs along with sketches of cats, generalize to sketches of dogs?
1. Introduction
Foundation models considered a decisive step towards more generic Al models (Bommasani al., 2021). For example, CLIP scaled the alignment of image-text pairs via a contrastive loss to millions of samples (Radford et al., 2021; Jia et al., 2021; Zhai et al., 2023). Unlike tradi- tional classifiers from the ImageNet era, which often ex- perience substantial performance drops under distribution shifts, CLIP demonstrates unprecedented generalization to -Out-of-Distribution (OOD)" data (Radford et al., 2021), I). However, what drives this improved OOD generalization?
To answer such questions, we construct fully controllable experimental conditions that allow precise and systematic manipulation of the domain mixtures and exposure to object classes in the training data (see Figure 1), while keeping all other variables, such as the CLIP model type, training process, and class distribution constant. Specifically, we augmented a base dataset consisting primarily of natural im- ages, such ImageNet- (Fang al., 2022), with non-natural samples from DomainNet (Peng al., 2019),
Equual contribution Universityy of Freiburg. Correspondence to: Elias Kempf <kempfe @eeffff@@es.u.. unii-reiburggdee. Simon Schrodi <schrodi@ess www.uniiiiiiuugdde
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We refer domain as group of images sharing common style, such as natural images, sketches, paintings, etc.

---

When and How Does CLIP Enable Domain and Compositional Generalization?
A
Domain Mixtures for CLIP Training
B OOD Evaluation
Domains
Natural-only
Leave-ouf-domain
Test Set
Natural
Test
Others
Included?
Natural-only
Lower Bound
Domain Generalization
yes
CG low-diversity
CG low-diversity
CG high-diversity
no
CG high-diversity
Classes
Leave-out-domain
Ci
15
30
Compositional Generalization
Performance
Figure 1: Training data setups and CLIP's generalization performance under varying domain mixtures and class exposure. A: We systematically varied the domain mixture and object class exposure in CLIP's training data across four controlled scenarios, while keeping other factors like dataset size and model choice constant. Specifically, CLIP models on (1) Natural-only-moon (2) Leave-out-domain- a diverse set of domains images
trained to establish lower bound on performance; excluding the test domain to assess CLIP's s domain with some test domain classes to evaluate its compositional plus some test domain classes to assess compositional generalization with higher domain diversity. CLIP models trained on diverse domains demonstrate stronger OOD generalization to held-out test domain classes
natural images (from classes
generalization; (3) CG low-diversit-
generalization with limited domain diversity; and (4) CG h diverse mix of domains
). compared to models trained only natural images fewer domains. Notably, CLIP can perform as well or better without exposure some test domain classes during training.
including the domains Clipart, Infograph, Painting. Quick- draw, and Sketch. By systematically including subsets of these domains and their classes, we study questions about CLIP's domain generalization and compositional general- ization capabilities. We complement these experiments with in-depth data-centric and mechanistic analyses to under- stand what changes in the CLIP model led to the improved generalization or failure thereof. Our experiments uncov- ered the following key findings:
tionally, it shares more embedding features between different domains. However, CLIP sometimes fails to generalize to certain domains. We provide a twofold explanation: (1) the inputs from these domains lack shared input features and, as result, (2) the model has limited shared intermediate features and 2ircuits between domains, constraining its ability to generalize effectively. We support this hypothesis through repre- sentational similarity analysis and introduce a related concept for circuits: mechanistic similarity, measuring similarity between circuits.
Domain diversity improves generalization: We reaf- firm the intuition that diversity of domains in the train- ing distribution is critical for both domain and compo- sitional generalization. However, CLIP only weakly generalizes, as there remains performance gap model that has seen similar samples during training.
This work presents a systematic study of CLIP's do- main and compositional generalization, unveiling both its capabilities and its limitations. For reproducibil- ity, our code is available at / com/ 1mb- fweeefrrruuggggr httgg//nddessdninnnnne el coipe--cciooooooo
Compositional generalization is challenging: Sur- prisingly, including a domain the training data does not always improve generalization to unseen classes within that same domain. However, ensuring sufficient class diversity within the test comainn-ddeale with overlap with the queried classes in avaluation-along with high domain diversity, can significantly reduce the aforementioned performance gap.
2. Related Work
OOD generalization of CLIP Unlike earlier models from the ImageNet era, CLIP models have been shown ex- hibit remarkable OOD generalization (Radford et al., 2021; Geirhos et al., 2021). Subsequent work has further enhanced
#Interconnected internal model mechanismsscomppoentt for performing specific computation task (Olah al., 2020).
Generalization requires sufficient feature and cir- cuit sharing: When CLIP generalizes well composi-
2

---

When and How Does CLIP Enable Domain and Compositional Generalization?
this capability by introducing stylistic diversity or encourag- ing the learning of domain-invariant representation (Huang Bose et al., 2024; Addepalli et al., 2024; Yu et al., 2024). These advances in OOD generalization have sparked research to uncover the underlying factors behind CLIP's strong OOD generalization.
(general) visual representation learning (Misra et al., 2017; Schott et al., 2022; Saranrittichai et al., 2022).
et al., 2023;
3. Problem Setup
Recent work highlighted that CLIP is trained on a substan- tial amount of non-natural images (Mayilvahanan et al., 2025, Table 2). but the role of these non-natural images in enabling CLIP's OOD generalization remains unclear. In this work, we aim to address this question by systemati- cally analyzing the effect of different domain mixtures in CLIP's training data (see Figure 1), allowing us to study CLIP's ability to generalize to entirely unseen domains (do- main generulization) and to novel combinations of known domains and classes (compositional generalization).
Fang et al. (2022) showed that CLIP's diverse training Distribtioon-ratae than its dataset size, language supervi- sion, or contrastive loss- is the primary factor of its OOD generalization. Similarly, Nguyen et al. (2022) found that dataset quality outweighs quantity. However, the precise characteristics of the training distribution contributing to CLIP's s generlization performance remained unclear.
While high train-test similarity was initially believed to be a key factor, Mayilvahanan et al. (2024) found its impact to be smaller than expected. Instead, other factors, such as the class distribution, were shown to play a more important role (Wen et al., 2024). Moreover, caption richness has been found to enhance CLIP's robustness (Xue et al., 2024; Wen et al., 2024), and CLIP's loss fosters the learning of dis- entangled representations, facilitating the generalization to unseen attribute-objeet combinations (Abbasi al., 2024). At the same time, other work revealed that CLIP exhibits behaviors resembling those of supervised classifiers. For example, CLIP fails to generalize when all non non-natural im- ages are removed from its training data (Mayilvahanan et al., 2025), and CLIP can be vulnerable to spurious correlations (Wang et al., 2024). While these works have significantly advanced our understanding of CLIP's OOD generalization, key questions remain. For example, "Can CLIP general- ize to an entirely unseen domain?" ? (domain generaliza- tion), "Can CLIP generalize to unseen class-domain combi- nations?" (compositional generalization), and which factors contribute to such generalization?
Notations Let Dg denote the base domain mostly con- sisting of natural images with image-text pairs (7º: T (red in Figure 1). Further, we consider m non-natural do- mains D, for r {1...., m) with image-text pairs ( IT) (blue, green, orange). Lastly, we consider the object classes C = fc,....,c,. cx for the images I which we divide into two disjoint subsets C₁ = {cI,-..,cx) (squares
and C2 = (c&+1..... Cn (circles ). We denote the subset of Dr that contains only classes from C" as D.
Training data setups Below, we specify the four training data setups; see Figure I for visual overview.
Natural-only (lower bound): We train only on our base image domain Do (e.g.. ImageNet- (Fang et al., 2022), see Section 4 for further details), consist- ing (almost) exclusively of natural images of all classes This condition serves as our lower bound and mirrors the training distributions considered in Fang et al. (2022); Mayilvahanan et al. (2025).
C
OOD generalization beyond CLIP The study of OOD generalization has been a focal point in the recent machine learning literature, covering various learning setups (refer to Table 2 of Gulrajani & Lopez-Paz (2021) for a compre- hensive overview). In this work, we focus on two specific setups: domain generalization and compositional general- ization. In domain generalization, models are trained on multiple domains and evaluated on an entirely unseen do- main (Blanchard et al., 2011; 1; Muandet et al., 2013; Gulra- jani & Lopez-Paz, 2021). This is known as "learning from multiple environments" in the causality literature (Peters et al., 2016; Arjovsky et al., 2019; Arjovsky, 2019; Richens & Everitt, 2024). Compositional generalization, on the other hand, examines whether models generalize to unseen com- binations of factors which were seen separately in training. Compositional generalization was recently studied for, e.g., (causal) generative models (Atzmon et al., 2020; Okawa et al., 2023; Wiedemer et al., 2023), object-centric models (Wiedemer et al., 2024), disentangled (Xu et al., 2022), and
. Leave-out-domain (domain generalization): We train on a variety of domains but hold out the test domain Dj: Ufzs Dj U Do lowing the classical domain generalization learning setup (Blanchard et al., 2011; Muandet et al., 2013; Gulrajani & Lopez-Paz, 2021). Note we could also in- terpret this as extrapolation in certain cases, i.e. does CLIP generalize to data outside the domain coverage seen during training?
); fol-
Definition 3.1. A model compositionally generalizes if it can accurately classify any new combinations of seen factors-here, classes and domains seen together dur- ing training.
Following this definition, we construct the training data setups for Compositional Generalization (CG) as follows:
CG low-diversity: We train on the base domain Do and a subset of classes C1 of the test domain D,: Do D.C
3

---

When and How Does CLIP Enable Domain and Compositional Generalization?
CG high-diversity: We train on the base domain Do diverse set of other domains Diti == Uffz / Dj con- taining all classes C, and subset of classes Ci of the test domain D,: Do U DJJ JDD D
Captions as base dataset. We discuss the results and key findings below.
To ensure the validity of our results and findings, we val- idated them across several alternative choices: (1) base datasets (CC3M, CC12M), (2) vision encoders (ViT-S-32, Swin-T), (3) contrastive loss choices (SigLIP: Zhai et al., 2023), and (4) vision-laguuage pre- pre-training methods (BLIP; Li et al., 2022). These additional results, which are consis- tent with those in Figure 2, are provided in Appendix B.1.
Test data Our test data for all training data setups consists of the same novel combinations of the classes C2 of the test domain Dj: D.= ). By keeping the test set fixed throughout all conditions, we ensure comparability across these setups.
The role of domain diversity By constructing domain mixtures and controlling for all other factors, such as dataset size or model choice, we are able to isolate the impact of domain diversity in Figure 2. In particular, Figure 2 reaffirms the hypothesis that domain diversity is key factor in enhancing CLIP's generalization: CLIP achieves both significantly better domain and compositional generalization in settings with high domain diversity (Leave-out-domain, CG high-diversity) compared to settings with low domain diversity (Natural-only, CG low-diversity).
4. Experimental Setup
Datasets We used either ImageNet-Captions (Fang et al., 2022), CC3M (Sharma et al., 2018), or CC12M (Chang- pinyo et al., 2021) as our base image-text datasets Do (red in Figure 1). Captions in Amageeet-Caationss are constructed using the title, tag. and description (if provided) to yield maximal descriptiveness. To mitigate the influence of class distribution shift, we augmented the base datasets with natu- ral samples from DomainNet-PR Real (Peng et al., 2019).
Does each domain contribute equally? While these re- sults demonstrate that domain diversity is critical for the (compositional) generalization of CLIP, they do not provide insights into the importance of each domain individually, since all domains are added at once. Thus, we successively added domains to CG low-diversity until arriving at CG high diversiivtt to assess their importance.
For the domain-specific image-text pairs D, (blue, green, orange in Figure 1), we used DomainNet's non-natural domains: Clipart, Infograph, Painting, Quickdraw, and Sketch (Peng et al., 2019). Since DomainNet provides captions, we created captions by using domain-invariant templates (e.g., an image of a {class}) or domain- specific templates (e.g., a {domain} of a (class)); see Appendix A.2 for further details. We used comparable final training dataset sizes across our different training con- ditions; refer to Appendix A.3 for details. Finally, the class choices for C and C2 are provided in Appendix A.4.
Figure 3 shows that some domains contribute more strongly to generalization, while other domains can even slightly reduce generalization performance on the test domain. In- triguingly, the impact of domain can change, depending which other domains are already included. For exam- ple, in Figure 3a, while Sketch and Quickdraw are the most beneficial domains to include initially for improving gener- alization performance on Clipart, once Sketch is added, it is more effective to first include the other domains (Paint- ing, Infograph) before adding Quickdraw. We hypothesize that this is due to the visual similarity between Sketch and Quickdraw- both consist of black-and black-and-whiteegrray white/gray images with object contours which may lead the model to learn similar features. Once such features are learned from one domain, the benefit of adding the other diminishes. Further investigation of this observation is left for future work.
Evaluation of CLIP models We evaluated CLIP mod- els in the classical zero-shot classification setting across all DomainNet classes C using the standard OpenAl tem- plates (Radford et al., 2021), extended with templates for the missing domains of DomainNet; see Appendix A.5 for further details. To mitigate the effect of class imbalance, we calculated the balanced top-1 I accuracy, which we will hereon refer to top-1 accuracy for brevity.
5. When Does CLIP Exhibit Domain and Compositional Generalization?
Overall, these observations suggest that while domain diver- sity is generally beneficial for generalization, the specific relationship between the added domain(s) and the test do- main plays critical role.
In this section, we trained CLIP models on our systemati- cally constructed training data setups, as described in the previous sections and illustrated in Figure I (refer to Ap- pendix A.6 for training details), to investigate when CLIP achieves domain generalization and compositional general- ization. Figure2 summarizes the results for CLIP models with 5e vision encoder and trained on ImageNet-
Domain diversity enhances domain gener- alization and compositional generalization. However, the relationship between domains also matters.
Finding 1:
4

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Natural-on only
Leave Leave-out-domes out-domain
CG low www.divenstyy
CG high-diveriity
Perfect generalization
Inlog dlocces bgp-))
13
Puinting Cussees clasees top-1)
Quicklow iussen top-))
12
Skesch (umoen claseo top- 11
23-
Clipan clasees
10
10-
10p-1)
Real
0op-11
(xp-13
Real (tug-3)
Figure 2: High diversity domain mixtures exhibit improved effective robustness. Each point represents the average balanced top-1 accuracy over three consecutive training epochs and three seeds, with higher opacity indicating later training epochs. High diversity domain mixtures (Leave-out-domain (red), CG high-diversity (purple)) have consistently higher generalization performance than their low diversity counterparts (Natural-only (blue), CG low-diversity (green)). These gains are especially pronounced in the Clipart and Sketch domains (leftmost and rightmost). However, for the Quickdraw domain (4th from left), generalization fails even in the high diversity settings-a limitation we will further investigate in Section 6.2. Other evaluation metrics (balanced top-5 accuracy and F1 score) yield consistent results (see Figure 8 in Appendix B).
Clipurt (unseen top-1)
Skerch (unseen classes, tap-1)
25
20
20
15
Clip
Inl Infogppp
Paining
Quidkltoee
Infograph
Paining
Quidklaas
Quidklzae
Quickilae
Shetch
Infgraph
Painiing
Quickioze
In Engraph
Paiating
Quickizan kinee
Inferrph
Quickitoo
Quickiton
Natual
Cliport
Skotch
+Paiming
-Inoogrph
Natural
Sketch
Clipart
Infograph
(a) Clipart.
(b) Sketch.
Figure 3: Certain domains contribute more strongly to generalization performance than others. When domains are added successively from CG low-diversity to CG high-diversity (from left to right), some domains significantly improve generalization performance (balanced top-1 accuracy) on the unseen classes of the test domain, while others provide only small gains or even slightly degrade it.
How close CL.IP the maximally achievable perfor- mance? In our previous experiments, observed that CLIP generalizes well given sufficient domain diversity. However, how close is CLIP to the maximally achievable performance if it had been trained on class-specific samples from the test domain?
ingly reveals the opposite: CLIP models trained on a subset of classes from the test domain (CG low-ttigh-diversio are often slightly outperformed by models that have not seen the test domain at all (Natural-only, Leave-out-domain)
Compositional generalization can be weaker than domain generalization.
Finding 2:
Table shows that, even in high diversity settings with better generalization, gap to this upper bound performance remains. We analyzed the number of test class samples required close this gap in Appendix B.3. We found that the number of required samples tends to scale linearly.
This finding surprising, since the test domain is entirely unseen and domain generalization can require extrapola- tion. In contrast, compositional generalization is expected to perform better as has partial exposure that domain. However, our results suggest that this intuition may not always hold.
Achieving good compositional generalization is challeng- ing While high diversity improves generalization, one may intuitively expect CLIP to generalize better to unseen classes within test domain it has seen some other classes of that domain during training. However, Figure 2 surpris-
To better understand this, we investigated the role of the test domain's 's chosen classes for training and the ones that are queried during evaluation. For example, CLIP may learn the shortcut that all sketches belong to the subset of the
5

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Table I: There is performance gap when CLIP has seen class-specific samples of the test domain. With suffi- cient domain diversity, CLIP generalizes well unseen the classes C2 of the Clipart and Sketch test domains D, How- ever, even in these cases, there remains gap in balanced top-1 accuracy compared models also trained on domain- specific samples of the classes, i.e., D. Appendix B.3 provides the results of the other domains.
Skatch (unsomm chasess top-
-00
Lonoveichess (LLI9)
Saamdoely ISLIP
CD londratti (CUP
23-
Satenloety (99)
CG tre IS9)
tudiebheaate ISV)
20
Perlet spaeniizake
Pelks pokcniizoon
60
50
-0
60 0
20
40
Real (top-3)
Rcal (top-1)
Training data setup (Figure 1)
Clipart
Sketch
Figure 4: Effective robustness of CLIP vs. supervised classifiers. Similar to CLIP, the robustness of supervised classifiers also increases with the domain diversity of the training data. However, CLIP consistently shows superior balanced top-1 accuracy. Refer to Appendix B.4 for the results on the remaining domains.
Leave-out-domain
27.4
30.1
CG high-diversity
27.6
28.1
w/ classes C2 (upper bound)
36.6 (+9.0)
44.3 (+16.2)
Table Seeing a subset of classes of the test domain can worsen compositional generalization. Including sub- set of sketches from DomainNet (CG low-thigh-diversityy) slightly decreases balanced top-1 accuracy on the unseen sketch classes C2 compared to not seeing that domain at all scave-out-domainn However, adding sketches of classes that do not overlap with DomainNet's classes, instead im- proves compositional generalization performance, suggest- ing that compositional generalization can be limited by (only) partial, suboptimal inclusion of the test domain.
Table 3: Domain diversity increases feature sharing in the embeddings. We report the percentage point increase in top-k shared SAE features, averaged over classes C2 and k {5, 10, 15, 20} (see Appendix C.I for details on the computation of shared features), when comparing low diversity (Natural-only) to high diversity (Leave-out- domain, CG high-diversity). Note that we used the CLIP models using CC12M as base dataset, since we found that SAEs extracted poor features for the models trained on ImageNet-Captions.
Training data setup (Figure 1)
Sketch
Natural-only
19.5
Clipart
Sketch
Leave-out-domain
30.1
Natural-only - Leave-out-domain +7.1
+4.1
CG low-diversity
19.2
Natural-only - CG high-diversity
+6.9
+3.4
w/ sketches of non-queried classes only
27.1 (+7.9)
CG high-diversily
28.1
w/ sketches non-queried classes only 36.9 (+8.8)
The role of language supervision To investigate the in- fluence of language supervision, we replicated our previous experiments with supervised ResNet50 classifier. Specifi- cally, trained the classifier the combined class distri- bution of ImageNet and DomainNet. Refer to Appendix A.7 for further training details.
seen classes, which becomes wrong for the unseen classes queried in evaluation. To test this hypothesis, we replaced DomainNet's sketches from the classes C1 in our previous CG settings with sketches from set of classes C3 which disjoint from C U22 and not queried in evaluation. For this, used -eakecht (Wang et al., 2019) and excluded all ImageNet classes that overlap with the classes from DomainNet; refer to Appendix A.3 for details on the class overlap.
Figure 4 shows that generalization performance of super- vised classifiers increases with domain diversity, similar to CLIP. However, CLIP consistently exhibits higher perfor- mance compared to the classifiers, which can be attributable to caption richness (Xue et al., 2024; Wen et al., 2024).
6. Why Does CLIP (Not) Generalize?
Table confirms that including sketches of non-queried classes improves compositional generalization. However, it also highlights a failure mode: while compositional gener- alization work for CLIP (to some extent, c.f., Table 1), partial exposure to classes of the test domain that overlap with the classes queried in evaluation significantly wors- ens compositional generalization. We further analyzed the severity this overlap on compositional generalization per- formance in Appendix B.2.
In this section, we investigate which changes in the CLIP model led to domain and compositional generalization, or lack thereof, observed in the previous section.
6.1. The Role of Visual Embeddings
Intuitively, we would expect CLIP to share more features in its visual embeddings across domains as generalization im-
6

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Table 4: Domain-speciffc captions do not explain CLIP's
UMAP component 2
UMAP componeet
Quickdraw Others
poor generalization performance to unseen Quickdraw classes in the CG high-diversity setting. When all domain- specific captions are replaced (second row), we observe increased alignment in the visual embeddings (Figure 5b); yet, this does not translate into improved generalization performance, as measured by balanced top-1 accuracy.
10
-10
10
UMAP component
UMAP component
Captions
Classes
(a) Nomain-invariant and spe- (b) Only domain-invariant cap- cific captions.
tions.
domain-specific
domain-invarian
seen
unseen
Figure 5: The separation of the visual embeddings be- tween Quickdraw and the images of other domains is due domain information in captions. However, the better alignment of the visual embeddings does not improve compositional generalization (Table 4).
50%
50%
50.7
1.7
0%
100%
46.2
0.7
proves. To test this hypothesis, we applied an unsupervised dictionary learning technique, i.e., Sparse Autoencoders (SAEs) (Bricken et al., 2023; Huben et al., 2024), to ex- tract interpretable features from CLIP's visual embeddings E I
sify the Quickdraw images of seen classes, indicating that is has learned something useful for the Quickdraw domain, yet CLIP performs poorly images of unseen classes.
SAE(a): == (g o @ (a),
(1)
where & is a ReLU non- non-linearity, and f and are the lin- ear encoder with weights W, E R and decoder with weights Wg 9 Rxx,, respectively. We trained the SAE with L2 reconstruction loss and L, sparsity regulariza- tion. Refer to Appendix C.1 for further technical details.
Are domain-specific captions the cause? We initially hy- pothesized that using domain-specific captions-our train- ing included both domain- innariann and domain-specifer captions (Section 4) 4)-might have inadvertently led CLIP to prioritize uniformity over alignment in its loss func- tion. We suspect this occurs due to the stark visual dif- ferences between Quickdraw images and those from other domains, making aligmment challenging. In order to mini- mize the total loss in spite of this, CLIP may adopt a short- cut: prioritizing uniformity, aligning Quickdraw image em- beddings with the text embeddings of the domain-specifer captions, while sacrificing alignment with Nomain--nvarian captions. Consequently, we would expect a clear separation between Quickdraw image embeddings and those from other domains- indeed, we observe this (Figure 5a). How- ever, this separation may come at a cost, limiting CLIP's ability to generalize across the Quickdraw domain, particu- larly to unseen classes.
After extracting interpretable SAE features, we measured the percentage overlap of the top-k most important SAE fea- tures between domain pairs for each class c E C2. Specifi- cally, to identify important SAE features, we counted how often each SAE feature appeared among the top-20 highest- activating SAE features across domain-class combinations. For each class, we then calculated the percentage overlap of the top-k (with & {5, 10, 15, 20}) of these top-20 most frequent SAE features between the test domain and each of the other domains. These percentage overlaps were fi- nally averaged over all values of k, all classes in C2, and all pairs that include the test domain. Further details on this computation and pseudocode are provided in Appendix C.I.
As a remedy, we trained CLIP using only domain-invariant captions. Although the visual embeddings now better aligned (Figure 5b), generalization performance remains poor (second row of Table 4). Thus, the domain invari- ance or specificity of captions cannot solely explain CLIP's poor generalization performance on unseen classes from Quickdraw.
As shown in Table 3, CLIP models trained on more diverse generalize better-exhibij more cross-domain feature sharing in their visual embeddings.
Finding 3: CLIP shares more features across domains in its embeddings as generalization improves.
6.2. Why Does CLIP Sometimes Fail To Generalize?
The role of shared intermediate features and circuits While visual embeddings appear aligned in Figure 5b, it does not necessarily imply that the computations leading up to these embeddings are also aligned. Thus, we hypothesize that insufficient sharing of intermediate representations and circuits (Hohman et al., 2019; Olah et al., 2020) within
Figure 2 shows that CLIP typically generalizes well compo- sitionally with high domain diversity, except for Quickdraw. is tempting to attribute this solely to the unique character- istics of Quickdraw images (Appendix A. However, the first row of Table 4 highlights that CLIP can correctly clas-
7

---

When and How Does CLIP Enable Domain and Compositional Generalization?
CLIP's vision encoder may explain its poor generalization to the unseen classes. For example, if CLIP learns separate circuit for Quickdraw images instead of sharing sufficient functionality across the domains, such a circuit may work for seen classes but will fail on unseen classes, since the unseen classes of the test domain can only be inferred with the help of the other domains.
Quickdrw
0.6
CKA similarity
0.G
Others
Jaccard index
0.4
0.2
0.0
Layer
Layer
(a) Representational similarity. (b) Amount of shared neurons.
To test our hypothesis, we used tools from representational similarity analysis to evaluate intermediate representations. Additionally, we introduce novel, related concept for cir- cuits: mechanistic similarity, which measures the similarity between circuits.
Quickdraw
Others
Weisfeiler-Lchman similarity (t)
0.218
0.28
(c) Circuit similarity.
Representational similarity analysis To assess whether CLIP learns similar intermediate representations for the same class across domains, we measured representational similarity using Center Kernel Alignment (CKA) (Kornblith et al., 2019) with the unbiased Hilleet-Schmiie Indepen- dence Criterion (HSIC) estimator (Song et al., 2012); refer to Appendix C.2 for technical details.
Figure 6: CLIP's intermediate representations and cir- cuits show a significant difference between Quickdraw and the other domains, even though final visual embed- dings can be aligned. We used the CLIP model trained on only domain-invarian captions from Figure 5b and Table 4 (second row). Scores are averaged over classes and higher scores mean higher representational similarity (a), more shared neurons (b), or more similar circuits (c).
Mechanistic similarity analysis While representational similarity captures alignment at the representational level, it does not reveal whether similar computations are imple- mented internally. To address this, we introduce a measure of mechanistic similarity, which quantifies the similarity of the underlying circuits, i.e., the sets of model components and their interactions responsible for a specific prediction. Our analysis proceeds in two steps:
captures more complex structural and hierarchical sim- ilarities. More details are provided in Appendix C.3.
Result Figure 6 confirms that the Quickdraw domain no- ticeably differs from the other domains, both in terms of representational and mechanistic similarity, supporting our hypothesis that (compositional) generalization requires suf- ficient sharing of intermediate features and circuits.
1. Circuit discovery: For each class-domain pair, we ex- tracted a circuit represented as directed acyclic graph. We first identified the top-k model components- specifically, axis-aligned neurons-based on their in- direct effect² (Pearl, 2001) on the model's predictions, following prior work (Vig et al., 2020; Schrodi al., 2022; Meng et al., 2022; Marks et al., 2025). These components serve as the nodes of the circuit. Next, for each component, we identified its k most influential predecessors (also based on indirect effect), which de- fine the edges of the circuit. Refer to Appendix C.3 for further details on circuit discovery.
Finding 4: Sufficient sharing of intermediate features and circuits is crucial for generalization to succeed.
7. Discussion
Compositional generalization Our results indicate that CLIP has (weak) ability for compositional generalization, which is influenced by the composition of its training data. Specifically, CLIP's compositional generalization perfor- mance worsens when a subset of classes of the test domain, that is queried during evaluation, is seen during training (Table 2). However, by including only classes that do not overlap with the queried classes in the training data, CLIP's compositional generalization significantly narrows the gap to the maximally achievable performance. In particular, per- formance unseen sketch classes significantly improved from 19.5% (Natural-only) to 36.9% (CG high-diversity w/ non-queried classes only. Table 2), approaching the max- imally achievable performance of 44.3% (Table 1). This result supports the hypothesis that compositional generaliza- tion could be indeed driver behind CLIP's generalization.
2. Circuit similarity: We then compared circuits across domains for each class-for example, comparing the circuit for "dog" of Quickdraw images vs. those of Cli- part, Infograph, Painting, Real (natural images), and Sketch images. To quantify similarity, we computed the layer-wise Jaccard index of the top-k components, capturing node overlap. To assess deeper structural sim- ilarity, we applied the normalized Weisfeiler-Lehmall subtree graph kernel (Shervashidze et al., 2011), which
Intuitivvlyy, the indirect effect quantifies how much each com- ponent contributes to the model's prediction.
8

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Our results also allow to reinterpret the main experiment from Mayilvahanan al. (2024, Table 1). In their experi- ment, the maximally achievable performance corresponds to the unaltered LAION-200M dataset, while the pruned ver- e.g., (near) duplicates of all
dent that our findings remain robust across dataset sizes, as demonstrated by consistent results in all three base datasets, with sizes ranging from approximately 0.5 M to 10 M sam- ples, in Appendix B.1.
sions of WA-1N--000---- ImageNet-Sketch classes were removed-can be interpreted analogous to the curated compositional generalization set- ting with high diversity from above. They observed only slight deterioration in performance despite these exclu- sions. Our result suggests that compositional generalization could be the key contributor of the sustained performance observed here, though further analysis is required to exclude other (unknown) factors.
Dataset quality Besides diversity, the quality of the datasets used for training also likely plays crucial role in achieving robust generalization. For example, Nguyen et al. (2022) showed that combining multiple datasets does not necessarily result in improved robustness; in fact, the ro- bustness of the best-performing dataset can even be diluted by lower-quality datasets. To date, dataset quality remains only loosely addressed in much of the current literature, largely due to the difficulty to quantify it. Thus, we leave further investigation of this aspect for future work.
Rethinking CLIP's OOD generalization Recent studies have concluded that CLIP's OOD generalization is largely driven by its vast and diverse training distribution (Fang et al., 2022; Mayilvahanan et al., 2025). Our experiments reaffirm this: While CLIP can weakly generalize (Figure 2), generalization gap persists when compared models that have seen class samples of the test domain(s) during training (Table 1). We believe it is worth studying how narrow this generalization gap can become and if it can ever be closed without access to (near) duplicate samples during training- both from empirical and theoretical perspectives.
Carbon emission estimate We conducted our experi- ments mainly on NVIDIA RTX 2080 GPUs and estimated the total GPU hours to be approximately 25 000. With car- bon efficiency of 0.321 kgCOpeq/kWh² total emissions are estimated to be about 1 725 kgCOzeq, using the Machine Learning Impact calculator (Lacoste et al., 2019).
8. Conclusion
In this work, analyzed when CLIP generalizes to unseen domains and subsets thereof using systematically created training data setups (see Figure 1), identifying domain di- versity as a prerequisite for both domain and compositional generalization. We showed that compositional generaliza- tion can fail in certain scenarios, i.e., even perform worse than domain generalization. We supported these findings with in-depth data-centric experiments and mechanistic anal- yses, offering insights into the internal workings that make generalization succeed or fail.
Mechanistic similarity analysis Mechanistic similarity analysis (Section 6.2) a general framework that first dis- covers circuits and then measures their similarity. In future work, we plan to apply this framework other problems, e.g., to investigate whether multi-lingua LLMs process dif- ferent languages in similar or very different ways.
Larger dataset sizes In this work, we disentangled the effects of dataset size and domain diversity- factors that are typically confounded and showed that domain diversity is a key driver of CLIP's generalization. However, dataset size alone may also impact CLIP's generalization performance.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
At the time of writing, the best of our knowledge, large-scale domain datasets comparable to the size and di- versity of DomainNet exists. To address this, future work could filter a large, diverse dataset, such as LAION (Schuh- mann et al., 2021), by domains. For example, the data filtering approach proposed by Mayilvahanan et al. (2025), which classifies images into natural and non-natural, could be extended classify various visual domains. Note that the existing filtered natural-only subset of LAION-200M curated by Mayilvahanan et al. could be directly adopted as the base dataset Do-
A Acknowledgements
This research was funded by the the German Research Foun- dation (DFG) under grant numbers 41 417962828, 539134284 and 499552394 (SFB 1597).
We would like to thank Evgenia Rusak, Thaddaus Wiedemer, and Prasanna Mayilvahanan for an insightful discussion.
Although, this would help overcome the searcity of domain- specific data, we lack the computational resources to con- duct such large-scale experiments. However, we are confi-
'Estimated carbon efficiency of the German power grid in 2024 according to https: / //www nowtri city. com/ country\ germany/.

---

When and How Does CLIP Enable Domain and Compositional Generalization?
References
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. In CVPR. 2023.
Abbasi, R., Rohban, M. H., and Baghshah, M. S. Decipher- ing the Role of Representation Disentanglement. Investi- gating Compositional Generalization in CLIP Models. In ECCV, 2024.
Devlin, J., Chang, M.-W., Lee. K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding. In NAACL, 2019.
Addepalli, S., Asokan, A. R., Sharma, L., and Babu, R. V. Leveraging Vision-Language Models for Improving Do- main Generalization Image Classification. In CVPR, 2024.
Dosovitskiy. A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.
Arjovsky, M. Out of Distribution Generalization in Machine Learning. Phd thesis, New York University, 2019. Avail- able at https::///.... a http:/aaxi...grga/////a///////// 02667.
Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L. Data Determines Distributional Robustness in Contrastive Language-Image Pre-training (CLIP). In ICML, 2022.
Arjovsky, M., Bottou, L., Guirajani, I., and Lopez-Paz, D. Invariant Risk Minimization. arXiv, 2019.
Atzmon, Y., Kreuk, F,, Shalit, U., and Chechik, G. A causal view of compositional zero-shot recognition. In NeurlPS, 2020.
J. F., Loftus, A. R., Todd, E., Brinkmann, J., Pal, K., Troitskii, D., Ripa, M., Belfki, A., Rager, C., Juang, C., Mueller, A., Marks, S., Sharma, A. S., Luc- chetti, F., Prakash, N., Brodley, C. E., Guha, A., Bell, J., Wallace, B. C., and Bau, D. NNsight and NDIF: De- mocratizing Access to Open- Foundation Model Internals. In ICLR, 2025.
Barthel, K. U., Hezel, N., Schall, K., and Jung, K. nav- igu.net: NAvigation in Visual Image Graphs gets User- friendly. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, 2023.
Blanchard. G., Lee, G., and Scott, C. Generalizing from Several Related Classification Tasks to a New Unlabeled Sample. In NeurlPS, 2011.
Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., and Brendel, W. Partial success in closing the gap between human and machine vision. In NeurlPS, 2021.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the Opportunities and Risks of Foundation Models. arXiv, 2021.
Gulrajani, and Lopez-Paz, D. In Search of Lost Domain Generalization. In /CLR, 2021.
Hanna, M., Pezzelle, S., and Belinkov, Y. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Find- ing Model Mechanisms. In Workshop @ ICML, 2024.
Bose, S., Jha, A., Fini, E., Singha, M., Ricci, E.. and Baner- jee, B. Stylip: Multi-scale style-conditioned prompt leamn ing for clip-based domain generalization. WACV, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In 2016.
Bricken, T., Templeton, A., Batson, J.. Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-oodds, Z., Tamkin, A., Nguyen, K. McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards Monosemanticity: Decomposing Language Models With Dictionary Leam- ing. Transformer Circuits Thread, 2023. https: put.ttt2.ppp//23233 wtwptnttiiieetttteeeeeeeeeeeeeee win.uureeeiiiie.eeeeee html.
Hohman, F., Park, H., Robinson, C., and Chau, D. H. P. Sum- mit: Scaling Deep Learning Interpretabilityy by Visualiz- ing Activation and Attribution Summarizations. TVCG, 2019.
Huang. Z., Zhou, A., Ling, Z., Cai, M., Wang, H., and Lee, Y. J. A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance. In ICCV, 2023.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12M: Pushing Web-Scale Image-Text Pre- Training To Recognize Long-Tail Visual Concepts. In CVPR, 2021.
Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse Autoencoders Find Highly Inter- pretable Features in Language Models, In ICLR, 2024.
10

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Hupkes, D., Dankers, V., Mul, M., and Bruni, E. Com- positionality Decomposed: How do Neural Networks Generalise? JAIR, 2020.
Mueller, A., Brinkmann, J., Li, M., Marks, S., Pal, K., Prakash, N., Rager, C., Sankaranarayanan, A., Sharma, A. S., Sun, J., al. The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal Interpretability. arXiv, 2024.
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT., Parekh, Z., Pham, H., Le, Q. Sung, Y.-H.. Li, Z.. and Duerig, T. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In ICML, 2021.
Nguyen, T., Raghu, M., and Kornblith, S. Do Wide and
Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. In ICIR, 2021.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Simi- larity of Neural Network Representations Revisited. In ICML,, 2019.
Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP. In NeurIPS, 2022.
Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. Quantifying the Carbon Emissions of Machine Learning. arXiv, 2019.
Okawa, M., Lubana, E. S., Dick, R., and Tanaka, H. Com- positional Abilities Emerge Multiplicativel:: Exploring Diffusion Models on a Synthetic Task. In NeurIPS, 2023.
Li, D., Li, J. Le, H., Wang. G., Savarese, S., and Hoi, C. "LAVIS: One-stop Library for Language-Viin Vision Intelligence". In ACL, 2023.
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom In: An Introduction to Circuits. Distill, 2020.
Li, J., Li, D., Xiong, C.,, and Hoi, S. BLIP: Bootstrap- ping Language-Iman Pre-training for Unified Vision- Language Understanding and Generation. In /CML, 2022.
Pearl, J. Direct and Indirect Effects. In UAI, 2001.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In ICCV, 2021.
Pearl, J. Causality. Cambridge University Press, 2009.
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment Matching for Multi-Source Domain Adaptation. In ICCV, 2019.
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. In ICLR, 2025.
Peters, J., Bühlmann, P., and Meinshausen, N. Causal In- ference by using Invariant Prediction: Identification and Confidence Intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2016.
Mayilvahanan, P., Wiedemer, T., Rusak, E., Bethge, M., and Brendel, W. Does CLIP's generalization performance mainly stem from high train-test similarity? In ICLR, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning Transferable Visual Models from Natural Language Supervision. In ICML, 2021.
Mayilvahanan, P., Zimmermann, R. S., Wiedemer, T., Rusak, E., Juhos, A., Bethge, M., and Brendel, W. In Search of Forgotten Domain Generalization. In ICLR, 2025.
Rao, S., Mahajan, S., Böhle, M., and Schiele, B. Discover- then-Name: Task-Agnostic Concept Bottlenecks via Au- tomated Concept Discovery. In ECCV, 2024.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and Editing Factual Associations in GPT. In NeurIPS, 2022.
Richens, J. and Everitt, T. Robust agents learn causal world models. In ICLR, 2024.
Miller, G. A. WordNet: lexical database for English. Communications of the ACM, 1995.
Saranrittichai, P., Mummadi, C. K., Blaiotta, C., Munoz, M., and Fischer, V. Overcoming Shortcut Learning in a Target Domain by Generalizing Basic Visual Factors from Source Domain. In ECCV, 2022.
Misra, I., Gupta, A., and Hebert, M. From Red Wine to Red Tomato: Composition with Context. In CVPR, 2017.
Schott, L., Kügelgen, J. V., Träuble, F., Gehler, P. V., Russell, C., Bethge, M., Schölkopf, B., Locatello, F., and Brendel, W. Visual Representation Learning Does Not Generalize Strongly Within the Same Domain, In ICLR, 2022.
Muandet, K., Balduzzi, D., and Schölkopf, B. Domain Generalization via Invariant Feature Representation. In ICMI., 2013.

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Schrodi, S., Saikia, T., and Brox, T. Towards Understanding Adversarial Robustness of Optical Flow Networks. In CVPR, 2022.
Wiedemer, T.,, Mayilvahanan, P., Bethge, M., and Brendel, W. Compositional Generalization from First Principles. In NeurlPS, 2023.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komat- suzaki, A. LAION-400M: Open Dataset of 400 Million Image-TText Pairs. Workshop NewrlPS, 2021.
Wiedemer, T., Brady, J., Panfilov, A., Juhos, A., Bethge, M., and Brendel, W. Provable Compositional Generalization for Object-Centric Learning. In ICLR, 2024.
Xu, Z., Niethammer, M., and Raffel, C. A. Compositional Generalization in Unsupervised Compositional Repre- sentation Learning: A Study on Disentanglement and Emergent Language. In NeurIPS, 2022.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Concep- tual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In ACL, 2018.
Xue, Y., Joshi, S., Nguyen, D., and Mirzasoleiman, B. Un- derstanding the Robustness of Multi-modal Contrastive Learning Distribution Shift. In ICLR, 2024.
Shervashidze, N., Schweitzer, P., Van Leeuwen, J., Mchlhorm, K., and Borgwardt, K. M. Weisfeiler-Lehman Graph Kemels. JMLR, 2011.
Yu, X., Yoo, S., and Lin, Y. CLIPCEIL: Domain Gen- eralization through CLIP via Channel rEfinement and Image-text aLignment. In NewrIPS, 2024.
Song, L., Smola, A., Gretton, A., Bedo, J., and Borgwardt, K. Feature Selection via Dependence Maximization. JMLR, 2012.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid Loss for Language Image Pre- Teairrainng In ICCV, 2023.
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic Attribu- tion for Deep Networks. In ICML, 2017.
Szabó, Z. G. The Case for Compositionality. The Oxford Handbook of Compositionality, 2012.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image trans- formers & distillation through attention. In ICML, 2021.
Udandarao, V., Prabhu, A., Ghosh, A., Sharma, Y., Torr, P. Bibi, A., Albanie, S., and Bethge, M. No "Zero- Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance. In NewrIPS, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u. and Polosukhin, Atten- tion All You Need. NeurlPS, 2017.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurlPS, 2020.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning Robust Global Representations by Penalizing Local Pre- dictive Power. NeurlPS, 2019.
Wang, Q. Lin, Y., Chen, Y., Schmidt, L., Han, B., and Zhang, T. A Sober Look at the Robustness of CLIPs to Spurious Features. In NewrlPS, 2024.
Wen, X., Zhao, B., Chen, Y., Pang. J., and Qi, X. What Makes CLIP More Robust to Long-Tailen Pre- Training Data? Controlled Study for Transferable Insights. In NeurlPS, 2024.
12

---

When and How Does CLIP Enable Domain and Compositional Generalization?
A. Further Details for Section 4
A.1. DomainNet Examples
Figure visualizes four examples from all six image domains (Clipart, Infograph, Painting, Quickdraw, Real, Sketch) from DomainNet (Peng al., 2019). Some domains are more visually similar than others. For example, sketches and Quickdraw images are typically gray, while paintings often contain similar level of image detail similar that of Real (natural) images.
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
aircraft carrier
A
axe
banana
BANANA
LION
lion
0
Figure 7: Random examples across the six domains DomainNet.
A.2. DomainNet Training Captions
The DomainNet dataset (Peng et al., 2019) does not include captions. Therefore, we created captions using the class names to train CLIP models on DomainNet data. For this, we used prompt templates similar Radford et (2021). Specifically, we used the following templates:
{domain) of {class).",
{class} [domain].".
{domain) depicting (class).",
(class} depicted in [domain].",
{domain) showing {class}.",
(class} is visible in {domain).".
For each DomainNet image sample, we randomly sampled one of the templates and inserted the corresponding class name for the placeholder (class). For the (domain) placeholder, we randomly chose with equal probability generic, domain-invann invain-invariint term, such as image or picture, or domain-speciffe term, such as clipart or painting. The terms are provided in Table 5.
13

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Table 5: Generie, domain-invariant and domain-specific terms. Generic, domain-invariant terms are shared across domains, while domain speciicc terms can be either its domain name or a synonym.
Domain
Terms
Generic
image, picture
Clipart
clipart, illustration infograph, informational chart
Infograph
Painting
painting, art quickdraw, doodle
Quickdraw
Real
photo, snapshot sketch, drawing
Sketch
A.3. Further Details on the Training Data Construction
Dataset Construction We created our training datasets based on a base dataset Do (i..,, ImageNet-Captionn CC3M, or CC12M) which provides a large collection of (mostly) natural images along with corresponding language descriptions. To mitigate effects stemming from to class shifts, we augmented the base dataset with samples from DomainNet-Real, which consists of natural images. Finally, from various non-natural domains of DomainNet D, with E {Clipart, Infograph, Painting, Quickdraw, Sketch) into the training data, as outlined in Section 3.
created different domain mixtures (see Figure 1) by incorporating subsets of samples
Subsampling To ensure fair performance comparisons between the different training data setups (see Figure 1), we applied subsampling to maintain comparable final dataset sizes when adding additional domains to the training data. We designed our subsampling method to preserve the original data distribution as much as possible. Specifically, domain D; contained twice as many samples as domain D, before subsampling, this ratio remained approximately constant afterward. Likewise, the class distribution within each domain was preserved much as possible.
Note that the non-natural domains in DomainNet vary significantly in size, e.g., Quickdraw has more than three times as many samples as Clipart. Thus, we chose to only keep dataset sizes fixed within the same test domain. That is, for a given test domain, CG low-diversity, CG high-diversity and Leave-out-domain datasets have the same size. However, dataset sizes may differ across test domains (i.e., CG low-diversity settings for different test domains are not necessarily of equal size). Since the Natural-only lower bound is independent of the choice of the test domain, contains slightly fewer samples than the other mixtures.
Joining the Class Distributions of ImageNet and DomainNet Both ImageNet-Caption and DomainNet provide class labels for training supervised classifiers. However, their class distributions differ significantly in diversity and granularity, ImageNet has nearly three times as many classes as DomainNet and is more fine-grained. For example, ImageNet distinguishes over 100 different dog breeds, whereas DomainNet has only a single dog class.
To address this, we created a mapping from ImageNet classes to DomainNet classes. Each ImageNet class was either mapped to single DomainNet class or left unmatched, and multiple ImageNet classes could be mapped to the same NomainNet class. We constructed this mapping manually based on class names, the WordNet hierarchy (Miller, 1995), and the NAVIGU image explorer (https: / //avvgguu... net/ imagenet, Barthel al. (2023)), ensuring that only semantically valid mappings were retained.
Our final mapping assigned 450 ImageNet classes to DomainNet, including 147 one-to-one mappings. Using this mapping, we merged the class distributions of ImageNet and DomainNet. Specifically, we relabeled the 450 mapped ImageNet classes with their corresponding DomainNet labels.
A.4. Choice of Classes C2
We carefully chose the subset C2 in a way that the classes are diverse and not biased towards any spurious features (e.g., color). We considered the 147 classes, with a one-to-one match in ImageNet (see Appendix A.3), as the possible candidates for C2. We selected about 10% of these candidates, i.e., 15 classes. For the selection process, we randomly sampled from
14

---

When and How Does CLIP Enable Domain and Compositional Generalization?
the of candidates. To ensure that adequately covered the different super-categories of DomainNet (e-g., furniture, mammal, tool), we kept only the first random sample for each category, rejecting further samples from the same category. We also manually rejected some samples if we considered them to be too similar to our existing selection. Our final selection of classes is
C2 {aircraft carrier, axe, banana, barn, bed, candle, lion, mountain, necklace, penguin, pizza, saxophone, television, tractor, traffic light).
(2)
and Cj are the remaining 330 classes of DomainNet.
A.5. DomainNet Evaluation Prompts
We evaluated the zero-shot performance of our CLIP models using the OpenAl templates from Radford et al. (2021). Since Painting and Sketch templates are already contained, we added the templates for the missing domain names:
"a clipart of the fclass).".
"a clipart of a {class}.".
"an infograph of the [class}.",
"an infograph of a (class).",
"a quickdraw of the {class}.",
"a quickdraw of (class).".
Following Radford et al. (2021), we created zero-shot weights for all 345 DomainNet Classes from these templates by taking the class-wise average over the text embeddings of all templates (marginalization to obtain the "true" object embedding) and normalizing afterwards. Formally, let {-,,..., Cr } be class, T be the set of templates, Le template with the name of class inserted, and g be the text encoder of our CLIP model. Assuming that the text encoder g produces fzrnoomal embeddings, we compute the zero-shot weights of class as:
पें Eter
(3)
Eier
A.6. CLIP Training Details
Following Fang et al. (2022), we trained CLIP models with an embedding size of 1024 with ResNet-50 (He al., 2016) and transformer text encoder (Vaswani et al., 2017) (12 layers with width of 512, attention heads, and context length of 77). We trained the models for 32 epochs with a batch size of 1024 with Adam (learning rate of 0.001, 31 0.9, 8g 0.999, == 1c-8, weight decay of 0.2) and cosine annealing learning rate scheduling with 500 warmup steps. We used the code from OpenCLIP (Cherti et al., 2023) (https: / /igttt tiu........ com. mm1 foundat iinattioon/// /ope/ clip, License: custom) for our training implementation, including OpenCLIP's default augmentations.
A.7. Supervised Classifier Details
Similar to Fang et al. (2022), we trained the supervised ResNer-50 classifiers (He et al., 2016) for 90 epochs with a batch size of 256 using SGD with Nesterov momentum, weight decay of le-4, momentum of 0.9, initial learning rate of 0.01 (they used 0.1) with step-wise decay by 0.1 at epochs 30, 50, and 70. We used the same image augmentations as for our CLIP models. We trained our supervised classifiers on the 895 classes-550 unmatched ImageNet classes combined with the 345 NetainNet classes Appendix A.3 for details
B. Additional Results For Section 5
In Figure 2, we compared the effective robustness trends of our different training setup across different domains. Table shows the final balanced t test accuracy (averaged over three runs) of our CLIP models with ImageNet-Captions as the base dataset and ResNet-50 as the vision encoder.
15

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Table 6: CLIP robustness results for the four training data setups (see Figure 1). We repeated CLIP trainings three times and evaluated the model's balanced top-1 accuracy on the unseen classes of the test domain. CLIP generalizes better with higher domain diversity. Intriguingly, CLIIP achieves superior performance when not seeing the domain at all vs. seeing subset of (c.f., Table 2). However, while diversity substantially improves CLIP's generalization performance, there remains performance gap to a model that has similar samples the classes C2 (c.f., Table 1). Figure 2 shows the respective effective robustness plots.
Data composition (Figure
Clipart
Infograph
Painting
Quickdraw
Sketch
Natural-only
20.3 + 0.7
11.8 + 0.2
34.1 + 1.4
0.8 + 0.1
19.5 - 0.7
Leave-out-domain
27.4 1.2
13.6 + 0.6
33.8 + 0.5
4.8 + 1.1
30.1 1.4
CG low-diversity
17.1 + 1.2
10.8 + 1.0
31.4 0.1
2.0 0.9
19.2 2.0
w/ classes C2 (upper bound)
37.2 = 0.8
21.5 1.8
4..5
06.0
50.3 - 1.2
CG high-diversity
27.6 1 1.5
12.7
2 2.0
34.6
1.7 0.7
28.1 0.8
w/ classes C2 (upper bound)
36.6 1.0
18.8 + 0.3
41.8 1.4
51.5
44.3 0.8
Natural-only
Leave-out-domaia
low-diversity
high-diversity
Perfect generalization
Integayhe torecon diwous. aopo))
Falnding (unseen claves top55)
(inscoo Quicklew choses, top-5
Sketch (trecem deoces, Spp-S)
10-
20-
Clipat
10-
10-
Real (tep-S)
Real (lep=S)
Reall (lop-5)
Real (lop-S)
Real (leg-5)
Chpprt (omseen F1)
60
Infograph Comeeres clances F))
Palidagg (unsoon claves,, Fi)
Skoch tameom cheops. F3
30
20-
20-
20
20
40
(e)
Real (5)
Real (F.)
Real (Fi
(eal
Figure 8: Effective robustness for additional evaluation metries. We re-visualized the results from Figure using balanced top-5 accuracy (first row) and macro Fi score (second row). Our findings remain consistent across both metrics.
Other evaluation metrics We also evaluated our CLIP models using balanced top-5 accuracy and macro Fi score ensure findings remain consistent across different evaluation metrics. Figure 8 shows the corresponding effective robustness plots.
B.1. Validity of the Results across Architecture, Dataset, and Loss Choices
We conducted the experiments from the main text (Figure 2 and Table 6) using ImageNet-Captions the base dataset and a ResNet-50 vision encoder. To ensure the consistency of our findings across different base datasets, vision encoder architectures, and contrastive loss functions, we performed additional experiments by systematically varying each of these components. In addition, we also investigated BLIP (Li et al., 2022), a different method for lision-language pre-training. Due to computational resource constraints, we repeated these experiments only for the Clipart and Sketch domains, where increasing domain diversity yielded the most significant robustness gains.
Architecture For the architecture experiments, we trained two CLIP configurations with different image encoder architec- tures. The first configuration used Swin-T (Liu et al., 2021), 1), the same text encoder as in our ResNet-50 experiments, and an embedding dimension of 512. The second configuration used ViT-S-32 (Touvron al., 2021), a slightly smaller text
16

---

When and How Does CLIP Enable Domain and Compositional Generalization?
encoder with width of 384, only six attention heads, and also smaller embedding dimension of 384. In addition, for ViT-S-32, we used slightly adjusted AdamW hyperparameters, i.e., 32 0.98 and 10-6. All other hyperparameters were consistent with the ResNet-50-based experiments (see Appendix A.6).
Table 7 and Figure 9 confirm that for cransoormrrbbasee vision encoders, robustness also improves with increasing domain diversity, as expected. Similarly, compositional generalization can perform worse than domain generalization.
Table 7: Results (balanced top-1 accuracy) when varying CLIP's vision encoder. We find similar trends across these
vision encoder choices.
Vision encoder
Training data setup
Clipart
Sketch
Natural-only
19.4
19.6
Leave-out-domain
27.1
31.8
ResNet-50
CG low-diversity
18.7
16.4
CG high-diversity
28.6
28.6
Natural-only
12.0
5.9
Leave-out-domain
15.1
8.2
VIT-S-32
CG low-diversity
12.7
5.8
CG high-diversity
13.8
8.3
Natural-only
17.1
11.3
Srave-out-domain
20.3
15.2
Swin-T
CG low-diversity
17.4
10.4
CG high-diversity
22.3
14.0
17.6
Skotth camson classes, top-1)
Naandands
Skench (omseron dlasses, Top-1 1)
20
Laresaideuh CG Calre-dissky
12.5
Cileghieeity
DG Lighthhhaaaz
(uncen
proeeiatee
Periec procaittation
10-
Clipart tomsen
7.5
2.5
(op
(top-))
Real (lo--))
Real (top-
(a) Clipart with ViT-S-32.
(b) Sketch with ViT-S-32.
(c) Clipart with Swin-T.
(d) Sketch with Swin-Ti T.
Figure 9: Effective robustness plots for different vision encoders. Refer to Figure 2 for the effective robustness plots for the ResNet-50 vision encoder.
Base Dataset For the base dataset experiments, we trained our ResNet-50 CLIP configuration using CC3M and CCI2M as the base datasets. Following Radford et al. (2021), we used maximum learning rate of 5e-4. We also adjusted the number of warmup steps to 2000 and the batch size to 2048. All other hyperparameters were consistent with the experiments from the main text (see Appendix A.6 for the hyperparameters).
Table 8 and Figure 10 confirm that both CC3M and CC12M exhibit the same robustness trends as our ImageNet-Captions models. However, the robustness gains are smaller compared ImageNet-Captions This is most likely due the reduced relative weighting of the DomainNet images with larger base dataset sizes, well as the higher inherent diversity of CC3M and CC12M. For example, both CC3M and CC12M include some non-natural images, which may diminish the effect of further increasing domain diversity. Note that compositional generalization seems to be slightly better than domain generalization for larger datasets, which we attribute to domain contamination (see Appendix B.2 why this may benefit compositional generalization). We leave further investigation into the effect of dataset size and diversity of the base dataset for future work (see Section for a discussion).
17

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Table 8: Results when varying the base dataset Da- We observe similar trends for CC3M and CC12M as for ImageNet- Captions. The only exception is that compositional generalization now tends to always work better than domain generalization for CC12M. We attribute this to a domain contamination of CC12M, i.e., manual inspection shows that CC12 1M contains a lot of Sketch and Clipart images.
Base dataset Do
Training data setup
Clipart
Sketch
Natural-only
19.4
19.6
Leave-out-domain
27.1
31.8
TaaggNet-CCaptionn
CG low-diversity
18.7
16.4
CG high-diversity
28.6
28.6
Natural-only
32.2
31.5
Leave-out-domain
31.6
35.1
CC3M
CG low-diversity
28.4
30.5
CG high-diversity
35.5
33.6
Natural-only
39.6
48.3
Leave-out-domain
46.1
51.2
CC12M
CG low- low-diversity
38.5
41.7
CG high-diversity
48.7
51.7
Sketch cumeeen closses, tap-1)
Clipurt (unseen clases, top-1)
Narabbekly
Corvecet tap-4)
Lanveze-dorad
Lanneor-toaak keedhhasstty (Yiiigh fioelly
341
10-
ITllegs deivityy
30
25
Pecken Pedken plasakariice
Redict pona Junce
40
Clipan (uncen
20
30
20
10
20
Real (top-) 1)
Real (lop- 1)
Real (top
Real (top 1)
(a) Clipart with CC3M.
(b) Sketch with CC3M.
(c) Clipart with CC12M.
(d) Sketch with CC12M.
Figure 10: Effective robustness plots for the different base datasets. Refer to Figure 2 for the effective robustness plots for the ImageNet-Caption dataset.
Contrastive Loss For the loss experiments, we trained our ResNet-50 CLIP configuration but replaced the original CLIP loss with the SigLIP loss (Zhai al., 2023).
Table 9 and Figure 1l confirm that the results for models trained with SigLIP are consistent with our observations from the experiments in the main text.
Beyond Contrastive Losses Finally, we conducted an experiment using BLIP (Li et al., 2022) instead of CLIP. Similar to CLIP, BLIP also a vision-language pre-training method that uses a contrastive image-text loss, but additionally includes an image-text matching as well as language modeling loss. We used the training pipeline provided by LAVIS (https:/ ///ittttu///// com/ salesforee LAVIS, License: BSD 3-Clause, Li et al. (2023)) to train BLIP in its base configuration, which uses V1T-B-16 (Dosovitskiy et al., 2021) pre-trained on ImageNet as the image encoder and text encoder initialized from BERT buse (Devlin et al., 2019).
Table 10 and Figure 12 show that the results for BLIP follow the same trends as our CLIP models, except for the overall higher performance due to the pre-trained image and text encoders.
B.2. Challenges of Compositional Generalization
In Section 5, we found that compositional generalization settings can suffer from an exposure to a subset of classes during training that are later queried in evaluation. This can make compositional generalization fail but can be alleviated by using domain samples from classes that do not overlap with the class distribution which is queried during evaluation. Table shows
18

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Table 9: Results when using a different loss function (SigLIP (Zhai et al., 2023)). The choice of loss function does not change the trends observed for CLIP's original contrastive loss.
Loss function
Training data setup
Clipart
Sketch
Natural-only
19.4
19.6
Leave-out-domain
27.1
31.8
CLIP
CG low-diversity
18.7
16.4
CG high-diversity
28.6
28.6
Natural-only
20.4
19.7
-eave-out-domain
28.1
26.1
SigLIP
CG low-diversity
19.2
16.4
CG high-diversity
25.7
27.5
Clipuri (onseen classes, wgp1)
25
28
Skotch (umseen classes,
20
13
(top
Real (lup-3)
Figure 11: Effective robustness plots for SigLIP. Refer to Figure 2 for effective robustness plots when using CLIP's
original contrastive loss.
that replacing all DomainNet sketches with sketches that do overlap with DomainNet's classes (see Appendix A.3), using ImageNet-Sketch, improves compositional generalization performance from 28.1 % 36.9%.
In practice, however, enforcing little no class overlap may not always be feasible; particularly in zero-shot settings where CLIP is applied to data and/or tasks that are unknown at training time. Therefore, we investigated the severity of this in more detail. To do this, we partitioned the set of classes C = C₁ U C2 U C3 in our test domain D; into three disjoint subsets (see Figure 13a):
C1: Classes that seen during training and queried during evaluation.
C2: Classes that are not seen during training and are queried during evaluation. Note that only classes from C2 are contained the test set D
C3: Classes that are seen during training and are not queried during evaluation.
This partitioning allowed us to systematically assess the impact of class overlap on compositional generalization by constructing training sets with varying mixtures of classes from C1 and C3-
For this experiment, we selected Sketch as our test domain." 5 The subsets Ci and C2 were defined as described in Section 3, meaning that C₁ C2 represents the class distribution queried during evaluation (i.e., all DomainNet classes). For C3. used classes from MeageNet-SSetch (Wang al., 2019) that do not overlap with the classes from DomainNet (see Appendix A.3 for more details). Note that throughout all these experiments, the class distribution in the other domains Djyi remained unchanged- that is, classes from Ci and C2 were included in training, while no classes from C3 were introduced.
As shown Table 2 and Figure 13b (rightmost bar), training solely on samples from C3 significantly improves compositional generalization performance compared to not seeing any test domain samples from D, (zero line) or only seeing samples from classes C1 of D. (leftmost bar). To further investigate this, we trained CLIP models using samples from C1 C3 of the test domain Dj and gradually reduced the number of classes from C₁ while keeping all other factors fixed, Figure 13b shows
We chose Sketch due to availability of large class distribution from ImageNet-Sketch.
19

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Table 10: Results when using BLIP (Li et al., 2022). Using BLIP as a pre-training method does not change the trends observed for standard CLIP. Note that the overall increased performance of BLIP is due to their usage of pre- trtirraned image and text encoders.
Training regime
Training data setup
Clipart
Sketch
Natural-only
19.4
19.6
Leave-out-domain
27.1
31.8
CLIP
CG low-diversity
18.7
16.4
CG high-diversity
28.6
28.6
Natural-only
37.1
35.7
Leave-out-domain
61.8
58.6
BLIP
CG low-diversity
35.0
41.6
CG high-diversity
55.6
61.2
Cliputt (omseen clasees top-1
rGligetimaty
Skerch (umoen
00
45-
80
35-
40
30
60
70
60
Real (top 1)
Real (top 1)
Figure 12: Effective robustness plots for BLIP. Refer to Figure 2 for effective robustness plots when using standard CLIP.
that reducing class overlap (moving from left to right) decreasing the number of C₁ classes seen during training- consistently improves compositional generalization performance. This reaffirms the detrimental effect of class overlap, as observed in Table 2, but also highlights that increasing class diversity can help mitigate its impact on compositional generalization performance.
Supervised classifiers We also investigated to what extent supervised classifiers are vulnerable to this bias. Table 11 confirms that supervised classifiers are also susceptible
B.3. Closing The Generalization Gap
Table 6 clearly shows that there a significant performance gap between the CG high-diversity with and without including comain-specific samples of our test classes C2, even for domains like Clipart and Sketch where generalization seems to work reasonably well. We conducted an interpolation experiment between the settings with and without such samples better understand how many of these samples are actually required to close this "generalization gap". To further investigate how the generalization capability of a model impacts the required number of samples, we also performed the same experiment for the CG low-diversity setting.
For given test domain D, we consider the number of training samples of our test set D to be 100% and then trained additional CLIP models in which we successively added 5%, 10%, 15%, 20% 40%, 60%, and 80% of training samples from DC Ca the training data. Note that we ensured that the overall dataset size remained unchanged.
Figure 14 shows that performance the classes seems follow roughly linear relationship with the number of samples allowed for all domains, except for the Quickdraw domain. For Quickdraw, the relationship seems to be log-linear instead, which may be due to the fact that CLIP does not generalize at all for Quickdraw. This observation also relates to the findings of Udandarao et al. (2024), who predict that a linear increase in samples leads only to a log-linear increase in zero-shot performance. Across all domains and settings, we observed that to achieve the maximally possible performance, 100% of the training samples from D. are required.
20

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Classes available for tminning Cj L/Cg
Impeoveseent Leave
C3
Queried classes C1
Queried
test set C22
1000 Cis
3055 CC HAKE CT
500 CI
200 Ciy
1055 C3
40. Cig
15
OS C3
Class Misture
(a) Classes from test domain.
(b) Effect different mixtures the test domain's classes Ci Cs seen during training.
Figure 13: Severity of class overlap on compositional generalization. a: We partitioned the classes in the test domain D; into three disjoint subsets: C Ci U U2 U3. During evaluation, both C1 and C2 included in the query set, allowing the model to predict any class from these subsets. However, the actual test set, D only contains samples from C2. In contrast, the classes in C3 are excluded from both the query and test sets. This partitioning allows us to investigate the severity of class overlap for compositional generalization performance. In particular, we found that including classes of C₁ led to a significant drop in balanced top-1 accuracy for compositional generalization- even performing worse than domain generalization (Figure 2). the other hand, replacing class samples from Ci with the ones C3 (which are not part of the query set) resulted varied the mixture of classes from C and C3- We find that reducing classes from C₁ (moving from left right), while keeping all other factors fixed, steadily improves compositional generalization performance (with the exceptions for the class mixtures 25% C₁ 100% C3 and class Ci 100% C3).
significant improvement in compositional generalization (Table 2). b: To investigate the severity,
Table 11: Supervised classifiers are also susceptible the seen class bias. Supervised classifiers' compositional generalization also deteriorates due partial test domain overlap and replacing them with samples from non-overlapping classes significantly improves compositional generalization.
Training data setup (Figure 1)
Sketch
Leave-out-domain
27.4
CG high-diversity
22.2
w/ sketches of non-queried classes only
30.1 (+7.9)
B.4. Role of Language Supervision
Previous studies comparing the robustness of CLIP models and supervised classifiers either examined models trained on different datasets or focused on low-diversity datasets consisting mostly of natural images, such as ImageNet-Captions (Fang et al.. 2022). Since both ImageNet-Captions and DomainNet provide class labels, we investigated how our domain mixtures affect the robustness of supervised classifiers and compared the results CLIP models.
Table 12 and Figure 15 show the results of our experiments supervised classifiers. We found that CLIP models consistently exhibit slightly higher robustness than their supervised counterparts, which may be due to the richness of captions (Xue et al., 2024; Wen et al., 2024). Interestingly, CLIP's advantages are more pronounced in compositional generalization settings (CG low/high-diveriiig However, in the domain generalization setting Leave-out-dommal supervised classifiers can sometimes achieve generalization comparable to CLIP.
21

---

When and How Does CLIP Enable Domain and Compositional Generalization?
direyy Winghehanty
27.5
Cd kes chindyy CD Cuiggerccy
Clipat fost claves (op-11
CGllg (aonsee
22.5
Phinting (test clo00s
60.0
17.5
27.5
35.0
15.0
12.5
Allowed Samples (8)
Allowed Samplox (9))
Samplex (16)
(a) Clipart.
(b) Infograph.
(c) Painting.
Quicktrue (ba1 (lat www.co..c.0. 10p-13
100
CC deen by OGlig-drintty
CG cuurdty CG tegebrcsity
50-
60-
Skatch (Bms
0.0
30-
20
Allowed (1mplex(99)
Allowed Stmples (18)
(d) Quickdraw.
(e) Sketch.
Figure 14: Interpolation between CG settings with and without domain-specific training samples of the test classes C2. For each interpolation, we fitted both linear and a log-linear regression model and visualized the fit with a lower mean squared error (MSE). Balanced top-1 accuracy on the test classes appears to follow a roughly linear relationship with the number of test samples included, except for Quickdraw, which shows a strong log-linear relationship.
Table 12: Performance comparison across supervised experiments. The robustness of supervised models (as measured by balanced top-1 accuracy) also increases with domain diversity. However, supervised classifiers generalize slightly worse than CLIP models (Figures 4 and 15).
Clipart
Infograph
Painting
Quickdraw
Sketch
Natural-only
17.6 + 1.6
10.9 0.8
32.6 1.6
0.7 0.1
15.0 + 1.1
30.8 + 2.1
14.5 + 1.2
35.7 + 1.2
10.1 + 0.5
27.4 + 2.1
CG low-diversity
13.2 = 0.6
9.7 0.5
25.1 # 0.9
0.0 0.0
12.0 + 1.1
CG high-diversity
25.0 + 1.6
12.2 + 0.7
28.6 + 0.4
0.7 + 0.4
22.2 0.7
C. Additional Technical Details and Results for Section 6
C.1. Technical Details on Sparse Autoencoders
Following (Bricken et al., 2023; Huben et al., 2024), we used a Sparse Autoencoder (SAE) to extract interpretable features from CLIP's visual embeddings a E IRP. The SAE is defined as follows:
SAE(a) (g o 0o (a),
(4)
where di a ReLU non-linearity, and f and g are linear encoder with weights W, E Rpxh or decoder with weights W, [Rxx,, respectively. We trained the SAE with an La reconstruction loss and L1 sparsity regularization:
C(a) ||a 0go (a)]l2 + All(o (a))l
(5)
where À governs the sparsity regularization strength.
We trained SAE's the activations of our CC12M CLIP m (see Appendix B.I). We used CC12M and the complete
'We also tried the CLIP models that trained with ImageNet-Captions as hase dataset but found that the SAE extracted poorly interpretable features.
22

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Clipar (unsess cleses. top-1]
Namsesb KtLas
Loncenddemaa KILI
ES highg daeede LLPP
Cd bnednasst (ELE)
Infograph (umsern clasers, top-l)
LowtosisheemnttL. Neainl-nte 159)
Nakril-ntt IULIP
CU dedeen (LUP
25-
Leseor (9V)
Natelcey gii
Ciigg asialyy (80) Petfet prreotutin
CG deaderyyyyyyy 3V
Loase ceceeese
CD Cuieed (So)
diress (99) Rebel peoziiatie
20-
Bnflst irave
Peciut grondiinaio
3-
Real (top-1)
Real 1ogg-1)
Keal 0op-113
Real (top-3)
Clipart.
(b) Infograph.
Paining (unseen clases. top-1)
Quickdmm (snsreen closess top-1)
15-
20-
20
90
20
40
0
20
40
403
40
Real (top-3)
Real (lop-1)
Reill (top-1)
Real (top-1)
(c) Painting.
(d) Quickdraw.
Nasak-iity (CLI
beleddonniyy (CLPS) bghh-rrre (CuI
Slotch top
disiniy
15
-01
Real (top-1)
Real (top-1)
(c) Sketch.
Figure 15: Effective robustness plots for CLIP vs. supervised classifiers. While the general trends are very similar between CLIP models and supervised classifiers, CLIP models typically generalizes better than the supervised classifiers.
DomainNet training set to train the SAE's to identify interpretable features. The hidden dimension h was set to 4096, i.e., 4x the embedding dimension of the CLIP model's 's output dimensionality. We trained the SAE for 200 epochs using a batch size of 4096. The regularization strength hyperparameter À was set to le-4. To alleviate the dying neuron problem, dead neurons were resampled every 500,000 training steps. Our implementation is based on the code published by Rao et al. (2024) (https: / i thub. com hwtp///weuoeppeeee/e/oo i c t saar/ di scover-then name, License: MIT).
We trained one SAE for our natural-only baseline, two SAE's for the leave-out-domain setting (with Clipart and Sketch as the test domain), and another two SAEs for the CG high-diversity setting (also with Clipart and Sketch as test domain). For each model. we analyzed the extent to which the top-k most activating SAE features are shared between the test domain (Clipart or Sketch) and all other domains (not just the training domains).
To identify the top-k SAE features, we computed the SAE hidden representations (after applying the non-linearity ạ) for each sample of each class and domain. We then selected the top-k SAE features (with & (5, 10, 15, 203)) per class-domain pair based on how frequently they ranked among the top-20 most activating features, i.e., largest activation magnitudes (see Algorithm 1).
To measure feature sharing, we then calculated the percentage overlap between the top-k SAE features of the test domain and those of each of the other domains. To yield a single overlap score per model, we averaged across classes C2, these pairs of domains, and the four values of & (see Algorithm 2). Finally, we calculated the increase of the overlap score of the corresponding leave-out-domale and CG high-diversity models over the natural-only baseline model.
23

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Algorithm 1 get topk features
input domain of interest Dj, class of interest e, number of features consider k 1: hist +- 0 e Ra
I/ h is the number of concepts
2: for each image I E D do
3: a - [I)
I/ CLIP image activation SAE concept activations
4: w + (-fsE((a))
5: for each index topk(w, 20) do
6: hist(d| hist|t] +1
// count top-20 activating concepts
7: end for
8: end for
output topk(hist, k)
// return top-k most frequent features
Algorithm 2 measure feature sharing
input test domain D;
1: S S
// initialize empty set of scores
2: for each E (5, 10, 15, 20) do
3: for each class E C2 do
4: F; f get.ttttppp.... deet.top_.feturess. [i, k) k)
I/ top-k features of test domain D;
5: for each other domain Dj 전 D; do
6:
Fj - get.tokk_featres
k)
// top-k features of domain D
s |F:NFll
7:
|/ percentage overlap of F; and F)
8:
5 SUSUUE {s}
9: end for
10: end for
11: end for
output S Σ
overlap averaged over domains Dj. classes c, and
ES
C.2. Technical Details on Center Kernel Alignment and Additional Results
Let X CCP and D E Cxp contain the C mean visual embeddings of each class for the domains D₁ D2- respectively. Then, compute the Gram matrices/kernels K = (X and L Y (YPPTT that contain the pairwise similarities of each pair of mean class embeddings. Note that we used linear kernel here, as commonly done in the representational similarity literature. Alternatively, we also tried non-linear kernel (i.e., RBF kernel) with similar results (see Figure 17). Center kernel alignment is defined by Kornblith et al. (2019) as follows:
HSIC L) / /HSIC(K, K K)HSIC L)
CKA(K, L)
(6)
where we used the used the unbiased Tillber-Schhiie Independence Criterion (HSIC) estimator (Song et al., 2012) following Nguyen et al. (2021), defined as follows:
HSIC(K, C(C-3) (rLL)) + () C-2
1
1'K11
-0-21"Kin 1"kki
2
where we set the diagonal elements of K and to zero in K or L, respectively.
We computed center kernel alignment between all pairs of domains for each class. Thereby, we assess the representational similarity for each class across the domains. For visualization, for each domain, we averaged over all classes to obtain representational similarity estimate. In the main text, we further averaged together the estimates for the non-Quickdraw domains ("Others"). The estimates for each individual domain is shown in Figures 16 and 17.
ADDITIONAL RESULTS
Figures 16 and 17 show results for all classes C. the classes seen during training C₁ and the unseen classes Cp for the CLIP model solely trained on domain- domain-invarian invariant captions from Figure and Table (second row). Interestingly, we find that the
24

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Quaillmee Ulben
Qhaiddew OAber
Qaddew
CXA sinnterit
CKA simmlantt
CKA sumilatte
0.5
Ohhes
0.0
Layer
Layer
Laya
(a) All classes.
(b) Seen classes only.
(c) Unseen classes only.
0.15
01.
CKA similaity
CKA similanty
CKA similarit
0.5
0.4
012
0.0
0.0
010
Layer
Layar
Laya
(d) All classes.
(c) Seen classes only.
(f) Unseen classes only.
Figure 16: Linear center kernel alignment (CKA) similarity. Quickdraw has the lowest representational similarity across domains. This particularly emphasized for the unseen classes C2 (c, f). We used the CLIP model trained on only domain-invarian captions from Figure 5b and Table 4 (second row). Figure 18 provides results for other CLIP models.
Quakdree
Quacidew
Qadidi
Olion
Orkers
Orhek
CKA sinntaiiy
CKA simmlarit
CKA summlarity
0.4
0.4
0.0
Layer
Layer
Laya
(a) All classes.
(b) Seen classes only.
(c) Unseen classes only.
0.0
0.5
Istrgrop Peinde
Dectator
Stokc
CKA similanty
CKA similanty
CKA similarit
01.0
0.0
0.2
Layer
Layar
(d) All classes.
(e) Seen classes only.
(f) Unseen classes only.
Figure 17: Non-linear center kernel alignment (CKA) similarity. The results follow the same pattern the linear CKA (Figures 6a and 16): Quickdraw is the most representational dissimilar domain. We used the CLIP model trained on only Nomain-invariant captions from Figure 5b and Table 4 (second row).
representational gap widens for the unseen classes Cg. Figure 18 shows the representational similarity for different CLIP models from Figure for the unseen classes C2. Similar to the above, the CLIP model evaluated on the Quickdraw domain cxhibits low representational similarity to models from other domains. In contrast, the CLIP models with the test domains Clipart, Infograph, Painting, and Sketch show higher or more comparable representational similarities.
C.3. Technical Details on the Circuit Similarity Analysis and Additional Results
Attributing the causal effects of model components is a key goal of (mechanistic) interpretability research (Meng et al., 2022; Marks et al., 2025; Mueller et al., 2024). We drew inspiration from them to analyze the level of sharing of the most
25

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Lipa
0.7
Diten
0.0
CXA similerity
CXA sioniaa
CKA simi
0.1
Layer
Layer
Laya
(a) Clipart.
(b) Infograph.
(c) Painting.
CKA similenty
0.5
CK. similarity
04
0.2
0.3
Layer
Layer
(e) Sketch.
(d) Quickdraw.
Figure 18: Linear center kernel alignment (CKA) similarity for different CLIP models. We used the CLIP models from the CG high-diversity setting in Figure 2 and report representational similarities on the unseen classes C22. The patterns for all classes C' and for the seen classes C1 are qualitatively similar. Among the domains, Clipart, Painting, and Sketch exhibit comparable levels of representational similarity, whereas Infograph and particularly Quickdraw show lower similarities.
important CLIP's model components, i.e., the axis-aligned neurons in its vision encoder, across domains. To do this, we first must attribute the components' importance via the indirect effect (Pearl, 2001).
I. Let m.g(Idde alean p be a p-dimensional neuron n in the L-th layer of model m for input image Letean Further, we define a patch E p as a corrupted baseline. The computation of the indirect effect is defined as follows:
mL do pacch) LL
I.n (lean)
where mI,e is the output logit for class c of the last layer L (note that we can obtain this logit through the dynamic zero-shot weights that be generated by CLIP's text encoder, see Appendix A.5) and do = a panch denotes the do-operator (Pearl, 2009) that intervenes on the computation of the CLIP model setting the activations a' parch
I.n
Since there are lot of neurons in CLIP's vision encoder, we sped up the computation through linear approximation using integrated gradients (Sundararajan et al., 2017), following Marks et al. (2025); Hanna et al. (2024):
1ggg
Σ
V al Ta Maa Ln cleam + (1 a) a)afiiia)))
(a
a )
1.m
(9)
where (0, ...., (...,,,,,,))) and we set a to We adapted the codebase from Marks et al. (2025) (https: reutuure--ireieeeee reu reuits, License: AIT)-using nns iitt KFuffno al., 2025, https://githu/ com/ htt//diiff www.tttaamnnnnnnneeeeee License: MIT) internally-for our implementation.
//github. com/ www.saprmaaaaaa/aaaaaaaaeo
sasssaapraaakea rks feature-ci
DISCOVERY OF CIRCUITS
We use above computation for the indirect effect to find the most important neurons and k' most important preceding neurons of each of those neurons for each class each domain. This approach is outlined below in more detail:
1. Identify the most important neurons: We directly apply Equation 9, We used N = 10 steps for the linear approximation and only retained the 10% most important neurons per layer. Note that these neurons represent the nodes of the graph representing the circuit.
26

---

When and How Does CLIP Enable Domain and Compositional Generalization?
Obiocs
Olkors
Other
Foccard inder
Joocard Lodes
0.3
Joacaed
0.2
Layer
Layer
Laya
(a) All classes.
(b) Seen classes only.
(c) Unseen classes only.
0..
0.5
Juccund index
Juceud Index
0.3 Jazzun
0.4
0.4
0.1
0.1
Layer
Laya
Laye
(d) All classes.
(e) Seen classes only.
Unseen classes only.
Figure 19: Amount of shared neurons. The Quickdraw domains shares the least neurons. This is especially apparent for the unseen classes C2 (c, f). Results are for the CLIP model trained on only domain-invarian captions from Figure 5b and Table 4 (second row). Figure 20 provides results for other CLIP models.
Table 13: Weisfeiler-Lehman similarities. Higher similarities mean higher circuit (graph) similarity. We used the CLIP model trained on only domain-invarian captions from Figure 5b and Table 4 (second row); results for other CLIP models are provided in Table 14. The Quickdraw domain exhibits the least degree of similarity, supporting our hypothesis that sharing of circuits is critical for generalization.
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
All classes
0.281
0.246
0.266
0.218
0.273
0.273
Seen classes only
0.281
0.246
0.265
0.218
0.273
0.273
Unseen classes only
0.278
0.258
0.274
0.211
0.272
0.275
2. Identify the k' most important preceding neurons of each neuron: We adapted Equation 9 to measure the effect of preceding neurons n of layer / < 1' on the activations of neuron n' in layer /'. Specifically, we replaced m.r,c by mp.. I.n. and measured the L2 change caused by the clean activations a and intervened activations a + a )a a patch We also set N to 10. Note that this will yield us edges for the graph representing the circuit. After computing the
In
Imm
indirect effects of all preceding neurons, only retained the k' most important edges.
MEASURING CIRCUIT SIMILARITY
We compared the circuit similarity of pairs of circuits (graphs) resorting classical graph similarity measures. Similar to the representational analysis, we computed the graph similarity measures between all pairs of domains for each class and averaged over all classes to obtain the final circuit similarity estimates. In the main text, we further averaged together the estimates of the non- non-Quichh -on-Quickdraw domains ("Others'). The estimates of each individual domain are shown in Figures 19d to 19f and Table 13.
Node-level similarity We computed the Jaccard index for the nodes NC Dj.dao NCsscii Dy daace follows:
Caceaaa olsse O. NC Da. chase
(10)
Casuuis D3 oasee U N NCeceen Dast
Intuitively, this quantifies the degree overlap between the most contributing model components for class c across domains Dj and Dz.
27

---

When and How Does CLIP Enable Domain and Compositional Generalization?
0.0
Neegrigh
Fuacord index
Juscard Index
0.3
Jaccard Lndex
0.2
Layar
Laya
(a) Clipart.
Infograph.
Painting.
00
Sknh Olko
Olbnn
Iacoediakk
0.5
Jaccord tades
0.
0.
0.3
0.2
that
(d) Quickdraw.
(e) Sketch.
Figure 20: Amount of shared neurons for different CLIP models. We used the CLIP models from the CG high-diversity setting in Figure and report results on the unseen classes C2. While Clipart, Infograph, Painting, and Sketch exhibit similar levels of neuron sharing, Quickdraw shows significantly less sharing.
Table 14: Weisfeiler-Lehmar similarities for different CLIP models. Higher similarities mean higher circuit (graph) similarity. We used the CLIP models in the CG high-diversity setting from Figure Results are for the unseen classes C2. The CLIP models with the test domains Clipart, Painting, and Sketch have higher degree of similarity than Infograph and Quickdraw.
Test domain
Test domain vs. Others
Others Others
Clipart
0.274
0.243
Infograph
0,246
0,250
Painting
0,263
0,248
Quickdraw
0.176
0.272
Sketch
0.259
0.242
Graph-level (structural) similarity Above simple node overlap cannot capture more complex structural and hierarchical similarities between circuits. Thus, we used the normalized Missfiiler-Lhhmal subtree graph kernel (Shervashidze et al., 2011). The Weisfeiler-Lehman graph kernel is popular graph kernel choice since (I) it can handle labeled, directed graphs of different sizes, (2) it is expressive, and (3) scales well to large graphs. The main idea of the Weisfeiler-Lehman kernel is the following procedure, given two labeled graphs, G1 and G2:
1. Multiset labeling and sorting: For each node n E G1, create a multiset (unordered set with duplicates allowed) consisting of the node's n current label and the sorted labels of its neighbors. Repeat this step for each node n' E G2-
2. Label compression: Assign a unique new label to each of these multisets using a hash function.
3. Counting occurrences: Count the occurrences of each compressed label obtain the feature vectors di (G1), (G())).
4. Relabeling: Replace the current node labels with the newly compressed labels.
We can repeat this procedure for h iterations and compute the similarity of graphs via:
K(G,, G2) = (o(G1), ((G2)) = Σ (G1) Φn (a(()) (Gz)
(11)
h
28

---

When and How Does CLIP Enable Domain and Compositional Generalization?
0.9
0..3
0.8
Faocan Index
Juccudd Lodes
0.5
0.5
0.1
O8ax
Thrsshois
Thresolk
Thichhlde
(a) Clipart.
(h) Infograph.
(c) Painting.
D-3
Jacend Indes
6.
index
0.0
0.7
0.0
Thesbokk
Throdalks
(d) Quickdraw.
(c) Sketch.
Figure 21: Amount of shared neurons for different CLIP models over various neuron thresholds. We used the CLIP models from the CG high-diversity setting in Figure 2 and report results on the unseen classes C2, averaged across all layers and evaluated over multiple thresholds. The patterns observed in Figure 20 remain consistent across these thresholds.
Finally, we normalize the kernel to obtain similarity score between 0 and
K(G1,G2) RGG VK(G1, G1) K(G2, G2)
(12)
For our analysis, we used = 3 iterations. Our implementation is based on the publicly available code from https:
//gittuul com/co//eeeoo/oo. wtt.coo//mmmmm wwweeemaaeeeeeeeeeeeeeeeeeeeeeee www.llejjsssp1111. License: MIT.
False positives certain cases, the Weisfeiler-Lehman kernel can incorrectly identify non-isomorphic graphs as isomor- phic. However, several factors reduce the chance of such false positives in our setting. Specifically, we used h which yields expressive graph kernel. In addition, node names in the circuits encode global topological information (i.e., layer and neuron indices), which further reduces the chance of false positives. Finally, verified that the circuits were already distinguishable at the node level (i.e., at the 0-th iteration of the Weiseeiler-Lemman kernel).?
ADDITIONAL RESULTS
Figure 19 and Table show results for all classes C, the classes seen during training C1, and the unseen classes C2 for the CLIP model only trained on covainninvariant captions from Figure 5b and Table 4 (second row). Neurons (nodes) are less shared and the cireuits are slightly less similar for the unseen classes Cz. Figure 20 and Table 14 extend this analysis to the different CLIP models from Figure 2 (results are the unseen classes C2). Across these models, consistently find that domains like Clipart, Painting, and Sketch maintain similar levels of neuron and circuit sharing, whereas Quickdraw exhibits less sharing. Figure 21 shows that our observations are robust across various neuron thresholds (we chose to retain the 10% most important neurons per layer in Figures 6b, 6c, 19 and 20 and Tables 13 and 14).
False positives can only occur if two graphs remain indistinguishable across all iterations Weisfeiler-L.cmma kernel.
29


Reviews:
[{"review_id": "vsuUZKCkmM", "reviewer": "Reviewer_jK9M", "summary": "The paper studies the generalization capabilities of CLIP in the Domain Generalization setting. Specifically, the authors study when and how clip exhibits domain generalization - when a model generalizes to unseen domains, and compositional generalization - when a model generalizes to classes from partially seen domains during training. To facilitate this study, the authors conduct carefully crafted experiments on CLIP with DomainNet and ImageNet to understand the influence of domain diversity, language descriptions, and shared representations on the generalization capabilities of CLIP.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and constructive feedback. Below, we address the remaining concerns and questions.\n\n> Can the CLIP visual features directly be considered for the top-k computation and further analyzes?\n\nYes, this is possible. However, we would like to note that CLIP’s visual features are in superposition and not directly interpretable w.r.t. an object class. For example, Schrodi et al [1] showed that some of CLIP’s visual feature dimensions have to do something with the semantically non-meaningful modality gap. Thus, recent work [2] and ours opt for dictionary learning methods like SAEs (i.e., linear vectors that need not be axis-aligned) to extract more interpretable and class-relevant features.\n\n> Missing details about the SAE experiment (including the questions raised by the reviewer)\n\nWe thank the reviewer for pointing this out. Below, we provide a detailed explanation of our SAE experiment, which we will add to the paper.\n\nWe trained a separate SAE on CLIP’s visual features for each of the following 5 models: a natural-only baseline, two leave-out-domain models (where either clipart or sketch is the test domain), and two CG high-diversity setting (also using clipart or sketch as test domain).\n\nFor each model, we analyzed the extent to which the top-k most activating SAE features are shared between the test domain (clipart or sketch) and all other domains (not just the training domains). To identify the top-k SAE features, we computed the SAE hidden representations (after applying the non-linearity) for each sample of each class and domain. We then selected the top-k SAE features (with $k \\in \\lbrace 5, 10, 15, 20\\rbrace$) per class-domain pair based on how frequently they ranked among the top-20 most activating features (i.e., largest activation magnitudes).\n\nTo measure feature sharing, we calculated the percentage overlap between the top-k SAE features of the test domain (clipart or sketch) and those of each of the other domains using the top-k SAE features that we got from above. To yield a single overlap score per model, we averaged across classes, these pairs of domains, and the four values of k. Finally, we calculated the deltas in Table 3 by comparing the overlap score of the natural-only baseline model with the corresponding leave-out-domain and CG high-diversity models.\n\n> False positives of Weisfeiler-Lehman (WL) kernel\n\nThank you for pointing this out. We will add this aspect. \n\nWhile the WL kernel can yield false positives, this is unlikely in our case: The node names in the circuits encode global topological information (i.e., layer and neuron indices), which reduces the chance of false positives. Besides that, we employed the 3-WL kernel, which is more powerful than, e.g., 1-WL, and also reduces the chance of false positives. \n\nBesides that, we verified that the graphs were distinguishable at the node level (0-th iteration of the WL kernel); as indicated by Figure 6b (the figure just shows aggregated results, but we verified it for each set of nodes of each pair of circuits). Note that false positives can only occur if two graphs remain indistinguishable across all iterations of the WL kernel, which is empirically not the case.\n\n> Revise some of the tables, captions, and experimental settings\n\nWe will revise them.\n\n--- \n\n[1] Schrodi, Simon, et al. \"Two effects, one trigger: on the modality gap, object bias, and information imbalance in contrastive vision-language models.\" ICLR 2025.\n\n[2] Rao, Sukrut, et al. \"Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery.\" ECCV 2024.", "claims_and_evidence": "Yes. All the claims made in the submission are supported by well-motivated and clear experiments.", "methods_and_evaluation_criteria": "The proposed method and evaluations use relevant benchmark datasets, and the authors have justified their choice of datasets and evaluation settings.\n\n- **Datasets**: For their experimental evaluation, the authors have chosen ImageNet-Captions, ConceptualCaptions 3M, or ConceptualCaptions 12M as their base dataset, along with domains from the DomainNet dataset for the domain-specific image-text pairs. Given that DomainNet remains the largest domain-shift benchmark for domain generalization, this choice is verified.\n- **Evaluation metrics**: The authors use balanced top-1 accuracy as their evaluation metric, which mitigates the effect of the long-tail nature of the DomainNet dataset (L192-195).\n\nSimilarly, authors have clearly outlined and justified their setups for evaluating the effect of shared representations on DG performance, and the behavior of CLIP on domains such as quickdraw via model circuit similarities across pairs of domains.", "theoretical_claims": "The paper is an empirical study of the generalization properties of CLIP. As a result, there are no significant theoretical claims or proofs in the submission.", "experimental_designs_or_analyses": "- **General setups:** The authors consider two experimental setups: domain generalization - where the model is required to generalize to an unseen domain, and compositional generalization - where the model has partial access to classes from the unseen target domain. The authors train CLIP in both of these settings by merging data from ImageNet-Captions and DomainNet. These experimental designs follow the general setup of generalization works.\n- **Experiments on the role of visual embeddings:** In Sec. 6.1, the authors discuss their experimental setup and results in determining the role of the shared visual embeddings across domains in facilitating generalization in CLIP. Specifically, the authors train a Sparse Autoencoder (SAE) on the CLIP visual representations across domains and consider the overlap between the top-k embeddings across domains. However, the details of this experiment are unclear (please see the weaknesses section below).\n\nOverall, the experiments and analyzes presented in the submission are valid for the study of CLIP's generalizability across domains.", "relation_to_broader_scientific_literature": "- The contributions of the paper are highly relevant to domain generalization literature. Specifically, several papers utilize the generalization capabilities of CLIP [1, 2, 3] for a typical domain generalization without delving into the analysis of CLIP's generalizability.\n- Moreover, the general consensus for the lack of performance of CLIP on the Quickdraw domain in DomainNet has largely been attributed to the lack of images from this domain in the pre-training data of CLIP. However, there has been no work that studies the behavior of CLIP with such domains, which this work explores in great detail.\n\n1. Addepalli, Sravanti, et al. \"Leveraging vision-language models for improving domain generalization in image classification.\" CVPR 2024.\n2. Huang, Zeyi, et al. \"A sentence speaks a thousand images: Domain generalization through distilling clip with language guidance.\" ICCV 2023.\n3. Yu, Xi, Shinjae Yoo, and Yuewei Lin. \"CLIPCEIL: Domain Generalization through CLIP via Channel refinement and Image-text alignment.\" NeurIPS 2024.", "essential_references_not_discussed": "The authors have referenced the relevant literature and discussed the same adequately.", "other_strengths_and_weaknesses": "### Strengths\n- **Relevance to prior works:** The major strength of this submission is its relevance to prior works that utilize pre-trained Vision-Language Models (VLMs) such as CLIP for domain generalization. Specifically, as mentioned above, most works in DG directly consider the generalizability of CLIP as a given and design frameworks using CLIP while the submission delves deeper into the mechanics behind CLIP's generalization capabilities. A noteworthy contribution is the submission's study on the quickdraw domain from DomainNet, which prior works often dismiss as unseen by CLIP.\n- **Experiments and results:** The submission presents well-motivated and simple experiments to analyze the properties of CLIP in the domain and compositional generalization settings. Specifically, the experiments on the role of shared intermediate features and similarity between model circuitry across domains is quite interesting.\n- Overall, the paper is well-written, easy to follow and understand. All the experiments are well-motivated, outlined clearly and analyzed well.\n\n### Weaknesses\n(a) As mentioned above, while the authors have presented an experiment to substantiate the claim that generalizable models share representations across domains, the experiment itself is unclear. \n- How is CLIP evaluated using this trained SAE? Are these evaluations conducted in the leave-one-out or CG settings? Which pairs of domains are considered in the setup, i.e., all domains or only the training domains?\n- How are the top-k features chosen across domains? What is the objective used to compute the top-k features (eg: cosine similarity, L2 loss, etc.)?\n- How is the performance delta collected (in other words, how do the authors evaluate the top-k representations to present the improvement using the shared embeddings)?", "other_comments_or_suggestions": "Some of the tables and their captions are unclear and can be better presented for enhanced readability. For example, Tables 3 and 4 do not mention the quantity being presented (whether it is the improvement in accuracy or the accuracy itself). Additionally, as mentioned in the weaknesses section, some experimental settings have not been clearly explained. The reviewer suggests the authors review these details and rewrite them for better understanding.", "questions_for_authors": "- In Sec. 6.1, the authors present an experiment to analyze the role of shared representations in generalization. Specifically, they train a SAE in the representation space and consider the top-k features. Can the CLIP visual features directly be considered for the top-k computation and further analyzes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aCD2H8BMWx", "reviewer": "Reviewer_jP1e", "summary": "This paper tries to find what factors affect the domain generalization and compositional generalization of CLIP. The empirical experiment show that domain diversity is essential for both domain and compositional generalization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. Below, we try to address the concerns. However, the review has been very brief, and some points remained unclear to us. We would greatly appreciate further clarification during the discussion phase so we can fully address them, if we have not already done so below.\n\n> Are the results due to domain diversity or just the sheer size of the dataset? (our interpretation of the reviewer’s concern)\n\nTo disentangle the effects of dataset size and domain diversity (two typically entangled factors), we conducted controlled experiments, where we fixed dataset size while varying domain diversity. This setup allows us to isolate the impact of diversity. For example, we observe that adding a single domain yields often negligible gains, while greater domain diversity significantly improves generalization performance (see Fig. 2).\n\nIn addition, recent work by Mayilvahanan et al [1] showed that “domain contamination [in web-scale datasets] contributes substantially to CLIP’s strong [generalization] performance”  [1, p. 9]. Our work goes beyond theirs by analyzing when and how different domain mixtures influence domain and compositional generalization, e.g., we investigate why CLIP sometimes fails to generalize even with high diversity, see our (mechanistic) analysis in Sec. 6.2.\n\n> No theoretical evidence to support its claim\n\nWe would like to clarify that this is an empirical analysis paper, with our claims grounded in controlled and well-motivated experiments – as also noted positively by both other reviewers. While we do not offer theoretical analysis, we follow the growing body of work that investigates model behavior through carefully controlled experiments, such as the popular “Physics of Language Models” series (https://physics.allen-zhu.com/).\n\n> DomainNet is a long-tail dataset, which may affect the claim \n\nDomainNet is long-tail; yet this mirrors the nature of web-scale datasets like LAION-400M (e.g., see Fig. 1a in [2]). As such, we view this as a strength of our experimental setup, which aims to closely replicate CLIP’s training while allowing for controlled dataset manipulations.\n\nTo mitigate the effect long-tail in the evaluations, we report *balanced* accuracy throughout (as noted positively by reviewer jK9M). Following reviewer H9Z3’s suggestion, we also verified consistency of our results for additional metrics, such as (standard) accuracy or F1 score.\n\n> Actually, improve domain diversity or learning shared information which could improve domain generalization have already been widely investigated in prior work.\n\nDomain diversity has been broadly studied, but we respectfully note that to the best of our knowledge, ours is the first systematic analysis of domain and compositional generalization in CLIP under OOD conditions. Besides this, we (i) analyzed the impact of partial class exposure on compositional generalization (Finding 2), and/or (ii) conducted mechanistic analyses demonstrating that feature and circuit sharing are needed for generalization (Findings 3 & 4). We would be grateful if the reviewer could provide references that have already explored this.\n\nThe references mentioned by the reviewer in “Essential References Not Discussed” enhance CLIP’s domain generalization via adapters [3] or style projectors [4]. However, they primarily focus on method improvements and do not offer in-depth analysis (as also positively noted by reviewer jK9M).\n\n> Include related works\n\nWe will add both references [3,4].\n\n> Figure 1A is confusing; please consider redrawing it and replacing the abstract symbols with actual image samples.\n\nWe used abstract symbols in Figure 1A to compactly highlight the difference between class sets $C_1$​ and $C_2$. We believe that actual image samples may obscure this important distinction. That said, if other reviewers share this concern, we are happy to revise the figure accordingly.\n\n--- \n\n[1] Mayilvahanan, Prasanna, et al. \"In search of forgotten domain generalization.\" ICLR 2025.\n\n[2] Wen, Xin, et al. \"What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights.\" NeurIPS 2024.\n\n[3] Yu, Xi, Shinjae Yoo, and Yuewei Lin. \"CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment.\" NeurIPS 2024.\n\n[4] Bose, Shirsha, et al. \"Stylip: Multi-scale style-conditioned prompt learning for clip-based domain generalization.\" WACV 2024.", "claims_and_evidence": "I'm not convinced that the experiments alone back the claims. CLIP's pretraining data is massive and very diverse, so it's unclear if the results are truly due to domain diversity or just the sheer size of the dataset. Additionally, the paper offers no theoretical evidence to support its claim.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "One issue is that DomainNet is a long-tail dataset, which may affect the claim.", "relation_to_broader_scientific_literature": "Actually, improve domain diversity or learning shared information which could improve domain generalization have already been widely investigated in prior work.", "essential_references_not_discussed": "Include two related works in the introduction—one that enhances domain diversity [1] and another that develops domain-invariant representations in CLIP [2].\n\n[1] Yu, Xi, Shinjae Yoo, and Yuewei Lin. \"CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment.\" Advances in Neural Information Processing Systems 37 (2024): 4267-4294.\n\n[2] Bose, Shirsha, et al. \"Stylip: Multi-scale style-conditioned prompt learning for clip-based domain generalization.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Figure 1A is confusing; please consider redrawing it and replacing the abstract symbols with actual image samples.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55XGE9wjne", "reviewer": "Reviewer_H9Z3", "summary": "This paper investigates the domain generalization and compositional generalization capabilities of CLIP models, focusing on how the diversity of training domains affects their ability to generalize to unseen domains and unseen class-domain combinations. The authors systematically construct training distributions with controlled domain diversity and object class exposure to evaluate CLIP's performance in various settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and constructive feedback. Below we address remaining concerns.\n\n> W1: Comparisons with other vision-language models (VLMs)\n\nWe focused on CLIP due to its wide adoption and use in prior related work (e.g., [1, 2]).\n\nWe had already verified the consistency of our results with SigLIP (which became more popular recently) in Figure 11 in Appendix B.1. As suggested, we additionally verified consistency of results with BLIP (which are also consistent). We chose to omit ALIGN due to its close similarity to CLIP.\n\n> W2: Discussion of dataset size and quality\n\nWe agree that both dataset size and especially quality influence generalization. We will include this to our discussion.\n\nRegarding dataset size: While Figure 10 in Appendix B.1 shows that large base datasets seem to improve generalization performance, there remains a gap to higher diversity settings. We would like to note that the higher generalization performance on CC12M may also be a confounding factor of a less clean and, thus, more diverse base dataset (see the main finding of [2]). As a result, the performance improvement when adding the diverse DomainNet samples is likely smaller because of this.\n\nRegarding quality: Unfortunately, quality remains only loosely addressed in much of the current literature (e.g., [3]), since it is not clear how to quantify it best. Recently, Schrodi et al [4] showed that an information imbalance leads to the modality gap. When we interpret information imbalance as one aspect of data quality, we might be able to indirectly measure this aspect of quality via the modality gap. However, we leave further investigation on how to quantify data quality for future work.\n\n> W3: Additional evaluation metrics\n\nAs suggested, we additionally evaluated with balanced top-5 accuracy and multi-class F1 score. The results are consistent with those based on balanced top-1 accuracy.\n\n> W4: Extensions to additional datasets beyond ImageNet-Captions and DomainNet\n\nFigure 10 in Appendix B.1 provides consistent results for the more diverse CC3M & CC12M datasets. However, CC3M & CC12M comprise images from a mix of domains, which creates confounders. Thus, for controlled experiments and clearer analysis, we primarily relied on the cleaner ImageNet-Captions dataset, following prior work [1]. The uncontrolled mix of domains is even more pronounced in the LAION dataset. Verifying our findings on LAION would therefore require robust automated class & domain annotation/filtering methods (which is technically challenging in itself) and large computational resources (which we lack).\n\n--- \n\n[1] Fang, Alex, et al. \"Data determines distributional robustness in contrastive language image pre-training (clip).\" ICML 2022.\n\n[2] Mayilvahanan, Prasanna, et al. \"In search of forgotten domain generalization.\" ICLR 2025.\n\n[3] Nguyen, Thao, et al. \"Quality not quantity: On the interaction between dataset design and robustness of clip.\" NeurIPS 2022.\n\n[4] Schrodi, Simon, et al. \"Two effects, one trigger: on the modality gap, object bias, and information imbalance in contrastive vision-language models.\" ICLR 2025.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "This paper puts forward four findings based on the experimental results, which are reasonable to infer and have sufficient evidence", "experimental_designs_or_analyses": "The authors designed a comprehensive set of experiments to investigate CLIP's domain and compositional generalization capabilities.  The key components of their experimental design include Controlled Training Distributions, the authors systematically varied the training data by constructing four different setups: Natural-only, Leave-out-domain, CG low-diversity, and CG high-diversity.  This approach allows them to isolate the effects of domain diversity and class exposure on CLIP's generalization performance.", "relation_to_broader_scientific_literature": "The generalization ability of CLIP under different conditions is systematically studied in this paper.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Adv:\n1.By constructing a variety of controllable training data distributions, the performance of CLIP in domain generalization and combinatorial generalization is systematically studied, and the generalization ability of CLIP is further revealed.\n2.The consistency of the experimental results is verified on different model architectures and a large number of data, which enhances the reliability and universality of the conclusions.\n3.The key effects of domain diversity and class exposure on CLIP generalization ability are revealed, especially the discovery that partial class exposure may weaken the combinational generalization ability. Four valid CLIP insights are proposed based on experiments.\n\nDis:\n1.The paper mainly focuses on the generalization ability of CLIP model, and does not compare with other similar visual-language models (such as ALIGN, BLIP, etc.). This makes it difficult for readers to fully understand CLIP's strengths and weaknesses in domain generalization and portfolio generalization.\n2.Despite the use of multiple datasets, such as DomainNet, these datasets may still be deficient in domain diversity and category richness, so the size and quality of the dataset may also have an impact on the generalization ability of the model, but this is less discussed in the article.\n3.The paper mainly uses top-1 accuracy rate as the evaluation index. It is suggested to introduce more evaluation indicators, such as top-5 accuracy, F1 score, etc., to evaluate the performance of the model more comprehensively.\n4.The current experiments focus on ImageNet-Captions and DomainNet datasets. It is recommended to extend the experiment to more datasets, especially those containing more diverse fields (such as LAION), to verify the generality of the conclusions.", "other_comments_or_suggestions": "Nothing", "questions_for_authors": "Nothing", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the generalization behavior of CLIP by varying training distribution and controlling diversity and object class exposure. The paper looks at domain generalization and compositional generalization, and shows that domain diversity is essential for both. This topic is of wide interest (generalizability of CLIP), and the paper includes a large set of experiments to provide more understanding about the limitations of generalization of CLIP. The paper also presents a mechanistic analysis suggesting that compositional generalization requires sharing of intermediate features and circuits. The reviewers have raised a few issues that the authors have addressed convincingly. The authors are requested to make relevant modification in the final version of the paper.
Decision: Accept (spotlight poster)