Is Large-scale Pretraining the Secret to Good Domain Generalization?
Authors: Piotr Teterwak
Kuniaki Saito
Theodoros Tsiligkaridis
Bryan A. Plummer
Kate Saenko
Venue: ICLR 2025 Poster (2025)
Keywords: Domain Generalization
Robustness
CLIP
Pretraining
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We show that current Domain Generalization methods don't generalize much beyond pretraining.

Abstract: Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. However, it remains unclear if DG finetuning methods are becoming better over time, or if improved benchmark performance is simply an artifact of stronger pre-training.  Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot performance, but we find the effect limited in the DG setting. Instead, we posit that  having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to introduce the Alignment Hypothesis, which states that the final DG performance will  be high if and only if alignment of image and class label text embeddings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it as an analysis tool of existing DG methods evaluated on DomainBed datasets by splitting evaluation data into In-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP. Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment. We release DomainBed-OOP at https://huggingface.co/datasets/PTeterwak/DomainBed_OOP.

Published as a conference paper at ICLR 2025
IS LARGE-SCALE PRETRAINING THE SECRET TO GOOD DOMAIN GENERALIZATION?
Piotr Teterwak', Kuniaki Saito', Theodoros Tsilikkaridis Bryan A. Plummer' Kate Saenko
Boston University
"MIT Lincoln Laboratory
{piotrt, kesaito, bplum, saenko) @bu. edu
tsili011, mit.m.itee...........
ABSTRACT
Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features Icarned from source data, and this has dramati- cally improved benchmark results. However, it remains unclear if DG finetun- ing methods becoming better over time, if improved benchmark perfor- mance is simply an artifact of stronger pre-training. Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot perfor- mance, but we find the effect limited in the DG setting. Instead, we posit that having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to in- troduce the Alignment Hypothesis, which states that the final DG performance will be high if and only if alignment of image and class label text embed- dings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it datasets by splitting evaluation data into in-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle DomainBed OOP, while recent methods excel on DomainBed-lll Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment. We release DomainBed-OO0 at /tcttps//huugiingg face. co/dato.t/..oooooooooo dataataaa dataaaaaooo sets/ P PTeterwak/Domai.ne
analysis tool of existing DG methods evaluated DomainBed
OCP
INTRODUCTION
Domain Generalization (DG) addresses the challenge of enabling AI models to generalize from known domains to unseen ones, a critical task given the inevitable distribution shifts between train- ing and real-world deployment (Saenko et al.] 2010). DG pipelines typically consist of three stages: pretraining model on a large, general dataset; finetuning the model with one or more source do- mains; and finally evaluating the model on target domains that are distinct from source domains. DG methods increasingly rely on huge-scale foundation models for initialization (e.g.. (Shu al. 2023] Cha et al. 2022; Addepalli al. 2024)). Simultaneously. finetuning has increasingly incorporated regularization to prevent catastrophic forgetting. As a result, remains unclear whether DG methods genuinely improving or if benchmark performance gains simply due to stronger pre-training, possibly with the target domains within the hundred million-scale pre-training data (Mayilvahanan al. 2025), combined with regularization.
In this work, we examine the reliance of recent state-of-the-art CLIP-based DG methods on pre- trained features. While prior studies have shown that perceptual similarity pre-training data plains zero-shot performance--refeeeed to as the Image Similarity Hypothesis (Mayilvahanan et al.) 2024 find this relationship to be limited in the DG setting. Despite evidence of target domains being present in pre- pre-training (Figure 4. we find that perceptual similarity alone does not fully ex- plain accuracy the DG context (Section 3). We argue that is not just the presence of similar data in pre-training that matters, but also how well this data was learned. To this end, we intro- duce the Alignment Hypothesis, which states that pre-trained alignment between image and class
Indicates equal contribution as senior author

---

Published as a conference paper at ICLR 2025
Source images
Target images
O
Label embedding
text
After Pretraining
After DG Method: Desired
After Method: Actual
Aligned:
"In Pretraining"
dog
dog C
dog
Misaligned: Out-Ot--rettaining
C
CC
C
i
Aligned: "In Pretraining"
cat
A
A
An overview of desired and actual behaviour of DG methods. 1) DG methods are ini- tialized with foundation models like CLIP. Pre-trained embeddings are relatively well aligned with ground truth labels on both source and target data for most samples (In-Pretraining, IP), but some samples are not well aligned (Out-of- pretroffirtrriing...mmmm OOP). 2) An ideal DG method would strengthen aligmment for both OOP and data with ground truth labels. 3) Our analysis shows that DG meth- ods only result in strong alignment for IP data, leaving OOP data misaligned (Figure
Figure 1:
embeddings is still predictive of DG performance even after source finetuning. We note that we do make assumptions of how or why alignment arose. In the Alignment Hypothesis, pre-training alignmnent is used as measure for how well a sample is Iearned. We find that performance for low alignment samples can be almost while performance for high alignment samples is close to perfect. These results confirm the Alignment Hypothesis. As illustrated in Figure I these findings suggest that current DG methods largely fail to learn new, general features from the source data when the pretraining does not already provide a strong alignment.
The confirmation of the Alignment Hypothesis gives us a tool to separate aligned and well learned in-pretraining (IP) data from misaligned and poorly learned out-of-pretraining (OOP) data for particular backbone, and we do so for five DG datasets with OpenCLIP- ViTT/B--1. resulting splits DomainBed-IP and DomainBed-OOG view of where current DG methods fail and where they succeed. We focus on CLIP-based meth- ods, as they used in state-of-the-art DG methods (Addepalli
call the
Evaluating on DommaiBedd-1P/OOp offers
al., 2024 Cho et al. 2023 Shu let al. 2023 Mao et al. 2024); we believe extensions to pure vision models such as DINOv2 (Oquab al. (023) represent interesting future work. We find that all methods, including those consid- ered state-of-the-art, perform poorly on OOP data, i.e., data that the pretrained backbone hadn't 't already aligned well. Furthermore, recent state-of-the-art methods do not outperform older methods on OOP data; CLIPood (Shu et al. (023) slightly under performs a combination of older methods (MIRO (Cha et al., (022) MPA (Arpit al., 2022)) on DomainBed-OOe
the same time, exist- ing DG methods show exceptional performance on Domain sometimes even outperforming an oracle model trained on the target domain. These results suggest that future research should aim to enhance DG methods on low-alignment data. In summary, make the following contributions:
Introduce the Alignment Hypothesis: We demonstrate that pre-training alignment be- tween image and class text embeddings is stronger predictor of Domain Generalization (DG) success than the previously proposed Image Similarity Hypothesis (Mayilvahanan) et al.) 2024). Based on this, we define In- In-Pretraining (IP) data well-aligned with pre- trained embeddings, and Ort-of-Pretraining (OOP) as data with weaker alignment.
Propose new IP/OOP evaluation framework: We demonstrate that splitting target data by its alignment with the pre-trained backbone can effectively test Out-oof-Petraiing (OOP) generalization. We release IP/OOP splits for DomainBed datasets.
Expose strengths and limitations state-of-the-art DG methods: Using DomainBed- IP/OOP find that leading DG methods perform well on data well-aligned by pre-training
2

---

Published as a conference paper at ICLR 2025
but struggle on misaligned samples, emphasizing the need for methods that move beyond reliance on pre-training.
2
RELATED WORK
Multi-Source Domain Generalization: Domain Generalization aims to mitigate the impacts of domain shifts between source (training) and target (deployment) domains. These can include sub- population shifts where all testing subpopulations are present in the training data but in different proportions (Dehdashtian et al. 2024). or it could be the case we consider in this work where the testing subpopulation is not at all present in the training subpopulation. Although we focus on the multi-source domain generalization task, where domains between train and test are completely disjoint, our analysis can be extended to other types of generalization. One standard approach is domain-invariant feature learning. which leverages domain labels to learn domain- invariant fea- tures. CORAL (Sun & Saenko, (016)) aligns second-order statistics, while DANN (Ganin et al. 2016) and ADDA (Tzeng et 2017) uses an adversarial loss. Gulrajani & Lopez-Paz (2020) show that ERM, which does not align features between domains, can outperform most prior work while being easier tune. Another common approach is domain-aware data augmentation to expand the training domain to become closer to or even overlap the target domain. Inter-domain mixup (Yan al.] 2020) blends images from different domains. Similarly, style transfer can diversify training images (Zhong et al. 2022). Deep ensembles are effective for domain generalization (Arpit et al., 2022). Since they are computationally inefficient for inference, many recent works average model
weights from either multiple finetuning runs or from a single training trajectory (Cha et al. 2021
Arpit et al., 2022, Rame et al.] 2022] Jain et al.] 2023 Li et al.] 2023) (Shu et al. (023). More recently, several methods perform regularized finetuning towards the initialization of pretrained model. This works under the assumption that pretrained features are useful for target data, and should not be unlearned. The general idea can be applied to weight space (L2SP (Xuhong et al.
2018)), feature space (MIRO (Cha et al. 2022 ), or output space (e.g., CAR-FT (Mao et al. 2024 CLiPood (Shu et al. 2023)).
Large-Scale Pretraining for DG: Recent DG literature (Cho et al. 2023 Cha et al.
2022 Adde-
palli et al. 2024 Mao et al. 2024| Arpit al., 2022) leverages large-scale pretrained initializations stronger than ImageNet (Russakovsky et al.] 2015). and CLIP (Radford et al. 202T is the most common choice. CLIP leverages a cross-domain contrastive loss to align images and captions. Due to the large scale of training data (typically at least 400 million samples) and the free-form nature of the text, CLIP enables effective zero-shot classification and learns features that generalize very well. Other choices for very strong pretraining include SWAG (Singh et al., 2022) and Dino (Oquab) jet al.] 2023). SWAG uses supervision from Instagram hashtags, while DinoV2 trained without text supervision and instead relies on augmentation-basel alignment. While our analysis focuses on image-text models like CLIP due to its popularity, the concept of alignment can extend to other types of pretraining models. We leave the exploration of this extension future work.
Impact of Data on Model Performance: Several recent studies have explored the influence of pre- training data model performance. Mayilvahanan al. (2024) investigated how the presence of perceptually similar images in CLIP (Radford et al., 2021) pretraining affects performance, intro- ducing the Similarity Hypothesis, which posits that nearest neighbor similarity is strongly correlated with zero-shot accuracy. Mayilvahanan el al. (025) show that domain contamination in the pre- training has substantial impact on DG performance. [Udandaroo et al. (2024) demonstrated that concept frequency in pretraining is correlated with zero-shot performance and introduced a dataset focusing on infrequent concepts. Fang et al. (2022) found that diversity in pretraining data is criti- cal for improving performance on benchmarks such as ImageNetV2 (Recht et al. (209)) ImageNet- (Hendrycks et al. (02T), ImageNet- Sketch (Wang et 2019). and ObjectNet (Barbu et al., 2019). However, these studies focus on the zero-shot setting, where models are evaluated without further training. In contrast, we examine the domain generalization setting, where pre-trained models are fine-tuned on source domains and tested on held-out target domains. Yu et al. (2024) recommend using self-supervised pre-training to avoid data label leakage. In contrast we study DG model be- havior in the more realistic setting of CLIP- prettrrata.nnn Our findings suggest that comparing target images to pre- pre-trained images, as proposed by Mayilvahanan et al. (2024), is less predictive of final DG performance than directly measuring the alignment between the image and its class embedding.
3

---

Published as a conference paper at ICLR 2025
a.) 1.-
b.)
10-
DG Method
(CVPS
V12+-50
Demaintet lerraincog
0.0
0.6
Domsinkke OfficHlome
Accurecy
0.4
0.2
0.2
0.775
0.800
3,825
0.850
3,075
900
3.525
0.950
0.30
0.20
0.25
0.35
Iaroettutiiniarppe. RySooo
AlgnmentScore
Figure 2: Comparing the Predictive Power of the Alignment and Image Similarity Hypotheses for Domain Generalization (DG). a.) Image Similarity Hypothesis: The cosine similarity be- tween a test image and its closest match from the pre-training set (Perceptual Similarity Score) shows relatively weak predictive power for accuracy. implying that visual resemblance alone is not fully indicative of downstream performance. b.) Alignment Hypothesis: In contrast, the cosine sim- ilarity between image and ground truth text-label embedding after pre-training (Alignment Score) is highly predictive of model accuracy after fine-tuning on five DG datasets, with Aligmment Score distributions shown in the colored histograms. This suggests that image-text pairs well-aligned dur- ing pre- pee-traienneee training result in better performance on target tasks.
ANALYZING THE ROLE OF PRETRAINING IN DOMAIN GENERALIZATION
This work explores Multi-Source Domain Generalization for classification, where samples from multiple source domains (e.g.. sketches, product photos) and a held-out target domain (e.g.. wildlife camera images) are annotated with both domain and class labels. We construct a training dataset by aggregating all sample-label pairs from all training domains d E (di..... dn J. denoted as
D {(x:,
YY,)).
We initialize classifier f with contrastively pre-trained vision-languaat model (e.g., CLIP) and finetune it on D. The scale of pre-training datasets is many orders magnitude larger than that of source datasets. Most methods fully fine-tune f, though LP-FT (Kumar al. 2021 fine-tunes the linear probe before the main network and Attention Tuning (Teterwak et al.) 2023] Touvron et al., 2022) only tunes attention layers. The performance is then evaluated a held-out testing domain diest- The key assumption is that drest has a different distribution from the source domains.
We aim analyze how reliant existing DG methods are on pre-training. A recent analysis of CLIP proposed the Image Similarity Hypothesis (Mayilvahanan et al., 2024), which supposes that high CLIP performance on a given test sample is a result of highly similar nearest-neighhor images in pre-training, and tested it on zero-shot classification tasks. They found strong correlation between neighbt similarity and zero-shot classification performance, but did not analyze OOD per- formance after fine-tuning. Therefore, we apply equivalent testing setup for the DG setting, where a pre-trained model is fine-tuned on source distribution and tested on a different target distribution. We find only limited influence of image similarity in Section 3.1 To better understand the role of pretraining in domain generalization, we introduce the Alignment Hypothesis, which we explore in detail in Section 3.2 We later use the Alignment Hypothesis to split DG datasets and analyze existing DG methods (Section
3.1 TESTING THE IMAGE SIMILARITY HYPOTHESIS
The Image Similarity Hypothesis Vahanlyahanan et al. 2024 posits that test performance im- proves when there are perceptually similar images in the pre-training dataset. The PerceptualSim- ilarityScore measures perceptual similarity and is defined the cosine similarity between target image I and its nearest neighbor I₁ in pre-training:
fI PerepttalSimilariySScor// = [[F(())I (I+II
(Eq. 1)
4

---

Published as a conference paper at ICLR 2025
Algorithm Evaluating the Image Similarity Hypothesis
Require: Target domain samples Durgets trained DG model M, pre-trained image encoder fr,
1: for each sample E Diarget do
2: Retrieve nearest neighbor of I in LAION-400M using fi features, assign to Ix 14) using Equation Eg
3: Compute Perceptuallimilarttysooore
4: Record correctness of (1)
5: end for
6: Bin samples based on PerceptualSimilarttySSooo
7: Compute DG accuracy within each bin
8: return Accuracy for each bin
where (--) denotes the dot product, and || denotes the Euclidean norm (magnitude). To evalu- ate the Image Similarity Hypothesis, we bin held-out target domain samples from five DomainBed datasets based on their PerceptualSimilartySScoo
and compute the accuracy of a Domain General- ization (DG) method independently for each bin. This procedure detailed in Algorithm | and visu- The SerceptualSimilarrtySSoeo is computed using approximate nearest neighbors over LAION-400M. (Schuhmann et al. 2021) with the CLIP-retrieval library (Beaumont 2022).
alized in Figure 2
Figure 2 a.) shows the results of this analysis of the recent, high-performing VL2V-SD (Addepalli ct al. 2024) on various DG datasets. While the Image Similarity Hypothesis is somewhat predictive of DG performance, its influence is not very strong. This suggests that perceptually similar pretrain- ing data alone may not guarantee high DG performance; additional factors, such as how effectively the nearest neighbors were aligned with the target concept, may also be significant.
3.2 INTRODUCING THE ALIGNMENT HYPOTHESIS
To find a stronger predictor of DG accuracy than perceptual similarity, we focus on how effectively pre-training captures the relationship between an image and its label. This leads us to propose the Aligument Hypothesis, which states that if an input image and its corresponding text label (e.g., 'A photo of a (cls}') are well-aligned the embedding space, final DG performance will be high. Crucially, alignment is measured before source fine-tuning while DG performance is measured after adaptation. This allows us to isolate the contribution of fine-tuning. Since models like CLIP opti- mize image-text pairs using contrastive loss, cosine similarity between image and text embeddings is an alignment measure well coupled to their training objective. Therefore, we use it as our metric of pre- pre-training generalization. More formally:
(T(T) AlignmentScore(1,7 = [f -1
(Eq. 2)
where fI) is the embedding of the image before finetuning on source, and fT" is the embedding of the text. Relative to directly using the contrastive loss value which also depends on negatives, PereepualSimilaritySoor has a scale which aligns across datasets (Appendix (B.6).
We verify the Alignment Hypothesis similarly to the Image Similarity Hypothesis, by binning sam- ples using the AlignmentScore and computing accuracy for each bin using VL2V-SD, We provide the same analysis for many more DG methods in the Appendix Figure [I In Figure b.). we can see that the Alignment Hypothesis explains DG performance after source finetuning, significantly more strongly than for the Image Similarity Hypothesis in Figure a.) This finding suggests that source men-ttning in DG, which aims to achieve high performance across all target samples, only succeeds those with high initial alignment.
2
RE-THINKING DOMAIN GENERALIZATION BENCHMARKING USING THE ALIGNMENT HYPOTHESIS
Knowing that the Alignment Hypothesis holds for contrastively trained image-text models (Section 3), we can it as tool to probe the performance of DG methods across different levels of pre- vre-training alignment. We apply this approach to five widely-used DomainBed (Gulrajani & Lopez- Paz 2020) DG datasets: VL.C (Fang al. 2013), PACS (Li 2017), OfficeHome (Ganin al.
5

---

Published as a conference paper at ICLR 2025
AlignmentScore
0.0 0.1
0.1 0.2
0.2
0.3 0.4
0.4 6.4-00
Dataset
Bird
Lipstick
Table
Erascr
DomainNet
Coyote
Dog
Bobcat
TerraIncognita
N/A
N/A
Person
Giraffe
House
Elephant
PACS
N/A
Chair
Dog
Person
Car
VLCS
N/A
Candles
Monitor
Bike
Knives
Clipboard
OfficeHome
Figure 3: Representative DomainBed dataset samples and their labels at various AlignmentScore values, At very low AlignmentScores, most labels (red boxes) are incorrect. At very high Align- mentScores, text present in the image corresponds to the label.
2016), Terralncognita (Beery et al. 2018), and DomainNet (Peng et al. 2019). This section dis- cusses how we create new splits for existing DG datasets using AlignmentScore. We start by com- puting AlignmentScore for all samples in 5 DG datasets (Figure 3) Based on our observation that some samples are mislabeled, we perform dataset cleaning (Section 4. Then we find an Align- mentScore threshold split DG datasets into well aligned In- In-Pretraining (IP) and poorly aligned out-of-Pretraining (OOP) evaluation subsets (Section 4.2) which we later use for evaluating DG methods (Section 5). In order connect the AlignmentScore with the DG method, backbone both for splitting the datasets into IP and OOP subsets and for training DG methods.
the same
4.1 DATA EXPLORATION AND CLEANING
We start by visualizing the data of all the datasets at various AlignmentScore values. We show some representative samples in Figure 3 At very low scores, we find that large fraction of labels incorrect (red boxes in Figure 3). Thus, divide the data into AlignmentScore intervals (e.g., 0.00-0.05, 0.05-0.10 10, and so on, up to 0.2) and randomly sample 100 instances from each interval for every dataset. This allows us systematically analyze the relationship between AlignmentScore and label accuracy across different score ranges. For each interval, we then count the fraction of mislabelled samples better understand how low AlignmentScores errors. We find that below an AlignmentScore of 0.15, label noise is unacceptably high, with all datasets suffering the most from mislabelling (Table 7 Appendix). Therefore, we discard all samples with AlignmentScore less than 0. in DomainBed-1P/OOF As shown in Table & (in the Appendix), we observe that the percentage of discarded samples due to mislabeling varies across datasets, with VLCS and DomainNet having the highest rates 12.41% and 7.64%, respectively.
associated with labeling
Furthermore, on the right side of Figure 3 than 0.4), images often contain text directly related to the label. However, goal is evaluate visual recognition rather than text recognition (OCR), and CLIP is known to have strong OCR abilities (Fort 2021). IP/OOP. As shown in Table the Appendix). only a small portion of data removed due to OCR filtering (0.00-0.155 15%% across datasets), but this issue may become more significant in future studies.
we observe that at very high AlignmentScores (greater
exclude all samples above AlignmentScore of 0.4 from DomainBed-

---

Published as a conference paper at ICLR 2025
4.2
DATA SPLITTING
After filtering, we focus on determining a threshold to split the dataset into In-Pretraining (IP) and out-of-Pretraining (OOP) subsets. We select 0.21 as the threshold, based on the trends observed in Figure b.), as this is the point where performance begins to improve significantly. indicating that existing methods become more effective. While this threshold represents somewhat subjective choice informed by observed patterns, we provide AlignmentScores in the released data, allowing researchers the flexibility to experiment with their own thresholds
Figure 18jin the Appendix shows how this split impacts the size and composition of each dataset. For example, clipart in DomainNet predominantly categorized as IP, likely due to its frequent presence on the internet and, therefore, web-scraped pre-training data. In comparison, Berralncognita-OOP is more balanced domains, but exhibits substantial class shift between IP and OOP splits (Figure [7 in the Appendix), meaning some classes are better aligned than others during pre-training.
We further investigate the distinctions between the IP and OOP subsets using VisDiff (Dunlap et al. 2024). LLM-based system that identifies differences between sets of images. For each combi- nation of dataset, domain, and the combinations due computational constraints- we independently sample up 30 images from both the IP and OOP subsets. VisDiff employs CLIP (Radford et al., (02T)) to compute an AUC- ROC score for the natural language differences proposed by the LLM, and we retain only those differences with an AUC score of 0.7 or higher. Results are presented in Appendix Table
for DomainNet, where we subsample roughly 15% of
Several clear patterns emerge from our analysis. Contextual environmental elements often over- shadow the primary object indicated by the text label. For example, in VLCS's SUN09 domain, the OOP subset for the car class frequently features images described as "historical architecture," while in Office Home the real-domain bed images are deemed OOP if they include "chiil-themed decor." These findings imply that object size may also play a role. In the car example, visible architecture implies that the car occupies only small portion of the image, and in the bed example, the pres- ence of child-themed decor indicates that the entire bedroom is visible rather than just the bed. To computed the object size of the ground truth labeled objects using bounding
test this hypothesis, boxes generated by the open-world object detector OWLv2 (Minderer et al., 2023). We found object size is correlated with the AlignmentScore (see Appendix Figure 10), with the sole exception being Terralncognita. This indicates that even the presence of small wildlife can substantially increase the AlignmentScore, and that the background vegetation does not present a significantly conflicting sig- nal that is sometimes present in other datasets. Overall, our findings suggest that pre-training which better represents scenes with multiple objects or concepts may improve benchmark performance.
5 EXPERIMENTS
We evaluate Domain Generalization (DG) performance across both imetraining-aaignen (IP) and pretraining-misalignn (OOP) data. We adhere to the DomainBed evaluation methodology, where one domain is chosen as the target, and the remaining act as source domains. To maintain a sufficient amount of training data, we train all DG methods on the original, unsplit datasets. We use hyper- parameter values recommended by the original implementation authors for each method.
After training, models are evaluated separately on IP and OOP subsets, as well the original, unsplit test domain. This allows us to measure how well each method generalizes to both pretraining- aligned and setraaning-misalignnnn
data. We follow the literature's standard practice of computing performance per target domain and averaging the results across all domains. This method ensures that any domain imbalances do not disproportionately influence the final performance metrics.
5.1 ALGORITHMS
Explicit Regularization towards Pretraining: Several recent DG methods leverage explicit reg- ularization towards the initialization. These methods generally operate either in weight space (by regularizing or freezing model parameters) or in feature space (by aligning internal feature rep- resentations with those of the pretrained model). MIRO (Cha et al,, 2022) minimizes the Mutual Information between DG model intermediate features and CLIP intermediate features. Attention Tuning Teterwaak et al. 2023 Touvron et al. 2022) freezes all parameters except those in the
7

---

Published as a conference paper at ICLR 2025
Multiheaded-Atte Attention Layers. VL2V-SD (Addepalli et al. 2024) self-distills linear combination of CLIP vision and text outputs into a model. CLIPood (Shu et al. 2023) regularizes both weights
and output features. Weights are averaged between the pre- pre-trained CLIP model and the fine-tuned
model, and outputs are regularized using a loss that incorporates information from the pre-trained text encoder. Linear Probe Fine Tuning (I.P-FT) (Kumar al. (202T) freezes the backbone, trains a linear probe, and then performs full finetuning. An untrained linear probe can cause finetuning to update the frozen backbone needlessly, potentially unlearning discriminative features. This biases the model towards pre-trained weights with smaller gradient updates.
Domain Invariance: A classic idea for Domain Generalization is Domain Invariance in the feature space, where the model learns only class-discrminattivn features shared among all training/source domains. CORAL (Sun & Saenko, 2016) matches the second moments of features across different domains and has been shown to be highly effective (Gulrajani & Lopez-Pazz 2020).
Studies by Izmailov et al. (2018) and Cha al. (20TT) have shown that flat minima generalize better than sharp minima, as they make loss values less sensitive to perturbations in the loss surface, resulting in smaller increases in loss during domain shifts. SWAD (Cha et al. 2021] Averages model parameters during training. determining the interval over which to average using validation loss over source domain. Model Parameter Averaging (MPA) (Arpit et al. 2022) Starts averaging model parameters after number of burn-in steps to find flat minima. SAGM (Wang al. 2023 is an optimizer that explicitly optimizes for flat minima.
Flat Optima:
We also train baseline and oracle model for lower-bound and upper-bound reference. The baseline model is an Empirical Risk Minimization (ERM) model that is finetuned on source domains and evaluated
Baseline and Oracle:
target domains, and has been found be effective for the DG task (Gulrajani & Lopez-Paz 2020). The oracle model is trained on an 80% training split of all domains and evaluated on 20% test split. The oracle model removes the OOD aspect of generalization and provides reasonable upper bound for DG methods. Finally, we add a zero- shot LAION-400M (Schuhmann al. 2021) pre-trained ViT-B/16 from OpenClip, which is the pre-trained model used in our analysis.
5.2 RESULTS
Table reports results for the poorly pre- pre-training aligned DomainBed-OOPF higly pre-training aligned DomainBed-IP, and standard DomainBed datasets. Underlined results highlight the best performance of any single method (excluding method combinations), while the bold numbers show the highest performance overall, excluding the upper bound and zeros-hot baseline. Overall, we find that AlignmentScore is highly predictive of performance. More detailed observations about the results follow, while a comparison to PercoptualSimilariiySoor can be found in Appendix BB.5
DG methods perform well on DomainBed-IP. In most datasets within DomainBed-IP, domain generalization approaches achieve excellent performance. On two out of five datasets ( Officeloome and PACS), the best DG method even outperforms the oracle. A notable exception is Terralncognita, where CLIPood scores 30% below the oracle, highlighting this dataset as a challenging outlier. Inter- estingly, all three datasets where performance exceeds the oracle have an average IP AlignmentScore of 0.28, while the others have a lower average AlignmentScore of around 0.26. Therefore, the un- derperformance of TerraIncognita may be partially explained by its lower IP AlignmentScore, sug- gesting that alignment plays a significant role in DG performance, even within the IP case. Another interesting observation is that on the IP subset, the zero-shot model can achieve performances greater than the finetuned models (for PACS and VLCS). This means that, for IP- IP-data, DG finetuning some- times causes more catastrophic forgeting than learning of new features We show additional analysis of pero-shot models in Appendix B.9
DG Methods leave much to be desired on DOmOinBed-OOP, but are still stronger than ERM: In DomainBed-OOP, we observe that even the top-performing DG methods struggle with low- alignment data. For example, CLIPood achieves 57.1% % accuracy, which is a significant drop com- pared to its performance on DomainBed-IP (84.7%) and DomainBed-All (78.1%) Despite this, DG methods still outperform Empirical Risk Minimization (ERM) by up to 10% on DomainBed-Onl OOP.. Thus, DG methods better equipped to handle domain shifts than ERM, possibly due to weak transfer of knowlegdge from pre-trained features. Nevertheless, there is still substantial room for improvement on low-alignment samples.
8

---

Published as a conference paper at ICLR 2025
Table 1: Benchmarking DG methods on DB-IP/OOP. Domain generalization methods excel on high-alignment (IP) datasets-often even surpassing the oracle-whil their performance notice- ably drops on low-alignment (OOP) data, though still outperforming ERM.
DomainBed-IP
DomainNet
OfficeHome
PACS
TI
VLCS
Average
OpenCI.IP 7S
74.9
88.9
98.5
36.8
95.9
79.0
CORAL
Sun & Sacnko 2016)
63.3
76.1
84.3
42.9
86.5
70.6
SAGM
Wang al. 2023
64.3
79.5
90.1
44.0
88.0
73.2
ERM
Gulrajani
Lopez- Paz
2020
63.1
78.1
87.1
42.0
85.3
71.1
LP-FT
Kumar el al.
2021
64.4
78.5
40.9
86.0
72.0
SWAD
Cha el al.
2021
72.3
84.8
94.6
52.7
88.5
78.6
MIRO
Cha al.
2022
72.4
88.8
97.6
58.9
91.0
81.7
VL2V-SD Addepali
2024
78.1
91.4
98.0
48.1
92.4
81.6
Attn Tune (Teterwak al
2023
69.2
84.8
53.0
88.7
78.4
MPA (Arpit et al. (0222)
73.6
85.1
95.4
54.4
90.7
79.8
CLIPODD (Shu of 2023
78.9
90.9
97.7
63.5
92.5
84.7
MIRO + SWAD
77.0
90.5
97.6
62.1
91.1
83.6
MIRO + MPA
78.2
90.7
98.1
62.6
91.0
84.1
Upper Bound (Target Finetune)
81.6
88.5
97.8
93.4
93.8
91.0
(a) Samples with high AlignmentScore values, indicating good pretraining alignment.
DomainBed-OOP
DomainNet
Officeloome
PACS
TI
VLCS
Average
OpenCLIP ZS
26.3
48.1
81.4
4.5
80.2
48.1
CORAL
Sun & Sacnko 2016)
22.3
42.6
74.1
16.0
74.0
45.8
SAGM
Wang al. 2023
23.0
44.5
74.2
19.3
73.3
46.9
ERM
Gulrajani Lopez- Paz
2020
22.3
42.9
16.5
76.4
47.0
LP-FT
Kumar cl al. 2021
22.7
78.6
23.1
70.7
47.7
SWAD
Cha et al.
2021
28.6
49.9
79.1
21.0
77.0
51.1
MIRO
Cha al.
2022
28.4
56.6
84.7
18.5
73.7
52.4
VL2V-SD (Adeealli
2024
31.8
56.6
85.0
15.9
79.1
53.7
Attn Tune
[Tteewwak et 202.1
26.8
20.3
76.1
51.8
MPA (Arpil et al. (0022)
29.6
$1.0
82.7
22.2
79.5
$3.0
CLIPOOD (Shu et al.
2023
33.9
63.9
87.2
19.9
80.7
57.1
MIRO + SWAD
32.0
59.0
85.4
21.1
78.9
55.3
MIRO + MPA
33.1
60.0
87.8
24.9
80.3
57.2
Upper Bound (Target Finetune)
48.8
61.9
92.9
83.22
92.4
75.8
(b) Samples with lower AligmmentScore values, representing cases where pretraining alignment weak.
DemainBed-All
DomainNet
OfficeHome
PACS
TT
VLCCS
Average
OpenCLIP ZS
59.5
85.4
97.0
33.2
82.4
71.5
CORAL
Sun & Saenko
2016)
50.6
73.2
83.2
39.6
78.5
65.0
SAGM
Wang al.
2019
51.5
76.4
87.5
41.0
80.4
67.3
ERM
Guirajani
Lopez- Paz
2020
50.5
75.0
85.2
39.0
77.9
65.5
LP-FT
Kumar et al.
2021
51.3
75.5
88.4
38.5
78.0
66.3
SWAD
Cha et al
2001
57.9
81.8
92.4
49.0
80.1
72.2
MIRO
Cha
2022
57.5
85.8
96.4
54.3
81.1
75.0
VL22--D Addepalli
2024
62.0
88.3
96.9
44.4
82.7
74.9
Attn Tune Telerwak et al.
2023
55.4
81.9
49.1
81.8
72.7
MPA (Arpit al. 2022
58.9
82.0
94.3
50.7
82,3
73.6
CLIPOOD (Shu et al.
2023
63.6
88.3
96.8
58.5
83.4
78.1
MIRO + SWAD
61.4
87.6
96.6
57.4
82.0
77.0
MIRO + MPA
62.4
87.9
97.2
58.2
82.8
77.7
Upper Bound (Target Finetune)
70.4
86.2
97.2
92.4
87.9
86.8
(c) Performance of DG methods unsplit DomainBed

---

Published as a conference paper at ICLR 2025
SOTA methods do not consistently outperform older methods OOP data: While CLIPood (Shu et al. 2023) clearly outperforms other methods on DomainBed-All. it performs com- parably to older methods on DomainBed-OOf For example, MIRO MPA (Cha et al. 2022 Arpit et al. 2022) achieves 57.2% on DomainBed-OOP, which is nearly the same as CLIPood's 57.1%
This suggests that CLIPood's primary advantage comes from well-aligned samples in DomainBed- IP. where obtains 0.5% better performance than the next-best method.
Model Parameter Averaging (MPA) boosts performance on OOP data: MPA obtains sig- nificant 6% gain over ERM on DomainBed-OOP. MPA is 0.5% better than MIRO on OOP data despite MPA being 2% worse than MIRO on IP data. When combined with MIRO, MPA delivers the best performance on DomainBed-OOl OOP, slightly surpassing CLIPood. This suggests that MPA can complement other regularization-ben -based methods like MIRO. On DomainBed-IP, MIRO + MPA is less than 1% away from CLIPood's performance, demonstrating versatility across both high- and low-alignment data. Interestingly, SWAD underperforms MPA on DomainBed- OOP by 2%, despite being conceptually similar. We attribute this to selecting the averaging interval the source data, which introduces overfitting to source domains.
5.3 DISCUSSION
As an increasing number of works in the Domain Generalization sub-field leverage pre-trained CLIP models for Domain Generalization benchmarks, it is important better characterize the impacts of pre-training on DG. We leave the reader with the following takeaways:
Pre-training Alignment Predicts DG Performance: Our study demonstrates that pre-training alignment, measured as the cosine similarity between image and text embeddings, is a robust predic- tor of DG performance. This holds true even after source fine-tuning, highlighting that the quality of alignment achieved during pre-training has significant impact on the generalization capability.
Current DG Methods Exploit Pre-training Rather Than Learning New Features: Our findings reveal a large difference in the performance of DG methods between metraining-alignee (IP) and vetraining-misalignnn (OOP) data. While state-of-the-ar methods achieve near-oracle performance on IP data, they struggle significantly on OOP data. This indicates that current methods primarily leverage on pre-trained features rather than learning new, generalizable features from source data. Consequently, their success is heavily tied to the quality of pre- pre-training, rather than fine-tuning.
The reliance on pre-trained alignment calls for reevaluation of DG benchmarks. Existing benchmarks often aggregate results across all target data, masking the limitations of DG methods on low- olw-aligmment samples. To address this, we propose splitting evaluation datasets into In- Pe-Peerrainnng (IP) and Out-of- Prtt-ff-retraining (OOP) subsets. This provides a clearer picture of where DG methods succeed and where they fail. We hope that our proposed DomainBBedTPP/OO splits will guide the development of future methods that are better equipped to handle low-aligmment data while maintaining performance on high- alggmentgnmnt samples.
Benchmarks Should Reflect Pre-training Reliance:
CONCLUSION
We systematically explore how Domain Generalization (DG) methods rely on pre-trained feature alignment from models like CLIP. We hypothesize that the alignment between image and text em- beddings during pre-training strongly predicts DG performance. Our experiments confirm this, showing that methods perform well on high-alignment samples (DomainBed-IP) but struggle on low-alignment data (DomainBed-OOP) near oracle-level on aligned data but see significant drops on misaligned samples. This suggests current DG methods rely on pre-trained features and fail to learn new, generalizable features from source domains. Moving forward, two paths emerge: developing DG methods that better learn gen- eralizable features, or focusing improving pre-trained backbones. While foundation models will continue to advance, there will always be specialized distributions where they fail. Therefore we take the stance that improved DG finetuning remains important avenure of research. We hope these findings inspire further research into improving generalization on low-alli www-aiigmeen data, pushing DG beyond reliance on pre-trained alignment.
Notably, state-of-the-an methods like CLIPood perform
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engi- neering under Air Force Contract No. FA8802-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
REFERENCES
Sravanti Addepalli, Ashish Ramayee Asokan, Lakshay Sharma, and R Venkatesh Babu. Leveraging vision-language models for improving domain generalization in image classification. CVPR, 2024.
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35:8265-8277 2022.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them.
www.ccom/ooooo. 504/ htt.a//b...ooooooool/lol///aeeee
2022.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV). pp. 456-473, 2018.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. European Conference on Computer Vision (ECCV), 2022.
Junhyeong Cho, Gilhyun Nam, Sungyeon Kim, Hunmin Yang, and Suha Kwak. Promptstyler: Prompt-driven style generation for source-free domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Sepehr Dehdashtian, Lan Wang, and Vishnu Naresh Boddeti. Fairerclip: Debiasing zero-shot pre- dictions of clip in rkhss. In International Conference on Learning Representations, 2024.
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E. Gonzalez, and Serena Yeung-Levy. Describing differences image sets with natural language. In Conference Computer Vision and Pultern Recognition (CVPR), 2024.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip). In International Conference Machine Learning, pp. 6216-6234, PMLR, 2022.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference Computer Vision, pp. 1657-1664, 2013.
Stanislav Fort. Pixels still beat text: Attacking the openai clip model with text patches and ad- versarial pixel perturbations, March 2021. URL htt.:/sss//saaalaaoooooo.....pp
aan__ddversariall exwmpexaeees..eeee html 1o/
2021/03/ 15/005////n __CIPP st ckers wttpecckess_adaddeeeeeeeee
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Nomain-adversarian training of neural networks. Journal of machine learning research, 17(59):1-35. 2016.
11

---

Published as a conference paper at ICLR 2025
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of gut-of-distributiin generalization. ICCV, 2021.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL hitps: /tt.//c.i.// wtt./or.g........ If you use this software, please cite it as below.
zenodo.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407. 2018.
Samyak Jain, Sravanti Addepalli, Pawan Kumar Sahu, Priyam Dey, and Venkatesh Babu. Dart: Diversifyaggregate-rraaa training improves generalization of neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16048-16059, 2023.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Con- ference on Learning Representations, 2021.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
Ziyue Li, Kan Ren, XINYANG JIANG, Yifei Shen, Haipeng Zhang, and Dongsheng Li. SIMPLE: Specialized model-sample matching for domain generalization. In The Eleventh International Conference on Learning Representations, 2023.
net net//orumm
iddBqrrPe__e5p
Xiaofeng Mao, Yufeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. International Journal of Computer Vision, 13265):1685-1700, 2024.
Prasanna Mayilvahanan, Thaddaus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Brendel. Does CLIP's generalization performance mainly stem from high train-test similar- ity? In The Twelfth International Conference Learning Representations, 2024. URL
http:::///peeeeveewe .nt./ http:a/wwwwweeeeeeoooeeeeee--e-. wttp:/oummmddttaaiiddooo
Prasanna Mayilvahanan, Roland S. Zimmermann, Thaddaus Wiedemer, Evgenia Rusak, Attila Juhos. Matthias Bethge, and Wieland Brendel. In search of forgotten domain generaliza- tion. In The Thirteenth International Conference on Learning Representations, 2025. URL
https: http:////peeeeeiw.ee...eee Low.
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocyy object detection.
Advances in Neural Information Processing Systems, 36:72983-73007, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2rX1v:2304.077193, 2023.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Peng,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8765 PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamoniy. Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for -ustr--ditt generalization. In NeurIPS, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1 11(())211122) 252, 2015. doi: 10.1007/s1 1263-015-0816-y.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: 11th European Conference Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp. 213-226. Springer, 2010.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. ArXiv, abs/2111.02114, 2021. URL https://api.
Isemnt org/Corpus http:/ooppps1::22133310000 241033103
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang. Jianmin Wang. and Mingsheng Long. Clipood: Generalizing clip to out-of-distributtion. In International Conference on Machine Learning, pp- 31716-31731. PMLR, 2023.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 804-814, 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 443-450. Springer, 2016.
Piotr Teterwak, Kuniaki Saito, Theodoros Tsiligkaridis, Kate Saenko, and Bryan A Plummer. Erm++: An improved baseline for domain generalization. arXiv preprint arXiv:2304.01973, 2023.
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek. and Hervé Jégou. Three things everyone should know about vision transformers. In European Conference on Computer Vision, pp. 497-515. Springer, 2022.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint 24X.v:2404.04125. 2024.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518, 2019.
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp. 3769-3778. 2023.
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pp. 2825-2834. PMLR, 2018.
13

---

Published as a conference paper at ICLR 2025
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint 2XXi:2001.00677/ 2020.
Han Yu, Xingxuan Zhang, Renzhe Xu, Jiashuo Liu, Yue He, and Peng Cui. Rethinking the evaluation protocol of domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 21897-21908, 2024.
Zhun Zhong, Yuyang Zhao, Gim Hee Lee, and Nicu Sebe. Adversarial style augmentation for do- main generalized urban-scene segmentation. Advances in Neural Information Processing Systems, 35:338-333 350, 2022.
A APPENDIX
A.I TRAINING AND EVALUATION DETAILS
We use slightly modified MIRO (Cha et al. 2022) codebase for training and evaluation. We use leave-one-out evaluation, where model trained on domains except the evaluation domain. We emphasize that we use DomainBed-IP and DodAinBed-OOP as evaluation data only, models are trained on full datasets.
OpenCLIP-ViT-B/IE (Ihaacco et al. 2021) trained LAION-400M (Schuh- mann al. 2021). I). We use default learning rate of 5e-5, weight decay of 0.0, batch size of 32 per-domain, Adam Optimizer, and no dropout for all methods.
For training,
as defined by (Cha et al.) 2022). This includes
For evaluation, unlike DomainBed, consider the entire test domain instead of an 80% random split. Following standard practice, we first compute accuracy for each domain, then average those ac- curacies to get dataset level statistics, and finally compute overall averages averaging across datasets.
For benchmarked methods, we also use typer-parameters found to be best in respective papers. For optimum patience parameter value of 3, overfit patience parameter value of 6, and tolerance ratio of 6. For MIRO, we use use regularizer loss weight of 1.0. For CORAL, we use CORAL regularizer weight of 1.0, following (Cha al., 2021). For LP-FT, train the linear probe for 600 steps before unlocking the full backbone. For Model Parameter Averaging, we burn the training for 600 steps before averaging iterates. For VL2V-SD and CLIPood, we directly use the author's implementation and hyper-parameters, except initializing with OpenCLIP (Ilarcco
SWAD,
al. 202T).
A.2 TRAINING COMPUTE
Each A6000 48GB GPU, trained for up to 12 hours per domain- combination.
B ADDITIONAL RESULTS
B. ALIGNMENTŠCORE VS ACCURACY
In Figure [1 we plot all benchmarked methods from the main paper, with x-axis corresponding to AlignmentScore, and the y-axis corresponding the Top-1 Accuracy. We normalize for dataset size, so that no dataset dominates the count. In Figures 12 through [6] we plot these statistics independently per dataset, and find the trends consistent across datasets.
B.2 TAR-DATAS BENCHMARKING RESULTS
We expand Table 1 in the main paper into per-dataset results in Table TO through 24
B.3 SIMILARITY OF TARGET TO PRE-TRAINING
To evaluate the Image Similarity Hypothesis, we retrieve the nearest neighbors from the Laion-400M dataset (Schuhmann al. 2021). This raises the question of how similar the target domains to
14

---

Published as a conference paper at ICLR 2025
PACS
DomainNet
TerraIncognita
Query
Retrieved
Query
Retrieved
Query
Retrieved
OfficeHome
VLCS
Query
Retrieved
Query
Retrieved
Figure 4: Nearest neighbors of target images in pre-training LAION data.
MMD Values for Different Domains PACS
Sepec
Figure 5: MMD between pre-training and target, source and target, and target and target for PACS. Target is more similar to pre-training than source. Despite this, Alignment is a better predictor of DG performance than perceptual similarity.
the pre-training data and whether the source domains might be even more similar. To investigate this, we compute Maximum Mean Discrepancy (MMD) distances between PACS domains and their nearest neighbors from Laion-400M, as shown in Figure 5 Our results indicate that target domains are, in fact, more similar to the pre-training data than source domains. We inspect nearest neighbors manually, and find even exact duplicates (Figure 4. Interestingly, while found not only domain- level duplicates but also exact matches in the pre-training data, the Image Similarity Hypothesis is ultimately less predictive than the Alignment Hypothesis.
B.4 OTHER BACKBONES
We benchmark 2 additional backbones(DINOv2 (Oquab al. 2023) and OpenAl CLIP) using the MIRO MPA Domain Generalization method, which we found to the strongest in our paper, on two datasets (OfficeHome and PACS). This consistency is likely due to the similar nature of the pre- training datasets, both sourced from web scraping and comparable scale. These findings reinforce that the usefulness of the DomainBed-IP/OOO split is not confined to specific backbone.
15

---

Published as a conference paper at ICLR 2025
Table 2: Benchmark results of different backbones on OfficeHome and PACS datasets.
Backhone
OfficeHome IP
PACS IP
OfficeHome OOP
PACS OOP
OpenCLIP- V 16
90.7
98.1
60.0
87.8
ELIP-ViT/B 16
88.1
97.6
57.0
86.9
Dino
87.4
97.1
58.9
85.0
Table 3: Splitting DomainBed by PerceptualSimilaritySsorr The differences between IP and OOP with this split are much lower than with Alignment Score.
Dataset
PACS
VLCS
Terralncognita
OfficeHome
DomainNet
Perceptual IP
97.2
74.8
60.6
86.8
61.4
Perceptual OOP
95.6
77.4
42.9
78.3
55.3
B.5 SPLITTING DOMAINBED USING PERCEPPUA..SMII.ARTTSSSOO
In (igureZa). we show that the slope of the relationship Top-1 Accuracy vs Perceptual Similarity Score is positive but shallow. This suggests that using PerceptualSimilarttySooor as an alternative to AlignmentScore for splitting DomainBed would not be very effective. To further prove this point, we split at SerceptualSmillartySSoooo value of 0.86 in Table 3 We can see the differences are not very large between OOP and IP, indicating that AlignmentScore a better thresholding metric.
B.6 COMPARING ALIGNMENTSCORE WITH ZERO-SHOT CLASSIFICATION CONFIDENCE SCORE
We also consider an alignment score which takes into account uncertainties, and compute a score using the confidence of the zero-shot classifier formed by the pre-trained CLIP model for each sample. Specifically, for sample with ground truth class c, we calculate the softmax over the logits output by the zero-shot classifier, and use resulting probability p(c) as the score. We refer to this as the Calibrated AlignmentScore and show the results in Figure Although the score predicts generalization for both OfficeHome and DomainNet, the scores have different seales for different datasets. In contrast out AligamentScore does align across datasets to greater degree (Figure
We explore the effect of averaging PercepuaaSimilarttySoorr and AlignmentScore in Figure We that there is not much of compositional effect, therefore stick with AlignmentScore as our generalization predictor.
B.7 IMAGE SIMILARITY HYPOTHESIS FOR SOURCE DATA
The main drawback of the Image Similarity hypothesis is that it does not consider how well the nearest perceptual neighbor is learned during pre-training. One reason for sample being poorly learned during pre-training that the pre-training caption not very relevant to the DG task. Source data is unlikely to have this issue, since source and target domains share labels. Therefore it is interesting to ask how strongly correlated the PerceptualSimilartySSooo is with DG accuracy when measured between source and target. Indeed However, simply using source-data to compute the PerceptualSimilarity results an incomplete understanding of the relationship between target data and the training procedure, due the lack of consideration of the pre-training. In fact, zero-shot models with NO learning from source are very performant (Table
seen in Figure 9 there is strong correlation.
B.8 TRAINING FROM SCRATCH
We focus analyzing the generalization capabilities of CLIP-based DG methods, these have been demonstrated to be the strongest in recent work. However,from -creertcc experiments are valu- able to measure how much finetuning source data can learn, and also trained ResNet-50 models from scratch, using the MPA (Model Parameter Averaging) method and present the results in Table We can see that performance very low even ALL BomainBed data (unsplit), further
16

---

Published as a conference paper at ICLR 2025
1.0
Top-1 Accuracy
0.6
0.4
Domainflet
0.0
Calibrated NignmentScore
Figure 6: Top-1 DG Accuracy vs calibrated alignment: We the confidence of the zero-shot classifier formed by the pre-trained CLIP models an alignment measure. Although the score predicts generalization for both OfficeHome and DomainNet, the scores have different scales for different datasets.
DomainNet
1.0
OfficeHomee
Top-1 Accuraty
0.6
0.4
0.0
AlgnmentSoore
Figure 7: Top-1 DG Accuracy AlignmentScore: The AlignmentScore introduced in our work have scales which are comparable across different datasets.
confirming that existing DG algorithms are poor at learning new knowledge from source data alone, and they rely strong pre-training for good performance.
B.9 ANALYSIS OF ZERO-SHOT BEHAVIOR
To better understand the behavior of Domain Generalization (DG) methods, we compute confusion matrices for zero-shot (ZS) and In-Pretraining (IP) and Out-of-Pretraining (OOP) subsets for the PACS dataset using CLIPood. The results are shown in 5
The confusion matrices reveal key insights about the behavior of DG methods:
For the IP subset. DG methods catastrophically forget a significant number of samples, flipping correct predictions made by zero-shot models incorrect ones. This suggests that DG methods can even have a negative value for IP data. Nevertheless, performance remains strong.
For the OOP subset, DG methods flip very few correct samples incorrect ones. However, the ofate-of-the-an (SOTA) method, CLiPood, is still unable to correct the majority of incorrectly classified samples in this subset.
17

---

Published as a conference paper at ICLR 2025
1.0
m
Top.1 Accurscy
0.6
W
0.2
NiggmmentScore
B. PercestuaiSmillrttyycrrr
A.
0.4
Score
Figure 8: Combining PerceptualSimilariy Score and AlignmentScore: We explore the effect of averaging PerceptualSimilarityS Score and AlignmentScore. There no visible additional signal from averaging. We therefore stick with AlignmentScore as our generalization predictor.
Officetame
3.7%
1.50
377
3.50
Figure 9: PerceptualSimilarity to Source for OfficeHome. DG performance is correlated with simi- larity source images.
Overall, in both IP and OOP cases, the results indicate that CLIPood and other DG methods rely heavily on pre-training. These findings underscore the limitations of current DG methods and high- light areas for improvement.
C ADDITIONAL DOMAIIBEDD-(PP/OOP) STATISTICS AND ANALYSIS
C.1 CONSTITUENT DATASETS
VLCS (Fang et al. 2013). DomainNet (Peng et al, 2019 OfficeHome (Ganin et al. 2016), PACS (Li et al. 2017), and Terralncognita (Beery al. 2018 Here we provide basic statistics of each.
We split 5 datasets in DomainBed-(1P/OO0)
VLCS has 5 classes and 4 domains: Caltech101, LabelME, SUN09, and VOC2007, with 10729 samples. The domain shift dataset source.
DomainNet contrains 345 classes and 6 domains:
clipart, infograph, quickdraw, real, and sketch. It has a total of 586,575 samples. The dataset shift is style.
OfficeHome has 65 classes and 4 domains: art, clipart, product, and real. The dataset shift is style.
TerraIncognita has 10 classes of wildlife cameras. There are 4 domains of different cameras and 24788 samples. The dataset shift is camera location.
18

---

Published as a conference paper at ICLR 2025
Table Training ResNet-50 from scratch. DG performance is very poor, showing DG methods rely on strong pre-training
DomainNet
OfficeHome
PACS
TI
VLCS
Avg
ResNet-50 from Scratch
7.6
17.2
36.3
23.8
55.2
28.0
Clip ViT-B/16
58.9
82.0
94.3
50.7
82.3
73.6
Table 5: Confusion matrix for the IP subset of the PACS dataset.
DG: Incorrect
DG: Correct
ZS: Incorrect
141
70
ZS: Correct
179
8231
PACS has 9991 sampes and 4 domains: arts, cartoon, photo, and sketch. There are classes, The dataset shift is style.
C.2 MISLABELING AND OCR RATES
In Table 7 we present the mislabelling rates at various AlignmentScore values. In Table we present how much data is removed due to mislabelling or OCR filtering.
C.3 CLASS DISTRIBUTION OF DOMAINBDD-(PPOOPP)
In Figures| 19 through 33 we provide class distribution statistics of different datasets before splitting and in IP and OOP splits. We find some interesting patterns. For example, in Office-Homee the OOP class is dominated by marker and toys, while the IP split has much more uniform distri- bution. Similarly, both PACS (DomainBed-OOO) and VLCS (DomainBed-OOPP) are dominated by person.
C.4 VIS DIFFERENCES BETWEEN DB-IP AND DB-OOP
We further investigate distinctions between the and OOP subsets using VisDiff (Dunlap al. 2024), an LLLM-based system that identifies differences between sets of images. For each combi- nation of dataset, domain, and class-except for DomainNet, where we subsample roughly 15% of the combinations due to computational constraints we independently sample up 30 images from both the IP and OOP subsets. VisDiff employs CLIP to compute an AUC-ROC score for the natural language differences proposed by the LLM, and retain only those differences with an AUC score of 0.7 or higher. The complete results are presented in Table 9 We find that contextual or envi- ronmental elements often overshadow the primary object indicated by the text label. For example, in the VLCS dataset's 's SUN09 domain, the OOP subset for the car class frequently features images described as "historical architecture," while in the Office Home dataset, real-doman bed images are deemed OOP if they include "child-theme decor."
C.5 RELATIONSHIP BETWEEN OBJECT SIZE AND ALIGNMENTSCORE
To test whether object size is correlated with AlignmentScore, we computed the object size of the ground truth labeled objects using bounding boxes generated by the open-world object detector OWLx2 (Minderer et al. 2023). Overall, object size is correlated with the AlignmentScore (see Fig- ure T0 with the sole exception being the Terralncognita dataset. This indicates that even the pres- ence of small wildlife can substantially increase the AlignmentScore, and that the background veg- etation does not present significantly conflicting signal that is sometimes present in other datasets. Overall, this suggests that pre-training which better represents scenes with multiple objects or con- cepts may improve benchmark performance.
19

---

Published as a conference paper at ICLR 2025
Table 6: Confusion matrix for the OOP subset of the PACS dataset.
DG: Incorrect
DG: Correct
ZS: Incorrect
98
62
ZS: Correct
15
890
Table Mislabeling rates across different AlignmentScore ranges.
Dataset
0.0-0.05
0.05-0.10
0.10-0.15
0.15-0.20
OfficeHome
100%
45%
28%
12%
PACS
46%
5%
0%
Terralncognita
55%
34%
23%
DomainNet
65%
35%
33%
11%
VLCS
33%
14%
9%
3%
Table 8: Percentage discarded samples due to mislabeling or label text in the image (OCR).
Dataset
Dropped Mislabelled
Dropped OCR
OfficeHome
0.92%
0.15%
PACS
3,05%
0.00%
Terralncognita
0,22%
0.00%
DomainNet
7.64%
0.03%
VLCS
12.41% %
0.00%
20

---

Published as a conference paper at ICLR 2025
0.230
Boundin fac Sre
0.200
Dounding Roe
0.175
0.2
0.150
0.375
0.200
0.225
0,250
0.275
0.300
0.325
0,350 6.375
AlgnmentScore
Algreaantocoee
(a) All datasets.
(b) OfficeHome.
0.15
0.175
0.150
0.15
0.125
Bourding Bix Stew
0.100
nounding
0.075
0.050
0.15
0.05
0.150
0.175
0.200
0.225
0.250
0.275
0,300
0.325
0.00
0.05
0.30
0.25
0.20
0.25
0.30
0.35
C.40
AlgnmentSeore
Algrmenntccre
(c) PACS.
(d) DomainNet.
0.12
Bairding Box Stow
0.10
0.05
Bounding
0.35
0.04
0.30
0.02
0.00
0.125
0.150
0.155
0.200
0.255 0,250 0.275
0.300
6.325
0.30
0.15
0.20
0.25
0.30
Algnmenttoooee
AlggmentSccoee
(e) Terralncognita.
(I) VLCS.
Figure 10: Overall relationship between ground truth object bounding box size and AlignmentScore across different DomainBed datasets. Overall the relationship is positive, indicating that object size plays a role in whether a data point is or OOP. The exception is TerraIncognita, indicating that camera trap backgrounds do not strongly conflict with the presence of the wildlife be classified. See Section C,5
21

---

Published as a conference paper at ICLR 2025
Table 9: VisDiff: Differences Identified for Image Sets. We use VisDiff (Dunlap al. 2024). an LLM-based system for describing differences between images sets, to identify differences in between IP and OOP images. We do so independently for each dataset, domain, and class grouping. Distractors unrelated to the class seem to be issue. For more details see Section C.4
Dataset Domain Class
Description of OOP Subset
VLCS_sUN09.car VLCS_SUNO9 bbrr
Historical architecture
variety of landscapes
VLCS VLCS_SUNO9.person
indoor water parks text about adoption and rescue services
VLCS www.aaaaalcccllcccccc _og
VLCS LabelMa chair
people walking city streets
VICS__aabelMe_dcog
gatherings of people paintings
MLCS.caltechh1lbbird
MLCS.Calteech101perraoo
murals on buildings
VLCs.VOc2007.dcgg
inside homes with multiple people presence of branded vehicles cooking areas double decker buses people near bodies water equestrian activities people engaging in activities
MLCS.Caltech101.car
VICS.VoC2007.car
VICS_LabeelWe.bird
VLCS.voC207-ppesson
VLCS bid
Gerraeiicoonttaaaoootttoo...bbb
sunny daytime scenes streams in wooded areas animal walking muddy areas cloud formations streams in the forest a dirt road in garden camera trap captures night vision middle of the night absence of man-made objects screens with green backgrounds crocodiles
ger2_iiccoontaaalccctt..111.....
serraiinoognttaallcccttioo......
geeraaahcoognttaaaaloatttooo111.
sersaiaccooniiaaallcttti........
Ber.elliccoplaaaallccatt........
ger.__hccopptaaallccttt........p
seraa.lcconnitaallocttiooo.....g
shadows in nature Idirt roads a black bear resting on a log muddy areas image of a crocodile deserts black bears
terra
seerraainoogntteee occatt...o... squirrel
cerreeiicoonnlaalllotttio......3 bobcat
serra.ihooppitaalloctttto.......
streams in a wooded area
aar.al/ccooniaaaalc0cti11oo..009 cat
waterfalls
Wooded areas crocodiles
Gecre.iiccognitaaalooctttoo.o... coyote
boats in water images of tapirs night natural landscape animals resting
Berre_aiccepitaaa.lcctttooo.....
terra_lcconttaallootiiioo.......
wildlife in water
office.hoomeAAtt.Paeee....l
animals hanging colorful backgrounds
office haa.ee.hhmee.clppaatteebbaaa
S.Eice.home.Prccuutt..dddd
of multiple beverage containers
office home Real Speaker office home Product Drill
jbl portable bluetooth speakers yellow and black color scheme advertising signage
22

---

Published as a conference paper at ICLR 2025
Table 9
Momaiit_Dmmain.Caee Class
Description of OOP Subset
office home Clipart Fan
close-up views
road cycling
circular objects textbooks with colorful covers
office Seriice.hom..liiee
office Saa.ce.homeRReal.5e
brushed nickel finish
office Nome_Clipar"s -eabbaar
singular computer components
office home Art. Art.Spakeee
toy-like objects
office affice..hom.PProcctt.Baadd
bluetooth devices
space-related elements asian clothing styles earrings with bottle and cork a promotional offer butterflies air purifiers
office sfiice..ome_11apat.DDe...mmmm
office office htw.eeaittcatt....ppoooo....
office home_Real,Flower.
affice.hom..Produe... home.Product.FFar
cartoon characters
office..homeAArt.Baccppaa.
toys with wings illustrations flowers pots focus packaging
office home Art Shelf
office siiiceehone.rroduutMMoueee
colorful electronics on display notebooks on a table
office htt..eemtttaattttsttttooo.oeeee.
office htt.ce/coom.rrccccc...pppbeeeeo
natural materials without additional items
office hal..e..omeerrduue..attteeeeeee
set of six objects
feminine imagery
office home Art Pan
eggs with faces drawn them 3D) rendering
office home Clipart Telephone www.cchhooe.clpppatMMiiiooo
technology themes close-up of a person engaged in an activity
office. office htw.eeeconccaaaa..ppee.........
Red color prominence
office htw.:aahhmmepcocuuccceeeoeooooee
octagon shapes
office hon.
close-up furniture shots
office hww.
dragon's 's head
3D rendering
office Sffice.homee@1ipprttthheee.
commercial products realistic images a bathroom setting a small white dog individual with headgear gel cushioning blue colored objects audio equipment black and white drawing
office. Sirice.homeeClipatt..77
office home how..Raal..urra.ns
office home
office hom.Artt.laassee
office off.ce.home.eeeeee home.Product.
office Sffiice.hom.RRealDriii...
S.ficc..ome.llipprtt.llsss
Srrice.Ioneeclppatt.lampahaaa
office home Real Bed
thild-themed decor
office home Frle Cabinet office Srrfice.homeeProduuttBBccke
people in professional setting
focus material texture
office settings with unusual elements
office wtw.:cehhom.eeeaBaateeeee
emphasis specific brand cans of angry birds tropic soda objects wheels
office home office home Trash Can office home Product Eraser
packages explicitly labeled 'pentel' laundry room book on a black background
afficce.hom.RRel__hhel
officehboneeClppart..oothbbsss office hriiceehhmm..rduutt....ee..
distinctive backgrounds sunglasses with red lenses sanitation theme
office ofieec.hom..llppattccaeeeed
23

---

Published as a conference paper at ICLR 2025
Table 9
Molaset_Dmmain.Caee
Description of OOP Subset
office home C1 Clipart Mop
detailed human figures
@fficc.home_Art.o0m
combination domestic and clinical spaces minimalist style
Srriice.hoe.AAr.fe.
digital artwork
futuristic design
office
aeriice.hom..ie home opppppop Laptop
fantasy or fictional themes
office Art. Bed
enchantment
offiice.home.ArtCCCrriiii
dark hair
offiice.homeRRealFFan
multiple ceiling fans
cocktail shakers
office hom.
blue and silver color palette gaming mouse included colorful stationery contemporary living spaces standing posture
office htw..eeeeoooooccc..eeeyarrrrrddd
office hwwe_Real
office how..eeehmeeeellccoooo
sffice.homeAArtPPP.
creative process
office Seiicc.home_Real.M.
clothes hanging
office home Art Scissors
video game characters
office wwoe.clipaart.55..
abstract shapes concrete structures printable charts with decorative backgrounds
office hareeheeee.tt.ebborrr
office www.:hooee.ea.......oo haa.cce.hmm.Reallclleeeeeee
office.home.Art.ca..... htt.eeeeeoo.ttt...llllloooor
decorative elements
Sriice.hmmeAAr.BB14
animal vehicle
office www.eeooooootttttecccaaaa
a single computer monitor
office home Clipart: Bottle
household container a printer Black high top sneakers monochromatic themes objects with vintage look objects associated with liquids abstract simplicity digital art objects placed against natural backgrounds animal skulls
office home Clipart Computer
office hww..eeeho.lllipttttuusss..
office.hoee.Art.tee AoteeArt.
office htw.eeehhooclllppttBeeeeee
office home home
office hww.
officce.home.Reea.PPush.....
cats
office Art. Lamp
intricate costumes unique artistic interpretations
office Art
office home.ArttR.Rfefflerratorr
toy girl in unusual setting
office www.ePooduttt
survival or tactical items image of protective gear abstract concepts calendar a caliper and a paper clip hammer head detail detailed human emotions a red background cakes with unique decorations White and brown plush toys red dress cooked fish
office office Rffice..oneeC11ipartttenn..e
sfiicehooeePrrddut..orebbbot
office home Product Paper Clip
office home Pome.ProducttHHameee
dfffice.home.Ciipart..eee
office how.eepaatapaaattcaiiii
office hww..roouuucc..ooooaaa
office home Art office home Real Pan
office.home_Ree. _
tabletop settings
afficce.home.rrt.Trresse
drinks on a colorful background
focus on object
office Row.ehooeee.eeeeeeeeeeeee
pink color theme
office S.ficeeloomnPPrccct..ie...C..e
green objects on white background
24

---

Published as a conference paper at ICLR 2025
Table 9
Dataint_Dmmain.om Class
Description of OOP Subset
office home Real Hammer
German craftsmanship
Siffice_hom_AArt...
illustrations
minimalistic design
illustrations of animals sport equipment
SeEiceehoneAArt-ellllheee
office home nome_RealFFolder
Serrice.home.ae
blue wall
office Pome.Proddbctt.ol..eee
focus on texture
objects on plain, uncluttered background black handle a single red rose mango juice with a straw people with hats focus on a single object fantasy elements
officce.hom_PPrcduuttiuuug
office hom.eeal....e..
office Saa.ceehome.Ar.Lamm..e
office home_clipart..ver
office www.ceehooe.AAr.t.. Art.Soon
cartoons food items
stiiee.homeelLppat.llamme
yellow boomboxes
office home wwwparttppa...eecc..cc.......eee Sink
human activity
effice-homer home.Clipart.Drill
digital art style
illustrations drawings clear and singular message a spool ribbon
office www..hhlppaattttttttttoa..m
office www.cehhhomcllippr..eeroooo
office Seriicehhmm.Ar.BBatteer..
graphic t-shirts coffee with food items inside signs with text multiple tv screens inflatable furniture conceptual illustrations cookies
Aotice.homeeAr_Muug
office home Clipart Bike office home Art TV
Sffice..hmeeAAt.Driill office
fire or flames women fantasy attire men in suits MSI branded devices
office home Art Hammer office hww.
officce.homeeProddicc..ooteteee
geographic representations a pair devices
Sfficce.oomeeG1lpattt.aap.ogg
office Www.eAtttteeaeeeee....ee
octopuses shoes whimsical objects ottomans
office Aot.
office www.eeeaeeeeeaai......
office htriice.hmeeClippr..tuuo
stacked objects man playing music
offiice.hhom_.Ar..Bucceett
PACS cartoon_pezson PACS art painting dog
images featuring cream hunting scenes
pacs.art.paintiinggiiree
deer in the artwork
PACS Rassaartppinniinggguiaar PACs-art www..aainiinggeeeeeeeeeeee
artwork displaying paint splatters
artistic depictions featuring abstract or imag- inative elements giraffes
cartoon.guitaa
hand-drawn illustrations
PACS_photo_dog
dogs in vehicles
PACS.photo.horse
non-living objects bulls
Pacs.rrt.paintinghhoeee
domaiin.net..llipart.shttt.
characters with speech bubbles technical illustrations
domainnnetLinfogrraooootttee domaihnnet.infforaah.aaaggg
different types of fish in infographics
25

---

Published as a conference paper at ICLR 2025
Table 9
Vataset_Domain.co
Description of OOP Subset
doma in net clipart bicycle
scenes involving flight
dommain.net_kkttch.sedl
monkeys as a subject
text with humor or messages
organized visual representation of navy
ships
black background
dommalnnnet..ketcchssbbmeeer
educational elements like drawing guides blue birds
dommainnnet..ll11ptt.peero
paintings on striped background large resorts images near a river
domaai..net-reaahhuuse
domalza.et.reeal.sopp.sigg
domaahh.net.clllparttmmous
cute small animals
domaihh.eet.raalhhcdeeege
objects a grassy area presence of dogs watercolor paintings steampunk design cocktails
domain..eet.rall.cat
domaaihh.eet..pannninng..bueettt
domaits..et.skketcc.w.ln.eeege_
lace tablecloths
domaiinnnet.Skech.bbus
text elements in illustrations floral arrangements
domait..net.clllaatt.ccerr
Airplanes flying in the sky repeated phrases or words a basketball on a wooden floor comparison charts
domaainh.ee..llipeet...ootter
domalihnnttskketcchbbaaaabblll
domain..eet.inffgrrpp..tte.
domain net sketch
teddy bear motifs
clt fish
black and white illustrations infographics focusing player statisties children in art circular tables clouds
dommalh.eet..aannnnnngbeeeegggg
d0maii..net..paittinn.haazp
domain.net.sketch.ant
squares and shapes specific locations informative visual aids stained glass birthday-themer images characters with beverages
dommainnneet.skecchhhhhuhhhh
domaata.net..llppettt.caddee
dbmaiihnnetccllpparbbbbbblll
a combination of different artistic styles mushrooms
domainn.eet.real.cloodd
illustrations of devices dairy farm flyer designs for personal adormment art creation process people creating art paintings with multiple subjects chef's hat frogs
domajishnet.cllppprt-cccpoooo
domaii..nett.painiiinnnnooohh.h
domaaii.net.paiininnggioooup
truck
domaiih.ne..lllpprt..ppaaag
d0maiih.eet.ceaa.eelghhtttt
scenes involving rivers water a video game setting
domais.neet.eeal.coonnaa
domainnnett.painninggoooo.
painting of people
indoor car scenes
modern info presentation template casual outdoors multiple animal species people indoors recipes
domaain.nett.cllpaett.vss
dombal..net.cliparrt.olphppa
domali..ett.eaallighhhnnngg
domaihhnet.infograbbbbaaaa
26

---

Published as a conference paper at ICLR 2025
Table 9
Dataint
Description of OOP Subset
domain net painting cooler
incomplete words objects depiction of people colorful graphic design infographics
domaalnnnet.sketcchrratttoo
dommala..etcclparr.ppeee......
commstn.eet_Lnffogppphh.aaagp
domaathhnetrrealssszingggeee
recipes with bacon
leather industry
domiain.net.reaaccarrrtt
hummus
murals
flyer designs
domaalhh.ett.kketchhbb.leeeeeer
set of hand drawn fruit cartoon animals flyers with promotional content mythological or fantasy elements fast food with ice cream organic farming elements unbrellas
comaii.net..kketchh.nnhtkk
domaii.nett.infforrphh.uuu
domaia_net_real_moon
dommain.nee..sketcllaaaabuugg
dommali.nett.nnffgppphh..mmm
domaal..net.ccllaatt.aaaaooo
domait..net.cliiartt..og
images featuring human characters educational posters
d0maia.net..ll1patffffe
vector illustrations vector illustrations people sitting desks
dommali.net.sketchhhboorggg.
dombala.net..keechhllaapooo
dommainheet.rrel..iireehhyyrrrr
airplane the sky cable connection jewelry
domais.nett.eaalmouueee
domaiiannet.reellhoorgg....
domaiinnetttaalntingggw.
paintings in gallery setting mixed media paintings illustrations of hats and sunglasses
Bommainneet-paiininngeeeeee
infographics with statistical data depictions of solitude
domaala.net.paaintiing..uu
demaiss.eet.palntinng..eee
military uniforms birds and flowers
domaiinnnet..seecch.ooteee
illustrations of chefs dynamic outdoor sports hand drawn illustrations sweet- sweet-themed infographics colorful designs
dommainnneet.real.iiipllaa.s
domaiia.net-Skettchhaaaaaaaa
coma::..ee.111f0gggph.........
domain.net.real.ovl
domain..ee..eaalbammn..
plate presentations
domiiie.ee..pannntnng..toaaaa
people in colorful dresses
domaihnnet.rraalllbbteer
multiple dishes presented together people drawing with different tools extreme sports guitar
domaain.net.sketchh.ffroog
domais.neet.reallmunnaaal
domais.nett.ell:crumss
Comainnnet inffgrraa.
knee
recovery processes coffee mugs images with helicopters first aid themes wheelchair-accessible features
domaihh.net.real.teepp..
domails.eet.re3.ccmmmrfgggg
dommal:.net.innfograpppaaaa
domaaiannet.iinfoggpph.baaat
artistic portraits people sitting together
nomaiinnettssketh
Rough outlines lizards and frogs
1ommiin.eet_skettc..crooooo
dommaiih.net..lllppr...egggoo
Christmas themes cartoons with both and dogs screenshots of video games illustrated people
domais.net.olliprtt.cat
domaih.net.reallgolf....u
dombit..eet.polnt.nngggppppgg
27

---

Published as a conference paper at ICLR 2025
Table 9
Description of OOP Subset
doma net clipart broom
dog in witch's s hat with pumpkin night scenes indoors
dommals.net..keetchaaaaalaaaa
nautical-themed accessories
Colorful paw prints colorful gumballs whimsical illustrations illustrations with human figures
domais.net.clipptttiddee:
commalhnnee.ccllipett..cccssss
images of airplanes rooms with large beds Everyday scenes involving musical themes promotional style imagery refrigerators
domaait.neet.eeal_chanddlllll.
domaal..net.skkttch.guuaaa
conbal..net..lippat.bbroooo.
domainh.eet_painnnnng..tttoo.
birds
commainnnetcccllpssrtchhhhhhhh
ceremonial events child safety colorful outdoor scenes green character in a bathroom scene
domaatt.eett_oaannnnnn....111111
domaata.net-eaalnnnngghhhhhttta
cartoon characters
people on piece of paper Venice
dommiia..et._painnt.ng.....gggg
domaiih.eet.cl11aatt.....l...
animals playing instruments abstract or surreal elements
domaain.net.reall.ooqqutt
domaiiannet.skeecch.ppp...d
comic book grading checkered patterns art restoration patterns and designs people in stylish outfits detailed sports statistics colorful designs watercolor
domaath.net.in1fggap....uuu
n st
clipart
flamingo
domaiin.net.real_knee
domaahh.eetiiiffpppph..asettttl
dommaaa.eet.aainnnng.aaaalloaaa
a person multitasking
abmaainnneet.reaaspeeccaba
tourist beaches
domaain.nee..lllpsrr..bbus
psychedelic imagery
images with variety of painting mediums decorative art
combais.eet_ppiinnnngg:.:..lelt
colorful designs scenes depicting historical or fictional char- acters a man in a hat vector illustrations with folders identification charts cartoon animal with flower oil paintings
Kompiannet.settc.bbard
domaiin.ne..clipatt..kyy
domait.net.clliaatt.nne
domain net paintiing stop sign
Bommii..net.setttchhaaaaealllll
vector illustrations on black background colorful clothing posters for events airplane flying
domain.net_reaaeelbov
dombain.net.infoggapp....ette
domain.net.clllpart.aasgmm
28

---

Published as a conference paper at ICLR 2025
1.0
ERM
SAGM
CORAL
0.8
Attn. Tuning
LP-FT
Top-1 Accuracy
MPA
0.6
MIRO
SWAD
DomainNet
VL2V-SD
OfficeHome
Clipood
0.4
PACS
MIRO SWAD
Terraincognita VLCS
MIRO MPA
Oracle (In-Domain) Finetuning
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Pre-Training Alignment (Higher Stronger)
Figure 11: Plotting AlignmentScore vs Top-1 Accuracy for all benchmarked methods, for all datasets together. Although some methods are stronger than others, all follow the same trend of increasing accuracy with increased AlignmentScore.
PACS
5000
ERM Baseline
SAGM
4000
CORAL
Attn. Tuning
0.90
Top. Accuracy
MPA
3000
0.85
Counts
MIRO
SWAD
VL2V-SD
0.80
2000
CLiPood
0.75
MIRO MPA
Baaclelin-Domalll finetuning
1000
0.15
0.20
0.25
0.30
0.35
Pre-Training Alignment (Higher Stronger)
Figure 12: Plotting AlignmentScore vs Top-1 Accuracy on the PACS dataset.
Table 10: Per-domain breakdown for OfficeHome (DomainBed-OOP)
Method
Art
Clipart
Product
Real
Avg
OpenClip ZS
49.4
24.6
$9.0
59.3
48.1
CORAL
33.0
29.4
50.0
58.0
42.6
SAGM
36.8
30.7
51.7
58.7
44.5
ERM
36.0
28.1
$1.3
56.0
42.9
LP-FT
37.2
29.7
$2.0
54.7
43.4
SWAD
43,9
37.1
56.0
62.7
49.9
MIRO
49.4
39.6
62.7
74.7
56.6
VL2V-SD
56.4
39.3
65.3
65.3
56.6
Attn. Tune
46.0
36.1
59.7
64.0
51.4
Model Parameter Averaging (MPA)
46.0
36.4
55.7
66.0
51.0
CLIPOOD
60.3
44.4
73.3
78.0
63.9
MIRO SWAD
55.2
44.7
63.3
72.7
59.0
MIRO MPA
53.5
44.1
65.7
76.7
60.0
29

---

Published as a conference paper at ICLR 2025
DomainNet
-100000
1.0
ERM Baseline
0.8
800000
CORAL
Attn. Tuning
LP FT
Accuracy
MPA
0.6
60000
Counts
MIRO
VLZV-SD
0.4
CLiPood
MIRO SWAD
MIRO MPA
OraclelIn-Domain) Finetuning
20000
0.0
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Pre-Training Alignment (Higher Stronger)
Figure 13: Plotting AlignmentScore vs Top-1 Accuracy on the DomainNet dataset.
OfficeHome
55000
1.0
SAGM
4000
CORAL
Altn. Tuning
0.8
LP-FT
Top-1 Accuracy
MPA
3000
0.7
Counts
SWAD
VL2V-SD
0.6
2000
CLiPood
MIRO SWAD
0.5
MIRO MPA
Oraclelin-Domain) Finetuning
0.3
0.15
0.20
0.25
0.30
0.35
Pre-Training Alignment (Higher Stronger)
Figure 14: Plotting AlignmentScore vs Top-1 Accuracy on the OfficeHome dataset.
Table 11: Per-domain breakdown for OfficeHome (DomainBed-IP)
Method
Art
Clipart
Product
Real
Avg
OpenClip 7S
88.5
78.0
95.4
93.9
88.9
CORAL
71.6
62.9
85.8
84.0
76.1
SAGM
75.7
70.8
87.0
84.5
79.5
ERM
74.1
67.4
86.7
84.1
78.1
LP-FT
75.2
66.9
88.9
83.0
78.5
SWAD
81.4
77.8
91.8
88.2
84.8
MIRO
88.3
81.1
93.7
92.1
88.8
VL2V-SD
91.0
83.8
96.7
94.2
91.4
Attn. Tune
86.0
75.1
89.5
88.7
84.8
Model Parameter Averaging (MPA)
83.6
91.1
88.5
85.1
CLIPOOD (Shu et al. 2023
92.3
81.3
96.0
94.2
90.9
MIRO SWAD
90.4
83.2
95.5
92.8
90.5
MIRO MPA
90.6
83.9
95.4
92.9
90.7
30

---

Published as a conference paper at ICLR 2025
Terralncognita
-5000
1.0
ERM Baseline
4400
CORAL
Attn. Tuning
LP FT
Top-1 Accuracy
MPA
3300
0.6
Counts
MIRO
VLZV-SD
CLiPood
0.4
MIRO SWAD
MIRO MPA
Oracle(in-Domain] Finetuning
1000
0.175
0.200
0.225
0.250
0.275
0.300
0.325
Pre-Training Alignment (Higher Stronger)
Figure 15: Plotting AlignmentScore vs Top-1 Accuracy on the Terralncognita dataset.
VLCS
-5000
1.0
ERM
SAGM
0.9
4400
CORAL
Altn. Tuning
LP-FT
Top-1 Accuracy
0.8
3000
MPA
Counts
SWAD
VL2V-SD
2000
CLiPood
MIRO SWAD
0.6
MIRO MPA
Oraclelin-Domain) Finetuning
0.5
0.10
0.15
0.20
0.25
0.30
Pre-Training Alignment (Higher Stronger)
Figure 16: Plotting AlignmentScore vs Top-1 Accuracy on the VLCS dataset.
Table 12: Per-domain breakdown for OfficeHome (DomainBed-All)
Method
Art
Clipart
Product
Real
Avg
OpenClip 7S
83.4
73.2
92.4
92.6
85.4
CORAL
66.6
60.1
83.0
83.0
73.2
SAGM
70.7
67.1
84.3
83.6
76.4
ERM
69.2
63.9
83.9
83.1
75.0
LP-FT
70.3
63.6
86.0
82.0
75.5
SWAD
76.6
74.3
89.0
87.2
81.8
MIRO
83,2
77.5
91.2
91.4
85.8
VL2V-SD
86.4
79.7
94.1
93.1
88.3
Atn. Tune
80.8
71.7
87.2
87.8
81.9
Model Parameter Averaging (MPA)
78.7
73.4
88.4
87.6
82.0
CLIPOOD
87.9
77.8
94.1
93.6
88.3
MIRO SWAD
85.6
79.9
93.0
92.1
87.6
MIRO MPA
85.6
80.5
93.0
92.3
87.9
31

---

Published as a conference paper at ICLR 2025
Table 13: Per-domain breakdown for Terralncognita (DomainBed-OOP)
Method
L100
L38
1.3
L46
Avg
OpenClip ZS
5.1
0.0
6.4
6.6
4.5
CORAL
13.9
8.1
21.1
21.1
16.0
SAGM
17.2
7.8
20.2
32.2
19.3
ERM
13.1
22.9
20.5
16.5
LP-FT
16.7
5.1
25.6
44.9
23.1
SWAD
20.7
8.2
26.9
28.2
21.0
MIRO
19.4
6.6
30.0
17.9
18.5
VL.2V-DD
9.3
5.6
20.2
28.4
15.9
Attn. Tune
20.5
10.7
24.8
25.3
20.3
Model Parameter Averaging (MPA)
21.7
8.6
30.9
27.6
22.2
CLIPOOD
37.9
11.3
21.6
8,9
19.9
MIRO SWAD
22.5
5.9
30.3
25.9
21.1
MIRO MPA
24.8
9.6
32.5
32.8
24.9
Table 14: Per-domain breakdown for Terralncognita (DomainBed-IP)
Method
L100
L38
L43
L46
Avg
OpenClip ZS
52.3
23.2
36.5
35.1
36.8
CORAL
46.4
35.3
56.9
32.8
42.9
SAGM
55.1
38.7
52.5
29.5
44.0
ERM"
37.4
38.0
55.2
37.5
42.0
LP-FT
50.7
36.5
58.1
18.2
40.9
SWAD
60.1
41.7
65.7
43.5
52.7
MIRO
69.5
50.2
69.7
46.1
58.9
VL2V-SD
52.8
38.4
57.6
43.8
48.1
Attn. Tune
52.4
44.0
68.0
47.5
53.0
Model Parameter Averaging (MPA)
63.8
41.4
67.9
44.7
54.4
CLIPOOD
77.7
56.4
69.0
51.0
63.5
MIRO SWAD
68.9
54.8
73.4
51.2
62.1
MIRO MPA
69.4
55.9
73.6
51.4
62.6
Table 15: Per-domain breakdown for TerraIncognita (AdmainBed-All)
Method
L100
L38
I.43
L46
Avg
OpenClip ZS
48.2
20.9
31.0
32.6
33.2
CORAL
43.6
32.7
50.4
31.8
39.6
SAGM
$1.9
35.8
46.6
29.7
41.0
ERM
35.3
35.3
49.3
36.0
39.0
LP-FT
47.8
33.5
52.1
20.5
38.5
SWAD
56.7
38.5
58.6
42.2
49.0
MIRO
65.2
46.0
62.4
43.7
54.3
VL2V-SD
49.1
35.3
50.8
42,5
44.4
Attn. Tune
49.7
40.8
60.1
45.6
49.1
Model Parameter Averaging (MPA)
60.1
38.3
61.1
43.2
50.7
CLIPOOD
74.3
52.1
60.4
47.4
58.5
MIRO SWAD
64.9
50.1
65.5
49.0
57.4
MIRO MPA
65.6
51.5
66.0
49.8
58.2
32

---

Published as a conference paper at ICLR 2025
Table 16: Comparison of Methods for DomainNet Dataset (DomainBed-OOP)
Method
Clp
Inf
Pnt
Qkdr
Real
Skt
Avg
OpenClip ZS
29.2
43.5
28.7
35.5
21.2
26.3
CORAL
34.9
19.2
26.9
5.7
24.6
22.8
22.3
SAGM
36.4
25.1
26.8
24.2
23.0
ERM
36.2
19.0
23.4
5.4
26.6
23.4
22.3
LP-FT
34.7
20.9
25.2
6.1
25.9
23.2
22.7
SWAD
42.5
28.3
33.4
7.6
30.2
29.5
28.6
MIRO
42.5
37.0
27.9
3.5
33.5
25.8
28.4
VL2V-SD
43.0
43.2
34.8
37.1
29.5
31.8
Attn. Tune
40.4
29.1
29.2
4.3
31.3
26.6
26.8
Model Parameter Averaging (MPA)
42.3
30.9
34.9
7.4
31.6
30.7
29.6
CLIPOOD
39.2
50.2
39.5
2.1
40.9
31.3
33,9
MIRO SWAD
46.
40.4
34.0
35.2
31.7
32.0
MIRO MPA
46.2
42.6
36.0
4.4
36.0
33.3
33.1
Table 17: Comparison Methods for DomainNet Dataset (DomainBed-IP)
Method
Clp
Inf
Pnt
Qkdr
Real
Skt
Avg
OpenClip ZS
84.0
84.1
85.5
23.5
91.4
81.1
74.9
CORAL
80.3
52.3
71.1
33.2
71.7
71.2
63.3
SAGM
81.7
52.8
70.2
34.5
73.4
73.1
64.3
ERM
80.6
53.4
69.0
31.3
73.0
71.6
63.1
LP-FT
80.0
56.5
70.5
34.0
73.9
71.5
64.4
SWAD
85.5
67.1
80.2
41.3
80.1
79.8
72.3
MIRO
85.4
77.7
79.1
34.6
83.2
74.2
72.4
VL2V-SD
88.3
85.6
84.6
38.0
88.3
84.0
78.1
Attn. Tune
83.5
66.1
77.3
80.3
75.5
69.2
Model Parameter Averaging (MPA)
85.8
69.7
81.6
41.4
81.4
81.4
73.6
CLIPOOD
86.6
85.4
87.6
38.0
91.0
84.7
78.9
MIRO SWAD
88.2
81.7
83.9
40.8
85.8
81.9
77.0
MIRO MPA
88.1
83.7
85.5
41.3
86.5
84.0
78.2
Table 18: Comparison of Methods for DomainNet Dataset (DomainBed-All)
Method
Clp
Inf
Pnt
Qkdr
Real
Skt
Avg
OpenClip ZS
74.9
49.6
68.7
12.7
85.7
66.0
59.5
CORAL
72.7
27.1
58.0
67.1
58.8
50.6
SAGM
74.1
27.4
56.9
20.9
68.9
60.6
51.5
ERM
73.1
27.4
55.7
19.0
68.5
59.3
50.5
LP-FT
72.4
29.3
57.1
20.7
69.2
59.2
51.3
SWAD
78.0
36.0
66.1
75.2
66.7
57.9
MIRO
78.0
42.9
64.0
20.0
78.3
61.7
57.5
VL22--D
80.5
47.8
69.5
21.6
83.1
69.8
62.0
Attn. Tune
76.1
36.0
63.0
19.2
75.4
62.9
55.4
Model Parameter Averaging (MPA)
78.3
37.9
67.4
25.2
76.5
68.2
58.9
CLIPOOD
78.4
52.9
72.6
85.9
70.7
63.6
MIRO SWAD
80.8
45.6
68.8
23.7
80.7
68.7
61.4
MIRO MPA
80.7
47.2
70.4
23.9
81.4
70.6
62.4
33

---

Published as a conference paper at ICLR 2025
Table 19: Per-domain breakdown for VLCS (DomainBed-OOP)
Method
Caltech101
LabelMe
SUN09
VOC2007
Avg
OpenClip ZS
100.0
77.1
81.6
80.2
CORAL
93.3
54.6
72.9
75.3
74.0
SAGM
80.0
76.5
78.0
73.3
ERM*
100.0
57.1
73.3
75.3
76.4
LP-FT
80.0
55.4
71.8
75.7
70.7
SWAD
100.0
51.2
79.1
77.7
77.0
MIRO
86.7
52.6
82.9
72.7
73.7
VL2V-SD
100.0
80.4
81.1
79.1
Attn. Tune
86.7
61.0
77.9
79.0
76.1
Model Parameter Averaging (MPA)
100.0
53.6
83.6
80.9
79.5
CLIPOOD
100.0
55.0
84.9
82.8
80.7
MIRO + SWAD
100.0
84.7
79.9
80.3
MIRO + MPA
100.0
54.6
83.4
77.4
78.9
Table 20: Per-domain breakdown VLCS (DomainBed-IPP)
Method
Caltech101
LabelMe
SUN09
VOC2007
Avg
OpenClip 7S
99.8
91.0
94.8
97.8
95.9
CORAL
98.2
83.5
84.5
79.8
86.5
SAGM
95.9
84.3
86.3
85.5
88.0
ERM*
96.9
83.7
86.1
74.6
85.3
LP-FT
97.1
82.8
80.7
SWAD
97.9
82.9
88.4
84.8
88.5
MIRO
98.9
83.3
93.5
88.2
91.0
VL2V-SD
99.5
83.9
91.8
94.6
92.4
Attn. Tune
98.0
84.7
85.7
86.5
88.7
Model Parameter Averaging (MPA)
98.4
83.0
92.6
88.8
90.7
CLIPOOD
98.9
83.6
93.5
94.0
92.5
MIRO SWAD
97.6
83.4
92.6
90.6
91.1
MIRO + MPA
98.0
83.2
92.9
90.1
91.0
Table 21: Per-domain breakdown for VLCS (DomainBed-All)
Method
Caltech101
LabelMe
SUN09
VOC2007
Avg
OpenClip ZS
99.8
72.7
71.0
86.0
82.4
CORAL
98.2
72.7
77.0
78.5
SAGM
95.8
68.8
76.2
80.8
80.4
ERM*
96.9
67.8
73.0
73.9
77.9
LP-FT
97.0
66.4
71.4
77.4
78.0
SWAD
98.0
77.7
80.3
80.1
MIRO
98.7
65.3
81.2
79.3
81.1
VL.2V-SD
99.5
66.6
78.8
86.0
82.7
Attn Tune
97.9
70.3
77.2
81.9
81.8
Model Parameter Averaging (MPA)
98.4
65.7
81.3
83.7
82.3
CLIPOOD
98.9
81.6
86.6
83.4
MIRO + SWAD
97.6
66.4
81.6
82.6
82.0
MIRO + MPA
98.0
67.4
82.2
83.6
82.8
34

---

Published as a conference paper at ICLR 2025
Table 22: Per-domain breakdown for PACS (DomainBed-OOP)
Method
Cartoon
Photo
Sketch
Avg
OpenClip ZS
95.7
37.7
81.4
CORAL
80.4
84.3
89.5
42.2
74.1
SAGM
78.1
54.7
74.2
ERM
75.9
86.8
91.6
53.2
76.9
LP-FT
74.2
85.9
97.9
56.3
78.6
SWAD
84.6
81.6
95.8
54.3
79.1
MIRO
94.2
94.1
97.9
52.8
84.7
VL2V-SD
93.5
97.9
85.0
Attn. Tune
94.0
91.4
95.8
55.7
84.2
Model Parameter Averaging (MPA)
94.0
88.9
97.9
50.2
82.7
CLIPOOD
96.0
96.8
97.9
58.3
87.2
MIRO SWAD
94.9
100.0
85.4
MIRO + MPA
95.3
95.4
100.0
60.3
87.8
Table 23: Per-domain breakdown for PACS (DomainBed-IPP
Method
Art
Cartoon
Photo
Sketch
Avg
OpenClip 7S
99.7
99.7
99.9
94.6
98.5
CORAL
83.3
89.8
91.8
72.1
84.3
SAGM
89.7
93.0
94.7
82.9
90.1
ERM
88.7
93.0
91.5
75.2
87.1
LP-FT
85.3
93.7
97.9
90.3
SWAD
96.4
93.0
97.8
91.0
94.6
MIRO
99.2
96.7
99.9
94.4
97.6
VL2V-SD
99.6
98.9
99.9
93.6
98.0
Attn. Tune
91.7
96.4
Model Parameter Averaging (MPA)
98.6
95.8
98.2
89.2
95.4
CLIPOOD
99.4
99.3
99.9
92.0
97.7
MIRO SWAD
99.1
97.9
99.9
93.3
97.6
MIRO MPA
99.0
98.5
100.0
98.1
Table 24: Per-domain breakdown for PACS (DomainBed-AAl)
Method
Art
Cartoon
Photo
Sketch
Avg
OpenClip ZS
97.8
98.7
99.8
91.6
97.0
CORAL
81.4
91.7
83.2
SAGM
83.4
90.4
94.6
81.4
87.5
ERM*
83.0
92.1
91.5
74.0
85.2
LP-FT
80.5
92.7
97.9
82.7
88.4
SWAD
91.5
97.8
89.0
92.4
MIRO
97.1
96.4
99.8
92.2
96.4
VL22--D
97.7
98.5
99.9
91.3
96.9
Atm. Tune
96.4
96.5
98.8
89.7
95.4
Model Parameter Averaging (MPA)
97.2
94.8
98.2
87.1
94.3
CLIPOOD
99.8
96.8
MIRO + SWAD
98.0
97.4
99.9
91.0
96.6
MIRO MPA
97.8
98.0
99.9
92.9
97.2
35

---

Published as a conference paper at ICLR 2025
DomainNet
VLCS
175000
Split Original
Split
Original
IP
IP
125000
OOP
2503
OOP
Count
100000
Count
2000
1500
50000
25000
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Cattech101
LabelMe
SUN09
voc2007
PACS
Terralncoginta
OfficeHome
Split Original
Split Original
Split
1383
Original
2.03
IP
230
Count
DOP
1ww.
Art Painting
Carbon
Photo
Stetch
Location 36
Location 43
Location 46
Ar
Clipart
Prodect
Roal
Location
DodainBed-(1P/OOP Statistics: Breakdown of DomainBed-IP and DomainBed-OOl counts, by dataset and domain. Overall, DomainNet and VLCS have the largest fraction of samples falling into DomOinBed-OOP.
Figure 17:
Terralncognica-11
Terralncognica OOP OOP
600
Count
400
4000
2000
200
coyote
cat
rabbit
dog
opossum
bobcat
bird
raccoon
empty
squirrel
coyote
rabbit
opossum
bobcal
bird
raccoon
empty
squirral
Figure 18: Class-distribution shift:
Terralncognita's class distribution differs between DB-IP and
DB-OOP, indicating that some classes were better aligned during pretraining.
36

---

Published as a conference paper at ICLR 2025
domain Class Distribution
Figure 19: Class distribution of DomainNet-IP Zoom on pdf for best viewing.
domain net OOP Class Distribution
Figure 20: Class distribution of DomainNet-OOP. Zoom in on pdf for best viewing.
domain_ne net Unsplit Class Distribution
2004
100
Figure 21: Class distribution of DomainNet before splitting. Zoom in on PDF before viewing.
37

---

Published as a conference paper at ICLR 2025
office home IP Class Distribution
100
250
200
100
50
Figure 22: Class distribution of OfficeHome-IP. Zoom in pdf for best viewing.
office home OOP Class Distribution
Figure 23: Class distribution of OOfP Zoom in pdf for best viewing.
office home Unsplit Class Distribution
300
250
200
150
100
Figure 24: Class distribution of OfficeHome before Splitting. Zoom in on pdf for best viewing.
38

---

Published as a conference paper at ICLR 2025
PACS IP Class Distribution
1500
Count
1000
500
horse
person
house
dog
giraffe
guitar
elephant
Figure 25: Class distribution of PACS-IP.
PACS OOP Class Distribution
400
Count
200
elephant
horse
person
house
dog
giraffe
guitar
Figure 26: Class distribution of PACS-OOP.
39

---

Published as a conference paper at ICLR 2025
PACS Unsplit Class Distribution
1500
Count
1000
500
horse
person
house
dog
giraffe
guitar
elephant
Figure 27: Class distribution of PACS before splitting.
terra incognita IP Class Distribution
6000
Count
4000
2000
coyote
cat
rabbit
dog
opossum
bobcat
bird
raccoon
empty
squirrel
Figure 28: Class distribution TerraIncognita-IP
40

---

Published as a conference paper at ICLR 2025
perra_incognita OOP Class Distribution
600
Count
400
200
coyote
cat
rabbit
dog
bobcat
bird
raccoon
empty
squirrel
opossun
Figure 29: Class distribution of Terralncognita-OOPP
terra incognita
Unsplit Class Distribution
6000
Count
4000
2000
coyote
cat
rabbit
dog
bobcat
bird
raccoon
empty
squirrel
ppossun
Figure 30: Class distribution of Terralncognita before splitting.
41

---

Published as a conference paper at ICLR 2025
VLCS IP Class Distribution
1500
Count
1000
500
0
car
chair
person
dog
bird
Figure 31: Class distribution of VLCS-IP
VLCS OOP Class Distribution
2000
Count
1000
car
chair
person
dog
bird
Figure 32: Class distribution of VLCS-0OP
42

---

Published as a conference paper at ICLR 2025
VLCS Unsplit Class Distribution
4000
Count
2000
car
chair
person
dog
bird
Figure 33: Class distribution of VLCS before splitting.
43


Reviews:
[{"review_id": "CGF5na7ilH", "reviewer": "Reviewer_Zx21", "summary": "This paper analyzes the reliance of DG works on pre-training particularly in the context of CLIP.  The authors propose the Alignment hypothesis which states that pre-trained alignment between image and class embeddings is helpful for domain generalization performance prediction even after a model is finetuned on source domains. For evaluation, the authors utilize a two-step method. First, they cleaned the data. Images with AlignmentScore < 0.15 are discarded. Similarly, high AlignmentScore (>0.4) is usually for samples that have label text in the image. Second, In-pretraining and Out-of-pretraining are determined based on a scoring threshold of 0.21, based on the authors' observations about the data.", "questions": "- IP/OOP splitting by using CLIP AlignmentScore does not make sense. The CLIP model is first used to calculate to split the data and then to predict the score. Is not it circular?\n\n- One assumption behind this work is that simple alignment between image and label embeddings is sufficient to measure the presence of data in pre-training. What is the justification for this assumption? If a model has learned well, it should have high similarity for examples that are not part of training data but are in the training distribution.\n\n\n- The authors state the following in the abstract: \"Not just the presence of data in pre-training but how well is it learned\", however, this part is not reflected later in the paper. How does IP/OOP reflect this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and generally easy to follow.\n\nThis paper presents some interesting insights: \n\n  1. Relation of low AlignmentScore and presence of label noise in the domain generalization datasets.\n\n  2. \"We find that all methods, including those considered state-of-the-art, perform poorly on OOP data\" which seems to suggest that DG methods do not learn anything beyond what has already been learned by CLIP during pre-training. This also aligns with some of the previous findings. \n\n  3. Model parameter averaging boosting performance. \n\n  4. The fact that a high score indicates text in the image, which is not a significant part of the data (0.00-0.15% across the dataset).", "weaknesses": "IP/OOP method of splitting data seems to be a bit circular. Some of the results are a bit difficult to interpret. For instance, Table 2 seems to represent the correlation of final performance and prediction with IP/OOP splits but presentations make it hard to understand. Further concerns are mentioned in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Br4EImsRyT", "reviewer": "Reviewer_BJqk", "summary": "The paper proposes a hypothesis stating that if the test images and their corresponding text labels (e.g., \"A photo of a {cls}\") are already well-aligned in the embedding space before fine-tuning, the model's domain generalization (DG) performance will be high after fine-tuning.\n\nUsing an alignment score, the target dataset is divided into two parts: in-distribution (IP) and out-of-distribution (OOP). The experiments show that DG methods perform well on high-alignment samples (DomainBed-IP) after fine-tuning but poorly on low-alignment samples (DomainBed-OOP). This suggests that current DG methods rely heavily on pre-trained features.", "questions": "I have several questions regarding the significance of the proposed Perceptual Similarity Score and the conclusions of this paper. Please see the details above.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I appreciate that the authors consider the impact of pre-training weights in the current DG evaluation protocol.\n\nI also value their effort to remove noisy data labels in standard DG benchmarks.\n\nThis paper presents a comprehensive set of experiments.", "weaknesses": "**Weaknesses**\n\n- Since the goal of CLIP's contrastive loss is to maximize the similarity between the ground truth text label embedding and image embedding, it's unsurprising that the alignment score (measured as similarity to the ground truth text embedding) correlates with final performance after fine-tuning. Additionally, the alignment score applies only to vision-language models, not to pure vision models, which is a limitation that should be mentioned in the paper.\n\n- When comparing the predictive power of the Alignment and Image Similarity Hypotheses for downstream tasks, it would be helpful to provide quantitative metrics showing how strongly each hypothesis correlates with performance.\n\n- In line 177, is there a typo where “before” should be “after” in “The cosine similarity between image and ground truth text-label embeddings before pre-training”? Also, just to confirm, are the alignment scores and similarity scores shown in Figure 2 calculated from the CLIP model immediately after pre-training on LAION, but before fine-tuning on the downstream DG benchmark?\n\n- I wonder if using a Perceptual Similarity Score with a well-chosen threshold could similarly divide the data into high-similarity samples (DomainBed-IP) and low-similarity samples (DomainBed-OOP). Would the same experimental results hold—that DG methods perform well on DomainBed-IP but struggle on DomainBed-OOP after fine-tuning?\n\n- Based on experiments showing that DG methods perform well on high-alignment samples (DomainBed-IP) but struggle on low-alignment data (DomainBed-OOP), the paper concludes that current DG methods rely on pre-trained features. While this is a useful insight, the conclusion is somewhat unsurprising, as it’s well-known that downstream task performance heavily depends on the diversity of the pre-trained dataset, whether in-distribution or out-of-distribution. For those interested in accurately measuring a DG method’s generalization performance, [1] suggests a new evaluation protocol isolating pre-training's impact. Alternatively, pre-training on a larger and more diverse dataset is likely the best approach for those focused on improving downstream generalization.\n\n[1] Rethinking the Evaluation Protocol of Domain Generalization. CVPR2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gCjCEVZZoE", "reviewer": "Reviewer_R32V", "summary": "This paper critically investigates the successes of recent DG methods that rely on pre-trained backbones, such as CLIP, with a focus on answering an important question: whether those successes are due to the effect of pre-training or to the algorithmic details of DG methods. To this end, the authors first consider the recently proposed image-similarity hypothesis, which posits that if the pre-training data contains images that are perceptually similar to the target data, then the zero-shot performance will be higher. However, the authors find that this does not fully explain the phenomenon and instead introduce the alignment hypothesis. They posit that if the alignment between image embeddings and text labels is high, with respect to the cosine similarity score defined in CLIP models, these alignment scores serve as a better indicator of generalization. They perform this analysis before fine-tuning and after fine-tuning, thus disentangling the effects of pre-training. Given this observation, they create a new benchmark that splits domain bed datasets into in-pretraining (IP) and out-of-pretraining (OOP) splits. \n\nThey then train a large number of DG algorithms from the domain bed framework and observe that most methods actually perform better in the IP condition and perform poorly on the OOP condition, where the performance gap between DG and oracle is quite high. They propose to release these datasets to provide the community with a new benchmark that encourages algorithmic innovations to improve generalization, rather than just using a more powerful backbone.", "questions": "Please see weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well-written and considers an important question of understanding the importance of pre-training backbones in modern DG approaches. Creating a new benchmark is an important contribution.", "weaknesses": "The paper could be improved, especially in Section 3. The methods for identifying image similarity scores and alignment scores need clearer explanations instead of the brief sentences currently provided. I suggest including an algorithmic table, as this is a key contribution.\n\nWhile I appreciate the current contribution, it seems limited since it only considers one variant of the DG problem. The community has explored a broader range of generalization tasks  including sub-population shifts. It will be important to discuss how this work fits into those contexts as well. Additionally, the current alignment score relies solely on cosine similarity loss. I believe it should also take into account the uncertainties exhibited by the model, which would help in making a better split. This consideration is important because it would factor in the calibration of the pre-trained models, which the current approach does not address.\n\nFinally, it is not very clear to me whether combining both image similarity and alignment scores would provide a better indicator or does alignment score already take that into account. \nAlso, I would recommend the authors toning down the statements like \"DG has been solved with datasets like PACS etc\". May be the takeaway that authors want to communicate is that some datasets should be dropped from future DG research and I think that is a better way to put it than the current style. As a community, DG needs new evaluation benchmarks that go beyond the now simple datasets and the excellent performance on these are hardly surprising given they have common categories that a pre-trained model would have been trained on. \n\nTypos:\n\nIn the section 3.1, the equation of  perceptual similarity score is missing $\\langle$. \n\nLine 368, it should be ``between IP and OOP`` instead of  ``between OP and OOP``", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rpOQkjlIOp", "reviewer": "Reviewer_DeUV", "summary": "In this work, the authors investigate the relationship between certain aspects of pre-training and domain generalization (DG). They observe that high DG performance is achieved only when there is strong alignment between image and class label text embeddings. Based on this finding and evaluations on splits with lower alignment, they conclude that current DG methods predominantly rely on pre-trained features and struggle to learn novel, generalizable features from source domains.", "questions": "Although the central idea of the work is interesting, I have several concerns with the work as follows:\n1. **CNN Evaluations**: It would be valuable to experiment with pre-trained CNNs and CNNs trained from scratch using some of the DG methods, then evaluate them on their corresponding IP/OOP splits. Do differences in IP/OOP performance persist when training CNNs from scratch as compared to adapting a pre-trained CNN? And therefore does a freshly trained CNN learn more generalizable features than a pre-trained one?\n\n2. **Zero-Shot Classification**: To better assess how pre-training alignment influences DG performance, it would also be helpful to conduct zero-shot evaluations on both the IP and OOP test sets of DomainBed using the same backbone as the DG methods. This could open up various observations, such as the proportion of zero-shot classified points in each split and the number of points that shifted from correct to incorrect (or vice versa) after DG. What additional insights about alignment and generalization could zero-shot evaluation on IP and OOP data reveal?\n\n3. **Revisiting the Image Similarity Hypothesis**: The image similarity hypothesis addresses the link between zero-shot generalization and similarity to the pre-training dataset but may not directly translate to DG. An analogous approach for DG might involve examining the perceptual similarity score between the target domain's evaluation set and the source domains' training set to see if it correlates with performance. Does this modified similarity hypothesis hold when applied to DG settings?\n\n4. **Clarifying Key Takeaways**: While the paper presents some interesting insights, the main takeaways risk being obscured by details. A dedicated discussion section could enhance clarity by addressing questions such as (but not limited to): What insights from prior work connect pre-training with DG? What are the primary takeaways of this study, and how do they interrelate, and how would you contextualize with findings from prior work?\n\nI am willing to adjust my score if these concerns are addressed adequately.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The observation that alignment in pre-training between image and text embeddings significantly correlates to domain generalization (DG) performance is a valuable contribution. \n2. The paper’s approach of dividing DomainBed datasets into In-Pretraining (IP) and Out-of-Pretraining (OOP) splits provides a novel lens for evaluating DG methods. This split reveals that while many models perform well on IP data, they struggle with OOP data, indicating that these models are not fully generalizing beyond pre-trained features.", "weaknesses": "Please see the questions section for more information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)