CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment
Authors: Xi Yu
Shinjae Yoo
Yuewei Lin
Venue: NeurIPS 2024 poster (2024)
Keywords: Domain generalization
CLIP
domain invariant
image-text alignment
Primary area: machine_vision

Abstract: Domain generalization (DG) is a fundamental yet challenging topic in machine learning. Recently, the remarkable zero-shot capabilities of the large pre-trained vision-language model (e.g., CLIP) have made it popular for various downstream tasks. However, the effectiveness of this capacity often degrades when there are shifts in data distribution during testing compared to the training data. In this paper, we propose a novel method, known as CLIPCEIL, a model that utilizes Channel rEfinement and Image-text aLignment to facilitate the CLIP to the inaccessible $\textit{out-of-distribution}$ test datasets that exhibit domain shifts. Specifically, we refine the feature channels in the visual domain to ensure they contain domain-invariant and class-relevant features by using a lightweight adapter. This is achieved by minimizing the inter-domain variance while maximizing the inter-class variance. In the meantime, we ensure the image-text alignment by aligning text embeddings of the class descriptions and their corresponding image embedding while further removing the domain-specific features. Moreover, our model integrates multi-scale CLIP features by utilizing a self-attention fusion module, technically implemented through one Transformer layer. Extensive experiments on five widely used benchmark datasets demonstrate that CLIPCEIL outperforms the existing state-of-the-art methods. The source code is available at \url{https://github.com/yuxi120407/CLIPCEIL}.

CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment
Xi Yu, Shinjae Yoo, Yuewei Lin"
Artificial Intelligence Department, Computing and Data Science Directorate Brookhaven National Laboratory, Upton, NY 11973
{xyul; sjyoo; yulin)@bnl.ge
Abstract
Domain generalization (DG) is a fundamental yet challenging topic in machine learning. Recently, the remarkable zero-shot capabilities of the large pre-trained vision-language model (e.g., CLIP) have made it popular for various downstream tasks. However, the effectiveness of this capacity often degrades when there are shifts in data distribution during testing compared to the training data. In this paper, we propose novel method, known CLIPCEIL, model that uti- lizes Channel rEfinement and Image-text aLignment to facilitate the CLIP to the inaccessible aut-of-distributt
test datasets that exhibit domain shifts. Specif- ically, we refine the feature channels in the visual domain to ensure they con- tain domain-invarian, invariant and class-relevant features by using lightweight adapter. This is achieved by minimizing the inter-domain variance while maximizing the inter-class variance. In the meantime, we ensure the image-text alignment by aligning text embeddings of the class descriptions and their corresponding image embedding while further removing the domain sppccffi features. Moreover, our model integrates multi-scale CLIP features by utilizing self-attention fusion module, technically implemented through one Transformer layer. Extensive ex- periments five widely used benchmark datasets demonstrate that CLIPCEIL outperforms the existing ofate-of thethh-artttttt methods. The source code is available at https://githur http::/ggttuuu..coouuxuoooeeeee httm//uux1100077/LPPPGC
Introduction
Machine learning models inevitably face the challenge of out-of-distributinn (OOD) generalization when encountering new tasks with different distributions from the training data. To mitigate this issue, extensive research has been dedicated to domain generalization (DG) [66], aiming to utilize knowledge from source domains to enhance the model's generalizability the test dataset with domain shifts.
Recently, the spotlight has been advancements Vision-language models (VLMs), like CLIP 41 which are trained on web-scale image-language pairs containing diverse range of domains and concepts from open world, exhibit exceptional zero-shot learning and transferability various However, despite their impressive zero-shot performance,
downstream tasks [26] 31 33
65)
supervised fine-tuning on specifc datasets remains essential for further improving performance on downstream tasks. However, recent works [27]55] 555 have pointed out that fine-tuning degrades the
CLIP's generalizability on the dfitt--diittitttio test datasets exhibiting domain shift. To tackle this challenge, various methodologies have been proposed. For instance, CoOp (8] and CoCoOp [67 models utilized the prompt learning. DPL. [62] learned lightweight prompt generator, while WiSE- FT (55) combined the original zero-shot and fine-tuned models. More recently. CLIPood 44
'Y. Lin the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

achieved tftte-t performance by employing the beta moving average and margin metric softmax to fine-tune the CLIP. It is noteworthy that these approaches do not explicitly guide the model to learn ionainninvariiat features, potentially capturing some domain-related information.
One prominent trend in Domain Generalization (DG) involves acquiring domain-invarian fea- tures across variance of source
Magritude of Domain Standard Deviations
Magnitude of Class Standard Deviations
CUP Zero-Shot
CLIP Zero-Shot
Number of Channels
CUPCEIL
Number of Channels
46
CLICCELL
domains [2 32 19 []. as it has been demonstrated that
40
feature representations are gen- eral and transferable to differ- ent domains if they remain in- variant across domains ( Intu- itively, the domain invariant fea- tures are intrinsic to the class while remaining insensitive to the domain changes. However, as shown in Figure [ (a), many
20
20
10
0.00
0.01
(a) 0.03
0.02
0.00
0.02 0.04
(b)
Figure 1: The feature channel sensitivity to domain and class shifts are quantified through employing the histogram of their standard deviations across different domains and classes. We analyze CLIP's image embeddings using the ViT-B/16 backbone OfficeHome [52 dataset. For each channel, the average outputs are computed across all samples from each domain/class, and the standard deviations are calculated on domain/class dimension.
CLIP visual feature channels hibit unstable activations across domains (illustrated by the blue histogram), indicating a lack of domain invariance. Similarly, shown in Figure (b), many CLIP visual feature channels show insensitivity, and thus indiscrimina- tive, to class variations. These observations prompt the question:
Can we enhance the pre-trained model's 's generalizability by excluding domain-specifet (sensitive) and class-irrelevant (insensilive) features?
To answer it, we conduct simple experiment using the pre-trained CLIP model on OfficeHome dataset. Given the original 512 CLIP visual feature channels, we select the ones with low domain variance and high class variance. We calculate the variance to different domains (Va) and classes (V)) for each feature channel, and then utilize criterion J = Va V. to select the top-Q (Q 400) channels with the smallest values. Assuming effective alignment of visual-language features CLIP, we use the same Q channels for text features. During inference, we simply use the inner product of the visual and text feature vectors, akin to the approach used in CLIP zero-shot [41]. As shown in Table I the simple feature channel selection improves the CLIP zero-shot generalizability.
Motivated by the above observations, we propose CLIPCEIL, a simple yet effec- tive method aimed promoting domain- invariant and class-relevant information within CLIP visual features from the per- spective of feature channels. Specifically, we freeze the CLIP visual and text encoders and exclusively train lightweight adapter for visual features, which fuses the multi-scale features, while minimizing the inter-domain variance and maximizing the inter-class variance. Furthermore, we establish alignment between image and text spaces by ensuring the consistency of direction among different classes in both the image and text domains. Our contributions are summarized as follows.
Comparison of channel selection (Q = 400) Table 1:
with the CLIP zero-shot Office Home benchmark
Model
P
Avg
CLIP full features
82.7
68.0
88.3
90.7
82.4
Channel-Selection
84.9
68.3
89.4
91.2
83.5
We propose to adapt CLIP through Channel rEfinement and Image-text aLignment (CLIPCEIL), ensuring the visual feature channels contain the domain-invariant and class- relevant information while preserving the image-text alignment.
Our model integrates multi-scale CLIP features by using self-attention mechanism, techni- cally implemented through one Transformer layer.
We comprehensively evaluate our proposed method on five benchmark datasets. The results demonstrate that our method achieves state-of-the- art performance.

---

2 Related Work
Vision-Lannguage Models (VLM). The VLMMs aim to link images and texts by embedding them into shared space for cross-model learning 45 [2]. Recently, equipped with advanced architecture (e.g., Transformer [51]) and trained on huge web-scale image-text pairs, the VLMs have attracted significant attention and demonstrated superior performance various downstream tasks like image classification, segmentation, object detection, and image-text retrieval. For instance, CLIP [4]] pre-trained 400M image-text pairs using contrastive loss, demonstrates outstanding zero-shot prediction capability. ALIGN [22], trained on 1.8B noisy image-text pairs with noise-robust con- four tasks simultaneously, achieving superior image-text retrieval performance. SLIP [36] incorporates superviino into contrastive learning, leading to more efficient pre-training. BLIP (3) and BLIP-2 [29 employ joint optimization with three objectives, achieving state-of-the-an performance on a wide range of vision-language tasks. Instead of developing new pre-trained model, our work aims leverage CLIP to enhance domain generalization performance.
trastive learning, ImageBERT (39, pre-trained
Domain Generalization (DG), DG aims to train model that generalizes well to the out-of-distribution test (target) domains, solely training on source domains. One typical way is domain augmentation. which either diversifies the source domain simulates the inaccessible test (target) domain conditions like domain randomization [25] 18 (0). adversarial data augmenta-
69]. Alternatielyy methods focus on the learning strategies, including ensemble learning 42 and meta-learning (2). Another prevalent
tion [53 64 58 and data generation [46 43
37
40
23
approach is representation learning, aiming capture the domain-invariant representations on source domains. (60) extracts the invariant semantic features by jointly learning the semantic and variation encoders. 337 learned style-invariant representation by reducing the intrinsic style from the class categories through the style-agnostic networks. [5] first disentangled the latent representations in dpmain-specific and domain-invariant and then concatenated them to make final decisions. Similarly, [59] proposed the information theory inspired disentanglement and purification loss functions to explicitly disentangle the latent feature in class-relevant and class- cass-iirreevvatt components. Most recently, Domp [17] dropped domain-specing channels during training by using additional domain discriminator networks.
In recent years, research has focused on enhancing the generalization of VLMs, like CLIP. Some studies learn the task- task-specific prompts [68 67 62). while others utilize the ensemble learning (55) or adapter learning [1]6]]] Despite the superior performance of large pre-trained VLMs, they still struggle with outoo--distriuution (OOD) generalization. Efforts have been made enhance their generalizability, e.g. StyLIP 4 and DPL (68] proposed the prompt learning approach for domain generalization. VL2V-SD M improved the OOD generalization of the VLM by visual-text alignment and visual encoder distillation. More recently, approaches like inference- time fine fine-tuning [63] or fine- tuning the entire visual encoder [35] 44 have been explored to further improve model generalizability, However, the former incurs an additional computational burden during inference, while the latter faces significant computational and storage challenges, requiring full CLIP--izeed model for each task. In contrast, our proposed model, once trained, does not require additional adaptation during inference, and we only need to store lightweight model for each task.
3 Methods
3.1 Problem Setup
This paper aims to improve the out-of-distributtin generalization through the pre-trained VLM. Let X' C R2 be the image space and y C R the class label space. A domain consists of data sampled from a joint distribution Pxy on X'
the context of domain generalization, we have K labeled training (source) domains {DD = {- v.") v } 1k=1 where ni is the number of samples in the tth domain, and each domain D associated with joint distribution pky- Note that each domain has
different joint distribution: Pky # Psy, 1 i + j < K. The goal of domain generalization is to train a model f X' + y from K training domain D. and achieve good generalization on an out-of- of-distribution inaccessible test (target) domain D, {x,, vt) ) where y" Y, and plest + Pxx for i E K).
3

---

Test Encoder
Prompt template "closs"
Maximize Image-Text
Similarity
Frozen
Tuned
*
Visual Channel Refinement
Dog
Horse
Photo
H
Visual Encoder
Projector
House
Pairs of Images from different classes different domains
Transformi
Image- Alignment
(())))
g(((x)))
yi
Prompt template
"horse"
Encoder
T
Tent
y1
O
Prompt template "dog"
T(y))
An overview of the proposed framework. We fixed the CL.P visual encoder 7 and text encoder and trained a lightweight adapter g during the training. The channel refinement ensures each feature channel contains domain-invariant (minimizing domain variance) and class-relevant (maximizing class variance) information. further align the image and text, we maximize the image-text similarity and minimize direction loss with the help of text class descriptions based on data pairs from different classes and domains.
Figure 2:
3.2 Framework Overview
The overview of our framework is illustrated in Figure. which consists of three primary components. The first one the lightweight adapter, depicted in the orange block of Figure 2 It fuses the multi- scale CL.I visual features and maps them to a latent feature space, aiming to enhance the model's generalizability. The second component visual channel refinement, which aims ensure the visual features contain domain-invariam and class-relevant features. As observed from Figure [ CLIP's visual features have numerous channels that exhibit sensitivity domain variations, which essentially domain-speciin features, as well as channels that exhibit insensitivity to class variations, which are essentially ftase-irrelevant features. that both features are often redundant and may hinder the model's 's generalizability. Our framework aim eliminate these undesirable features by minimizing the feature variance across domains and maximizing feature variance across classes. The third one is the image-text alignment component. The feature channel refinement module, working solely the image space, has the potential disrupt the well-aligned image-text feature space from CL.IP. Therefore, realigning the image and text spaces becomes necessary. Specifically, we introduce the direction loss to minimize the difference between the direction of two image features and that of their corresponding textual features. We describe each component of our framework thoroughly in the subsequent sections.
the context of domain generalization, it argued
3.3
Adapter g
A CLIP's visual encoder consists of several vision transformer layers and a final project layer, as depicted blue block in Figure Let 7 denote the visual encoder within CLIP. Given an image x, its visual features in CLIP are represented as 7(x) = ff
Here, f IRd signifies the
feature map derived from the [cls] token in the 1hh layer, with dimension of d, where L stands the number of transformer layers. Additionally, / ffinal RDD represents the ultimate output of CLIP's visual encoder, obtained by passing the feature map of the last layer o& through inherent MI.P projector. In this paper, we use ViT-B/16 as the visual encoder backbone with the number of transformer layers L = 12, the feature dimensions d 768 and D 512.

---

We aim to enhance the visual features' resilience the domain shifts. Therefore, we propose lightweight adapter y that consists of Trans- former layer [51] and an MIP projector, specif- ically utilizing the self-attention mechanism to integrate visual features from different Trans- former layers in the CLIP encoder and map these features to a latent feature space that ben-
Image
Class tokee
Visual Encoder
Projector
efits the model's 's generalizability. Specifically, the multi-scale features {fx [ are fed into Transformer layer Tr, the feature obtained from each layer treated as a token. The feature extracted from the [cls] token in the output of Tr is considered as the fusion of multi-scale features. This fused feature is then directed into a single-layer MLP projector Pr, which maps it from dimension d to D. Finally, both the output of Pr and the CLIP final feature ffinal fused by residual connection to obtain ultimate visual embedding Zx. More formally, it is formulated as follows:
Transformer
Figure 3: The architecture of the adapter go.
Xx - (x(()) = Pr (Tr ((&)))) + +f Final
where represents all the learnable parameters within the adapter g.
3.4 Channel Refinement
To extract domain-invariant and class-relevant features, while eliminating those that are domain- specific and class-irrelevant, we design a channel refinement loss based on two criteria, 1) inter- domain variance: domain-invariant features should exhibit minimal changes across different do- mains, implying smaller inter-domain variance; 2) inter-class variance: class-relevant features should change across different classes, while the changes are expected large possible to have more discriminative ability, indicating they should have larger inter-class variance.
Inter-domain Variance. It measures changes in feature channel across domains. Given the 7th input image from tth domain, x its refined fea-
ture z go (I((x))) and we de- note its m dimension zx, gfm) As chonnel
Channel-utie wile overage acrous sumples
d
d
dv
shown in Figure. 4 we first put fea- tures from all the images from the same domain together, i.e., each col- umn indicates the feature of one im- age. Then, we calculate the (m)
P
dbmuiiss
2m
channet
refers to the mht channel-wise aver- age value of all the samples in the kimo tth domain: z,() = Σm 2x;
Chamnst-wohe averape
Average
Variance
Domatn de
where n4 is the number of samples in
the 1.th domain. Finally, inter-domain variance of the m th channels is calcu- lated as follows:
Figure 4: Diagram of calculating the channel domain sensi- tivity across different domains.
K Σ(Z [Z(m)
V (m)
-z Z(m))2
(2)
where K is the number of domains, different domains.
(m) represents the average output at mh channel across
5

---

Inter-class Variance. It measures changes in a feature channel across different classes. Similarly to inter-domain variance, we use the same way to compute the inter-class variance, formulated in Eq.
L
((m)
m)) =!
(3)
where L is the number of classes and Z
denotes the channel-wise average value of all samples from hth category, where nı is the number of samples in the hth category, and ffw)
xx
(m)
denotes the refined feature from 7th input image in gth category. Ze (m) represents the average output
mh channel across different classes.
To ensure the image feature channels contain both domain-invariant and class-relevant information, minimize the inter-domain variance to eliminate the domain-specifer information and maximize the inter-class variance to capture more discriminative class-relevan information. Our channel refinement loss combines the above two criteria in the following way:
1 D
((())
Liel
Σ log 1+
D
[.(m)
where D refers to the number of feature channels.
3.5 Image-Text Text Alignment
The adapter ge maps features from the CLIP's image embedding space I to the refined image embedding space 2, aiming for capturing domain-invariant and class-relevant features. However, this mapping may disturb the well-alignment between image spaces I and text spaces T provided by CLIP, leading misalignment between Z and T spaces. Therefore, it is necessary re-align the refined image space and text space T. attain this objective, we first simply employ the standard CLIP loss formulated follows:
Lee Cross-entropy (Sottaaa/lol ((xx)) Tyl.
(5)
where 4." inner product, T, (t,,)) denotes the text embedding text prompt by of class y.
However, the standard CLIP loss only aligns image embedding with the correct text embedding on per-sample basis but overlooks the potential relationship between samples. Thus, we propose to explore semantic structure information to strengthen the image-text aligmment. Inspired by prior work [3 aim align the pairwise directions in the image and the text spaces. To this end, we first normalize the pairwise distance in image and text space and then directly minimize their cosine similarity. For pair training samples {(x;, Vi), (xj, my)), the direction loss is defined
go (o -90(())))9(((())) (xx,)) (T(x:) 3o (T(x))
Tg. T yj T
Ldir =
T
To further remove the Nomain-speciffe information in the image space, we sample the pair data from different domains and different classes and align them with the direction of the corresponding classes in the text space. Since the language embedding of the class is naturally domain-invariant, Thus, if the output of ga ([(x;) between them will not align with the corresponding class text direction. Therefore, the direction loss strengthens the image-text alignment by exploiting semantic structure information as well as removing domain-specific information in the image space.
ge (I(x))) contains any domain-speciff information, the difference
3.6 Training and Inference
We aggregate all the losses to our overall objective defined as follows:
min 0
L LCE + Leef + Lairs
(7)

---

Algorithm Training Procedure of CLIPCEIL
Input: Pre-trained CLIP image encoder I, text encoder T, adapter go, initialized with ERM.
I: for / E [1, N] do
2: Sample data {(x. y)} from the source domain set S.
3:
Calculate Channel Refinement loss Liet (Eq. 4 and Cross-Entropy loss Lce (Eq. 5
4:
Sample the pair data {x:: M1), (xj, my)} from the source domain set S, where and x, are from different domain and y/ o y/-
5:
Calculate Direction loss Ldir (Eg. 6 on above pair data samples.
6: Update with total loss (Eq. with Beta Moving Average (BMA).
end for return: 98-
7:
where 0 is the parameters of trainable adapter go- We show the overall training procedure of the proposed CLIPCEIL method in Algorithm
To incorporate prior knowledge of CLIP, during the inference stage, ensemble the fine-tuning model's 's prediction and CLIP zero-shot prediction to obtain the final classification logits. The logits of sample x; are formulated as follows:
logitsx, = -1( w (x)(((x)))
(8)
where W .T...... (T)), Cs the number of classes.
4 Experiments
This section showcases the superiority of our method across five widely used DG benchmark datasets. Furthermore, we carry out detailed ablation studies to determine the impacts of different loss terms, the channel refinement strategies, and the architecture of adapter g.
4.1 Datasets and implementation details
We evaluate our proposed method on five standard DG benchmarks: PACS (88 contains 9991 images of categories from 4 domains; VLCS [48] comprises categories from 4 domains, 10,729 images in
total; OffeceHome [52] contains 15,579 images 65 categories from domains; TerraIncognita [2] contains 24,788 images with 10 categories from 4 domains; DomainNet [8] is more recent and
the largest one among all five datasets, which contains 0.6 million images in 345 categories from domains. We utilize the CLIP pre- mre-trained model with the VIT B/16 [10 backbone. More results of other backbones are in Appendix C.1 We fixed the image and text encoders and solely trained adapter g during training. To avoid the influence of different template prompts, the output of the text encoder is calculated by the average of 80 template prompts from ImageNet [41] In all experiments, we use the open-source code DomainBed [16] and follow the train-validate--
split of each dataset on the DomainBed benchmark. Following the literature, we train our model with 5000 iterations on PACS, VLCS, OfficeHome, and TerraIncognia datasets and 15000 iterations on the DomainNet dataset. Our model is selected based on the source domain validation set. All experiments are conducted on the NVIDIA A100 GPUs. All the results were averaged after five runs with different random seeds. More detailed information are in Appendix
4.2
Main Results
We evaluate our CLIPCEIL model against the state-of-the-art (SOTA) approaches on five standard benchmark datasets. We initially compare with CLIP zero-shot, which serves as pre-trained vision- language baseline model without any training, which outperforms of-e-oof-the-rr ResNet-50 based models, e.g., SAGM [54] and DomainDrop [17]. demonstrates the superior of the pre-trained VLMs. We further compare with standard linear probing, which learns a single-layer linear classifier upon CLIP encoder, and three SOTA VLMs based models, i.e., the mutual-information regularization based MIRO (7) model, the prompt learning based DPI. [62 and StyLIP 4 models. To extend the comparison, adapt three widely-used prompt learning models, i.e., CoOp (68), CoCoOP (67).
7

---

MaPLE [4], and one adapter-based method CLIP-Adapter [1]] which are originally designed for few-shot learning, the DG task using the same experimental setting on the DG benchmark. Furthermore, to ensure fair comparison with methods that fine-tune the entire visual encoder such as CLIPood [4]. CAR-FT 35 and UniDG [33, we train our CLIPCEIL similarly, which we term CLIPCEIL++ Note that UniDG [63] is an inference-time fine-tuning model, which adapts the model
with additional information from the target domain.
Table Comparison our proposed method with the State oftte-of-the-art methods on the DomainBed benchmark. denotes ResNet-50 backbone; denotes frozen CLIP ViT-B/16 encoder; denotes fine-tuning the entire CLIP ViT-B/16 encoder, * denotes the two rounds inference- fine-tuning. Red and indicate the best performance in each group.
Model
Venue
PACS
VLCS
OfficeHome
Terrainc
DomainNet
Avg
SAGM [54]
CVPR
86.6
80.0
70.1
48.8
45.0
66.1
DomainDrop 17
ICCV'23
89.5
78.3
71.8
44.4
Zem-SSht
96.2
81.7
82.4
33.4
57.5
70.2
Lin.Pro
96.5
82.6
80.4
73.5
CoOp [68]
IJCV'22
96.0
81.1
83.5
47.0
59.8
73.5
CoCoOp [67
CVPR'22
95.7
$3.1
$4.3
50.4
60.0
74.7
CLIP-Adapter 15
UJCV224
96.4
84.3
82.2
59.9
MaPLE 24
CVPR'23
97.6
85.1
83.4
60.4
DPI. (62
2023
97.3
84.2
56.7
75.0
StyLJP 4
WACV'24
98.1
86.9
84.6
62.0
CLIPCEIL
Ours
97.6
88 = 0.4
85.4 0.2
53.0 + 0.3
62.0 0.1
77.3 0.2
MIRO 7
ECCV 22
95.6
82.2
82.5
54.3
54.0
73.7
CLiPood
ICML'23
97.3
85.0
87.0
60.4
78.6
CAR-FT 33
IJCV'24
96.8
85.5
85.7
61.9
62.5
78.5
UniDG 663
arXiv'23
96.7
86.3
86.2
62.4
61.3
78.6
VLV2-SD 1
CVPR '4
96.7
83.3
87.4
58.5
62.8
77.7
CLIPCEIL++
Ours
97.2 0.1
85.2
0.5
87.7
0.3
62.0 = 0.5
6..6
79.1
0.2
As illustrated in Table 2 our proposed CLIPCEIL exhibits significant improvement over the CLIP Zero-Shot and achieves the best average performance five benchmark datasets among all the compared methods. Specifically, CLIPCEIL exceeds the second-best method DPL [62] by 2.3% average, CLIPCEIL++ exceeds the second-best method CLIPood 44 by 0.5% average. The results prove CLIPCEIL's effectiveness enhancing the model generalization through capturing domain-invariant and class-relevant features. More detailed break-down results are in Appendix
4.3 Ablation Studies
4.3.1
Effectiveness of each loss term
Firstly, conduct the ablation study to examine the efficacy of each loss (i.e., channel refinement loss Lref, and direction loss Lair) in our overall objective function. Cross- loss CCe is very standard and thus include it by default, similar multi-scale fusion, which will be investigated in Section 4.3.3 Table B the results of different CLIPCEIL variants with the pre-trained ViT- B/16 model the OfficHHome dataset. As shown in the table, utilizing multi-scale information alone can enhance performance compared to the CLIP Zero-Shot. Integrating Lref leads to further enhanced performance, indicating the effectiveness in channel refinement loss to capturing domain-invariant and class-relevant information. Similarly, the improved performance of adding Ldir suggests that the direction loss contributes enhancing As result, combining all three components results in the best performance, showing that each loss works as an indispensable component for achieving superior generalization of the framework.
features through the help text description.
Table 3: Ablation study of each loss in our objective function on OfficeHome dataset.
Model
C
P
R
Avg
Zero-Shot
82.7
65.0
88.3
90.7
82.4
+Multi-scall
82.0
69.6
90.6
90.4
83.2
+Multi-scale+&.od
83.5
70.6
91.3
90.7
81.1
+Multi-scale+&/:
83.9
70.8
91.8
91.2
84.4
CLIPCEIL (Full Model)
86.0
71.2
92.2
92.3
85.4

---

To further demonstrate the corporation of each loss term, we visualize the image features of the CLIP pre- rre-trained model and our proposed CLIPCEIL on the OfficeHome dataset in Figure Different colors represent different classes or domains. As illustrated in Figure 5 (a) and (b). the image features extracted by CLIPCEIL exhibit more discrimination than the CL.IP pre-trained model, proving the effectiveness of CLIPCEIL in capturing the class-relevant features. Meanwhile, the image features corresponding to different domains extracted from CLIPCEIL are distributed almost equally across all classes, demonstrated in Figure (d), indicating that CLIPCEIL definitely extracts domain-invariant features. In contrast, image features from the CLIP pre-trained model are located in various places across different domains, shown in Figure 5 (c), suggesting that it still contains domain-specine information. The visualization of other datasets can be found in Appendix B.2
-2
Zero-shot CLIPCEIL
(c) Zero-shot domains
CLIPCEIL domsins
Figure 5: I-SNE A9 visualization on image features CLIPCEIL and CLIP pre-trained models across different classes and domains. Different colors indicate different classes domains
4.3.2 The effectiveness the two criteria in channel refinement loss
Our proposed channel refinement loss Lret is based on two criteria, namely inter-domain vari- ance and inter-class variance. To demonstrate the effectiveness of these criteria, we conducted experiments on all five datasets. In Figure. O the results show that combining inter-domain vari- ance with inter-class variance (represented by the darkest bars) results in better performance than using either criterion alone. This indicates that the two criteria can be effectively blended and class-relevant
100
IntenClas
Inten-Demwin
Bath
80
Accuracy
60
and both Nomain-invarian information complement each other and are es- sential to enhance model's generalization abil- ity. More detailed breakdown results are in Ap- pendix B.3
Officeoome
DomainNet
Figure The average accuracy bar of the different channel refinement strategies.
4.3.3 Architecture of adapter g
We investigate the structure of adapter by comparing the efficacy of multi-scale and bypass connections. As indicated in Table 4 integrating both multi-scale and bypass connections yields the most optimal performance. This can be attributed to two main factors: (1) The multi-scale approach captures a wide range of image features from both lower and higher levels, making it more generalizable than solely using the final layer output. (2) The bypass design preserves the original CLIP pre-trainee pre-trained knowledge and easier to optimize. More ablation studies for different adapter architecture and integrating Multi-scale text features are in Appendix C.2 and
Table 4: Ablation study of different adapter architectures.
Multi-scale Bypass
A
C
P
R
Avg
X
K
83.2
69.6
90.3
91.6
83.5
/
84.0
70.2
91.0
91.8
84.3
x
83.8
70.5
91.7
92.0
84.6
/
86.0
71.2
92.2
92.3
85.4

---

5 Discussion: Potential Data Leakage in CLIP on DomainBed Benchmarks
This section discusses the possibility of data leakage when fine-tuning the pre-trained CLIP model on DomainBed benchmarks. A primary concern is whether the DomainBed datasets truly represent out- of-distribution (OOD) data for CLIP, given its extensive pretraining on 400 million image-text pairs. We argue that the data distributions differ significantly: DomainBed datasets, such DomainNet, display distinct characteristics like imbalance and long-tailed distributions, in contrast to the balanced nature of CLIP's pretraining dataset [4] [56]. Furthermore, CLIP's zero-shot performance benchmarks like Terralncognita and DomainNet highlights that certain domains (c.g., Infograph and Quickdraw in DemainNet, and camera-trap images in Terralncognita) remain underrepresented in the CLIP pretraining corpus. These observations suggest that the distribution, style, and specific content of CLIP's pretraining data diverge meaningfully from those in DomainBed, potentially mitigating concerns about data overlap and preserving the intended OOD nature of DomainBed benchmarks.
6 Conclusion
In this paper, we introduced the CLIPCEIL model to enhance the generalizability of the pre-trained CLIP model the test datasets undergoing domain shifts. Specifically, we proposed a lightweight adapter for the refinement of visual feature channels to ensure the inclusion of domain-ime invariant and class-relevant information, which is achieved by minimizing inter-domain variance while maximizing inter-class variance. We maintained image-text alignment by aligning image features with the text features of their corresponding textual descriptions, concurrently eliminating domain-specific features. Comprehensive experiments on five benchmark datasets illustrated that CLIPCEIL surpasses the existing state-of-the-ar methods.
Limitations. Since calculating inter-domain variance involves multiple domains, CLIPCEIL currently only applies to multi-source domain generalization. Exploring its applicability single-source domain generalization is deferred for future investigation.
Acknowledgments
This work was supported by the U.S. Department of Energy (DOE), Office of Science (SC), Advanced Scientific Computing Research program under award DE-SC-0012700 This work was supported by the Laboratory Directed Research and Development (LDRD) Program (24-063 and 25-006) of Brookhaven National Laboratory under U.S. Department of Energy Contract No. DE-SC0012704. We are grateful to the anonymous reviewers for their valuable feedback and constructive suggestions, which have significantly improved this paper.
References
[1] Sravanti Addepalli, Ashish Ramayee Asokan, Lakshay Sharma, and R Venkatesh Babu. Leveraging vision-language models for improving domain generalization in image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 23922 229322, 2024.
[2] Sara Beery, Grant Van Hom, and Pietro Perona, Recognition terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018.
[3] Shai Ben-David John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
[4] Shirsha Bose, Ankit Jha, Enrico Fini, Mainak Singha, Elisa Ricci, and Biplab Banerjee. Stylip: Multi-scale style-conditioned prompt learning for clip-hased domain generalization. In Proceedings of the IEFF/CVF Winter Conference Applications of Computer Vision. pages 5542-5552, 2024.
Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specific features enhance domain generalization. Advances Neural Information Processing Systems, 14-21189-212011 2021.
[6] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han- Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405 11222005-22488,, 2021.
10

---

[7] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In European Conference Computer Vision, pages 440-457. Springer, 2022.
[8] Chia- Yuan Chang, Yu-Neng Chuang. Guanchu Wang, Mengnan Du and Na Zou. Dispel: Domain generalization via domain-specifie liberating. arXiv preprint arXiv:2307.07188. 2023.
[9] Ching-Yau Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via invari-in representations. In Proceedings of the 37th International Conference on Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pages 1188-19944 PMLR, 2020.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
[11] Lisa Dunlap, Clara Mohri, Devin Guillory, Han Zhang, Trevor Darrell, Joseph E Gonzalez, Aditi Raghu- nathan, and Anna Rohrbach. Using language to extend to unseen domains. In The Eleventh International Conference on Learning Representations, 2022.
[12] Andrea Frome, Greg S Corrado. Jon Shlens. Samy Bengio. Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances neural information processing systems, 26, 2013.
[13] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan- nada: Elip-guided domain adaptation of image generators. ACM Transactions Graphics (TOG), 414::1111, 2022.
[14] Peng Gao. Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li. and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, pages 1-15, 2023.
[15] Peng Gao. Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-595, 2024.
[16] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.0143/ 2020.
[17] Jintao Guo, Lei Qi, and Yinghuan Shi. Domaindrop: Suppressing -enniiive channels for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[18] Narges Honarvar Nazari and Adriana Kovashka. Domain generalization using shape representation. In European Conference on Computer Vision, pages 666-670. Springer, 2020.
[19] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. Uncertainty in Artificial Intelligence, pages 292-302. PMLR, 2020.
[20] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
[21] Maximilian Isse. Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322 222-348. PMLR, 2020.
[22] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision laioun-anguuage representation learning with noisy text supervision. In International conference on machine learning, pages
PMLR, 2021.
[23] Juwon Kang, Sohyun Lee. Namyup Kim, and Suha Kwak. Style neophile: Constantly seeking novel styles for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7130-7140, 2022.
[24] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modaa prompt learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19112-1922, 2023.
[25] Rawal Khirodkar, Donghyun Yoo, and Kris Kitani. Domain randomization for scene-specific car detection and pose estimation. In 2019 JEEE Winter Conference on Applications of Computer Vision (WACV), pages 1932-1940. IEEE, 2019.

---

[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.00 0264333..0..3..... 2023.
[27] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribation. arXiv preprint arXiv:2202. 10054, 2022.
[28] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pages 5542-5550 2017.
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2:
Bootstrapping language -aggggee-mmage pre- are-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[30] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imag image pre- vre-trainin for unified jsion-languuae understanding and generation. In International Conference on Machine Learning, pages I 12888-1 12900. PMLR, 2022.
[31] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
[32] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 024-639, 2018.
[33] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-iman pre-training paradigm. arXiv preprint ar.iv:2110.05208, 2021.
[34] Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. arXiv preprint
11X10:1711.05100,2111
[35] Xiaofeng Mao, Yufeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. International Journal of Computer Vision, pages 1-16, 2023.
[36] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In Eumopan Conference on Computer Vision, pages 529-544. Springer, 2022.
[37] Hyconseob Nam, HyunJac Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[38] Xingchao Peng, Qinxun Bai, Xide Xia. Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1405-1415, 2019.
[39] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale seak-supervised image-text data. arXiv preprint arXiv:2001. 07966,, 2020.
[40] Fengehun Qiao and Xi Peng- Uncertainty-guided model generalization to unseen domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6790-6800 2021.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[42] Mattia Segu, Alessio Tonioni, and Federico Tombari. Batch normalization embeddings for deep domain generalization. Pattern Recognition, page 109115, 15, 2022.
[43] Yang Shu, Zhangjic Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain- augmentd meta-learning In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9624-9633, 2021.
[44] Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to out-of-distribution In International Conference Machine Learning, 2023.
12

---

[45] Richard Socher, Milind Ganjoo, Christopher D Manning. and Andrew Ng. Zero-shot learning through cross-modal transfer. Advances in neural information processing systems, 26. 2013.
[46] Nathan Somavarapu. Chih- Yao Ma, and Zsolt Kira. Frustratingly simple domain generalization via image stylization. arXiv preprint arXiv:2006.1110 11200, 2020.
[47] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schncider, Wojeiech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSU international conference intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[48] Antonio Torralba and Alexci A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528, 2011.
[49] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 9(11), 2008
[50] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5018-5027, 2017.
[53] Riccardo Volpi, Hongseok Namkoong, Ozan Sener. John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing unseen domains via adversarial data augmentation. Advances neural information processing systems, 31, 2018
[54] Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 077699-778. 2023.
[55] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Korblithh, Rehecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[56] Hu Xu, Saining Xie, Xiaoging Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Weang-WWnnte Li, Gargi Ghosh. Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint 00Xiv:339916677 2023.
[57] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang. and Qi Tian. A fourier-based framework for domain generalization. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14383 14392, 2021.
[58] Fu-En Yang. Yuan-Chiz Cheng, Zu-Yun Shiau, and Yu-Chiang Frank Wang. Adversarial teacher-student representation learning for domain generalization. Advances in Neural Information Processing Systems, 34:19448-1998 19468,,99,,,001 2021.
[59] Xi Yu, Hlian-Hsin Tseng, Shinjae Yoo, Haibin Ling, and Yuewei Lin. Insure: information theory inspired disentanglement and purification model for domain generalization. IEEE Transactions on Image Processing, 2024.
[60] Hanlin Zhang. Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf. and Eric P Xing. Towards principled disentanglement for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8024-8034, 2022.
[61] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.0300 2021.
[62] Xin Zhang, Shixiang Shane Gu, Yutaka Matsuo, and Yusuke Iwasawa. Domain prompt learning for efficiently adapting clip to unseen domains. Transactions of the Japanese Society for Artificial Intelligence. (866):B-MC2 1, 2023.
[63] Yiyuan Zhang, Kaixiong Gong, Xiaohan Ding, Kaipeng Zhang, Fangrui Lv, Kurt Keutzer, and Xiangyu Yue. Towards unified and effective domain generalization. arXiv preprint arXiv:2310.1000 2023.
13

---

[64] Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entropp adversarial data augmentation for improved generalization and robustness. Advances Neural Information Processing Systems, 33:14435- 14447, 2020.
[65] Ye Zheng, Xi Huang, and Li Cui. Visual language based succinet shot object detection. In Proceedings of 29th ACM International Conference on Multimedia, pages 5410-5418, 2021.
[66] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[67] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Puttern Recognittinn (CVPR). 2022.
[68] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for visoun-aanguage models. International Journal of Computer Vision, 130(9))2337-2383 2022.
[69] Kaiyang Zhou, Yongxin Yang, Qiao, Tao Xiang. Domain generalization with mixstyle. International Conference on Learning Representations, ICLR. OpenReview.net, 2021.
14

---

The appendix is organized into the following sections:
Table of Contents
A Dataset and implementation details
B Full results
B.1
Domain Generalization henchmarks
B.2 Visualization of visual features
B.3
Ablation studies on channel refinement criteria
C Additional experiments
C.1
Performance on difterent backbones
C.2 Ablation studies for Adapter g
C.3 Apply multi-scale mechanism on text encoder
A Dataset and implementation details
We evaluate our proposed method on five conventional DG benchmarks. PACS [28] contains 9991 images of 7 categories from 4 domains: photo (P), art-painting (A), cartoon (C) and sketch (S), OfficeHome (52) contains 15,579 images in total with 65 categories from 4 domains of styles: Artistic (A), Clip-Art (C), Product (P) and Real- World (R). TerraIncognita 2 contains 24,788 images with 10 categories from 4 domains, i.e., four different locations where the images are taken. VLCS [48] comprises 5 categories from domains, VOC2007 (V), LabelMe (L), Caltech (C) and Sun (S), and 10,729 images in total. DomainNet (38] is a more recent and the largest dataset used in domain generalization task. In total, it contains 0.6 million images in 345 categories from six clipart, infograph, painting, quickdraw, real, and sketch.
domains:
We the CLIP pre-trained model with ViT-B/16 the image encoder. We freeze both image and text encoders during the training and only train a lightweight adapter consisting of one transformer layer and single-layer MLP projector. The structure details of adapter are reported in Table 6 To avoid the influence of different template prompts, the output of the text encoder is obtained by averaging 80 template prompts ImageNet [41 represented in Table
Our optimizer is
AdamW [34] with a weight decay of 5e 4, and the learning rate is initialized to 5e 5, gradually decreasing by using the cosine annealing scheduler. We train the model for 5000 iterations for all the datasets except for DomainNet (38) with 15000 iterations. We adopt a batch size of 32 for all datasets, and all images are randomly resized and cropped to 224 224. Following the same training process of CLiPood 44). we utilize the beta moving average (BMA) to update our parameters during the training. All the default configurations are shown in Table 5 All experiments are conducted on GPU server equipped with 4 NVIDIA A A100-SXN 100-SXX4-80GB GPUs, although only 2 were used for this paper. The server also has an Intel Xeon Gold 6336Y CPU @ 2.40GHz with 24 cores and 48 threads, 1 TB of memory. Our CLIPCEIL model is implemented and evaluated with Python 3.8. 13, Py PyTocch 1.8.0, Torchvision 0.9.0, and CUDA 11.1.
Table 5:
Default configurations for the experiments.
Default Settings
Value
optimizer
Adam [34]
base Ir
x 10
weight decay scheduler
x 10 -4
cosine decay
batch size
32 RandomResizedCrog
augmentation
# iterations
5000
15

---

Table 6: Structure details of Adapter g.
Transformer (Tr)
Projector (Pr)
Width
Head
Layer
Input
Output
Layer
786
1
786
512
1
Table 7: 80 template prompts on the ImageNet
Template Prompt
bad photo of O}. photo many (}.
the origami Q
the O in video game. sketch of O..
sculpture of O
a photo of the hard to see 0.
doodle of the 0 origami O..
low resolution photo of the(?. a rendering of a 0.
low resolution photo of O. the toy O.
graffiti of O.
a bad photo of the 0.
rendition of the 0.
cropped photo of the O.
photo of the clean O.
a tattoo of f}.
photo of a large().
the embroidered O.
rendition of O..
a photo of a hard to see 0.
photo of a nice O.
bright photo of O. a photo of a clean O. photo a dirty O.
photo of a weird 0 blurry photo of a O. cartoon O}.
dark photo of the O. drawing of O.
art of a O. sketch the 0.
photo of my O.
embroidered O.
the plastic O.
pixelated photo of a0. tap of the O.
photo of the cool O.
close-up photo a 0.
jpeg corrupted photo of the 0.
black and white photo of the 0. a painting the O.
good photo of O.
plushie O..
painting of O. a pixelated photo of the 0. sculpture of the O.
photo of the nice O. photo of the small Q. photo of the weird O.
a bright photo of the 0. cropped photo of 0. a plastic O.
the cartoon O.
art of the O. drawing of the O. photo of the large O.
photo of the dirty 0.
a jpeg corrupted photo of a 0. blurry photo of the 0.
black and white photo of a O.
the plushie O. dark photo of O. itap of a O..
a photo of the O. good photo of the 0. a rendering of the O. video game.
graffiti of the O. toy O.
a photo of one 0.
itap of my O.
a doodle of 0.
photo of a cool 0.
close-up photo of the O. photo O.
photo of a small O. tattoo O.
B
Full results
B.1
Domain Generalization benchmarks
In the main paper, we report the average accuracy across each dataset. the supplementary. we provide comprehensive breakdown of results for each domain on PACS 28 in Table VL.CS 48
Table 9 OfficeHome [52) in Table TO Terralncognita [2] in Table TI and DomainNet 38 in
Table 12
We present the results reported in the original papers comparison methods. For some
16

---

methods, such as CoOp [68] and CoCoOp [67 where the original papers do not report results under the domain generalization setting, we reimplement them for unified comparison. As presented in tables, CLIPCEIL outperforms methods with ResNet pre-trained model by large margin, indicating that vision-language models pre-trained on huge web-scale image-text pairs provide promising way to boost OOD generalization. It also outperforms SOTA using CLIP models ie., MIRO ( and DPL [62]. In general, our method achieves the best performance on most domains, and our overall average performance each result CLIPCEIL, we report the average results and the standard deviation five runs with random seeds.
total of five benchmark datasets exceeds other SOTA DG methods. For
Table Detailed comparison our proposed method with the State-of-the-ee methods on the PACS dataset. denotes the models that utilize the ResNet-50 backbone, and the rest utilize CLIP ViT-B/16 backbone.
Model
Venue CVPR 23
Art
Cartoon
Photo
Sketch
Avg
*SAGM [54
86.6
* *DomainDrop
17
ICCV'23
98.0+0.2
89.8+0.4
84.2:10.4
86.0
89.5
CLIP Zero-Shot
97.3
99.1
99.9
88.3
96.2
Lin, Probing
97.6
98.9
99.9
89.7
96.5
ERM [50
96.5
95.3
96.2
86.5
93.7
MIRO
ECCV'22
95.6
CoOp
68
IJCV'22
98.3
98.8
99.7
87.3
96.0
CoCoOp 67
CVPR'22
97.6
98.6
99.7
87.0
95.7
DPL 62
2023
97.3
CLIPCEIL
Ours
98.3 0.1
99.6
100.0 0.0
92.3 0.2
97.6 0.1
Table 9; Detailed comparison of our proposed method with the State-of-the-ar methods on the VLCS dataset. * denotes the models that utilize the ResNet-50 backbone, and the rest utilize CLIP ViT-B/16 backbone.
Model
Venue
Caltech
LabelMe
Sun
Pascal
Avg
* "SAGM [54
CVPR223
80.0
Drom 17
ICCV'23
98.911.2
64.0+1..3
76.4
73.7+1.2
78.3
CLIP Zero-Shot
98.9
65.5
77.6
84.5
81.7
Lin.Probing
99.2
68.1
83.6
79.6
82.6
ERM [50
97.2
67.1
80.4
86.2
82.7
MIRO 7
ECCV'22
82.2
CoOp
68
IJCV'22
97.9
65.5
76.6
84.3
81.1
CoCoOp 67
CVPR'22
99.8
67.0
78.5
87.1
83.1
DPL 62
2023
84.3
CLIPCEIL
Ours
0.0
80.5:10.6
85.7+1..2
0..4
88.4L10.4
B.2 Visualization of visual features
To further demonstrate the effectiveness of CLIPCEIL, we visualize the image features of CLIP pre-trained model and our proposed CLIPCEIL. We show the t-SNE figures across different domains and classes on PACS, VLCS, and TerrIncognita in Figure Figure 8 and Figure respectively. Note that the OfficeHome results have been reported in the main paper. It is clear see that the image features extracted by CLIPCEIL exhibit more discrimination with respect to different classes than CLIP pre-trained model. Mcanwhile, CLIPCEIL's image features corresponding to different domains appear in most classes. This proves that CLIPCEIL's image features contain domain-invariin invariant and class-relevant information.
B.3 Ablation studies on channel refinement criteria
To demonstrate the effectiveness of our channel refinement strategy, we compare it with other methods that either consider the inter-domain inter-class variance criterion. The main paper illustrates the
17

---

Table 10: Detailed comparison of our proposed method with the State-of-the-an methods the OfficeHome dataset. * denotes the models that utilize the ResNet-50 backbone, and the rest utilize CLIP ViT-B/16 backbone.
Model
Venue
Art
Clipart
Product
Real
Avg
*SAGM
CVPR 23
70.1
DDomainDrop
17
ICCV'23
0..3
6..4
79.1+:0.3
80.2++0.2
71.8
CLIP Zero-Shot
82.7
68.0
88.3
90.7
82,4
Lin.Prohing
81.6
65.7
87.3
87.1
80.4
ERM [50]
80.2
65.1
85.7
83.1
78.5
MIRO
ECCV'22
82.5
CoOp 68
IJCV'22
82.8
69.7
91.0
90.6
83.5
CoCoOp 67
CVPR'22
83.9
70.0
91.4
91.9
84.3
DPL 62
2023
84.2
CLIPCEIL
Ours
0.2
7..
92.2
92.3
85.4
Table 11: Detailed comparison of our proposed method with the State-of-the--e methods on the Terralncognita dataset. denotes the models that utilize the ResNet-50 backbone, and the rest utilize CLIP ViT-B/16 backbone.
Model
Venue
L100
L38
L43
L46
Avg 48.8
*SAM 14
CVPR 23
*Domain Drop 17]
ICCV'23
CLIP ZeroSShtt
51.2
23.4
29.9
29.1
33.4
Lin.Probing
49.7
55.3
51.4
44.2
50.2
ERM (50)
60.3
53.5
51.2
44.0
52.3
MIRO
ECCV'22
54.3
CoOp 68
IJCV'22
41.4
53.7
48.9
44.6
47.0
CoCoOp 67
CVPR'22
50.7
56.0
51.9
44.0
50.4
DPL 62
2023
52.6
CLIPCEIL
Ours
63.7.L0.3
5..0.10.2
0.6
0..
53.0
average accuracy bars of different channel refinement strategies across each dataset. Here, we provide comprehensive breakdown of results for each domain on five DG datasets in Figure 10
C Additional experiments
C.1 Performance different backbones
ViT-B/16 the backbone. To further explore performance across different architectures, we conducted additional experiments with ResNet-50, ViT-B/32, and ViT-L/14 on the OfficeHome dataset. The process for extracting latent representations differs between ResNet and ViT- based backbones. For ResNet, we extract latent features from the feature map and apply Attention Pooling to transform the 2D feature map into ID vector. These vectors from different layers are then passed into the adapter's s Transformer layer, The results, summarized in Table [3] show that CLIPCHII consistently outperforms zero-shot predictions VIT backbones and other BesNeet-ased models, highlighting its strong generalization ability across different architectures.
In main experiments,
C.2 blation studies for Adapter
We conducted ablation studies to explore the effects the Transformer layer in the adapter g. this study, we replaced the Transformer layer with Average Pooling and one-layer MLP projector and used simple adapter y. i.e., one-layer MLP, that did not consider multi-scale information. As shown in the orange block in Table 14] the Transformer layer outperformed the other fusion strategies, indicating necessity. The pink block of Table 14 suggests that the inclusion of the reference loss Cret and the directional loss Lair alongside the simple adapter g leads still improves the performance.
18

---

Table 12: Detailed comparison of proposed method with the State-of-ther methods the DomainNet dataset. denotes the models that utilize the ResNet-50 backbone, and the rest utilize CLIP ViT-B/16 backbone.
Model
Venue
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Avg
SAGM [54]
CVPR'2 ICCV'23
45.0
*DomainDrop
Ti
62.9=0.3
21.6+0.11
50.7+00.2
14.8=0.3
62.74:0.1
53.5+00.6
44.4
CLIP Zero-Shot
71.3
47.4
66.4
14.2
83.4
63.1
57.5
Lin.Probing
71.1
46.9
66.7
15.4
83.1
57.6
ERM (50)
64.2
43.1
61.2
14.3
80.1
60.3
53.8
MIRO 7
ECCV'22
54.0
CoOp 68
1ICV222
75.1
49.5
69.6
15.8
81.7
66.8
59.8
CoCoOp
CVPR
74.8
51.9
69.2
16.0
80,9
60.0
DPL 662
2023
56.7
CLIPCEL
Ours
77.1=0.1
52.1=0.1
71.40011
17.0=0.2
85,410,1
69.120.1
62.0=0.1
(a) Zero-shot actoss classes
(b) CLIPCEIL across classes
(c) Zero-shot across domains
(d) CLIPCEIL across domains
Figure 7: t-SNE [49] visualization image features of our proposed CLIPCEIL and CLIP pre-trained
across different classes and domains on PACS dataset. Different colors indicate different classes or domains
C.3 Apply multi-scale mechanism on text encoder
To investigate the effectiveness of the multi-scale mechanism the text encoder. We conducted experiments to incorporate a multi-scale adapter into the text encoder. As shown Table 15] using both visual and text adapters did not perform well as only using the visual adapter. This may due to the increased complexity of optimizing both adapters simultaneously. It also suggests that focusing on image feature adaptation more crucial for domain generalization tasks since the semantic gap between visual features in pretrained and custom datasets is larger than that of text features.
19

---

Zero-shot
CLIPCEIL acioss Zero-shot CLIPCEIL
Figure t-SNE 49 visualization on image features of our proposed CLIPCEIL and CLIP pre-trained across different classes and domains VLCS dataset. Different colors indicate different classes or domains
(a) Zero-shot classes
(b) CLIPCET acmoss classes
(c) Zcro-shot across domains
(d) CLIPCEH across domains
Figure t-SNE 49 visualization on image features of our proposed CLIPCEIL and CLIP pre-trained
across different classes and domains on Terralncognita dataset. Different colors indicate different classes domains
HiasDorulh
Sert
320
KIAPR2 15
(a) PACS
(b) OfficeHome
Derran
koth
70-
65
255
(c) VLCS
(d) Terralncognita
Intar Dumah
0p
105
Palns
62kc
Real
Se4th
(e) DomainNet
Figure 10: Full accuracy bar results of different channel refinement strategies on the five DG datasets.
20

---

Table 13: Performance with different backbones on OfficeHome datasets.
Model
Art ResNet-50 Backbone
Clipart
Product
Real
Avg
SAGM
70.1
SWAD 6
66.1
57.7
78.4
80.2
70.6
Domain Drop
67.3
60.4
79.1
80.2
71.8
DISPEL
71.3
59.4
80.3
82.1
73.3
CLIP Zero-shot
74.6
49.5
79.4
83.5
71.8
CLIPCEIL
76.9
54.3
85.0
86.3
75.6
ViT-based Backbone
CLIP (VIT-L/14) Zero-shot
89.8
74.8
93.6
94.1
88.1
CLIPCEIL (ViT-L/14)
91.1
79.6
94.8
95.1
90.2
CLIP (VIT-B/32) Zero-shot
82.7
61.8
86.6
88.6
79.9
CLIPCEIL (ViT-B/32)
84.2
66.4
90.0
91.5
83.0
Table 14: Performance of linear layer adapter g on OfficeHome dataset with ViT-B/16 backbone.
Model
A
C
P
R
Avg
CLIP Zero-shot
82.7
68.0
88.3
90.7
82.4
One linear projector
84.0
69.8
90.2
90.8
83.7
One linear projector + +ref Lair
85.0
70.6
91.7
91.8
84.8
Average-pooling
84.2
68.6
90.8
91.3
83.7
Two-layer MLP
85.5
70.2
90.7
91.6
84.5
CLIPCEIL (we Transformer layer)
86.0
71.2
92.2
92.3
85.4
Table 15: Performance comparison with text encoder adapter with ViT-B/16 backbone.
Model
A
C
R
Avg
Visual + text multi-scale adapter
85.7
70.5
92.0
91.8
85.0
CLIPCEIL (Only visual multi-scale adapter)
86.0
71.2
92.2
92.3
85.4
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
Yes, in both abstract and introduction, we described our contributions and Scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes, we discussed the limitations of our work at the end of "Conclusion"
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Yes, provide detailed information about the parameter setting and configu- ration the section 4.1 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, the source code is available at attap:://github.coom/uuuut CLIPCEIL, while all the data we use is publiely available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
coblic//guides/Coeeetuiiiss
casssiinnooyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppppclggui.aeeeaeccuuea
Saaeee//CoSSSubisssS.S.. www.sssoiccgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we provide detailed information about experimental setting the sec- tion 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we provide the mean and standard deviation of the five runs with different random seeds for all the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide the detailed information about the computational resource we used in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics h http:///wwrrrr..cccooooccccccaaa
Answer: [Yes]
Justification: Yes, our research is with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As our model is designed for efficient fine-tuning the pre-trained vision language model, we do not anticipate any ethical or social impacts this point.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As our model designed for efficient fine-tuning the pre-trained vision language model for the image classification task, we do not anticipate any risk at this point.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we cited all the existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "iVtcumryur", "reviewer": "Reviewer_cn9a", "summary": "The paper introduces a novel method called CLIPCEIL, designed to improve the performance of CLIP on unseen test datasets with domain shifts. The approach refines visual feature channels to maintain domain-invariant and class-relevant features using a lightweight adapter. This involves minimizing inter-domain variance and maximizing inter-class variance. Additionally, CLIPCEIL ensures image-text alignment by matching text embeddings of class descriptions with corresponding image embeddings while eliminating domain-specific features. The model also integrates multi-scale CLIP features using a self-attention fusion module implemented through a Transformer layer. Extensive experiments on five benchmark datasets show that CLIPCEIL outperforms current state-of-the-art methods.", "questions": "1. Why do authors list the venue in the table? It's quite strange I have to say.", "limitations": "The author addressed limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured, good written, and easy to follow.\n2. Analysis is thorough and extensive and could bring some insight to readers.", "weaknesses": "1. This paper lacks novelty overall. CLIP adapters for few-shot transfer learning have been well-studied these years. This paper might get accepted if submitted 2 years ago. The novelty of this work compared with other adapters' work is a little bit incremental.\n2. Some famous adapters for CLIP are not cited, like Vt-CLIP (Qiu L, Zhang R, Guo Z, et al. Vt-clip: Enhancing vision-language models with visual-guided texts[J]. arXiv preprint arXiv:2112.02399, 2021.) or cited but not compared, like Tip-Adapter (Zhang R, Fang R, Zhang W, et al. Tip-adapter: Training-free clip-adapter for better vision-language modeling[J]. arXiv preprint arXiv:2111.03930, 2021. ) \n3. Performance gain compared with other methods is not significant enough to verify the effectiveness of the design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's careful comments and provide our responses below.\n\n### Weakness\n\n>**[W1]:** Overall novelty.\n\nThanks so much for your comments. First, we would like to emphasize the difference between **few-shot learning** and **domain generalization**. Although they are related, they have distinct differences. In domain generalization, the key assumption is that there are no target domain examples available during training. This means the model must learn to generalize to entirely unseen domains based solely on the information from the source domains. In contrast, few-shot learning typically focuses on task adaptation, where there are, although only a small number, target domain examples available (actually not only available but also labeled ) during training. This allows the model to fine-tune its knowledge and adapt to the new task with minimal data. This paper specifically targets domain generalization task.\n\nSecond, we want to mention that there are **two main directions** in fine-tuning pre-trained large vision-language models (e.g., CLIP): (text and/or visual) **prompt learning** and **adapter techniques**, and thus the adapter itself is not our contribution. Our model, along with CLIP-Adapter, Tip-Adapter, etc. all fall under the adapter technique. We want to emphasize that our contributions lie in **proposing novel loss functions tailored specifically for the DG task**, compared to other adapter based models that target few-shot learning or other tasks.\n\n>**[W2]:** Some famous adapters for CLIP are not cited, like Vt-CLIP, or cited but not compared, like Tip Adapter.\n> \nThanks so much for your suggestions. We will add Vt-CLIP to our related work section. Since Vt-CLIP and TiP-Adapter are both for the **few-shot learning** setting, comparing them in the domain generalization setting is difficult.\n\n>**[W3]:** Performance gain compared with other methods is not significant.\n>\nThanks so much for your comments. Our CLIPCEIL achieves the best average performance on five widely used domain generalization datasets with both the frozen encoder and fine-tuning the entire visual encoder settings. Considering the frozen encoder is a more realistic setting in practice, CLIPCEIL exceeds second-best by **$2.3$%** on average. \n\n### Question\n\n>**[Q1]:** Why do authors list the venue in the table?\n>\nThanks for your question. Our original purpose is to make it easier for readers to identify when the comparison methods were published (highlighting the latest progress in this area) and where they were published (demonstrating that they are state-of-the-art models by referencing the top conferences or journals).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vj7B4amAFW", "reviewer": "Reviewer_k7YB", "summary": "The paper addresses Domain Generalization (DG) by leveraging the superior generalization abilities of CLIP. While most prior works that utilize CLIP focus solely on the adaptation of CLIP for the given downstream task, this work investigates the domain-specific properties of CLIP. Specifically, the authors demonstrate that a meticulous channel selection on CLIP’s image embeddings to exclude the domain-specific channels can improve the zero-shot performance. Motivated by this observation, they propose CLIPCEIL - a simple approach that aims to enhance CLIP’s generalization properties by focusing on domain-invariant parameters through a transformer-based adapter. CLIPCEIL learns refined features through attention on multi-scale features from CLIP’s image encoder. These refined features are learned through channel refinement and image-text alignment on the downstream dataset. Experiments on datasets from the DomainBed benchmark demonstrate the effectiveness of the approach.", "questions": "1. Following the point about channel refinement strategies from the Weaknesses section, could there be an alternative strategy where inter-domain refinement is sufficient? Based on the results from Fig. 6, it appears that sufficiently diverse source domains (as in DN) enable this strategy. Thus, could this strategy be realized by using augmentations on the source domains?\n2. Can the multi-scale mechanism in the adapter module also be extended to the text encoder? Would that enable better image-text alignment since the proposed approach would then align the refined image-text features rather than aligning the refined image embeddings with CLIP’s vanilla text embeddings?", "limitations": "The authors have adequately addressed the limitations of the work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Presentation:** The paper has been presented well overall, making it easy to understand. The authors first motivate the core idea of the paper through a simple zero-shot experiment on the channels of CLIP’s image features, followed by a clear outline of the proposed method. The authors also present various visualizations demonstrating how the proposed approach improves the CLIP baseline.\n\n- **Motivation:** Rather than leveraging CLIP as is (often done by prior works), the authors investigate and improve its generalization properties through a simple and efficient channel refinement strategy.", "weaknesses": "### (a) Proposed Method\n\n- **Novelty:** The idea and motivation behind the proposed approach are the same as DomainDrop [14]. The histogram analysis of the channels in a pre-trained model in the current paper is the same as in [14]. Thus, [14] is an important paper that needs to be discussed in more detail. Additionally, some of the ideas in CLIPCEIL are similar to the following works, [R8-R11]. The authors should discuss these works in the “Related Works” section and how CLIPCEIL differs from the ideas in these works. \n\n- **Channel refinement strategies:** Fig. 6 shows that the various channel refinement strategies are quite similar in performance on most datasets except PACS and TI. This gives rise to the following question - are both inter-task and inter-domain refinement required for effective DG? With sufficiently diverse domains, could the inter-domain refinement be sufficient? Especially in the case of DN which has several diverse domains, the performance gap is negligible. How would the refinement strategies scale with more source domains or with domains having more diversity? The paper does not address this aspect. *(An example can be the difference between the PACS and DomainNet datasets. Even if we consider the same number of source domains in these datasets, DomainNet has infograph, painting, quickdraw, etc which are more diverse than the domains in PACS. How would this affect the training?)*\n\n### (b) Experiments\n- There seems to be a misunderstanding of how MIRO [5] works, according to the way Table 2 has been presented. MIRO trains the entire CLIP backbone and does not freeze it. However, Table 2 indicates that the backbone is frozen for MIRO. Thus, MIRO should be moved to the blue section in Table 2.\n\n- SAGM [15] and DomainDrop [14] have been shown on ResNet-50 while all the other results are presented on ViT-B/16. The authors should present these results on CLIP ViT-B/16 for a fair comparison.\n\n- The authors need to compare with additional prior works mentioned in the missing references (below).\n\n- There is no analysis on the weights for the loss terms in Eq. 7. Does equal weightage give the best performance across all datasets or is there a certain set of weights that work best? The authors should provide this analysis for a better understanding of the method.\n\n### (c) Missing references\n\n- [R1] Zanella, Maxim, et.al “Low-Rank Few-Shot Adaptation of Vision-Language Models”, CVPR 2024.\n- [R2] Cha, Junbum, et al. \"Swad: Domain generalization by seeking flat minima.\" NeurIPS 2021.\n- [R3] Bose, Shirsha, et.al, “STYLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalization”, WACV 2024.\n- [R4] Wang, Zhengbo, et.al, “A Hard-To-Beat Baseline for Training-free CLIP-based Adaptation”, ICLR 2024.\n- [R5] Kan, Baoshuo, et.al, “Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models”, ICCV 2023.\n- [R6] Khattak, Muhammad Uzair, et al. \"Maple: Multi-modal prompt learning.\" CVPR 2023.\n- [R7] Addepalli, Sravanti, et.al, “Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification”, CVPR 2024.\n- [R8] Singha, Mainak, et.al, “AD-CLIP: Adapting Domains in Prompt Space Using CLIP”, ICCV 2023.\n- [R9] Chang, Chia-Yuan, et.al, “DISPEL: Domain Generalization via Domain-Specific Liberating”, CVPR 2023.\n- [R10] Yu, Ding, et.al, “Domain Generalization by Learning and Removing Domain-specific Features”, NeurIPS 2022.\n- [R11] Hu, Xuefeng, et.al, “ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation”, WACV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's detailed comments and offer our responses below.\n\n### Weakness\n#### [Proposed Method]\n>**[W1]:** The idea and motivation are the same as DomainDrop [14]. Some of the ideas in CLIPCEIL are similar to the following works, [R8-R11].\n\nThanks for your insightful comments. The technique used in the DomainDrop [14] paper differs from CLIPCEIL. DomainDrop **explicitly drops the domain-specific feature channels** identified by a domain discriminator. However, each feature channel may include **both domain-specific and domain-invariant information**. Thus, directly dropping entire feature channels can lead to the **loss of class-relevant information** for downstream tasks and may not be optimal. In contrast, CLIPCEIL implicitly drops the domain-sensitive information by **minimizing the inter-domain variance and maintaining class-relevant information** as much as possible by maximizing the inter-class variance. Moreover, CLIPCEIL also proposed the Image-text alignment that **specifically designs to the vision-language model**, compared to DomainDrop only utilizes visual features.\n\nThe histograms in Table 1 are the **visualization tool** adapted from DomainDrop [14]. Our observations align with those in the DomainDrop paper; features extracted from pretrained models without specific adaptation for the domain generalization task exhibit large variances across domains. Moreover, we also plotted the channel histogram across classes, and noticed that certain channels are insensitive to class variations, motivating our loss function to maximize inter-class variance.\n\nWe will include [R8-R11] in our revision, but want to emphasize the **difference between CLIPCEIL and these models**. [R8] is a prompt learning-based method for domain adaptation by learning domain-agnostic tokens from multi-scale visual styles, whereas CLIPCEIL is an adapter-based method. [R9] learns a mask to explicitly filter out the domain-specific feature elements, which can be problematic as one element can contain both shared and specific information, similar to DomainDrop. [R10] uses multiple domain-specific classifier heads to remove domain-specific features. [R11] removes class-agnostic visual information by projecting all the visual embeddings onto the span of text embedding, focusing on source-free domain adaptation. In contrast, our method removes the domain-specific and class-agnostic information by minimizing the domain variance and maximizing the class variance, inspired by our channel histogram observation.\n\n>**[W2] & [Q1]:** Channel refinement strategies, esp. domain refinement.\n\nThank you for your insightful perspective. We would be happy to discuss the channel refinement strategies with the reviewer. We conducted a simple experiment to show how domain variance dynamically changes during training in **Figure 1 in the rebuttal PDF**. Diverse datasets (e.g. TI and DN) start with large variances, which are reduced to reasonable levels using our domain variance loss term. Figure 1 in the main text also demonstrates this. Of course, there are other options, such as adversarial learning, information theory based, etc, which are the main focus in the domain generalization area., we welcome the opportunity to discuss these further with the reviewer.\n\n\n#### [Experiments]\n>**[W1]:** Misunderstanding of how MIRO [5] works.\n> \nThanks for pointing it out. We will move it to the blue section.\n\n>**[W2]:** Fair comparison with SAGM and DomainDrop.\n\nThanks for the insightful suggestion. Our original purpose in listing SAGM and DomainDrop was to demonstrate that the CLIP ViT backbone can outperform the best ResNet backbone models and that CLIPCEIL is built upon this superior backbone. \n\nTo fairly compare with the ResNet backbone models, we conducted experiments on **CLIPCEIL using the CLIP ResNet-50 backbone**. **Table 2 in the rebuttal PDF** demonstrates that CLIPCEIL outperforms SAGM and DomainDrop with the ResNet-50 backbone on OfficeHome datasets.\n\n>**[w3]:** Comparison with additional prior works.\n\nThank you for your suggestions. References [R1, R4, R5] pertain to **few-shot learning**, while [R8, R11] address **domain adaptation** problems. We did not include these as comparing models focused on other tasks within the domain generalization setting, which is the primary goal of our paper, is challenging. \n\nWe have included references [R3,R6] (which are already included in main text), and [R7] with the ViT-B/16 backbone (**see Table 1 in the rebuttal PDF**), as well as references [R2,R9] with the ResNet-50 backbone (**see Table 2 in the rebuttal PDF**). Our CLIPCEIL outperforms the prior works with different backbones.\n\n>**[w4]:** Weights analysis for the loss terms.\n\nThank you for your suggestions. To avoid hyper-parameters ($\\alpha$ for $L_{ref}$, $\\beta$ for $L_{dir}$) searching for different datasets, we set $\\alpha=1$ and $\\beta=1$ as default. Following the Reviewer's suggestion, we investigated hyper-parameter sensitivity by tuning one parameter at a time while keeping the other at 1. **Figure 2 in the rebuttal PDF** demonstrates that $\\alpha=1$ and $\\beta=1$ yields the best accuracy and that CLIPCEIL achieves stable performance with $\\alpha\\in[0.6,1.2]$ and $\\beta\\in[0.6,1.2]$.\n\n### Questions\n\n>**[Q1]:** See [ Proposed method] [W2]\n\n>**[Q2]:** Can multi-scale adapter be extended to the text encoder?\n\nThank you for your insightful suggestion. We conducted experiments to incorporate a multi-scale adapter into text encoder. As shown in **Table 3 of the rebuttal PDF**, using both visual and text adapters did not perform as well as only using the visual adapter. This may be due to the increased complexity of optimizing both adapters simultaneously. It also suggests that focusing on image feature adaptation is more crucial for domain generalization tasks, since the semantic gap between visual features in pretrained and custom datasets is larger than that of text features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BYWUaOzMPy", "reviewer": "Reviewer_dJc5", "summary": "The paper addresses the challenge of domain generalization for CLIP. To tackle this, the authors introduce CLIPCEIL, a method that enhances CLIP's performance on unseen test datasets with domain shifts, which employs Channel rEfinement and Image-text aLignment techniques to refine visual feature channels, ensuring they are domain-invariant and class-relevant. It aligns text embeddings with corresponding image embeddings and removes domain-specific features. Additionally, CLIPCEIL integrates multi-scale CLIP features using a self-attention fusion module implemented through a Transformer layer.", "questions": "- How many data points were used in the experiment in Table 1?\n- CLIP pre-training essentially has no concept of domain. The proposed method, when fine-tuned on a specific dataset, such as Office Home, will essentially introduce dataset bias. In other words, the domain-invariant feature obtained by the Office Home fine-tuned model is only valid for the Office Home data and may not be valid for another dataset. Therefore, I am curious about how the model obtained by this method on Office Home performs on other datasets, such as ImageNet.\n- Why do Table 4 and Table 5 demonstrate different performances of models with only Multi-scale employed?\n- Can you experiment with $g$ as a relatively simple architecture, e.g., just a linear layer, and see how it differs from the Transformer layer? Also, how about $L\\_{ref}$​ and $L\\_{dir}$​ incorporating $g$ as such a relatively simple architecture—do they still boost performance?", "limitations": "See previous responses.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper is presented very clearly, is well-structured, and is generally easy to follow and understand.", "weaknesses": "- The adapter $g$ is a model-specific design, intended only for CLIP with ViT as the backbone. This prevents the proposed method from being used with CLIP which has ResNet as the backbone.\n- The lightweight adapter $g$ is implemented using a Transformer layer and an MLP projector. This lacks clarity on how the Transformer layer is necessary.\n- The problem setup (line 131) demonstrates that the goal is to train a model $f$ in the source domain and expect it to perform well in the target domain. This needs to be more specific, such as whether it involves training from scratch or fine-tuning a pre-trained model.\n- CLIPCEIL currently only applies to multi-source domain generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's detailed comments and offer our responses below.\n\n### Weakness\n>**[w1]:** The adapter is designed for ViT, and is hard to use for ResNet backbone.\n\nThank you for your insightful comments. Our proposed method can also be **extended to the ResNet backbone**. The primary difference lies in the way we extract latent representations, while other components, such as the architecture of the adapter $g$, remain unchanged. For ResNet backbone, we use the latent feature map as the latent feature and apply Attention Pooling to convert the 2D feature map into a 1D vector. The vectors of different layers are then fed into the Transformer layer of adapter $g$. We conducted experiments using the ResNet-50 backbone, and our method **outperformed other ResNet based models**, as shown in **Table 2 in the rebuttal PDF**.\n\n>**[w2]:** How the Transformer layer is necessary.\n\nThank you for your insightful comments. We conducted an ablation study to investigate the Transformer layer. We replaced the Transformer layer with Average Pooling or an MLP. As shown in the **orange block in Table 4 of the rebuttal PDF** , the Transformer layer outperformed the other fusion strategies, indicating its necessity.\n\n\n>**[w3]:** Training from scratch or fine-tuning a pre-trained model.\n>\nThank you for your clarifying question. Our proposed method builds upon a pre-trained CLIP model, which we then fine-tune on the source domains.\n\n>**[w4]:** CLIPCEIL currently only applies to multi-source domain generalization.\n\nThank you for your insightful comments. We also put this in the limitation section of the main text. However, CLIPCEIL model can be **adapted to single source domain generalization** by simply removing the domain variance term in the loss function. We conducted experiments on ImageNet (w/ 1000 classes) using the same few-shot single source domain generalization setting as CoOp. **Table 5 in the rebuttal PDF** indicates that CLIPCEIL outperforms \"CoOp\". \n\n### Question\n>**[Q1]:** How many data points were used in the experiment in Table 1?\n\nThank you for your clarifying question. Table 1 in the main text presents the results of a simple proof-of-concept experiment based on our observations. This experiment is **training-free**, with no data points used for training, and is tested on the OfficeHome test set. We first calculate the domain variance in text embedding channels across 80 prompt templates and the class variance across 65 classes. We then select the channels with larger class variance and smaller domain variance. Assuming effective alignment of visual-language features in CLIP, we use the same selected channels for visual embeddings. During inference, we use the inner product of the visual and text feature vectors, similar to the approach used in CLIP zero-shot.\n\n\n>**[Q2]:** How the model trained on OfficeHome performs on other datasets?\n\nThank you for your insightful perspective. The domain generalization task aims to enhance the model generalizability, and as such, the domains in the benchmarks designed for domain generalization are already very diverse. Moreover, domain generalization assumes that the source and target domains **share the same label space** (i.e., the categories are consistent across different domains), which is not the case with different benchmark datasets. Therefore, a model trained on one benchmark dataset (e.g., OfficeHome) typically is very difficult to test on another one (say PACS). \n\nNevertheless, we are intrigued by the reviewer's suggestion about evaluating the performance of a model trained on one benchmark dataset on another dataset. To investigate this, we identified a common category, \"person\", within the PACS and VLCS datasets. We then conducted an experiment by testing a model trained on the PACS dataset in the \"Labelme\" domain of the VLCS dataset. The results below shows the performance drop but **we would be happy to discuss this interesting result with the reviewer**.\n\n| **PACS (leave Sketch)**|  **VLCS (leave Labelme)** |\n| :-----:  | :----:  |\n| $80.9\\%$  |$92.7\\%$|\n\n>**[Q3]:** Why do Table 4 and Table 5 demonstrate different performances?\n\nThank you for your clarifying question. Since the content in Table 5 is irrelevant, we assume that the reviewer referred to Table 3 and 4. Table 3 in the main text presents the ablation studies for different loss terms, while Table 4 focuses on the ablation studies for various adapter architectures. This distinction may be somewhat confusing, and we will address this in the revised paper. The term \"Multi-scale\" in Table 3 indicates training CLIPCEIL using **only $\\mathcal{L}_{\\rm{CE}}$**. In contrast, \"Multi-scale\" without \"bypass\" in Table 4 signifies training CLIPCEIL **with all loss terms except for the bypass connection**. Thus, they have different performances.\n\n>**[Q4]:** Can you experiment with as a relatively simple architecture?\n\nThank you for your insightful suggestion. We conduct the experiment with a simple adapter $g$, **without considering the multi-scale information**. $\\textit{i.e.,}$ one-layer MLP projector, and evaluate the impact of our proposed loss. The **pink block in Table 4 of the rebuttal PDF** indicates that incorporating $\\mathcal{L}_{\\rm{ref}}$ and $\\mathcal{L}_{\\rm{dir}}$ with a simple adapter $g$ still improves the performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "inh8ljhYwe", "reviewer": "Reviewer_wrXY", "summary": "The paper tackles the issue of domain generalization for vision-language models like CLIP. The authors propose a new and simple method which is divided into multiple stages, to mitigate the performance gap for this problem setup. Their method achieves State of the art performance on the benchmarks evaluated.", "questions": "1. The objective of the Image-Text alignment appears similar to contrastive loss functions. According to Table 3, this component is the most crucial part of their method. Did the authors consider ablating this component with other contrastive loss objectives or alignment methods?\n2. Line 292 states that using \"multi-scale information alone can enhance performance compared to CLIP zero-shot,\" and Table 3 also mentions this observation. However, I am unclear on what the authors mean by \"multi-scale information.\" Could they provide further clarification?", "limitations": "After reading the paper, I realize that although the authors achieve state-of-the-art performance, it requires significant compute resources. Could the authors provide a comparison of their method against previous state-of-the-art methods, such as CoOP, in terms of computational efficiency?", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written, simple, and easy to follow, making it an enjoyable read.\n2. The method's motivation is well-founded, aiming to exclude domain-sensitive and class-irrelevant visual features, which makes the approach straightforward to understand.\n3. The authors conduct thorough ablations for each component of their proposed method and training pipeline, clarifying the importance of each element to the overall approach.", "weaknesses": "1. In many Domain Generalization tasks, the community has evaluated ImageNet and its variants. One of the popular baselines, CoOP (compared by the authors), does this as well. I am curious why the authors did not present these comparisons, especially since they use the 80 prompt templates initially designed for ImageNet.\n2. It appears that the improvements diminish once the model is fine-tuned, with the most significant gains observed when the backbones are kept frozen.\n3. It would have been helpful to see the model's performance on other ViT backbones, as the authors primarily focus on the ViT-B/16 model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's detailed comments and offer our responses below.\n\n### Weakness\n\n>**[W1]:** Present the comparisons on ImageNet\n\nThank you for your insightful suggestion. We conducted experiments on ImageNet (w/ 1000 classes) using the same setting as CoOp. Since the setting is the **single source domain**. We modify our loss and remove the domain variance term in the $\\mathcal{L}_{\\rm{ref}}$. Similar to CoOp, we train our model on **few-shot** ImageNet (16 samples per class) and test the model on different variants of ImageNet datasets. **Table 5 in the rebuttal PDF** indicates that our CLIPCEIL outperforms both the \"CLIP Zero-Shot\" and \"CoOp\". \n\n>**[W2]:** It appears that the improvements diminish once the model is fine-tuned\n\nWe appreciate the reviewer's insightful question and are eager to engage in further discussion. One possible explanation for the observed phenomenon is that when the backbone is frozen, we can only adjust the adapters and loss functions, meaning that all the improvements come from these adjustments. In contrast, fine-tuning the entire backbone allows us to leverage a large number of model parameters to enhance performance, which might limit the potential benefits of the adapters and loss functions.\n\n>**[W3]:** It would have been helpful to see the model's performance on other ViT backbones.\n\nThank you for your insightful suggestion. We conducted experiments **using different ViT backbones, i.e., ViT-B/32 and ViT-L/14**, on the OfficeHome dataset. The performance results are presented in **Table 6 of the rebuttal PDF**. It shows that CLIPCEIL consistently outperforms the zero-shot prediction on these two ViT backbones, demonstrating its generalization ability on different architectures.\n\n### Question\n\n>**[Q1]:** Did the authors consider ablating this component with other contrastive loss objectives or alignment methods?\n\nThank you for your insightful suggestion. We try another contrastive loss \\textit{i.e.,} SLIP, which combines CLIP loss and SimCLR. We conducted experiments by replacing $\\mathcal{L}_{\\rm{CE}}$ with $\\mathcal{L}_{\\rm{SLIP}}$. As shown in **Table 7 of the rebuttal PDF**, combining our proposed loss with $\\mathcal{L}_{\\rm{SLIP}}$ achieves the best performance on the OfficeHome dataset, indicating the effectiveness of our proposed loss.\n\n**[SLIP]** SLIP: Self-supervision meets Language-Image Pre-training, ECCV 2022.\n\n\n>**[Q2]:** What do the authors mean by \"multi-scale information?\n\nThank you for your clarifying question. \"Multi-scale information\" refers to the latent representations obtained from different transformer blocks at various levels within the CLIP visual encoder. While the term \"multi-scale information\" is typically associated with CNNs and might not be entirely accurate for ViTs, we use it to convey the concept that features are derived from multiple levels.\n\n### Limitations\n>**[L1]:** Could the authors provide a comparison of their method against previous state-of-the-art methods, such as CoOP, in terms of computational efficiency?\n\nThank you for your insightful question. We measured the GPU memory usage and average training time per step for various methods on the OfficeHome dataset using a single A100 GPU. The table below indicates that CoOp is the most memory-efficient and fastest model, while CLIPCEIL offers comparable computational efficiency to CoCoOp and ERM.\n\n| **Model**    | **GPU Memory** | **Time/step** |\n| --------     | --------       | --------      |\n| ERM          | $20.8$G          | $1.56$s         |\n| CoOp         | $2.6$G           | $0.53$s         |\n| CoCoOp       | $25.4$G          | $1.73$s         |\n| CLIPCEIL     | $26.5$G          | $1.87$s         |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Reply

We would like to express our sincere appreciation for all reviewers' invaluable feedback and comments. Below are the general replies to the common concerns and the summary of the additional conducted experiments.

First, we would like to clarify the difference between few-shot learning and domain generalization. In domain generalization, no target domain examples are available during training, requiring the model to generalize based on source domains alone. In contrast, few-shot learning involves a small number of labeled target domain examples for task adaptation. **This paper specifically targets domain generalization.**

Additionally, there are **two main directions** to fine-tuning pre-trained vision-language models like CLIP: **prompt learning and adapter techniques**. Our model falls under adapter techniques, similar to CLIP-Adapter and Tip-Adapter. Our key contribution is **proposing novel loss functions specifically for the domain generalization task**, unlike other adapter-based models that focus on few-shot learning or other tasks.


### Additional experiments (order of the response to reviewers)
In response to the reviewer's comments, we conducted thorough additional experiments, enhancing the paper from the following aspects:
* Fair comparison with SAGM and DomainDrop by using ResNet-50 backbone.
* Weights analysis for the loss terms.
* Apply multi-scale mechanism to text encoder.
* Ablation Study for the Transformer layer in Aadpter $g$.
* Evaluate the CLIPCEIL on the ImageNet datasets.
* Performance on other ViT backbones.
* Ablation Study on other alignment methods.
* Computational resources comparison.

Meta-review: The final ratings are 6, 6, 5, 4. 

That said, there seems some discussion on **what is considered unseen**, which seems to have been resolved based on the rebuttal that the clip original dataset is very skewed anyway. AC does think that a better word should be used besides "unseen" since technically, even if the dataset is very skewed, we can't say it is unseen per se. There were also some concerns on novelty, which the authors provided a channel refinement explanation, which seems satisfactory to the AC and reviewers. 

The largest concern is it remains somewhat unconvincing that this approach can enhance CLIP's domain generalization ability in the way suggested by the title. Table 2 does show CLIP performs much worse than CLIPCEIL on those domain-adaptation/gen datasets. 

AC does agree that the paper may not offer significant insights or inspiration for those interested in the generalization of foundation models. However, there are supports among the reviewers that this paper has some merits from a domain generalization perspective. As such, the AC urges the authors to change the title, which is a bit misleading.

Overall, there is more positive support than not, and the AC decides to accept the paper after careful consideration. However, the AC strongly requests the authors to re-word "unseen" and the title to reflect a more domain generalization perspective to avoid misleading the community.
Decision: Accept (poster)