Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization
Authors: Jixuan Leng
Yijiang Li
Haohan Wang
Venue: Accepted by TMLR ()

Abstract: Domain Generalization (DG), a crucial research area, seeks to train models across multiple domains and test them on unseen ones. In this paper, we introduce a novel approach, namely, Selective Cross-Modality Distillation for Domain Generalization (SCMD). SCMD leverages the capabilities of large vision-language models, specifically CLIP, to train a more efficient model, ensuring it acquires robust generalization capabilities across unseen domains.
Our primary contribution is a unique selection framework strategically designed to identify hard-to-learn samples for distillation. 
In parallel, we introduce a novel cross-modality module that seamlessly combines the projected features of the student model with the text embeddings from CLIP, ensuring the alignment of similarity distributions.
We assess SCMD's performance on various benchmarks, where it empowers a ResNet50 to deliver state-of-the-art performance, surpassing existing domain generalization methods. Furthermore, we provide a theoretical analysis of our selection strategy, offering deeper insight into its effectiveness and potential in the field of DG.

Published in Transactions on Machine Learning Research (04/2024)
Choosing Wisely and Learning Deeply: Selective Cross- Modality Distillation via CLIP for Domain Generalization
Jixuan Leng University of Rochester
Meng3@@u.rochester. edu
Yijiang Li Johns Hopkins University
y155566kjhu.edu edu
Haohan Wang
haohanw@rllinoiss.ooo edu
University of Alinois Urbona- Chanpppgn
Reviewed on OpenReview: https: openreviee.........emmee net/ forum? id= 4KLuepp6mA
Abstract
Domain Generalization (DG), a crucial research area, seeks to train models across multiple domains and test them on unseen ones. In this paper, we introduce a novel approach, namely, Selective Cross-Modality Distillation for Domain Generalization (SCMD). SCMD leverages the capabilitics of large vision-language models, specifically CLIP, train more efficient model, ensuring it acquires robust generalization capabilities across unseen domains. Our primary contribution is unique selection framework strategically designed to identify hard- to-learn samples for distillation. In parallel, we introduce a novel cross-modality module that scamlessly combincs the projected features of the student model with the text embeddings from CLIP, ensuring the alignment of similarity distributions. We assess SCMD's perfor- mance on various benchmarks, where it empowers a ResNet50 to deliver state-of-the performance, surpassing cxisting domain generalization methods. Furthermore, we provide theoretical analysis of our selection strategy, offering deeper insight into its effectiveness and potential in the field of DG.
1 Introduction
Imagine a young pianist learning to perform a complex piece of music. Initially, she listens to her experienced piano teacher's rendition, absorbing the nuances and techniques. Yet, the most crucial learning comes from recording and listening to her own performances. When she listens her own recordings, she can identify where her performance differs from the perfect melody she has in mind the ideal, real-world performance. These differences are more valuable for her improvement than simply comparing her performance to the ideal. This is because these disparities directly reflect her unique challenges, and addressing them (with the teacher's help) will have the most direct impact on improving her performance.
This learning scenario akin the process of Knowledge Distillation (Hinton et al. 2015). The simpler problems that the pianist initially tackles resemble the casily learnable features of a datasct, which a student model can grasp without assistance. The complex music pieces, however, symbolize the hard-to-grasy grasp parts of the data, where the student model benefits immensely from the teacher model's guidance.
As the pianist matures artistically, she discorns that mastering known pieces alone does not capture the full essence of her musical journey. Wanting to elevate her compositions, she begins to intertwine lyrics into her melodies. This is not a mere juxtaposition heighten the clarity and expressiveness of her music. In the same vein, the harmonization of visual content and natural language in CLIP Radford et al. (2021) is not just for the sake of fusion, but because language significantly augment the nuances of visual recognition.
words and tunes; it is a realization that lyrics can

---

Published in Transactions on Machine Learning Research (04/2024)
Deep learning models, much like the pianists, may falter when faced with unfamiliar terrains or "genres" in the form of out-of-domain data, despite their proficiency in various lasks. This situation frequently arises in real-world applications. Numerous algorithms have been developed to ensure consistency across distribu- tions [Ben-David et al., 2010; Ben-Davvdd ct al.) 2006) and regularize the models to learn domain-invariant Teatures (Luu el al. 2022 Huang et al. 2020 2022); ; but they often yield only modest. improvements /Gulrajani & Lopez-Paz 2021 over traditional Empirical Risk Minimization (ERM) technique (Vapnik) 1998).
Inspired by the pianist's pursuit of harmony between melody and lyrics, and her introspective approach to identify discrepancies in her performance to perfect her craft, work similarly seeks to focus challeng- ing aspects of training data and incorporate semantic information to better capture the domain-invariant features. In this paper, we present the Selective Cross-Modality Distillation (SCMD) framework.
Rather than relying on the soft target distribution from the teacher model, SCMD emphasizes the discrepan- cics, spocifically, the gap between the student's performance and real-world expectations. Just as the pianist hones in on the variations between her rendition and the ideal composition, SCMD selects hard-to-leane samples in the training data, targeting those for knowledge distillation. This approach, we believe, not only enhances the learning process but also equips the student model with robust feature representations, crucial for navigating unfamiliar terrains.
We have chosen to utilize CLIP as a key component of our approach, not only for its ability to combine visual and linguistic information but also for its proficiency in matching images with textual descriptions. This special capacity of CLIP enhances our framework, offering a more comprehensive knowledge base from which student models extract and learn.
Contributions:
We present the Selective Cross-Modality Distillation (SCMD) framework, an innovative departure from traditional methods, emphasizing adaptive sample treatments over the uniform approaches commonly found in current knowledge distillation techniques.
We emphasize the importance of complex and hard-to-learn training samples and provide a compre- hensive theoretical foundation for our selection strategics.
We propose a cross-modality distillation module within our SCMD framework to leverage the unique capabilities of CLIP, seamlessly integrating linguistic comprehension with visual perception for a more nuanced learning paradigm.
We substantiate the effectiveness of our SCMD method through empirical evaluations, demonstrating its superior performance and capability to establish new standards on various benchmarks.
2 Related Work
2.1
Domain Generalization
Domain Generalization (DG) Muandet et al. 2013) has recently emerged as key area in machine learning, focused on training models with data from multiple source distributions for application on a distinct target distribution.
DG is generally classified into two facots: data augmentation and invariance regularization (Liu et al. 2023
Data augmentation thread (Shankar et al.] 2018 Yue et al.) 2019 Gong et al. 2019 Zhou et al. 2020 Huang
et al.
2021 Wang et al.) 2022b) refines the inputs to facilitate learning of generalizable representations.
Invariance regularization techniques Li et al.
2018b
Wang et al. 2017
Akuzawa et al.
2020 Ding et all
2022 Han et al. 2021 Wang et al.] 2022a Meng ct al.
2022 Lee et al. 2022 Ge ct al. 2021 Lu et al.
2022 aim for domainn-nvvriint feature representations to en enhance generalization. In addition, some mods
employ learning strategies like meta-learning (Li et al.
2018a) to simulate domain shifts during training,
improving the model's adaptability unseen domains. Recent studics indicate that weight averaging can further boost DG task performance (Cha et al. 2021 Arpit et al. 2022). contributing to more stable out-
of-domain test. results.
2

---

Published in Transactions on Machine Learning Research (04/2024)
Several previous studies have leveraged vision-language models to enhance Domain Generalization (DG) performance, closely aligning with our contribution. For instance, Domain Prompt Learning (DPL.) (Zhang et al. 2021) employs a lightweight prompt adapter to automatically generate a prompt estimating domain- specific features from unlabeled examples across distributions. Howover, these generated prompts often lack
clear semant ic meanings, potentially limiting their elfectiveness in certain conlexts (Zhou al., 2022). Other research (Li et al. 2022) dispatches appropriate pretrained models, including CL.P,, to each sample basod on their veneralization ability. Another approach (Cha et al., 2022 reformulates the DG objective using mutual information with oracle models, inchuding CLIP.
Recent work (Huang et al. 2023) introduces RISE, method that utilizes both absolute and relative distances for distilling CLIP for domain generalization. Building on this foundation, our approach prioritizes the identification and selection of hard-to-learn samples for knowledge distillation. This combination of ideas, though seemingly straightforward, represents new direction that has not been extensively explored in prior rescarch.
2.2 Contrastive Language-Image Pre- Training
CLIP (Radford et al. 2021), a vision-language model utilizing contrastive loss to align visual and text encoders within a shared feature space, has recently garnered significant attention.
Pretrained on 400 million image-text pairs (Radford et al. 2021
CLIIP effectively aligns semantic meanings between images and sentences, demonstrating its promise for generic visual representation learning and zero-
shot transfer via prompt (Radford et al. 2021 Jia et al. 2021 Yang et al. 2022b Yao et al. 2022 You
ett al.]
2022 Notably, CLIP matches a full-supervised ResNet1 101 model's 's performance with a 76.2% top-1
accuracy rate on the ImageNet (Deng et al. 2009 validation set and even outperforms it on the ImageNet Sketch Dataset with 60.2% accuracy rate.
These results highlight CLIP's exceptional capabilities in tasks like image classification. More importantly, CLIP's extensive pretraining not only bolsters its standalone performance but also demonstrates its poten- tial to transfer extensive knowledge to other architectures, offering new possibilities for enhancing model generalization across diverse domains.
In this paper, we present a novel method to distill knowledge from CLIP, a multi-modal vision-language model, into a single-modal student model. By transitioning from multi-modal to single-modal dist insiilaaiion, we aim to enhance the student model's domain generalization, opening up new avenues for leveraging these potent models.
2.3 Knowledge Distillation
Knowledge distillation, introduced by Hinton et al. (Hinton et al., 2015), is a pivotal technique for balancing model performance and computational complexity by training smaller student network with the soft output of a larger teacher model. This approach has spurred extensive rescarch in model compression (Cheng al. 2017 and knowledge transfer (Tan et al. 2018).
Numerous distillation techniques have emerged, including beature-based knowledge transfer methods (Ronero et al. 2015a Bengio et al.) 2013 Zagoruyko & Komodakis 2017 Kim et al., 2018a Heo et all, 2019a) that align embeddings from certain layers, (Zhang et al.][2018 that train teacher and student models concurrently. The design of loss functions operating on the outputs both models has also been a significant rescarch
area, with notable methods including I, (Kim et al.] (018b)), l2 (Chen et al. 2020
Passban et al. 2021
Wang et al. 2020). Maximum Mean Discrepancy (MMD) (Huang & WWan 2017]
KL divergence CCen
ct al. 2018 Passalis et al. 2020a b), and cross-entropy losses (Xu et al. 2020 Liu ct al. 2019
However, most studies focus on honnlogous-architeeuuu relatively untapped. Recently, Liu et al. (Liu et al. 2022) made significant progress in this area by mapping a CNN's fcature space into a transformer's attention and fcature space using a partially cross-attention pro- jector and group-wise linear projector. With cross-view robust training scheme, they achieved remarkable performance on ImageNet (Deng et al. 2009).
distillation, leaving cross-architeclun distillation
3

---

Published in Transactions on Machine Learning Research (04/2024)
Our approach diverges from conventional methods, introducing fresh perspectives knowledge distillation through two key innovations. Firstly, implement novel selection mechanism that identifies hard-to- learn samples, which enhances the student model's depth of understanding. Secondly, we leverage CLIP's multi-modal capabilities by employing a cross-modality module. This strategy not only facilitates a profound transfer of both visual and linguistic knowledge but also significantly enhances the domain generalization capability of the student model.
3 Methods
In this section, we provide a detailed description of our Selective Cross-Modality Distillation Framework, which leverages a pretrained CLIP model (Radford et al. 2021 with fixed weights as the guiding teacher model.
CLIP Img Encoder
Student Backbone
Logits
Classifier
Logits
Feature Projector
Labels
img Feature
Proj
Fox
Teacher Logits
CM Logits
Some
Text Feature
CLIP Text Fenture
This photo of
Encoder
Figure 1: SCMD that features selection mechanism to focus on hard-to-leary samples and a cross modaltyy module that projects the student's feature into CLIP multi-modal space for alignment.
3.1 Vanilla Knowledge Distillation
The vanilla knowledge distillation process is
0KD arg min
Σ
L(o(x;), f(x;;
(EX..)
(1)
+x+H(f(x;0)),)
where is the teacher model, f(; 0) is the student model. Y) is the standard dataset that is used for training the student model. L is generic distance function (can be the KL divergence between soft target distributions), H. represents a generic loss function (usually cross-entropy loss), and R is an arbitrary designed regularization. and correspond to certain layers or attention maps.
3.2 Selection Mechanism
S x=x; X where I i:: H(f(xx) 2T
(2)
In the preceding equation, X denotes the batch of samples, x, an individual sample, and y, its true label. The set consists of selected samples. The function H(f(x; y.) computes the cross-cntropy loss for the i-th sample, while contains indices of samples in the batch with cross-entropy loss exceeding the threshold

---

Published in Transactions on Machine Learning Research (04/2024)
Cross-entropy loss quantifies the divergence between the predicted probability distribution and the actual labels. High cross-entropy indicates challenging samples model uncertainly. In our optimization, we use this loss to identify hard-to-learn samples. During each forward pass, samples with higher losses are selected for adjustment, a methodology also adopted prior works (Huang et al. 2022 Byrd & Lipton 2019 Chang
et al. 2017 Katharopoulos Fleuret, 2018).
The uniqueness of our method lies not the recognition of hard-to-leam samples but in its integration within knowledge distillation. By doing so, harness the teacher model's rich knowledge more efficiently, opti- mizing the learning of the student model. We explore the theoretical foundation of our selection mechanism in Section !
The "hard-to-learn" lag for the samples can change each iteration. However, to ensure holistic learning across the entire training dataset, we switch to full-batch training for the final k% of the training epochs.
3.3
Module
Various feature-based knowledge distillations have been explored (Romero et al. 2015a Bengio et al. 2013 Zagoruyko & Komodakis] 20177 Kim et al.] 20188a Heo et al. 2019a): however, direct aligmment of clas- sification features often presents challenges. To address this, exploit. the robust cross-modal alignment
capabilities of the CLIP (Radford et al. 2021 and employ cross-modality distillation strategy.
In the context of our method, the student features are transformed into the CLIP's (Radford et al. 2021 multi-modal space using linear projection. This is analogous to how CLIP projects its image features into a multi-modal space to achieve alignment with text emboddings. This transformation bridges the semantic gap between the student model and the teacher model, Iacilitating a more ellective knowledge-trand process. After projection, we calculate the scaled pairwise cosine similarity with the text embeddings derived from the CLIP (Radford et al., 2021 model.
Our cross-modality loss is expressed as follows:
LCM = DRI ((x")))
where p' is the soft target distribution of CLIP and (p") = oly W(e(x;;b)) prext; =
(3)
In this equation, y is a scale factor, which adjusts the magnitude the projected student feature. e represents the backbone of the student model. linear projection W applied to the student feature e( 0.), and Òrext represents the text embedding of CLIP. a the softmax function parameterized by the distillation temperature T.
In order to generate unbiased text features using CLIP, use a generic template: "this is a photo of a {class}". This method helps us avoid incorporating any human prior knowledge about the dataset, ensuring that the feature generation process remains objective and is not influenced by any preconceived human understanding of the data.
During the inference phase, we omit the fcature projector and rely solely on the student model's backbone and its associated classifier for generating predictions, introducing no additional computation overhead.
3.4 SCMD
Figure illustrates the overall framework of our proposed method.
5

---

Published in Transactions on Machine Learning Research (04/2024)
The final training objective can be summarized as follows:
fcCMD = arg min
E {=H(f(x(()))x))
((X.))
+AzLlogits A3LCM
(4)
where Llogits = DDu((''lp") and x, and y;
are the selected samples and corresponding labels
4 Theoretical Evidence for Selection Strategy
To consistent with the notation, we let (X, Y) denote the standard data set and (x) be one of the samples. We let denote distribution and P denote the distribution distributions. We let f(; denote the student model, & denote the teacher model, and r(-) denole the risk. For convenience of notation, we allow r(.) to be parameterized by distribution by dataset.
When r(-) is parameterized by datasct, we have the empirical risk as
r(X, n
Σ
O))(
(xx.)))(X.Y)
where L is generic loss function.
When r(-) is parameterized by distribution, we have the expected risk as
r( r(P) = =[xx,r)--((UUxx))...
For simplicity of discussion, we only use r(P,c) to denote robustness performance when do not need to specify how the test distribution deviates from the training distribution.
Assumption 1. for any data pair (x) studied in the contert of this paper, there is gold standard labeling function (albeil unknown) that y: f(x:).
We believe this assumption is fundamental for numerous works studying the robustness behaviors of models with respect to feature perturbations, especially in the context of OOD robustness, where the test dataset is manually collected rather than generated by noise addition. Intuitivelyy this assumption stipulates that a musical piece recognized in the training phase must also be identified as the same piece in the testing phase, despite substantial shifts the performance style or instrument used. In other words, these variations representation, akin to distribution shifts in data, should not alter the fundamental recognition of the piece, preserving the semantics of the data.
Lemma 4.1. Given Assumptions A1 such that there a gold standard labeling function for source and target domains. For two arbitrary distributions P and P,
r(P)) (PP) t
where tu denotes the lotal varialion.
Proof. We leave the proof in Appendix
Lemma 4.2. Given the assumplion that samples are independent and identically distrübuted, hypothesis space O and any d > O, with probability at least 6, 6, we have
to r(P') (()) (( 0,
where we let Y(X,Y)e denote the number of sample sizes the finite dataset (X, Y)P, ; E a vanilla term that connects the wumber of samples and hypothesis space with generalization error bound.

---

Published in Transactions on Machine Learning Research (04/2024)
Proof. We leave the proof in Appendix A
The above results demonstrate that empirical robustness is determined by three factors: the divergence between training and test distributions, the measurable empirical error on the training distribution, and a technical term influenced by sample size and hypothesis space. Therefore, the critical term that will bound the robustness performance how the training distributtion deviates from the testing distribution. This intuitively gives us the idea that training with the distributions that are the most similar the test distribution will benefit the model most.
The above results apply to arbitrary distributions P ~ P. However, this does not necessarily encode the characteristics of the cases we are studying; some samples are hard for the model to learn.
To address this, we consider datasets generated by multiple distributions, some of which present more challenging learning scenarios. We represent these as a set P, consisting of m distributions, i.c., P (Pi, P2..... ............. Each data point considered be sampled from these distributions. For the convenience of discussion, we use lu(P', P) to denote the average divergence between the distributions within the set. (P(PPP):==> tu(P', PP)mm, YP. P.
Finally, we use s() to denote the distribution selection mechanism, and compare two selection mechanisms: selecting the hard-to-learn samples (denoted as 81) and selecting random samples (denoted as 82).
Lemma 4.3. p is continuous and has a finite expected value; for the two selection mechanism that are formally defined as
(t) sup P), P)) =0
PEP
for a fized testing dataset Pl, with the assumption that tw(P, P') = tv( + to V'), P we have
Ep to((0(11(P)) P')] s Ep (P(s P')]
Proof. We leave the proof in Appendix
Our result compares the upper-bounded differences between the two training distribution selection strategies, and our results suggest that selecting the hard-to-learn samples will lead to tighter generalization error bound.
Another important factor to note is that, given assumption AD and Lemma 4.1 and 4.2 the selection strategy applicable to our theoretical discussion (i.e. (P(s supper to (pP, P)) is only when selecting the hard- to-learn samples according to the label of the samples (thus cross-entropy loss). Other selection strategies such as selecting based on KL-divergence or distillation loss (experimented in Section 6.2) despite might following similar goal, does not strictly match our theoretical discussion with will likely lead to an error bound in between $1 and s2. Therefore, with the support of the theoretical discussion, argue that the most effective hard-to-loarn selection mechanism is to be based on cross-cntropy loss.
Another possible question is lhat the assumption Ju(P,P") = lv(P, + ("), P might appear strong. In fact, the proof will hold with any assumptions that describe the concept that the more different one distribution from the average of the training set, the more it will benefit the testing distribution. In the typical domain generalization setting, where there are no guaranteed connections between training and testing distributions, we believe this one of the practical assumptions we can consider, also widely used in
practical by other domain generalization literature (Huang et al. 2022 Byrd & Lipton 2019 Chang et al.
2017 Katharopoulos & Fleuret 2018
5 Experiment
In this section, we demonstrate the effectiveness of our proposed method using the DomainBed Guulaajan & Lopez-Paz 2021 benchmark and compare to the current state-of-the-art DG techniques.
7

---

Published in Transactions on Machine Learning Research (04/2024)
5.1 Experimental Setup
We adhere the protocol out in (Gulrajani & Lopez- Pazzz--azz 2021) for our experimental setup and assess the
performance of SCMD using VLCS (Fang al. 2013). PACS (Li et al.) 2017). OfficeHome (Venkateswara
et al. 2017), Terralncoguita (Beery et al. 2018 and DomainNet (Peng et al. 2019). It is noleworthy that
CLIP does not perform well on the Terralncognita [Beery et al. 2018 dataset, and we will explore these results in the discussion section.
Due to the intensive computing requirements of DomainBed's (Gulrajani & Lopez- search protocol, we take a more simplified approach. We restrict our research five distinct hyperparameter combinations, each tested three times. We assign 80% of the data for training and 20% for validation, choose the model based the training-domain validation performance, and report the results the held-out domain.
2021
hyperparameter
To ensure fair comparison with other methods, we employ ResNet50 (He et al. 2016) pretrained on ImageNetlk (Deng et al.) 2009) as the student model and CLIP ViT-B/32 (Radford et al. 2021 as the teacher model, aligning our approach with existing research in the field.
Consistent with findings from previous studies (Cha et al. 2021 Arpit et al,, 2022, we also incorporate weight averaging into experiments access SCMD performance. This technique has been shown to mitigate the discrepancy between waining-domaile validation performance and out-of-domain test performance.
Eight RTX 3090 GPUs are utilized for all experiments.
5.2
Experimental results
Algorithm
Ens MA
VLCS
PACS
OffHome
TerraInc
DNet
Avg
Teacher (CLIP with VIT-B/32)
No
78.4 1 0.0
94.7 + 0.1
79.6 1 0.1
19.0 1 0.1
54.0 1: 0.0
65.1
ERM [Vapnik 1998
No
77.5 +
85.5 = 0.2
66.5 + 0.3
46.1
40.9 0.1
CORAL Sun & Saenko 2016
No
78.8 + 0.6
86.2 = 0.3
68.7 0.3
47.6 1.0
41.5 0.1
64.6
VREx Kruoger et al 2021
No
78.3
1 0.2
84.9 = 0.6
66.4 0.6
46.4 + 0.6
33.6
2.9
61.9
RSC Huang et al. 2020)
No
77.1
+ 0.5
85.2 = 0.9
65.5 0.9
46.6 + 1.0
38.9 + 0.5
62.7
ERM + SWAD Chha et al.
2021
Yes
79.1
+ 0.1
88.1 0.1
70.6 - 0.2
50.0 + 0.3
46.5 - 0.1
66.9
CORAL + SWAD Chha et al 2021
Yes
78.9
+ 0.1
88.3 = 0.1
71.3 + 0.1
51.0 + 0.1
46.8 - 0.0
67.3
AdaClust (Thomas et all 212
No
78.9
0.6
87.0 1 0.3
67.7 1 0.5
48.1 0.1
43.3 0.5
64.9
MIRO | SWAD Cha et al. 2022
2022
Yes
0.2 79.6
88.4 - 0.1
72.4 1 0.1
52.9 1 0.2
47.0 1 0.0
68.1
EoA Arpit et al.
Yes
79.1
88.6
72.5
52.3
47.4
68.0
Model rata(Greed
Ramm al. 2022
Yes
78.7 + 0.2
90.5 + 0.2
73.4 + 0.3
49.2 0.9
47.7 + 0.0
67.9
Model rata(Uni)
Rame et
Yes
78.3
89.8
73.5
52.0
47.7
68.3
SCMD (ours)
Yes
80.9 1 0.2
90.1 0.0
74.8
0.1
51.3
0.2
48.4
0.0
69.1
Table 1: Performance benchmarking on 5 datasets of the DomainBed benchmark. The gray background shows our proposed method. Experiments report the performance based training-domain validation accuracy follow (Gulrajani & Lopez-Pac 2021). 'Ens/MA' stands for ensemble ensemble/moving moving average. (best in bold and second underlined)
Table shows that our proposed method achieves the best performance on the DomainBed benchmark. outperforms existing methods on all datasets, with Model Ratatouille (Ramé et al. 2022) coming in second.
Model Ratatouill (Ramé et al. 2022) utilizes technique that adjusts the model multiple extra tasks to obtain different weight initializations. These weights are then adjusted for the desired tasks, and the final model is created by taking the average of these weights. This is exemplified by Model Ratatouille (Uniform), which averages a total of 60 models to achieve the final result, as shown in the Table.
In contrast, our proposed method uses a single teacher model and evaluates performance using just one student model. Furthermore, our method is orthogonal existing DG methods, potentially providing additional avenues and perspectives in the broader landscape of DG research.
8

---

Published in Transactions on Machine Learning Research (04/2024)
6 Ablation Studies
We conduct thorough analysis of SCMD by breaking it down into its components and examining cach using the PACS dataset. To evaluate the effecliveness of the proposed cross-modality module and the selec- tion mechanism, we follow the same standardized hyperparameter search protocol as the main experience, ensuring consistency and comparability.
6.1 Impact of the Cross-Modality Module
Table (Top Section) presents the comprehensive results of our method alongside its different variations.
"Vanilla KD* (Hinton et al. 2015) denotes the conventional knowledge distillation technique
where the KL divergence between the predicted distributions of the student and teacher models is minimized.
"SCMD (logits)* is the combination of the selection mechanism and the minimization of KT. divergence.
"SCMD (logits + CM)" represents the full version of our method, including all our proposed
components.
We have included weight averaging into the Vanilla KD to guarantee a fair comparison and show the effec- tiveness of our proposed components.
As shown in the top part of Table 2 our selection mechanism alone leads 0.5% improvement in com- parison to the average performance of Vanilla KD. Additionally, cross-modality (CM) module further boosts the performance by 0.6%. When both are combined, our proposed methodology offers significant increase in performance, surpassing Vanilla KD by total of 1.1%. These results demonstrate the combined power and effectiveness of our proposed approach.
Algorithm
Avg
Vanilla (Hinton et 2015
89.0 0.3
SCMD (logits)
89.5 0.2
SCMD (logits CM) (full mothod)
90.1 0.0
SCMD (no selection)
89.4 0.3
SCMD (selection based on KL)
89.6 0.3
SCMD (selection based on distill loss)
89.8 0.2
SCMD (selection based on focal loss)
89.2 0.4
SCMD (selection based on CE loss)
90.1 0.0
Table 2: Performance evaluations: (Top) Impact of the cross-modality module in SCMD the PACS dataset. (Bottom) SCMD performance with different strategies for selecting hard-to-learn samples the PACS dataset. (best in bold)
6.2 Empirical Validation of Our Theoretical Analysis
As illustrated in Table 2 (Bottom Section), we employed various selection strategies for the samples.
"no selection" represents the baseline scenario where the entire training dataset is used without any hard-to-learn sample selection.
"selection based KL refers to sample selection based the KL-divergence between the predicted distributions of the student and teacher models.
"selection based on distill loss implies that samples are chosen according to the distill loss, as defined in Eq 4
"selection based on focal loss" implies that samples are chosen according to the focal loss et al. 2017
Lin
"selection based CE loss" denotes our proposed selection strategy.

---

Published in Transactions on Machine Learning Research (04/2024)
It is evident that any selection strategy except "selection based on focal loss" yields better results than the "no selection" baseline. Our proposed "selection based on CF loss" approach is the most successful on the PACS dataset, outperforming "selection based on KL" by 0.5%, "selection based on distill loss" by 0.3%, and the no selection strategy by 0.7%. It is worth noting that the "distill loss" (Eq 4 includes the cross-entropy loss, which could explain why performance similar to "selection based on CE loss", albeit slightly lower.
These results provide empirical support to our theoretical proposition: "Other selection strategies such selecting based on KL-divergence distillation loss despite might following a similar goal, do not strictly match our theoretical discussion which will likely lead to error bound between $1 and s2." Therefore, with the support of the thcoretical discussion and empirical evidence, we argue that the most effective hard-to-learn selection mechanism is to be based on cross-entropy loss.
7 Discussion and Limitations
7.1 Impact of Prompts Variations
Prompt
Avg
In order to reduce bias in the feature extraction process with CLIP, we template that does not contain any human- derived insights, which is: "this is photo of 0". This template anchors the feature generation process in way that is not dependent on any particular domain, thus avoiding the impact of any human preconceptions.
this is art of a ()
88.9 0.4
this is sketch 0
89.6 0.2
this is cartaon
0
88.7 0.4
photo of
89.9 0.2
this is photo of O
89.9 0.4
O
88.8 0.2
Our experiments show that the prompt "photo" was the most effective for optimizing performance. We also found that slight changes to the prompt, such as "a photo of O and "this is a photo of a ()", ; had little effect on the success of the distillation process. This demonstrates the resilience of feature distillation to minor changes in the prompt structure. Table 3 provides further details on the ablation studies.
Table 3: The performance of SCMD on PACS with different prompts, using the same hyper- parameters for three trials.
7.2 Analysis on Terrainc Performance
Table 4: Impact of KD on Terralncognita.
CLIP itself exhibits sub-optinal zero-shot performance, which becomes evident when we distill it into ResNet50 using the TerraIncognita dataset, as detailed in Table [ Despite this challenge, our approach still offers advantages; notably, SCMD-no-KD outperforms the baseline ERM method. This suggests that pre- creditinig the CLIP model through fine- tuning before distillation could be beneficial strategy to euhance performance for similar tasks.
'MA': Moving Average. 'SCMD no KD variant when KD not used.
Algorithm
MA
Terra
Avg
ERM
46.1 + 1.8
SCMD (without KD) SCMD
Yes
53.1 + 0.5
Yes
51.3 + 0.2
7.3 Experiments on various student models
We investigate the effect of different teacher models and model sizes by applying SCMD to the PACS dataset. We use 1esNet152 and ResNet18 and follow the same experimental setup and hyperparameter search protocol in our previous experiments.
Table 2 demonstrates that our SCMD approach consistently outperforms Vanilla KD across various scenar- ios, including when different vision encoders from CLIP, such as RN101, are used as the teacher model. Specifically, SCMD achieved an improvement of approximately 0.6% over Vanilla KD. This result high- lights the versatility and elfectiveness of our method in both cross-architecture and distillation scenarios.
Our approach demonstrates significant enhancements in performance, yiclding noteworthy improvement of 3.4% over the ERM technique and 0.8% over Vanilla KD when using ResNet 152 the student. model.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm
CLIP
Student
Avg
ERM (reprddud))))))
ViT-B/32
94.7 1. 0.1
ERM (reproddue)))
RN101
94.9 + 0.1
Vanilla KD
87.9 + 0.3
RN101
RN-50
SCMD
88.5 = 0.2
Vanilla KD
89.2 0.3
VIT-B/16
RN-50
SCMD
89.5 0.3
ERM (reproduced)
88.7 0.5
Vanilla KD
VIT-B/32
RN-152
91.3 0.1
SCMD
92.1 0.2
ERM
Ye al. 2021
81.5 = 0.0
RSC
(uung al.
2020
82.8 + 0.4
IRM
Arjovsky
2019
81.1 0.3
VIT-B/32
RN-18
MMD
Li al.
20186
81.7 0.2
Vanilla KD
84.8 0.3
SCMD
85.0 0.2
Figure 2: Evaluation of SCMD's performance across various student and CLIP model architectures on the PACS dataset
Even with smaller model like ResNet18, our method maintains strong performance relative to other DG methods, showing marginal improvement of 0.2% over Vanilla KD. This slight difference may be attributed to the substantial capacity gap between ResNet18 and CL.PP
8 Conclusion
In this paper, we introduce Selective Cross-Modality Distillation (SCMD) for Domain Generalization, a novel approach that enhances the traditional knowledge distillation framework. Our method is designed to supplement existing DG techniques. We introduce a cross- modaliyyyy module that leverages the robust cross- modal alignment capabilities of CLIP. Central SCMD selection mechanism that is both theoretically grounded and empirically validated through extensive experimentation. Our results demonstrate the efficacy of our proposed method, underscoring its potential to advance domain generalization.
References
Kei Akuzawa, Yusuke Twasawa, and Yutaka Matsuo. Adversarial invariant feature learning with accuracy constraint for domain generalization. In Machine Learning and Knowledge Discovery in Databases: Eu- ropean Conference, ECML PKDD 2019, Würzburg, Germany, September 16 20, 2019, Proceedings, Part II, pp. 315-331. Springer, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. ArXio preprint, abs/1907.02893,, 02893,2883... 2019. URL https://arxix org/ abaa 0tt//9997.0000 028993
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Neurol Information Processing Systems, 35:8265 8277, 2022.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456 473, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Percira. Analysis of representations for domain adaplation. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann (eds.), Ad- vances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Confer- ence on Newral Information Processing Systems, Vancowver, British Columbia, Canada, December 4- 2006, pp. 137-144. MIT Press, 2006. URL https: https://procceedings www..us....ppppprr..... Lib34//aabbcee774425111888 Abstrrrctt..tttt....o..mmo. html
11

---

Published in Transactions on Machine Learning Research (04/2024)
Shai Ben-David John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798 1828, 2013.
Jonathon Byrd and Zachary Chase Lipton. What is the effect of importance weighting in deep learning? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 872-881. PMLR, 2019. URL http://prooceednngs mlr hal...rres/vv9//
byrd19a. I
Junbun Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: domain generalization by seeking flat minima. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Newral Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 22405 22418, 2021. URL htttps://proceedingg.. neuriii neurips.. cc/
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In Computer Vision ECCV 2022: 17th European Conference, Tel Avio, Israel, Oclober 23 27, 2022, Proceedings, Part XXIII, pp. 440 457. Springer, 2022.
Haw-Shiuan Chang, Erik G. Lcarned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1002-1012, 2017. URL https::/ neur httppppper//00//aaaa///217111330
Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding. IEEE Transactions on Neural Networks and Learning Systems, 32(1):25 35, 2020.
Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Darkrank: Accelerating deep metric learning via cross sample similarities transfer. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 pp. 2852 2859. AAAI Press, 2018. URL https: http:///wwwaaaa.rooooo////////// www./os////ooo.ooooooooo.o....oo index. . php/ AAATA AAAAA///AAA///pppp//////////////
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. ArXiv preprint, abs/ 1710.09282, 2017. URL
09282
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248 255. IEEE Computer Society, 2009. doi:
10.1109/CVPR.2000 wtt.e2209..20022. URL littps://oot htt/doi.org//// org/10...000 1109/CVPR,
.5006488
Yu Ding, Lei Wang, Bin Liang, Shuming Liang, Yang Wang, and Fang Chen. Domain generalization by learning and removing domain-specifie features. ArXio preprint, abs/2211.071011, 2022. URL https:
sarxiv.org//bs//22123
Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datascts and web images for softening bias. In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, pp. 1657 1664. IEEE Computer Society, 2013. doi: RLL1109/CCC..0111
URL URLLhttps:// http::://dooooog///// 11w.//Ic1..icc/ 2013.
Songwei Ge, Haohan Wang, Amir Alavi, Eric Xing, and Ziv Bar-Joseph. Supervised adversarial alignment of single-cell raa-ee data. Journal of Computational Biology, 28(5):501-55) -513, 2021.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. DLOW: domain flow for adaptation and gen- eralization. In IEEE Conference Compuler Vision and Paltern Recognalion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2477 2486. Computer Vision Foundation / LEEE, 2019. doi: URL
10.1109 10.1109/CVPR.221 20VPR.2019.02555
thecvf wtt.t/tccocccoceeeeeeeeeooeeeeee www.coottettctc//ooooootoo.o S_tte__v_______ SVPP__201/httm//ong
DLOW DLOW_Domaie Domain Flow for Adaptation and General CVPR, 2019 Bactl__ppeeehhr_ html
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net net, 2021. URL
Beining Han, Chongyi Zheng, Harris Chan, Keiran Paster, Michael R. Zhang, and Jimmy Ba. Learn- ing domain invariant representations in goal-conditioned block mdps. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 94: Annual Conference on Newral Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 764-776, 2021. URL httpps://procceedngss nww.gg....ui.... httl
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pallern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: /ttps://doi.org org/ 10. 1109/CVPR. 2016.90
URL
Bycongho Hco, Jecsoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In 2019 IEEE/CVE International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pp. 1921-1930. IEFE, 2019a. doi: 10.1109/ (CCV.2019.00 10000.. URL https://doi. org/10. 10/ 1109/ICCV 2019.00201
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with adversarial samples supporting decision boundary. In The Thirty- Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirly-Firsl Innovalive Applications of Artificial Inlelligence Conference, TAAT 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honotulu, Hawaii, USA, January 27- February 1, 2019, pp. 3771-3778. AAAT Press, 2019b. doi: (0.1609/aaat v33i01 33013777.
URL https: /ttp::///oooooroo/// 1609//0..6001//0 11.6699aaai..
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. ArXiv preprind, abs/1503.0254 2015. URL /tttts:///axiiii www./ogg///////eee////oo
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. FSDR: frequency space domain randomization for domain generalization. In IEEE Conference on Computer Vision and Pattern Recognation, CVPR 2021, virtual, June 19-25, 2021, pp. 6891 6902. Computer Vision Foundation / IEEE, 2021. doi: URL luttps://opeaccee Huang_FSDRR_ FSDR_ Perequency_Spalte Domain Randomization for Domain Generalization CVPR_2021,
10.1109 (9.1009CVPPRAR CCVPR44437 010446437.2021.000 2021.00622.
thecvf.ee.ccc com/ wwtpa///oaaeee//////////taaaaaaa
paper html
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. Ar preprint, abs/ 0ts//707..02111 2017. URL hhtts::// http:///aaax....o///////70/000/0 01219
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Selfcchalongging improves cross-domain general- ization. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16, pp. 140. Springer, 2020.
Zeyi Huang, Haohan Wang, Dong Huang, Yong Jae Lee, and Eric P Xing. The two dimensions of worst-case training and the intograted effect for out-of-domain generalization. ArXio preprint, abs/ 2200.444444884 2022. URL
org.abs.////////ter 2204. 24384
Zeyi Huang, Andy Zhou, Zijian Ling, Mu Cai, Haohan Wang, and Yong Jac Lec. A sentence speaks a thousand images: Domain generalization through distilling clip with language guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11685-1 11695, 2023.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuat Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4904 4916. PMIR, 2021. URL htttp://proceedingg mlr, ptwsss/v39/ html
Angelos Katharopoulos and François Fleuret. Not all samples are created equal:
Deep learning with importance sampling. In Jennifer G. Dy and Andreas Krause (ods.), Proceedings of the 35th Interna- tional Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2530-2533 PMLR, 2018. URL ceriiiinnnmmmmm....mmmmmm. mlr. wla...pes//vve v80/ pttpsss//00/kaaaarrooollll... html
http://prooceddings
Jangho Kim, Seonguk Park, and Nojum Kwak. Paraphrasing complex network: Network compression via factor transfer. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (ods.), Advances in Neural Information Processing Systems 31: An- vual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 2765-206200 2774, 2018a. URL https::/pproceedingg www.:gg....rrri..ccccccco /ccprrppp.cc/ppppr//2/////////
6496777668ecc330b6e77776066883.. abbrr//abbbbbaeaaaatataat..... Abstract. html
Jangho Kim, Seonguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biancha and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: An- vual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 2765 2774, 2018b. URL
G.Bbb//eeesacc00ddee333663333333 htt.aaabbbbeeaaaaabaabbaatt. html
David Krueger, Ethan Caballero, Jörn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distribution generalization via risk extrapolation (rex). In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5815 5826. PMLR, 2021. URL httpp://proceedings. mlr. (tress/v13//rruuee2211 html
Kyungmoon Lee, Sungyeon Kim, and Suha Kwak. Cross-domain ensemble distillation for domain general- ization. In Computer Vision-ECCV 2022: 17th Europeon Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Parl XXV, pp. 1-20. Springer, 2022.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In JEEE International Conference on Compuler Vision, ICCV 2017, Venice, Italy, Oclober 22-29, 2017, pp. 5543 5551. IEEE Computer Society, 2017. doi: 10.1109/1CCV htt0//CCCV.211..5.. 2017.591. URL https
1109/ICCV 591
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learning for domain generalization. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thérty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (TAAT-18), and the 8th AAAT Symposium on Educational Advances in Artificial Intelligence (EAAI--1)) New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3490 3497. AAAI Press, 2018a. URL https: h/wp:///wwwwwww wwwp:wwwaao aaai, org/
www.oo/gg///ooooooeeeeeeoooooooe hw/////a/.ndnceeee/ee//e////e.ec whwpppppppaap http:/AATAAAAAI///A// patp://aeeeee//r/r/// P
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE Conference on Compuler Vision and Pallern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 5400-5449. IEEE Computer Society, 2018b. doi: 10. 1100/CVPR.201..0655 URL httptttt///ppeccceesstt thecvf. Generalization With Bitss__VP__a00 BB____VVVR______Pp1 2wa___211____e____ html
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dachong Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pp. 2018c.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Ziyue Li, Kan Ren, Xinyang Jiang, Bo Li, Haipeng Zhang, and Dongsheng Li. Domain generalization using pretrained models without fine-tuning. ArXio preprinl, (hss/2203.04600, 2022. httpt:t/aaaaxii.....
04600
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980 2988, 2017.
Haoyang Liu, Maheep Chaudhary, and Haohan Wang. Towards trustworthy and aligned machine learning: A data-centric survey with causality perspectives. arXiv preprint 1rx 2023.
Junjie Liu, Dongehao Wen, Hongxing Gao, Wei Tao, Tse Wei Chen, Kinya Osa, and Masami Kato. Knowl- edge representing: efficient, sparse representation of prior knowledge for knowledge distillation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Yufan Liu, Jiajiong Cao, Bing Li, Weiming Hu, Jingting Ding, and Liang Li. Cross-architocture knowledge distillation. In Proceedings of the Asion Conference on Compuler Vision, pp. 3396 0416-,311---1 2022.
Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, and Xing Xie. Domain-invarianl feature exploration for domain generalization. ArXiv preprint, abs/ abs/2207.12020, 2022. URL https://arxiv: www.:o////aaaoeccccoooo.oo.. ab///2//////2/// 2207.
12020
Rang Meng, Xianfeng Li, Weijie Chen, Shicai Yang, Jie Song, Xinchao Wang, Lei Zhang, Mingli Song, Di Xie, and Shiliang Pu. Attention diversification for domain generalization. In Computer Vision-ECCV 2022: 17th Europeon Conference, Tel Aviu, Israel, Oclober 23 23,27, 2022, Proceedings, Part XXXIV, pp. 222-340.55 Springer, 2022.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant. feature representation. In Proceedings of the 30th International Conference on Machine Learning, 1CML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 10-18. Jorg. 2013. URL httt::://proceedinggee mmr...g.srrr....mmmmmm wwt.:eeree/emmaeaeeess 11tmm
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillaioon. In IEEE Conference Computer Vision and Pattern Recognátion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3967 3976. Computer Vision Foundation / IEEE, 2019. doi:
20VPR.2019.00400 URL http: stttp://openccees thecvf. wtt.cooccccoooc hata/content_CtVi @VPP__2199htmm/Part Relational wwwwaladnnnwweeeaaaaaaaaee www.://eeeeeiiiiiilliiii...o Distillation, CVPR 2019 _acle__pppe..__ html
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using informa- tion flow modeling. In 2020 IEEE/CVE /CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seatile, WA, USA, June 13-19, 19, 2020, pp. 2336 2345. IEEE, 2020a. doi: 10.1109/VVP142600.2220 00241. URL https: //do: /ttp:////ooooo/////////oooo http://1.11111..111 1119////////0/ 2020. 00241
Nikolaos Passalis, Maria. Tzelepi, and Anastasios Tefas. Probabilistic knowledge transfer for lightweight deep representation learning. IEEE Transactions on Newrol Networks and Learning Systems, 32(5):2030 2039, 2020b.
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh,, and Qun Liu. ALP-KD: attention-based layer pro- jection for knowledge distillation. In Thitty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Artificial Intelligence, TAAT 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-
9. 2021, pp. 13657- 13665. AAAT Press, 2021. URL https://ojs aaai wwwooorggg...eee.ooeoooeooee.e.e php/ Att.//AAAIaarlee//////a view/17610
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In 2019 IEEE/CVT International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1406-1415. IEEE, 2019. doi: www.0/9/CCV..2...00 URL http:httts///ee http:////....011..1111////////// 010100000/110222200000000099999
15

---

Published in Transactions on Machine Learning Research (04/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Leurning Research, pp. 8748 8763. PMLR, 2021. URL http://proceedings mlr. wwwpss/vss/...peeep.. http//////rrrdd1121../ html
Alexandre Ramé, Kartik Abuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Recy- cling diverse models for out-of-distribution generalization. ArXiv preprint, abs/2212. 1011241.,,,4, 2022. URL
attps://arxiin wwwpaawwww....aaaaaaa 2212. 10445
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Galla, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015a. URL http://arxiv org/ abs/ aba/a/a/////////////////////////
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnels: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015b. URL htttp://arxiv.orgg wwwg/ba///aaaooooooooooggge ht//a/b///1111.55
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedingss OpenReview.nnt, 2018. URL
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 443 450. Springer, 2016.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In Artificial Neural Networks and Machine Learning- -CCAN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pp. 270 279. Springer, 2018.
Xavier Thomas, Dhruv Mahajan, Alex Pentland, and Abhimanyu Dubey. Adaptive methods for aggregated domain generalivation. ArXiv preprint, abs/2112.04766, 2021. URL https: http://axivv.or//abb/////// org/abs//112 04766
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing Conference on Computer Vision and Pattern network for unsupervised domain adaptation. In 2017 IEEE
Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5385-5394. IEEE Computer Society, 2017. doi: 10.1109/CVV CVPR22017.552. URT. https:: http///i..ro//..///////////1111e 1109//109...../ CVRR. 2017. .572
Haohan Wang, Aaksha Meghawat, Louis Louis-Philippe Philippe Morency, and Eric P Xing. Select-additive learning: Improving generalization in multimodal sentiment analysis. In 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 949 954. IEEE, 2017.
Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribution generalization with causal invariant transformations. In TEEE/CVF Conference on Computer Vision and Pallern Recognilion, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 375 385. IEEE, 2022a. doi: 10.1109/CVPRSS 2022.00047. URL
org/ 10. 1109/
2022.00047
Xiaobo Wang, Tianyu Fu, Shengcai Liao, Shuo Wang. Zhen Lei, and Tao Mci. Exchusivtty-consisteney regularized knowledge distillation for face recognition. In Computer Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part XXIV 16, pp. 325 342. Springer, 2020.
2020: 16th European
16

---

Published in Transactions on Machine Learning Research (04/2024)
Zhuo Wang, Zezheng Wang, Zitong Yu, Weihong Deng, Jiahong Li, Tingting Gao, and Zhongyuan Wang. Domain generalization via shuffled style assembly for face anti-spoofing. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 4123 4133, 2022b.
Kunran Xu, Lai Rui, Yishi Li, and Lin Gu. Feature normalized knowledge distillation for image classifica- tion. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 664-680. Springer, 2020.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in teatebee space. In Proceedings of the IEEE/CV //F Conference on Computer Vision and Pattern Recognation, pp. 19163-19173 2022a.
Jinyu Yang. Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng. Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognâtion, pp. 15671-15680, 2022b.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: fine- fine-graine grainer interactive language-image tre-tiainnn In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
OpenReview.nel, 2022. URL htttpp//ppeaaeeeeeeeeeee//oodddp www.net/foooumiidcpppceeeee
Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. Ood- bench: Benchmarking and understanding out-of-disttiibttion generalization datasets and algorithms. ArXio preprint, abs/2106. 03721,, 2021. URL
03721
Haoxuan You, Luowci Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-inage training In Compuler Vision-ECCY 2022: 17th European Conference, Tel Aviu, Israel, October 23 23- 27, 2022, Proceedings, Part XXVII, pp. 69 87. Springer, 2022.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto L. Sangiovamn&Vimenntlll Vincentelli, Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consistency: Simulation-tto-rall generalization with- out accessing target domain data. In 2019 IEEE/CVI International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pp. 2100-2111 2010-210..-11 IEEE, 2019. doi: (1A.L////1CCC22221.0020
URL https://doi: org/10. 1110/ICICVV 2ww....................... 2019. 00219
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenRe- view.net, 2017. URL https::/ httpp:://pperrevier. wttp://wwwiioeoeeoooooeeo//---e. saajj
Xin Zhang, Shixiang Shane Gu, Yutaka Matsuo, and Yusuke Iwasawa. Domain prompt learning for efficiently adapting clip to unseen domains. arXiv e-prints, pp. arXiv-2111 2111, 2021.
Ying Zhang. Tao Xiang, Timothy M. Hospodales, and Huchuan Lu. Deep mutual learning. In 2018 IEEE Conference on Computer Vision and Pollern Recognition, CVPR 2018, Sall Lake Cily, UT, USA, June 18-22, 2018, pp. 2018.00454. URL http:// thecvf com/content, cvpr_2vi8_1../eee haai8/httm/Zhangg Deee_DDep___aa_ Mutual_ Learning_ CVPR_ 2018 Bac--B_pppr___ html
IEEE Computer Society, 2018. doi: 10.1109/CC 1.1109/CVPR.
Kaiyang Zhou, Yongxin Yang, Timothy M. Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAT Symposium on Educational Advances in Artificial Intelligence, EAAT 2020, New York, NY, USA, February 7-12, 2020, pp. 13025-13032. AAAI Press, 2020. URL https://aaai: org//js// index. pitp/AAA//arriill//veeee/////
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, (3309))2337-2344 2022.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Appendix
A Theoretical Evidence for Selection Strategy
To be consistent with notation, we let (X, Y) denote the standard dataset, and (x, as one of the samples. We let. P denole a distribution and P denote the distributiin of distributionss. We let f(=;0) denote the student model, & denote the teacher model, and r(-) denote the risk. For the convenience of notations, we allow r(-) to be parameterized by distribution by dataset.
Lemma A.1. Given assumptions A1 such that there is a gold standard labeling function for source and target domains. For two arbitrary distributions P and P.
r(P') + to
where to denoles the total variation.
Proof. Recall that assuming the same labelling function, let and o' be the density functions of P and p
((P')=r(P)) = -MP) |r(t+rr(PPP))))p r(())
Sr(P) / [x(x - d(xx f(x;0) y dx
Sr(P) + (p( P)
Lemma A.2. Given assumption that sumples are independent and identically distributed, hypothessi space ④ and any 8 0, with probability least 1 5, we have
to ((P') (r()),((XXV))) ()
where we let \)x denote the number of sample sizes in the finite dataset (X, Y),, & is vanilla term that connects the number of samples and hypothesis space with generalization error bound.
assuming that samples are independent and identically distributed, have {()(x,)),, 0) 2R(C)+ (log 1/0)/2n where R(2) stands for Rademacher complexity and L = (lo 0 e θ} where lo is the loss function corresponding to the student model f(:
Proof. Recall that
r(P') Sr(P) to(P',P) S <r((X, Y))) tu((", I P) E((XX.)) 0,0)
This is an direct application of Lemma A.1 and generalization error bound.
The above results demonstrate that empirical robustness is determined by three factors: the divergence between training and test distributions, the measurable empirical error on the training distribution, and technical term influenced by sample size and hypothesis space. Therefore, the critical term that will bound the robustness performance how the training distribution deviates from the testing distribution. This intuitively gives the idea that training with the distributions that are the most similar to the test distribution will benefit the model most.
The above results apply to arbitrary distributions P ~ P. However, this does not necessarily encode the characteristics of the cases we are studying: some samples are hard for the model to learn.
To address this, we consider datascts generated by multiple distributions, some of which present more challenging learning scenarios. We represent these as set P. consisting of distributions, i.c., (Pi, Pa...., P..). Each data point is considered sampled from these distributions. For the convenience
18

---

Published in Transactions on Machine Learning Research (04/2024)
of discussion, use tv(P), P) to denote the average divergence between the distributions within the set. to(P) (
Finally, we use s() to denote the distribution selection mechanism, and we compare two selection mechanisms: selecting the hard-to-leary samples (denoted as $1) and selecting random samples (denoted as $2). Lemma A.3. P is continuous and has fonite expected value; for the two selection mechanism that are formally defined as
(t), = sup to(P, P))
PEP
for a ftxed lesting dalaset. P', with the assumption that Lu(P, Pt) + to we have
Ep to(s P') Ep (u(s)(P))
Proof. Based on our definition of 51 and $2;
(ett(ss(P), Pp
And based on our assumption that tu(P, P') = to to P'), we have
Ep
(s)) tu (P), P')
Ep inf to (o PP) Eptu(P, P') 0
PEP
Our result compares the upper-bounded differences between the two training distribution selection strategies, and results suggest that selecting the hard-to-learn samples will lead to tighter generalization error bound.
Another important factor to note is that, given assumption Al and Theorem 0.1, the selection strategy tv(P, sup is only when selecting the
applicable to our theoretical discussion (i.e. (u(s = supper hard-to-learn samples according to the label of the samples (thus, cross-entropy loss). Other selection strategies such as selecting based on KL-divergence or distillation loss despite might following a similar goal, do not strietly match our theoretical discussion, which will likely lead to an error bound in between 81 and $2. Therefore, with the support of the theoretical discussion, argue that the most effective hard-to-learn selection mechanism is to be based on cross-entropy loss.
Another possible question is that Assumption le(P, P') Poo P') might appear strong. In fact, the proof will hold with any assumptions that describe the concept that the more different one distribution is from the average of the training set, the more it will benefit the testing distribution. In the typical domain gencralization setting, where there are no guaranteed connections between training and testing distributions, we believe this is one of the practical assumptions can consider, also widely used in practical context Chang et al. 2017
by other domain generalization literature (Huang al. 2022 Byrd & Lipton 2019
Katharopoulos & Fleuret 2018
B Method
B.1
Algorithm
B.2 Selection Mechanism
(5) In the preceding equation, X denotes the batch of samples, x; an individual sample, and y, its true label. The set consists of selected samples. The function H(f(x; y,) computes the cross-cntropy loss for the i-th sample, while contains indices of samples in the batch with cross-entropy loss exceeding the threshold
S = E X where I = i: (x ).)) 2T
19

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm Selective Cross-Modality Distillation
Input: Dataset (X,3) of size n; Percentile of hard-to-learn samples per batch p; Percentile of full-batch training к; Batch size 77; Maximum number of iterations T; Feature Projector P; pretrained teacher model Φ; Student model 0 (randomly initialized)
Output: Trained student model 0
while - ' do
while S (1 k)T do Identify top p percentile samples with highest Cross-entropy loss based on Eq 2
For selected samples, compute student features and project to CLIP's multi-modal space via P.
Distill knowledge from the teacher model to the student model 0 using Eq 4
end while
Distill knowledge from & to 0 across the entire batch and calculate the final loss with Eq 4
end while
return optimized student model 0
B.3 Cross- Modality Module
LCM = DKL [(p")))
where p' is the soft target distribution of CLIP and
(6)
(p")' = o (e(exx;;))) T
In this equation, y is a scale factor, which adjusts the magnitude the projected student feature. e represents the backbone of the student model. linear projection W is applied the student feature e( 0,), and Grext represents the text embedding of CLIP. a the softmax function parameterized by the distillation temperature T.
B.4 SCMD
SCMM = arg min
Σ x
((X.) +AgLlogits + A3CCM
(7)
where flogits = DKL. (pr) and x; and y; are the selected samples and their corresponding labels
C More Analysis
C.1 Parameter Sensitivity Analysis
To reiterate the definition of the parameters for clarity: for implemennatton purposes, we set to be the percentile of samples selected as hard-to-learn, with T 0.25 indicating the selection of samples with a loss exceeding the 75th percentile. Meanwhile, & denotes the fraction of steps before the transition to full-batch training occurs; for example i: 0.25 implics that this shift happens during the last 25% of the stops.
The table presents the performance of our model across various domains of PACS adjust the values of k and t, keeping all other parameters constant during the experiment.
k Sensitivity (with - 0.2): The model's performance remains relatively stable across an array of k values, as evidenced by the average accuracy consistently around 90.0%.
0.25): Having very few samples is not helpful to the learning process. And a slight decrement in the model's performance is observed as the value of increases, after reaching 0.2. This trend can be ascribed to the dilution of the impact of hard-to-learn samples as their proportion increases.
T Sensitivity (with
20

---

Published in Transactions on Machine Learning Research (04/2024)
meterr Seadiiityy Domain RACS
Damult
Acnsage
613
85.
Vatar 501
Dumaain
Denwi
homais
Toandeer Verue (kd))
Neer de wak 1982
Nesendio Walic (1O)
Pwerde YOu C09t
Neonecto Vive (1K0)
Figure 3: Sensitivity analysis for and
Specifically, as more samples are classified as hard-to-learn (with a higher r), the advantage of focusing on these challenging samples is somewhat mitigated, subtly influencing the overall model performance.
C.2 Are high-loss samples outliers?
Exclude Top-K
Domain
Domain
Domain
Domain
Average
93.0
86.8
99.0
81.1
90.0
92.5
86.8
99.2
80.8
89.8
92.1
86.0
99.3
80.3
89.4
93.0
86.4
99.3
80.6
89.8
86.4
98.8
80.8
89.6
99.1
79.7
88.9
Table 5: Performance upon the exclusion of top hard-to-learn samples
As depicted in Table 5 our empirical analysis confirms the critical role of the highest-loss samples in influ- encing the model's overall performance. The data reveals a clear pattern: excluding these high-loss samples consistently leads to a reduction in performance metrics. Specifically, the deliberate removal of the top 1 and top highest-loss samples results in a quantifiable decrease in model efficacy, with average performance dropping from 90.0% to 89.8%, and then 89.4%. These findings highlight that high-loss samples are not merely outliers but are crucial for the model's learning process, significantly enhancing its ability generalize and perform accurately across various domains.
C.3 Training Time Analysis
Figure 4: Average step time per domain for different algorithms on PACS
21

---

Published in Transactions on Machine Learning Research (04/2024)
The results indicate that the average training overhead of SCMD is comparable to that of Vanilla KD. This is attributed to our training strategy that focuses hard-to-learn samples for the majority of the duration, transitioning to full-batch training only in the last & steps. This method balances the overall training overhead.
C.4 Compare with other KD methods
Algorithm
MA
Avg
FitNet Romero ct al.
2015b)
Yus
1.2 88.4 i 0.2
BSS
(Hoo ct al 20196
Yus
89.3 1 0.1
RKD
Park et al. 2019
Yes
87.4 = 0.2
Vanilla (Hinton et al.
2015
Yes
89.0 = 0.3
SCMD
Yes
90.1 = 0.0
Table 6: SCMD other KD on PACS. "MA": Moving Average
At the core of SCMD is the knowledge distillation process. To evaluate the effectiveness of SCMD, we conduct comparative experiments with other knowledge distillation methods on the PACS dataset. The results in Table demonstrate that SCMD outperforms these contemporary techniques.
D Full Results
Full details of the results from Table D in the main paper are provided in Table
D.1 hyperparameters search space
We adhere to the experimental setup described in the DomainBed Gulrajani & Lopez-Paz 2021
paper.
The specifics of our setup are outlined below:
Data Split: We partition datasets into 80% training and 20% validation sets. Model selections are
based on training domain validation performances, and we report on the corresponding test domain.
hyperparameters: Although many hyperparameters follow (Gulrajani & Lopez- Paz ations are documented in Table 7
2021 devi-
Batch & Decay: We adjust batch size and weight decay following the guidelines of Cha al.
2021).
Dropout: The ResNet dropout rate set to 0 to mitigate excessive randommess.
Learning Rate: We abandon the rate of 10-5 because it converges too slowly, and instead focus on rates of 10-5 and 5 10-5
Table g outlines the search space specific to our algorithm's hyperparameters. We consistently set ' the balance factor for cross-cntropy, to 1, while conducting random sweops for the other weight factors.
Parameter
Default
DomainBed Guurajani & Lopez- Paz 2021
SWAD Cha et al. 2021
Ours
batchsize
(3.6.5)) (-
32
32 (3--5, 5e-5]
learning rate
5e-5
[1e-5, 3e-5, 5e-5]
ResNet dropout
[0.0, 0.1, 0.5]
0.0, 0.1, 0.5)
0
weight decay
(-
[1e-4, 1e-6]
[1e-4, 1e-6)
Table 7: Unconditional hyperparameter search space. (U and list indicate Uniform distribution and random choice, respectively)
22

---

Published in Transactions on Machine Learning Research (04/2024)
Parameter
Default Value
Sweep range
A1 (for CE loss)
Ag
0.5
U(0.5, 1.0)
0.5
U(0.5, 1.0)
last k s_eecch
0.25
U/002, 0.4)
hard-to-learn sample percentage
1/3
0.25, 0.3] 0.22,
temperature
3.0
U(2.0, 5.0)
Table & Algorithm-specif hyperparameter search space. (U and list indicate Uniform distribution and random choice, respectively)
VLCS
CLIP VIT-B/32
Student RN50
A
C
P
S
Avg 80.9 1 0.2
SCMD
98.8 + 0.1
64.6 +
0.4
78.2 + 0.3
81.9 1 0.3
PACS
L
S
Avg 1
SCMD
VIT-B/32
RN50
92.9 0.3
86.0
0.1
SCMD
ViT-B/32
RN152
94.0
0.2
89.1
85.9
0.6
SCMD
VIT-B/32
RN18
84.7 0.1
80.7 0.5
96.2 +
78.6 - 0.6
85.0 + 0.2
SCMD
RN101
RN50
90.2 + 0.8
83.4 0.3
99.1 +
81.1 = 0.8
88.5 + 0.2
Vanilla KD
VET-B/32
RN50
91.2
0.2
85.2
0.3
99.4
80.3 = 0.9
89.0 1 0.3
Vanilla KD
VET-B/32
RN152
93.8 0.1
87.3
0.4
84.7
0.2
91.3 0.1
Vanilla KD
VIT-B/32
RN18
0.5
1
78.5 0.6
1
Vanilla KD
RN101
RN50
0.2
80.8
0.5
OfficeHome
C
P
Avg 74.8 + 0.1
SCMD
VIT-B/32
RN50
72.7 + 0.0 58.7 + 0.1
82.4 + 0.1
85.5 + 0.1
Terralncognita
L100
L38
L43
L46 46.6 1 1.0
Avg 51.3 1 0.2
SCMD
VIT-B/32
RN50
52.9 1 1.1
45.3 0.1
60.5 1 0.5
DomainNet
clip
info
paint
quick
real 70.4 i 0.0
sketch 57.4 + 0.1
Avg 48.4 + 0.0
SCMD
VIT-B/30 B/32
RN50
66.2 + 0.1 25.2 1 0.0
56.9 i 0.2
14.3 i 0.2
Table 9: Detailed Performance of SCMD and Vanilla KD
23


Reviews:
[{"review_id": "yQ2tHHK9ts", "reviewer": "Reviewer_9y2v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The knowledge distillation with a multi-modal foundation model is a popular idea that enables zero-shot visual recognition including image classification, object detection, semantic segmentation, and visual grounding. In many use cases, the users do not have sufficient computing resources, studying efficient and effective knowledge distillation methods is a right research direction. \n2. Knowledge distillation via two classification heads (student’s classifier and CLIP-style classifier with text encoder). This is also an interesting construction. \n3. The authors claimed that hard sample mining is effective for knowledge distillation. \n\n**Weaknesses**\n\n1. Heuristics in training scheme. Focusing important samples (e.g., hard sample mining) usually can be more elegantly achieved by new loss functions. For instance, hard negative mining has been actively studied in the context of object detection but dense object detectors often use the focal loss, which is properly modified to address imbalance problem. \n2. Efficiency and complex pipeline. The pipeline seems complex and no analysis of computational cost is provided. Filtering/selecting hard samples may require additional neural network inferences and the step may make the pipeline less GPU-friendly. Training time comparison will be useful for readers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors studied a knowledge distillation method for domain generalization leveraging cross modal foundation model, CLIP. This paper proposes a new distillation pipeline equipped with two new modules: hard sample mining and cross-modal module. In addition to the cross-entropy loss, distillation using CLIP-style prediction by the innerproduct between student’s image features and CLIP text features is a different additional component. Lastly, the authors theoretically analyzed the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Comparison of computational overhead in training / training time \n2. Sensitivity analysis for hyperparameters (e.g., sample selection threshold)\n3. Table 2, SCMD (final version) has 0 variance but other baselines have some variance. Could you add discussion about this? \n4. Table 5, the orders of methods are confusing. Could you place the proposed method on top or at the bottom?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K0YIXNm5tx", "reviewer": "Reviewer_cdPz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n\n* The introduction starts with an analogy to explain the concept of the proposed method. A quote from the analogy is ‘lyrics can heighten the clarity and expressiveness of her music.‘ This also raises a question, however, as introducing lyrics too early can discourage the student with their inability. How does one discern when is the right moment to introduce new complexity?\n\n* The related work gives a strong overview of the literature.\n\n* The paper makes a bridge between the field of hard-to-learn samples and the field of knowledge distillation.\n\n# Weaknesses:\n\n* In Table 1, the improvements are generally less than 1 point percent. Could the authors clarify why they think that the added complexity is worth the small increase in performance?\n\n* The title of the paper is ‘Choosing Wisely’, but the motivation for ‘selecting hard examples’ is missing, except for the high-level analogy in the introduction. If the motivation is to reduce the overall required amount of computation, then the computation time should be reported experimentally. For example, Table 1 could report empirical runtimes. If, on the other hand, the motivation is to select hard examples to improve learning, then I would expect a study on that. For example, how do different selection criteria differ in their influence on the final performance? Or if there is a fraction of hard-to-learn samples, it would be interesting to see how varying that fraction influences results.\n\n* Where should the line be drawn between hard-to-learn samples and plain outliers? Moreover, where should the line be drawn between hard-to-learn samples and incorrectly labeled data? We know from the literature that popular large-scale datasets could contain incorrectly or inconsistently labeled data [1].\n\n* The experimental results could have been stronger with an ablation of the selection criterion. As the title of the paper is ‘Choosing wisely’, it is a natural question to ask how wisely one should choose. Equation 2 introduces a parameter tau that seems to be a threshold about how low the cross-entropy can be. A very instructive experiment could be to sweep different values of tau and study domain generalization performance. If computation is a bottleneck, one could select a fixed number of samples from the cross-entropy ordering (e.g. temperature 1), and interpolate that to random sampling (e.g. temperature 0). \n\n[1] Beyer, Lucas, et al. \"Are we done with imagenet?.\" arXiv preprint arXiv:2006.07159 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of domain generalization. In domain generalization, one wants to learn about data from one domain and achieve good performance on unseen data from a new but related domain. The proposed method selects hard-to-learn samples from the source domain in order to learn better generalization characteristics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Change requests:\n\nPlease address the questions asked in the strengths & weaknesses section.\n\nNotation: The capital letter P is used for both the linear projection and for the data distribution P_(X,Y) (when in boldface). They are hard to discern.", "broader_impact_concerns": "Domain Generalization has a net positive research impact", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ca2qiINYI9", "reviewer": "Reviewer_v6HT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although backed by strong assumptions and incalculable metrics (e.g., TV), the authors have attempted to explain why their methods work well.\nHowever, I am doubtful whether this method fairly compares with other baseline methods shown in Table 1, as all other methods do not employ a powerful pre-trained foundational model (i.e., CLIP).\nI note that few papers published in ICCV 2023 [1,2], which employed CLIP for the task of domain generalization, did not compare their method with existing DG approaches head-to-head.\nMoreover, the performance of these two approaches is remarkably higher than that of the proposed method, with the caveat that they used ViT-B/16 CLIP, whereas this submission used ViT-B/32 CLIP.\nSince the submission claims to be about domain generalization methods, I believe there is room for improvement.\nThe authors should clearly denote why this method is tailored for domain generalization, not feature distillation, as I think the logical steps in this paper could be a better fit for the topic of feature distillation.\n\n[^1]: A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance  \n[^2]: PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Selective Cross-Modality Distillation for Domain Generalization (SCMD) to train an image encoder that generalizes well to unseen domains.\nThe authors claim their proposed methods—hard-to-learn sample selection and a cross-modality module—excel at training a model for the DomainBed benchmark, which evaluates the domain generalization capability of models.\nTheir methods differ from existing ones by exploiting a pre-trained CLIP model.\nThe ablation study showed that each method contributed to the performance boost. The paper also provides assumption-driven theoretical proof explaining why SCMD outperforms other baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please clarify what the unique contribution of this submission is, compared to the two papers I mentioned in the previous section.\nGiven that the additional computational burden of using ViT-B/16 CLIP should be minimal, results using ViT-B/16 CLIP as a teacher model should also be included for a comprehensive analysis.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows an effective CLIP-based domain generalization:

-It introduces a novel distillation pipeline leveraging the CLIP model, showcasing a new approach to domain generalization that harnesses cross-modal capabilities.
-With the introduction of hard sample mining and a cross-modal module, the paper contributes new methods to the knowledge distillation field, evidenced by performance improvements on benchmark tasks.
-Comprehensive ablation study validates the effectiveness of each proposed component, providing clear evidence that they contribute to the model's enhanced performance.

Initially, the reviewers raised concerns about insufficient comparative experiments, unaddressed computation efficiency, and hyperparameter selection. After the response, the authors have addressed all the concerns. All reviewers recommend acceptance.
Decision: Accept (poster)