AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation
Authors: Zihao TANG
Zheqi Lv
Shengyu Zhang
Yifan Zhou
Xinyu Duan
Fei Wu
Kun Kuang
Venue: ICLR 2024 poster (2024)
Keywords: out-of-domain knowledge distillation
mixup learning
domain shift
uncertainty
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Due to privacy or patent concerns, a growing number of large models are released without granting access to their training data, making transferring their knowledge inefficient and problematic. In response, Data-Free Knowledge Distillation (DFKD) methods have emerged as direct solutions. However, simply adopting models derived from DFKD for real-world applications suffers significant performance degradation, due to the discrepancy between teachers' training data and real-world scenarios (student domain). The degradation stems from the portions of teachers' knowledge that are not applicable to the student domain. They are specific to the teacher domain and would undermine students' performance. Hence, selectively transferring teachers' appropriate knowledge becomes the primary challenge in DFKD. In this work, we propose a simple but effective method AuG-KD. It utilizes an uncertainty-guided and sample-specific anchor to align student-domain data with the teacher domain and leverages a generative method to progressively trade off the learning process between OOD knowledge distillation and domain-specific information learning via mixup learning. Extensive experiments in 3 datasets and 8 settings demonstrate the stability and superiority of our approach.

Published as a conference paper at ICLR 2024
AUG-KD: ANCHOR-BASED MIXUP GENERATION FOR OUT-OF-DOMAIN KNOWLEDGE DISTILLATION
Zihao Tang, Zheqi Lv, Shengyu Zhang
Yifan Zhou
Zhejjang University
Shanghai Jiao Tong University gen i uszouuyi f com
(tangzi hao, zheqi 1v, sy e edu.cn
Xinyu Duan
Fei Wu & Kun Kuang"
Huawei Cloud
Zhejiang University
duanxinyuehuawel. com
{wufei, kunkuang)ezju. edu. en
ABSTRACT
Due to privacy or patent concerns, growing number of large models are released without granting access to their training data, making transferring their knowl- edge inefficient and problematic. (DFKD) methods have emerged as direct solutions. However, simply adopting models derived from DFKD for real-world applications suffers significant per- formance degradation, due to the discrepancy between teachers' training data and real-world scenarios (student domain). The degradation stems from the por- tions of teachers' knowledge that are not applicable the student domain. They are specific to the teacher domain and would undermine students' performance. Hence, selectively transferring teachers' appropriate knowledge becomes the pri- mary challenge in DFKD. In this work, we propose simple but effective method and sample-specife anchor to align
response, Data-Free Knowledge Distillation
AuG-KD. It utilizes an uncertainty-guite student-domain data with the teacher domain and leverages a generative method to progressively trade off the learning process between OOD knowledge distil- lation and domain-specific information learning via mixup learning. Extensive experiments in 3 datasets and 8 settings demonstrate the stability and superiority of our approach. Code available at https://gthab..com//Skknnnaaaaa
INTRODUCTION
With the surge of interest in deploying neural networks on resource-constrained edge devices, lightweight machine learning models have arisen. Prominent solutions include MobileNet (Howard
cet al. 2019), EfficientNet Tann & Le 2019), ShuffleNet (Ma et al., 2018), etc. Although these models have shown promising potential for edge devices, their performance still falls short of pectations. In contrast, larger models like ResNet (He et al., (016) and CLIP (Radford et al.) 2021), have achieved gratifying results in their respective fields (Wang et al., 2017 Tang al. 2024). To further refine lightweight models' performance, it is natural to ask: can they inherit know ledge from larger models? The answer lies in Knowledge Distillation (Hinton al. 2015) (KD).
Calderon et al., 2023) leverages massive training data to transfer knowledge from teacher models T students S. guiding S in emulating I's prediction distribution.
Vanilla KD (Kim et al.) 2023
Although these methods have shown remarkable results in datasets like ImageNet (Deng et al. 2009) and CIFAR10 (Krizhevsky) 2009), when training data is unavailable due to privacy concerns (Troong 2021 or patent restrictions, these methods might become inapplicable.
et al.
To transfer T"s knowledge without its training data, a natural solution to use synthesized data samples for compensation, which forms the core idea of Data-Free Knowledge Distillation (DFKD) (Binici al.) 2022) Li et al. [023] Patel et al.) [023] Do et al.. [022] Wang et al. (023a). These methods typically leverage 7"s information, such output logits, activation maps, intermediate outputs, etc., to train a generator to provide synthetic data from normally distributed latent variable.
"Shengyu Zhang and Kun Kuang corresponding authors.

---

Published as a conference paper at ICLR 2024
The distillation process is executed with these synthesized data samples. However, DFKD methods follow the Independent and Identically Distributed Hypothesis (IID Hypothesis). They suppose that T's training data (teacher domain D2) and the real application (student domain D,) share the same distribution (Fang et al.) 2021b). In case the disparity between these two distributions cannot be neglected, these methods would suffer great performance degradation. Namely, the disparity is denoted as Domain Shift while the distillation without I"s training data under domain shift is denoted as Out-of-Domain Knowledge Distillation (OOD-KD). In Figure [ we demonstrated the difference among KD, DFKD, and OOD-KD problems, where KD can access both D, and D,, while DFKD can access neither Dr or D,- OOD-KD can access D,. but has no prior knowledge of D.. Moreover, KD and DFKD problems require the IID assumption between D, and D.. which can hardly satisfied in real applications. Here, OOD-KD problem designed for address the distribution shift between D, and Da. Although domain shift has garnered widespread attention in other fields (L et al., 2023; 2024 Zhang et al. 2023c Huang et al. 2021; Lv et al., 2022). there's no handy solution in OOD-KD (Fang et al., 2021a). MosiacKD Fang al. (02la) is the state-or-the-a art method for addressing OOD-KD problem, but it mainly focuses on the improvement of performance in Dr, ignoring the importance of D, (i.e. out-of-domain performance). Recently, some studies propose cross-domain distillation (Li et al. 2022] Yang et al., (022) for OOD-KD, but these methods require grant access to D,, which is impractical in real applications.
Unavailable
Unavailable
Teacher Domain Dj
Only Test. Available
Student (Test Scene)
KD (IID)
(II
OOD-KD
Figure 1: Differences between KD, DFKD, and OOD-KD KD problems.
In this paper, we focus on the problem of OOD-KD, and address this problem we are still fac- ing the following challenges: (i) How to selectively transfer teachers' knowledge. In OOD-KD problem, the difference of the joint distribution P(X, Y) between teacher domain D, and student domain D, creates a significant barrier. Since T is optimized for D,, faced with data in D., T is likely to give inaccurate predictions or fail reflect the precise relationships between classes D., impeding S's performance unavoidably. (ii) The absence of 7's training data makes OOD-KD extremely challenging. As T's training data act as the carrier of knowledge in vanilla KD, without it, knowledge transferring becomes troublesome. In contrast, data in the application scenes are easy to obtain. It is important to notice that their domain- -spciffc information is applicable to D., utilized properly, it is able to benefit S's training.
To tackle these challenges, we propose a simple but effective method: Anchor-Based Mixup Gen- erative Knowledge Distillation (AuG-KD). Our method utilizes uncertainty-drie -driven and sample- specific anchor to align student-domain data with D, and leverage a generative method to progres- sively evolve the learning process from OOD knowledge distillation to domain-specific information learning. Particularly, AuG-KD consists of 3 modules: Data-Free Learning Module, Anchor Learn- ing Module, and Mixup Learning Module. Data-Free Learning Module bears semblance vanilla DFKD, tackling the absence of D.. Anchor Learning Module designs an uncertainty-awarc An- chorNet to map student-domain samples "anchor" samples in D,, enabling T to provide proper knowledge for distillation. Mixup Learning module utilizes the "anchor" samples to generate a series of images that evolve from D, D., treating them as additional data for training. As the module progresses, T becomes less certain about them while the domain-specific information grad- ually becomes important, balancing OOD knowledge distillation and domain-specific informa- tion learning ultimately. Extensive experiments attest the excellent performance of our proposed method. In essence, our contributions can be briefly summarized as follows:
We aim at an important and practical problem OOD-KD, To the best of our knowledge, we are the first to provide a practical solution to it.
We propose simple but effective method AuG-KD. AuG-KD devises a lightweight An- chorNet to discover a data-driven anchor that maps student-doman data to D,. AuG- KD then adopts novel uncertainty-aware learning strategy by mixup learning, which pro-
2

---

Published as a conference paper at ICLR 2024
gressively loosens uncertainty constraints for a better tradeoff between OOD knowledge distillation and fomain-speciffe information learning.
Comprehensive experiments in 3 datasets and 8 settings are conducted to substantiate the stability and superiority of our method.
2 RELATED WORK
Since OOD-KD is novel problem, focus on the concept of Knowledge Distillation first. KD a technique that aims to transfer knowledge from large teacher model to an arbitrary student model, first proposed by Hinton al. (015). The vanilla KD methods either guide the student model to resemble the teacher's behavior on training data (Bucila et al.] (006) or utilize some inter- mediate representations of the teacher (Binici et al. 2022: Romero al. 2015 Park et al.) 2019). In recent years, knowledge distillation has witnessed the development of various branches, such as
Adversarial Knowledge Distillation (Binici et al. 2022 Yang et al.) 2023). Cross-Modal Knowl-
edge Distillation Li ct al. 2022 Yang et al. 20022). and Data-Free Knowledge Distillation (Li ct al.
2023
Patel et al. 2023 Do et al. 2022 Wang et al. 2023a).
Recently, data-free methods (DKFD) have garnered significant attention. DFKD typically relies on teacher models' information such as output logits and activation maps to train generator for com- pensation from normally distributed latent variable. Besides, there are also some sampling-base based methods utilizing unlabeled data (Chen et al., 202T, Wang et al.) 2023b). However, the effective- ness of DFKD methods is based on the assumption of the III Hypothesis, which assumes that student-doman data distributed identically to that in D,. This assumption does not hold in many real-world applications (Arjovsky al. 2019 Zhang et al. 2020 Liu et al. (023). leading to signifi- cant performance degradation. The violation of the IID Hypothesis, also known as oft-of-domain or
domain shift, has been extensively discussed in various fields (Huang al. [021] [iang al. [022]
Sagawa et al. 2020 Zhang et al.] [202bb 2023b; Qian et al.) a (022). However, little attention has been paid to within the context of knowledge distillation (Fang al. 2021a). MosiacKD (Fang
eet al. 20 first proposes out-of-Domain Knowledge Distillation but their objective is fundamen-
tally different from ours. They use OOD data to assist source-data-free knowledge distillation and focus on in-domain performance. In contrast, we use OOD data for better out-of-domain perfor- mance. IPWD (Niu et al. 2022) also focuses on the gap between D; and D.. However, different from DOD-KD, they mainly solve the imbalance in teachers' knowledge Some studies discuss the domain shift problem in cross-time object detection (Li et al. 2022 Yang et al., 2022), but grant ac- cess to D.. which is impractical in real-world scenarios. These studies try to figure out the problems in the context of knowledge distillation. However, they either discuss a preliminary version of the problem lack rigor in their analysis. In summary, it is crucial to recognize that there is growing demand for solutions to OOD-KD, while the research in this area is still in its early stage.
3 PROBLEM FORMULATION
To illustrate the concept of Out-of-domain Knowledge Distillation, we focus on its application in image classification. In this work, the term "domain" refers to set of input-label pairs denoted as D {xx,, w.) Here, the input x, X C RC represents an image with H x W dimensions and C' channels, while the corresponding label is denoted as y E Y C 1.... .K- 1}= : [K), where K represents the number of classes. Vanilla KD methods guide the student model S(:; 0,) to imitate the teacher model T(::0, 0z) and learn from the ground truth label, formatted as:
0_ = arg min P(a.a)-P,, DKL 0,) |I 0,), + CE(S(I;0.).))
(1)
where CE refers Cross Éntropy, Ps is the joint distribution in D.. In the context of OOD-KD, the teacher domain D. and the student domain D, differ in terms of the joint distribution P(X, Y). For instance, in Dr, the majority of the images labeled as "cow" might depict cows on grassy landscapes. On the other hand, the ones in the student domain D, could show cows beaches or other locations. Unavoidably. T not only learns the class concept but also utilizes some spurious correlations (e.g., associating the background "grass" with the cow) to enhance its training performance.
However, as the occurrence of spurious correlations cannot be guaranteed in the target application, blindly mimicking the behavior of T is unwise. Hence, the key challenge lies in leveraging the
3

---

Published as a conference paper at ICLR 2024
Module I: Data-Free Learning
Ubberv Acthation Map
Legend
Imagge
Geanskk
Jigle
Fixed Modkle
Lamraiie Madio
image
Makkad
Tuinad Modio
Wise Maligiation
Stadent Madil
Module Anchor I Learning
Module Mixup Learning
Nomalls
Iragee
Teater Made (lanestalke)
Stador Dosads trages (M.P)
Stakas Danan Imaage (4,9)
Genenaer
Mhop Data trovider
(1- f-1
IT: Tayer
Insae
Ancherhii
Stadcet Modld
logkk
ued for the smullls louna dustibaion
Figure 2: Overview of our proposed method, consisting of three major modules.
teacher's knowledge effectively, accounting for the domain shift between D, and Dj (Zhang et al., 2022 2023a 2024a Bai al., 2024). Vanilla methods bridge this domain shift with the assistance T training data. However, in JOD-KD, due to various reasons (privacy concerns, patent protec- tion, computational resources, etc.), quite number of models are released without granting access to their training data and even some of the models are hard to adapt. This situation further amplifies the difficulty of the problem. Hence, we present the definition of OOD-KD herein.
Problem Definition: Given an immutable teacher model with its parameter 8g and labeled student-domain data D, = {(r,, } whose joint distribution P(X, Y) differs from that of D, but the label space is the same (Y, = Y.), the objective of DOD-KD is to train student model 0l))0)) only with access to D, and T, leaving the teacher model 7 unchanged in the overall process.
4 METHOLOGY
To address OOD-KD, we propose simple but effective method AuG-KD, Generally, AuG-KD is composed of three modules: Data-Free Learning Module, Anchor Learning Module, and Miuxp Learning Module, as is vividly shown in Figure In certain module, the green blocks inside are trained with the help of fixed red blocks. The overall algorithm utilizes these 3 modules sequentially. For space issues, we leave the pseudo-code of our overall method in Appendix A In the following sections, we provide detailed descriptions of each module.
4.1 MODULE 1: DATA-FREE LEARNING
To leverage 7"s knowledge without access its training data, DFKD methods are indispensable. This module follows the vanilla DFKD methods, training a Generator G(:;; 0,) + from normally-distribune
latent variable 2n .N(0,1) 1) under the instructions of the teacher model T. The generated image is denoted as æ G(zn; 0y), while the normalized version of it is 5 N(x). y/ arg max T 0.) means the labels predicted by T. The dimension of Z is denoted as Ng. H refers to the Information Entropy. AM stands for Activation Map, which observes the mean and variance of the outputs from BatchNorm2d layers.
(x) ( (z; 0.) T():0))
(2)
LCE(=0) (T;
(3)

---

Published as a conference paper at ICLR 2024
Lgenerator = B (Boo~N(0011)
LKL + LCB + ag H(T(;;))) + AM(T(2;
(4)
8g = aaggmmn,, Ligeerrator
(5)
Meanwhile, an additional encoder ((;;0): X,Y + 2 is trained, keeping 0g fixed. It absorbs the generated image z (22p; 8g) and the label y = arg max T T(;; input and outputs the related latent variable 2 = E(x, y; 0e) with Eq. 6 where MSE represents the mean squared error.
6. arg min Lencoder arg min E (o,))) MSE(= 2) a. DKI || 20] D.
(6)
0. When training the encoder, the student model S is trained simultaneously with Eq.
= arg min Lsstudeut = arg min No- 0,
0,
4.2 MODULE 2: ANCHOR LEARNING
Anchor Learning Module trains AnchorNet (m, th; 0a) to map student-domain data to the teacher domain. It consists of a class-specific mask mn(:0o))
+ (0,1)% and a mapping function (.) Z + Z, which trained concurrently in this module. and v are integrated into lightweight neural network AnchorNet as shown in Figure 2 Detailed implementations of them provided in Appendix [A] This idea draws inspiration from invariant learning (Creager et al. 2021 Kuang et al. 2018), which is proposed especially for the problem of domain shift. IRM (Arjovsky al. 2019) assumes the partial invariance either in input space or latent space, implying the presence of some invariant factors across domains despite the domain shift. In this work, we assume that a portion of the latent variable = exhibits such invariance:
Assumption Given any image pair ((x1. 31), (22, V2)) that is identical except for the domain- specific information, there exists class-specific binary mask operator ml=;0,) ())= Y ~ {0. 1). that satisfies the partial invariance properties the latent space under the Encoder (.)) X, X,Y
Z, as shown in Eq. The mask masks certain dimensions in the latent space to zero if the corre- sponding component in is set to or preserves them if set to
11 my: 0a)) @ E(I1, 0.) = (1 - m (y2: 0.)) @ E(a2, y2; 02;
(8)
@ in Eq is the element-wise multiplication operator. Assumption sheds light on the method of effectively transferring 7's knowledge: just to change the domain-specifie information. With it, we can obtain the invariant part in the latent space of an arbitrary data sample. we change the variant part, we can change the domain-specific information and thus can change the domain of the data sample to Dr. As a result, T can provide more useful information for distillation. To direct vi to change the comain-sspecife information and map the samples Dr. we introduce the uncertainty metric Ua: T) which draws inspiration from Energy Score (Liu et al. 2020 formulated as:
Ulx;T) = -t log
exp
where t is the temperature and (x()) denotes the 7th logits image x output by the teacher model T. U(x; T) measures T's uncertainty of an arbitrary image The lower the value of U U(:; T) is, the more confident is in its prediction.
To preserve more semantic information during the mapping, include the cross-entropy between T's prediction on the mapped image and the ground truth label in the loss function of AnchorNet, shown in Eq. 11 where x' G(=;;0,) and z' m(y: 6.) @ (1(;; 0a) + (1 m(y: 0.)) represent the resultant images and latent variables after mapping individually. We denote x' as "an- chor". These anchors are in the teacher domain D,. T' is hence more confident about its prediction on them and can thus provide more useful information for distillation.
For simplicity, the portion of invariant dimensions in is preset by Qg- Linv regulates it based on the absolute error between the ( no-m and the desired number of ones in the mask m.
Lin(yy) (v) [(1 (..) N2 ||ma(s; 0g 11
(10)
6,- arg min Lanchor arg min LL (x,,))>, (ux'; + Linv (v) + B. CE(T((((0).)))
(11)
0,
5

---

Published as a conference paper at ICLR 2024
T- 1-0.5
I- 0.5
[-0.75
[-1
Figure 3: Different mixup samples generated Module 3 for DSLR Office-31, controlled by the stage factor [0, 1]. The value of determines the proximity of the samples to D, and D.. A smaller value of f indicates that the samples are closer to the teacher domain Dr, while a larger value of indicates that the samples are closer the student domain Ds.
4.3 MODULE 3: MIXUP LEARNING
This module is process of knowledge distillation using D, and mixup data provided by AnchorNet. To be specific, for an arbitrary image x D.. Encoder E(:; 02) encodes it to 2 and AnchorNet (m, si; (a) maps it to z'. Mixup Learning utilizes the mapping from z' to to generate series of images that evolves during the training process. The evolution is governed by a stage factor f. which is given by monotonically non-decreasing scheduler function F(;; b) N + [0, 1]- Parameter a controls the rate the change of mixup images, while determines their starting point. These parameters adhere the property F(a- EEpoch; a, b) and F(0; a, b. where :Epoch represents the total number of training epochs. The mixup samples are formulated as shown in Eq. Figure 3 vividly illustrates the mixup samples provided.
12
(12) from D2 to D,. Consequently. at the beginning of training, the teacher model T exhibits more certainty regarding the samples and can provide more valuable knowledge. As training progresses, T becomes less certain about its predictions, which thus encourages the student model learn more from the student-domain data.
Im (1 f) G((1(() As the latent variable evolves from to 2, the mixup samples
+ fg ff
Table 1: Test accuracies (%) of distilling ResNet34 to MobileNet- V3-Small on Office-31. The row "Settings" implies the arrangement of domains. For instance, "Amazon, Webcam-<DSLK indicates that T is trained on Amazon and Webcam, S is to be adapted on DSLR. The first row of Teacher T's performance on D., while the second row is that on D,. The "+" mask signifies that these methods are fine-tuned on the student domain after applying the respective methods.
Office-11: resmet34 + mobilenet v3 small
Settings
Amazon, Wehcam - DSL.R
Amazon, DSLR Webeam
DSLR,
Acc
Acc0.
Ace@5
Acc
Acc@.
Acc@5
Acc
Acc@3
Ace
96.2
97.3
Teacher
67.1
82.6
88.0
60.0
77.5
82.5
15.2
26.1
36.0
DFQ+
80.4:::.7
93.3::..1
96.412.1
86.545777
97.54-2.0
99.0:::.0
46.61:
67.642.4
76.54:999
CMI+
67.1-t3.5
86.6:24.3
92.9=13.0
70.0:1:.3
88.0=5.11
94.3+:2.11
35.9
56.1-15.1
65.0+:5.6
Deeplav+
65.9:16.3
84.71499
90.6:13.88
70.0:1:.4
91.51005
9481:1
36.51:4.4
$6.11:5.1
66.31::333
w/o KD
90.2.13.7
96.0.1.9
98.311077
72.5..3.6
79.9
7SKT+
33.34:5.9
15.
65.9-111.5
33.0-18.11
14.3
66.811622
23.74:5.3
42.747.1
53.74-5.9
PRE-DFKD+
68.3:+19.55
87.8
91.8-11.3
66.5+:20.99
82.0=17.3
88.9112.9
28.4-:33
46.4-19.0
55.94:20.8
Ours
84.3331
94.9-12.6
97.6:10.8
87.817.66
96.341.88
99.5:10077
58.813377
73.7.12.1
79.74:1.5
EXPERIMENTS
5.1
EXPERIMENTAL. SETTINGS
The proposed method is evaluated on datasets Office-31 (Saenko et al. 2010). Office-Home Vennateswara al. 2017), and VisDA-2017 (Peng et al. 2017). These datasets consist of mul- tiple domains and are hence appropriate to our study.
Office-31 31 This dataset contains 31 object categories in three domains: Amazon, DSLR, and Webcam with 2817, 498, and 795 images respectively, different in background, viewpoint, color, etc.
Office-Home Office-Homm is a 65-class dataset with 4 domains: Art, Clipart, Product, and Real- World. Office-Home comprises 15500 images, with 70 images per class average.

---

Published as a conference paper at ICLR 2024
Table 2: Test accuracies (%) distilling ResNet34 to MobileNet-V3-Small on dataset Office-Home A, C, P, and R refer to Art, Clipart, Product, and Real-World respectively.
Office-Home: resmet34
mobilenet
Settings
ACP-R
ACR-P
APR-+C
CPR-A
Aec
3
Acc
Acc03
Ace85
Ace
Acet5
Ace
Ace@3
Acc85
Teacher
89.4
93.3
94.5
88.6
92.5
93.8
66.7
75.8
79.6
90.9
94.4
95.4
30.3
46.7
55.2
35.8
53.4
60.8
24.7
40.6
18.7
19.6
31.2
39.1
D
133
$1.7+114
60.77117
000238
758829
8188226
506228
61.7=28
75221.6
210034
318833
40.3225
CMI+
16.4+12
29.0+0044
$7.040.7
188+15
63.9
703+-166
16.3+-19
51:2=20
584+1.7
1344300
21.4-277
21.52288
Deeplaws
15.4::7
28.6:1188
36442.
47842.
62.9222
707422
369225
525514
60.529
130421
22.3-144
27.542.1
52.54:38
48.44:8
57.6333
599420
71:2114
8266288
4999:4
6700=16
13
168421
28.911
36442.5
7SKT-
15.5133
29.7:2:44
38.5444.5
119956
23.5295
12.02:099
784229
19:525.
2755667
78231
17.621
26.72:30
PRE-DFKD+
22.34377
36.925.0
44.955.2
$4.499
52.421122
60.5110
58.4779
17.9
6542111
9.427
20,42377
27.55:00
Ours
852:125
53.4:12.0
68
653311
793:114
84.1220
534130
70.3:144
76.614
212144
3344:38
41.71:46
Table 3: Test accuracies (%) of distilling ResNet34 to MobileNet- V3-Small on dataset VisDA-2017.
Methods adapted the validation domain and tested on the test domain. The first column of Teacher is the results in the train domain, while the second row is those in the validation domain.
Vi5DA-2017 (train validation): resmet34
mobilenet small
Settings
Teacher
DFQ+
CMI+
Deeplavt
w/o KD
ZSKT+
PRE- DFKD+
Ours
Acc
53,41:00
1.
47.610.9
50.71009
48.41135
$4.9110
0.35
100.0
34,5
80.2:10.6
77.2=111
75.5
78.7+.0.7
77.541.66
81.4
82.1-0.2
A
100.0
54.7
89.0=1044
88.1=007
87.320.6
89.3+:04
88.9:112
90.62:0.
91.3-0.11
VisDA-2017 VisDA-2017 is 12-class dataset with over 280000 images divided into 3 domains: train, validation, and test. The training images are simulated images from 3D objects, while the validation images are real images collected from MSCOCO (Lin et al. 2014).
Main experiments adopt ResNet34 (He et al. 2016) as the teacher model and MobileNet-V V3-SSmll (Howard et al. 2019) the student model. Usually, teacher models are trained with more data samples (maybe from multiple sources) than student models. To better align with real-world utilized for training the teacher model T, except for domain
scenarios, all domains that is reserved specifically for adapting the student model S. Since Office-31 and Office-Home do not have official train-test splits released, for evaluation purposes, the student domain D. of these two datasets is divided into training. validation, and testing sets using a seed. with proportions set 8:1:1 respectively. As to VisDA-2017, we split the validation domain into 80% training and 20% validation and directly use the test domain for test. The performance of our methods is compared with baselines using top 1, 3, and 5 accuracy metrics.
Given that OOD-KD is a relatively novel problem, there are no readily available baselines. Instead, we adopt state-of-the-arr art DFKD methods, including DFQ (Choi et al. 2020), CMI (Fang et al.) 202 DeepInv (Yin et al. 2020). ZSKT (Micaelli & Storkey 2019 and PRE-DFKD (Binici et al. 2022). and fine-tune them on the student domain. One more baseline "w/o KD" to train the student model S without the assistance of T, starting with weights pre-trained on ImageNet (Deng cct al. 2009). To ensure stability, each experiment is conducted five times using different sceds, and the results reported as mean + standard variance.
Due to limited space, we leave hyperparameter settings, full ablation results, and combination with other baselines (like Domain Adaptation methods) to Appendix B Bland C
5.2 RESULTS AND OBSERVATIONS
Our main results are summarized in Table 2 and 3 Extensive experiments solidly substantiate the stability and superiority of our methods. this section, we will discuss the details of our results.
Larger domain shift incurs larger performance degradation. It is evident that all the teacher models experience significant performance degradation when subjected to domain shift. The extent of degradation is directly proportional to the dissimilarity between the student domain D, and the teacher domain D, For instance, in Office-Home Art is the most distinctive domain, it is signif- icantly different from other domains. As a result, in the CPR setting, the performance of the teacher model T exhibits the largest decline, with approximate 70% drop absolutely. The same
7

---

Published as a conference paper at ICLR 2024
phenomenon can be observed in VisDA-2017, where the train domain is 3D simulated images, but the others are real-world photographs. Moreover, the problem of performance degradation can be amplified by the imbalance amount of training data between T and S. Usually, we assume that teacher models are trained with more data samples. When the assumption violates, like DW > A in Office-11, where the Amazon domain is larger than the sum of other domains, the issue of performance degradation becomes more prominent.
DFKD methods are unstable but can be cured with more data samples. It is worth noting that the standard variance of each method, in most settings, slightly high. This observation can be attributed to both the inherent characteristies of DFKD methods and the limited amount of data for adaptation. As DFKD methods train a generator from scratch solely based on information provided by the teacher model, their stability is not fully guaranteed. Although the remedy to it goes beyond our discussion, it is worth noting that as the amount of data increases (Office-31 (5000) to VisDA- 2017 (28000)), these methods exhibit improved stability (Office-31 (7.6) to VisDA-2017 (0.3)).
AnchorNet DOES change the domain of data samples. To make sure AnchorNet does enable T to provide more useful information, we observe the mixup data samples in Mixup Learning Module under the setting Amazon, Webcam - DSLR (AW + D) in Office-31, 1, shown in Figure These domains exhibit variations in background, viewpoint, noise, and color. Figure | gives a few exam- ples in Amazon (the right up) and DSLR (the right bottom). Obviously, Amazon differs from other domains - the background of the samples in it is white. In AW + D, when f = O the images are closer to D., with white backgrounds (Amazon has more samples than Webcam). As f goes larger, the mixup samples get closer to D,, depicting more features of DSLR.
5.3 ABLATION STUDY
To further validate the effectiveness of our methods, we perform ablation experiments from three perspectives: Framework, Hyperparameter, and Setting, In line with our main experiments, each experiment is conducted five times with different seeds ensure the reliability of the results. For simplicity, we focus on Amazon, Webcam + DSLR setting in Office-31.
(a) Ablation study on the framework our method.
Framewurk Ablation: Amazon, Webcam + DSLR
(b) Ablation study (Ace) on different + S pairs.
Ace
Acc@3
Acc
Setting Ablation:
Amazon, Webcam - DSLR
33.755.1
56.1188.1
68.62:311
pair
(34-
550 - r18
(34
(34
MI-M2+MB Mixup)
46.47 =0.66
80.1
36.2f 14.0)
92,3444.1
27.5 +100
96.4-2211
w/o KD
DFQ+
8044::57 63.54:79
8433--4.8
87.51:577
79.6-1-.8
86.11224
85.14-4.7
90.11:6
83.514411
94.5119
96.12:200
PRE-DFKD+
69.3
79.115566
87.9:15.1
83.91:19
MI+ME
341 11.61
3.21 #1:2
0.3. 1014
Oars
84.3-:311
88.2--4.3
88.61-..1
91.8::3.5
MI-M24M3
0.87 1:103
84.313311
DAC 4:0.5)
940426
1,t +1.14
97.62088
5.3.1 FRAMEWORK ABLATION
Here, we evaluate the effectiveness of our modules. Framework ablation studies traditionally involve masking parts of the proposed modules for experimental purposes. Yet, it is essential to recognize: 1. Module 1 is fundamental to our method and is non-removable; 2. Module 2 serves to support Module There is no need to test the results only with Module 1&2. Hence, our investigation focuses on the outcomes absent Module 2, and absent both Module 2 and 3, denoted as M1+M3 and M1, respectively. Additionally, our analysis dives into Module 3, where we omit the mixup samples to evaluate their critical role, denoted as M1+M2+M3 (w/o Mixup). It's worth noting that there is no need to add one more setting w/o M2 & Mixup here since it makes no difference to M1+M2+MB (w/o Mixup). Consequently, we get three distinct ablation scenarios: MI, MI+M3, and M1+M2+M3 (w/o Mixup). To be specific, in MI, we directly choose S's best checkpoint in Module 1 and test it on D.- In M1+M2+M3 (w/o Mixup), the model trains solely D.. In MI+M3, we mask AnchorNet by equating its output 2' with its input and then proceed with the method.
The results are presented in Table The performance improvement between MI and M1+M2+M3 (w/o Mixup) mainly stems from the supervision of D,. As MI is a simple DFKD, the striking per- formance gap underscores the urgent need for solutions to OOD-KD. The considerable enhancement
8

---

Published as a conference paper at ICLR 2024
101
4001
10h
150
XM
120
1.00
1/0
200
120
100
.00
(a) Acc@3 on b, a is fixed individually.
(b) Acc on a, is fixed individually.
4471
000
440
679
000
679
101
400
E00
6:00
380
03
(c) Acc@5 5 on b, a is fixed individually.
(d) Acc@5 on o, is fixed individually.
Figure 4: Grid study on hyperparameter a and b in Module 3. The red line b = 1.0, meaning no mixup data. The blue line portrays the performance of various b settings. The light blue area symbolizes the range encompassing mean std.
evidences the efficacy of Module and remedying domain shifts. The rise in average accuracy coupled with reduced variance firmly attests to the significance of each component in our method.
5.3.2 SETTING ABLATION
In this study, we change the T - S pair in our experiments. We additionally employ ResNet50 - ResNet18 (r50 r18), ResNet34 ShuffleNet-V--X0-- (r34 - sf) and ResNet34 EfficientNet- BO (r34 + ef) in this study. The ResNet50 - R8sNet18 pair is a commonly used evaluation pair in traditional distillation methods, while ShuffleNet and EfficientNet are well-known lightweight neural networks suitable for edge devices. These pairs are compared with several effective baselines in our main experiments. The results this study are displayed Table [4b] which confirm the effectiveness of our methods across different teacher-student distillation pairs.
5.3.3 HYPERPARAMETER ABLATION
In this ablation study, our primary focus is on two hyperparameters, namely a and b in Module 3, which govern the speed and starting point of the mixup data samples. We perform grid study on the values of and within their domain [0, 1], with a step size of 0.05. Since = 0 is useless but causes the division-by cisiion-byzzer by-zero problem, we set the minimum value of a to step size 0.05.
Detailed results are depicted in Figure Due to the limited space, we present only portion of assignments here, with more results included in Appendix C The red line in the figures represents the baseline, wherein no mixup data but only raw images are provided. Notably, the blue line consistently surpasses the red line over the majority of the range, testifying to the effectiveness of our method. Both Figure 4a and 4c demonstrate slight decrease in performance b increases, suggesting that an excessively large assignment of is not preferred.
CONCLUSION
In this work, we dive into the problem of transfer teachers' proper knowledge to students. Further, propose a simple but effective method AuG-KD. It utilizes a data-driven anchor to align student-domain data with the teacher domain and leverages generative method to progressively evolve the learning process from OOD knowledge distillation to domain-speciin and superiority of our approach. However, it is worth emphasizing that the research on OOD-KI still its early stages and considered preliminary. Therefore, we encourage further attention and exploration in this emerging and practical field.
Knowledge Distillation to selectively
information learning. Extensive experiments validate the stability

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Science and Technology Major Project (20222D0119000 the National Natural Science Foundation of China (62376243, 62037001, U20A20387), Scientific Research Fund of Zhejiang Provincial Education Department (Y202353679), and the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010),
REFERENCES
Martín Arjovsky, Léon Bottou, Ishaan Guirajani. and David Lopez- Paz. Invariant risk minimization. CoRR, abs/1907.022 .2293,, 2019. URL http:://aax: org/ abs 02893
Shuanghao Bai, Min Zhang, Wanqi Zhou, Siteng Huang, Zhirong Luan, Donglin Wang, and Badong Chen. Prompt-based based distribution alignment for unsupervised domain adaptation. In Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI 2024). AAAI Press, 2024.
Kuluhan Binici, Shivam Aggarwal, Nam Trung Pham, Karianto Leman, and Tulika Mitra. Robust and resource-efficient data-free knowledge distillation by generative pseudo replay. In Thirty- Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on In- novative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium Educa- tional Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 6089 6096. AAAI Press, 2022. URL pattps://ojpp aaai. org/ berggiinder index.nii.. php/ AAAI/
art secicle/viesi ew 20056
article
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Tina Eliassi- Rad, Lyle H. Ungar, Mark Craven, and Dimitrios Gunopulos (eds.), Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadel- phia, PA, USA, August 20-23, 2006, pp. 335-541. ACM, 2006. doi: 10.145/1150402..15444.. 1150464
URL https: //do:///oiiio org/ 1145/
Nitay Calderon, Subhabrata Mukherjee, Roi Reichart, and Amir Kantor. A systematic study of knowledge distillation for natural language generation with pseudo-target training. In Anna Rogers, Jordan I.. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 14632-146599 Association for Computational Linguistics, 2023. URL http::::///aaaahhooooaoa
wwg//c23. acl long. 818
Hanting Chen, Tianyu Guo, Chang Xu, Wenshuo Li, Chunjing Xu, Chao Xu, and Yunhe Wang. Learning student networks in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 6428- 6437. Computer Vision Foundation / IEEE, 2021. doi: 10. 10.1099CVVPR46437.2222 00636. URL Stto__tttteettettrsst_aata
thevv com http://tteeenttt///1///////
Chen wacterrinn__Stuudee
Networks Sttt__th___________________ee___ wwild__ciiee 202P _apper__ppp_.__ .tml
Yoojin Choi, Jihwan P. Choi, Mostafa Kl-Khamy, and Jungwon Lee. Data-free network quantization with adversarial knowledge distillation. In 2020 JEEE/CVF Conference on Computer Vision and
Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 3047- 200.99080220006663 CVPRW 2020/htm Adversarial Knowledge_
3057. Computer Vision Foundation / IEEE, 2020. doi: 10.1 109/C URL F
Choi Data Free
Network http:/ooo__uaniiiatiriaae Distillation_Cver 2twp_22200pppehhhe_1 html
Elliot Creager, Jörn-Henrik Jacobsen, and Richard S. Zemel. Environment inference for in- variant learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Interna- tional Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 2189-2010 2200. PMLR, 2021. URL html
http:://proceedingss ml wrw.sprssssee vv33://vvv///c// http://///reeee2///
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Lcage-Scale Hierarchical Image Database. In CVPR09, 2009.
10

---

Published as a conference paper at ICLR 2024
Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. nource-free domain adaptation via distribution estimation. In /EEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 7202-7212 IEEE, 2022. doi: 10.1 ww9p///PP222..22022222.
URL ptl.http:: | 1tt.:////0..111/////5522226222
Kien Do, Hung Le, Dung Nguyen, Dang Nguyen, Haripriya Harikumar, Truyen Tran, Santu Rana, and Svetha Venkatesh. Momentum adversarial distillation: Han- dling large distribution shifts in data-free knowledge distillation. In NeurlPS, 2022. URL http:://oapes..nnattee/ cetppp/peereppp..cppppeeeeeeee http://aapr___eee__eeeeeeeeeeeee hallsspaperr2222/ 411
Cobbbtrrcct-Coffeeeneee html
Gongfan Fang, Yifan Bao, Jie Song, Xinchao Wang, Donglin Xie, Chengchao Shen, and Mingli Song. Mosaicking to distill: Knowledge distillation from out-of-domain data. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (cds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 11920-11932, 2021a. URL htpp:://p///rro www.gggnn...r..... (ttp:///aa.00030022555544aaaa300
-ww.addd-bbstttt...ommm html
Gongfan Fang, Jie Song, Xinchao Wang, Chengchao Shen, Xingen Wang, and Mingli Song. Con- trastive model invertion for data-free knolwedge distillation. In Zhi-Hua Zhou (ed.), Proceed- ings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Vir- tual Event / Montreal, Canada, 19-27 August 2021, pp. 2374 ijcai.org, 2021b. doi: 0..44663/jcai.2221/333 (tItttsss: doi org/ 0 9633 24433/1jccii.... /tt.eaa..2022332 2021/327
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778 IEEE Computer Society, 2016. doi: 10.1109/CVPR.201622
CoRR, Abs/1503.02231, 2015. URL BoLLhhtps://doii.
11w.//VPR....... 2016. 90
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. .rg.
org abs 111./115300.225
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chiee Chich Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Search- ing for mobilenetv3. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1314-1324. IEEE, 2019. doi: 100.0004... htl.ttts:: tittts::/cot //ooi. org 10. 1 109 ICCV 2019 9. 00140
Jaxxig Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical con- trastive learning for unsupervised domain adaptation without source data. In Marc" Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 3635-3649) 2021. URL https: httpss::/pocceeeddeeee neuries tttccc//c/ee wwtppppr///aeecppcccc 22/2//222//11 2011/hash//hh/h /hash/ 083.55eeee33885558 wwweeec80affffff 1.80af1411880 84e5 Abst- htm
Nazmul Karim, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi. Han-pang Chiu, Supun Sama- rasckera, and Nazanin Rahnavard. C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24120-24131 June 2023.
Sunok Kim, Seungryong Kim, Dongbo Min, Pascal Frossard, and Kwanghoon Sohn. Stereo con- fidence estimation via locally adaptive fusion and knowledge distillation. IEEE Trans. Pat- tern Anal. Mach. Intell., (33)))66726338833 2023. doi: 10. 1..10/9AM....2222277766. URL
http:///...o//////////// 1tt.9/10...099PPAAAAMMM TPAMI. 2022. 3207286
Alex Krizhevsky, Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL wtw.:/eaaaturee2000-r........
https: httt:///www... cs wwwottorroooo.oo
11

---

Published as a conference paper at ICLR 2024
Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown envirouments. Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference
Knowledge Discovery & Data Mining, KDD 2018, London, UK, BRRhttss:
August 19-23, 2018, pp. 1617-1626. ACM, 2018. doi: 10.1 10.114533219819.20 7/doi. org. 10. 11 9, 3220082
Jogendra Nath Kundu, Suvaansh Bhambri, Akshay R. Kulkarni, Hiran Sarkar, Varun Jampani, and R. Venkatesh Babu, Concurrent subsidiary supervision for unsupervised source-free domain adaptation. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX, volume 13690 of Lecture Notes in Com- URL
puter Science, pp. 177-194. Springer, 2022. doi: 10000/978-33003-2005555 https: wtt.//do...oo
Jingru Li, Sheng Zhou, Liangcheng Li, Haishuai Wang. Jiajun Bu, and Zhi Yu. Dynamic data- free knowledge distillation by easy-to-hard learning strategy. Inf. Sci., 642:119202, 2023, doi: 10.1016/j ino..//..i........... 202.3 19202. URL
11000.1111110 ins 2023.119202
Miaoyu Li, Yachao Zhang, Yuan Xie, Zuodong Gao, Cuihua Li, Zhizhong Zhang, and Yanyun Qu Cross-domain and cross-modal knowledge distillation in domain adaptation for 3d semantic seg- mentation. In João Magalhães, Alberto Del Bimbo, Shin ichi Satoh, Nicu Sebe, Xavier Alameda- Pineda, Qin Jin, Vincent Oria, and Laura Toni (eds.), MM '22: The 30th ACM International Con- ference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, pp. 10.1145/3503161.33322 3554999.. URL http::://////t //ts:://dooo /oi. org.or/////// 10. 11 35031 3347990
Shin'ichi
ACM, 2022. doi:
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 6028-6028 PMLR, 2020. URL http://proceeddnng mll.. press / vl1 9/1iang20a.... aatmamn20aa..mll11...
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Trans. Pattern Anal. 8617, 2022. doi: 10. 10.1009/......21....11
Mach. Intell.,
URL https:
//doi. 10. 11 1001 TPAMIA 2021 310339C
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014
13th European Conference, Zurich, Switzerland, September 1-12, 2014, Proceedings, Parí V.
volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014, doi: 1007/ 978-3-3199 1000-10002-11 11.48.
URL http:::/oo....cr/00000000////// 319-
48
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based detection. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020. /tt.:///.5eeebbbaa14477aaaa3333.
neurips htce/paper/222///aaeo
html
Yifei Liu, Yiquan Wu, Yating Zhang, Changiong Sun, Weiming Lu, Fei Wu, and Kun Kuang. ML- LJP: multi-law aware legal judgment prediction. In Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang, Makoto P. Kato, Josiane Mothe, and Barbara Poblete (eds.), Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pp. 1034. ACM, 2023. doi: 011144//539688.3591777 URL htt.hhttss:/ doi. wrw.orgg110. 10100.1145 145 3339613336611 8. 8..111..55993 3591731
Zheqi Lv, Feng Wang, Shengyu Zhang, Kun Kuang, Hongxia Yang, and Fei Wu. Personal- izing intervened network for long-tailed sequential user behavior modeling. arXiv preprint arXiv:2208. 09/30, 2022.
12

---

Published as a conference paper at ICLR 2024
Zheqi Lv, Wenqiao Zhang, Shengyu Zhang, Kun Kuang, Feng Wang. Yongwei Wang, Zhengyu Chen, Tao Shen, Hongxia Yang. Beng Chin Ooi, et al. Duet: A tuning-free device-cloud collabo- rative parameters generation framework for efficient device model generalization. In Proceedings of the ACM Web Conference 2023, pp. 3077-3085, 2023.
Zheqi Lv, Wenqiao Zhang, Zhengyu Chen, Shengyu Zhang, and Kun Kuang. Intelligent model update strategy for sequential recommendation. In Proceedings of the ACM Web Conference 2024, 2024.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Vittorio Ferrari, Martial Hebert, Cristian Sminchis- escu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Mu- nich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes Springer, 2018 doi:
00109997883-0000122------ https: /owii doii org/ 10. 1007 978 03000 01264-9226299
in Computer Science, pp.
URL
Paul Micaelli and Amos J. Storkey. Zero-shot knowledge transfer via adversarial be- lief matching. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' -'lché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9547 9557, 2019. URL httpsss/pprorccddogoooooo neurips http://ss..eeaaaooaoooo///////// cc/papper/219/haat. 2 fe66a3772272b6011111111111666661
html
Yulei Niu, Long Chen, Chang Zhou, and Hanwang Zhang. Respecting transfer gap in knowledge distillation. In NeurlPS, 2022. URL Utt.hctpp://appers.n.pp.
html
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/9, Long Beach, CA, USA, June 16-20, 2019, pp. 3967-30 -976. Computer Vision Foundation IEEE, 2019. doi: 10.1 URL CVPR.2019.000 lat onal Know hwwoov.edeeeDDssiillaiiooo...... http:/PPar_Relaaiinneera
Gaurav Patel, Konda Reddy Mopuri, and Qiang Qiu. Learning to retain while acquiring: Combating distribution-shift in adversarial data-free knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 7786-7794, June 2023.
Jiangho Pei, Zhuqing Jiang, Aidong Men, Liang Chen, Yang Liu, and Qingchao Chen. Uncertainty- induced transferability representation for source-free unsupervised domain adaptation. 1EEE Trans, Image Process. 32:2033-2048, 2023. doi: 10.1109/TIP2023335555... URL https:
7/doi. 10. 1109/TIP 2023. 0...2...2880. 3258753)
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.
Visda: The visual domain adaptation challenge. CoRR, abs/1710.06924, 2017. URL http:
//arxiv. wawlvvorg/abbs. /1tp///1111..m 06924
Xufeng Qian, Yue Xu, Fuyu Lv, Shengyu Zhang, Ziwen Jiang, Qingwen Liu, Xiaoyi Zeng, Tat- Seng Chua, and Fei Wu. Intelligent request strategy design in recommender system. In KDD '22 The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3772-3782. ACM, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- chine Learning Research, pp. 8748-8763. PMLR, 2021. hRRhhtp: / mmr. ftess//139/radffoft
html
13

---

Published as a conference paper at ICLR 2024
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
6550
Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Amoo Solin. Uncertainty-guie source-free domain adaptation. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.). Computer Vision ECCV 2022 17th European Conference, Tel Avię, Israel, October 23-27, 2022, Proceedings, Part XXV, volume 13685 of Lecture Notes in Computer Science, pp. 537-555 Springer, 2022. doi: 10. 978-3-031- 19.00011-19069)..11 31. URL https: /doi org/ 000.0.0000000000 1001100079907 3 031 19806-10-- 9
31
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category mod- els to new domains. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios (eds.), Com- puter Vision ECCV 2010, 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV, volume 6314 of Lecture Notes in Com- doi: 10. 1000/777 3- 642- 15561-1),16. URL
puter Science, pp. 126. Springer, 2010.
https:// org / 10. 1007/9 --1-8-8---22111-----1-- 16
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpps://ppenevveew met/foruue www.mndddryxxuxxxv
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105 6114. PMLR, 2019. URL
mmrr wwwpss/vve// tanl 9a. html
Zihao Tang, Zheqi Lv, Shengyu Zhang, Fei Wu, and Kun Kuang. Modelgpt: Unleashing Ilm's capabilities for tailored model generation. arXiv preprint ar Xii::2402.124000 2024.
Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, and Nicolas Papemot. Data- free model extraction. In IEEE Conference on Computer Vision and Pattern Recog-
nition, CVPR 2021, virtual, June 19-25, 2021, pp. 4771-4780 Computer Vision
Foundation / IEEE, 2021. doi: 10.1 0..1090CVPR44437.22110000 004744. URL https:
//openaccess.co thecvf. com/ www.tttnt//VP/20////ttmm///////e Mode 1attell_xxtttttttteeee oolooo___ooo wt_eeiiin________ee 20ae222221___ppp___1 html
Data -
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, (86):2577 2008 URL http::// fmwr. wttpppppes/.v/g/////
mardeermattennaa.co 1
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In 2017 JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu. HI, USA, July 21-26, 2017, pp. 5385- 5394. IEEE Computer Society, 2017. doi: 10. 10.1109/CVPR.2017..22.. URL wtttt::///// //doi.
org #rg/10.10 11..0.110/CC// CWRR. SWW.PP..201....7
Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, and Shiqiang Yang. Community preserving network embedding. In Satinder Singh and Shaul Markovitch (eds.). Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence. February 4-9, 2017, San Francisco, California, USA, pp. 203- 209. AAAI Press, 2017. doi: 10. (0.1609/AAALV311) 10488. URL httpp::/
doi www/1orr/10... 1 609 aaai v3111.1.1.. 10488
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbal- anced data classification. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2. 2019, pp. 5016-5000000 5025. IEEE, 2019. doi: https: //o.i .rg /0. 1 1109/ TCCV, 201 9.9.9 00551
10. 11110/CC.20019000
14

---

Published as a conference paper at ICLR 2024
Yuzheng Wang, Zhaoyu Chen, Dingkang Yang. Pinxue Guo. Kaixun Jiang, Wenqiang Zhang, and Lizhe Qi. Model robustness meets data privacy: Adversarial robustness distillation with- out original data. CoRR, abs/2303. 11611, 2023a. doi: 10448500aXX1..2033.11111 URL
https://doo wtw.:ioorgg1.... 220 11611
Yuzheng Wang, Zhaoyu Chen, Jie Zhang, Dingkang Yang, Zuhao Ge, Yang Liu, Siao Liu, Yunquan Sun, Wenqiang Zhang, and Lizhe Qi. Sampling distill: Knowledge transfer from open- world data. CoRR, abs/2307. 1it/22377.16611, 2023b. doi: (0.4850/ARXXI.230.6660000 URLhttps:
org/10. 4 4000.08850/rra 2307. 16601
Lijin Yang, Yifei Huang, Yusuke Sugano, and Yoichi Sato. Interact before align: Leverag- ing cross-modal knowledge for domain adaptive action recognition. In IEEE/CVF Confer- Computer Vision and Pattem Recognition, CVPR 2022, New Orleans, LA, USA, June IEEE, 2022. doi: 10.1109/VVPR52688.2000 20222.1131.. URL
18-24, 2022, pp.
https: wttp:/oooooooooo O. 1109 CWWRS268RR228 2222. 01431
Shiqi Yang. Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Exploit- ing the intrinsic neighborhood structure for source-free domain adaptation. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 29393-29405, 2021. URL
neurriripppsssnnspp......... cc/ wttpc//ppppp/211111 wat.teegttBbbbaaat......o...aa.e html
hash/ tass:/55eceeee33SEcc4555222e22
Yingguang Yang. Renyu Yang. Hao Peng. Yangyang Li, Tong Li. Yong Liao, and Pengyuan Zhou. Fedack: Federated adversarial contrastive knowledge distillation for cross-lingual and cross-model social bot detection. In Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Car- los Castillo, and Geert-Jan Houben (eds.). Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX. USA, 30 April 2023 4 May 2023, pp. 1314-1323, ACM, 2023, doi: 10.1 10.1145//543507:35835500 URL https: //o.i. org/10 1 041434435/377777 3583500
Hongxu Yin, Pavlo Molchanov, José M. Álvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 8712-8721. Computer Vision Foundation IEEE, 2020. doi: 0101099CYYR24220022000000000
URL
/ttp::://oppccccscseeeee thecvf.
ceam/content_coo CVPR_ 2020 2ww.22200002/// Bttp//Yii_Deeammmaeeee Dreamirinnnn tariiinnttttoooo to Distill Data-Free. PatePrreea
via www.Deeeeeeeiiioooo CVPR______________ CVPR_002 pwpprpppp......... html
Min Zhang, Siteng Huang, Wenbin Li, and Donglin Wang. Tree structure-aware few-shot image classification via hierarchical aggregation. In Proceeding of the 17th European Conference on Computer Vision, ECCV, 2022.
Min Zhang, Junkun Yuan, Yue He, Wenbin Li, Zhengyu Chen, and Kun Kuang. Map: Towards balanced generalization of iid and ood through agdel-agm adapters. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11921-11931, 2023a.
Min Zhang, Haoxuan Li, Fei Wu, and Kun Kuang. Metacoco: A new few-shot classification bench- mark with spurious correlation. In Proceedings of the Twelfth International Conference on Learn- ing Representations (ICLR 2024)., 2024a.
Shengyu Zhang, Tan Jiang, Tan Wang, Kun Kuang, Zhou Zhao, Jianke Zhu, Jin Yu, Hongxia Yang, and Fei Wu. Devlbert: Learning deconfounded visio-linguistie representations. In MM '20: The 28th ACM International Conference on Multimedia, pp. 4373-4382 ACM, 2020.
Shengyu Zhang, Fuli Feng, Kun Kuang, Wenqiao Zhang, Zhou Zhao, Hongxia Yang, Tat-Seng Chua, and Fei Wu. Personalized latent structure learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b.
Shengyu Zhang, Qiaowei Miao, Ping Nie, Mengze Li, Zhengyu Chen, Fuli Feng, Kun Kuang, and mettarrpprssnttatioss for target-unknown cross-
Fei Wu. Transferring causal mechanism domain recommendation. ACM Trans. Inf. Syst., 2024b. doi: 10.1145/3643807. https:
7/doi. htt../////...11555.1...
Just Accepted.
15

---

Published as a conference paper at ICLR 2024
Wenqiao Zhang, Zheqi Lv. Hao Zhou, Jia-Wei Liu, Juncheng Li, Mengze Li, Siliang Tang, and Yuet- ing Zhuang. Revisiting the domain shift and sample uncertainty multi source active domain transfer. arXiv preprint a7X1v2331.11905, 2023c.
16

---

Published as a conference paper at ICLR 2024
A METHOD DETAILS
Our proposed method consists of three main modules. Data-Free Learning Module serves as the cornerstone of the entire approach. It trains teacher domain D,'s s generator G(:; Z X and encoder (1;; 0e) X, Y + Z and warmups the student model S in advance. In Anchor Learning Module, mapping established within the latent space Z that aligns the distribution of D, to that of D, taking into account the uncertainty metric provided by T'. Finally, in Mixup Learning Module, the mapping obtained in Anchor Learning Module is employed generate synthetic data of Dr and mixuped with D.. The pseudo-code of our proposed method is displayed in Algorithm)
Algorithm 1: Pseudo-code of our proposed method
Input: Student domain data Ds, Batch size b, Latent size N₂
Output: Optimized 5(::0s)
Learning Module, trains G( B(; Sample vvll from normal distribution, with size (10b, N₂);
x D
for i + 1 to tepoch do
Sample 20 from normal distribution with size (b, N2):
Compute /'generator and update 0g:
for + to do
Compute Lencoder Latudeut and update 0.. O;;
end
Evaluate G(: ((: with 2val, the best parameter; end
Anchor Learning Module, trains (m, 0a) for i +- 1 to tepoch do
for (x, y) E D training set do
E(x, y; 0,);
+ m(y 0.) O (2; Ba) + (1- m(y; 0a)) 2
x' 6g) end
Compute Lanchor and update 0g
Evaluate (m, sh; 02) with D,'s 's validation set, save the best parameter;
end
Mixup Learning Module, trains S(: for i + 1 to depoch do
for (Ir, v) E D, 's training set do
f F(i 1; b);
Im < - (1- (1
+ f. x;
(x. v) 4- (oll=m)llly)/ Concatenate two batches
Compute Isttudet with newly get (æ, y) and update A
end
Evaluate S(::0,) with D,'s validation set, save the best parameter;
end
In Anchor Learning Module, we integrate the mask operator m and the mapping function i into lightweight neural network AnchorNet. Concretely, the network is implemented with Pytorch as shown in Code[A] In forward pass, we first embed the class label to get the class-specific mask and then map the latent variable back to D,. To retrain we combine the mapped latent variable with the original one with the help of the class-specific mask.
information during mapping,
class AnchorNet (nn Module) **"AnchorNet
Args:
latent_siie size (int): Latent dimensionality
num_classes (int): Number of classes
17

---

Published as a conference paper at ICLR 2024
The AnchorNet module zakes an input tensor and a label tensor as input.
It embeds the class labels, generates mask based on the embedding ng, masks the input, and passes it through CNN module.
The CNN module consists ID convolutional and linear layers.
The weights are initialized from a uniform distribution in init
The forward pass:
1.
Embeds class labels
2. Generates mask from label embedding
3. Masks input tensor
4.
Passes masked input through CNN module Returns masked output and mask tensor
5.
def init__ssef,, satent_size: int, num_classes: int): super (). init_
selffnum_classes num_classes
= nn.Linear (num_classes, latent_size)
gelf.mask = Sequential
nn. Linear (latent_size, latent_size),
nn Linear (latent_size, latent_size),
nn.Linear (1ateet_sizeete latent_sizer
nn.Sigmoid
Lambda (lambda x: 0.5), nn. Sn.ooftignn() (), nn.ReLU()
self module = nn Sequential V1ew -1),
(1, 4, 3, 1, 1),
m.BBatchNomld((4)
In. (),
(n 8, 3, 1, 1),
m.Bttchhormld((),, nn. I. nn Convl 4, 3, 1, 1),
nn BattchhNNorld(1), nn LeakyReLUU)) View(-1),
nn. Liner (4 latent_size, latent_size),
... (initializations)
def forward (self, inputs:
Tensor, **kwargs) -> Tuple [Tensor, Tensor]
= weelf.eeeed_ci clabedd_lassssee (one_hot (kwargs['labell) mask self.mase (y)
(' lalablls' ], self. wum.muummee num_cllsssss)
in inputs mask
z = self. self.module module masked_inputs)
return masked _inkut_inppusseee z + (1 masked _nneuts) inputs, mask
18

---

Published as a conference paper at ICLR 2024
In Mixup Learning Module, the generation of mixup samples is controlled by monotonically non- decreasing scheduler function F(;; a, b) N - [0,1], which is parameterized by a and b. Parameter controls the rate of the change of mixup images, while b determines their starting point. These parameters adhere to the property F(a- FFocch; a, b) = and F(0; a. b) b. The idea of scheduler function draws inspiration from Curriculum Learning (Wang et al. 2019). All of experiments directly adopt the simplest linear scheduler fuunction:
1 min(max(x,, 0), [Epoch) + b
F(2za, =
:Epocc
B EXPERIMENT DETAILS
Each experiment is conducted using single NVIDIA GeForce RTX 3090 and takes approximately 1 day to complete.
B.1 HYPERPARAMETERS AND TRAINING SCHEDULES
We summarize the hyperparameters and training schedules of AuG-KD on the three datasets in
Table
Table 5: Hyperparameters and training schedules of AuG-KD.
Datasct
Parameters
Setting
GPU
NVIDIA GeForce RTX 3090
Optimizer
Adam
Learning Rate (except Encoder)
Learning Rate (EncoderiGPT
le
Offic-31
Office-Home
Ng
256
VisDA-2017
Image Resolution seed
32x32
2022,,,022...,,,..00 .2205)
Og
20
0,00022
On
0.25
0.1
Notably, the temperature of the KI--divergence in Module 3 is set to 10. As to baselines, we adopt their own hyperparameter settings. During the fine-tuning stage of each baseline, the standard setting involves 200 epochs, with a learning rate of 1e-3 and weight decay of le-4. Slight adjustments for optimal results are granted.
Module is determined by two significant hyperparameters a and b, which control mixup data's evolution speed and starting point. In the section of the ablation study, have demonstrated that most - b settings are effective. For reproducibility, we provide detailed a assignments in our main experiments 6
Table Detailed a b assignments in our main experiments. The column Setting gives the domains T and S use. In Office-31, means Amazon, W means Webcam, and D means DSLR individually. In Offec--Home, A means Art, means Clipart, means Product, and R means World indi- vidually.
a Setting in Main Experiments
Dataset
Setting
b
AW-D
0.6
0.2
Office-31
AD-WW
ACP-RR
Ofice-Home
ACR-
0.8
0.2
APR->C
0.8
0.2
CPR A
0.8
0.2
VisDA-2017
19

---

Published as a conference paper at ICLR 2024
color, symbol
o, circle
1, square
2, diamond
0.8
3, triangle-up
4, star
star-square
Second t-SNE
0.6
6, circle
7, square
8, diamond
9, triangle-up
10, star
11, star-square
12, circle
0.2
13, square
14, diamond
0
0.2
0.4
0.6
0.8
First t-SNE
Figure 5: Visualization Results
C ADDITIONAL RESULTS
C.I VISUALIZATION ON MASK PROVIDED BY ANCHORNET
In Module 2, AnchorNet integrates the mask operator m and the mapping function & into lightweight neural network. The mask class-specific and plays a crucial role in retaining domain-invariant knowledge in the latent space. To vividly demonstrate the effectiveness of the mask, we conduct t-SNE (van der Maaten & Hinton 2008) on the latent variables and the masked version of them. To be specific, we use AnchorNet and Encoder trained under the setting AW-D Office-31 and select 32 images for each class (31 classes in total). For each image (x, y), we encode it to get the latent variable
Exx; 02), and obtain the class-specift mask m(y; 9²). Next, we obtain the masked latent variable 2' = (1 my 0a)) @ 2. The t-SNE results on and 2' are displayed in Figure and 6 Each displays distribution of data points plotted against t-SNE components. The points are colored and shaped differently to represent different classes within the latent space.
In Figure 5) the distribution is quite mixed, with no distinct clusters or separation between the different classes. In contrast, in Figure after applying mask operation on the latent variables. there appears to be a more distinct separation between different classes. Clusters of the same shapes and colors are more evident, indicating that the mask operation has enhanced the class-specifis knowledge within the latent space.
C.2 FULL ABLATION STUDY RESULTS
In previous sections, we thoroughly examined the impact of various assignments of and b on the overall performance. For the sake of limited space, we only demonstrate part of the results previously. Full results provided in Figure 1 The red line in the figures represents the baseline, wherein no mixup data but only raw images are provided. These results are in alignment with the observations before. Notably, the blue line consistently surpasses the red line over the
20

---

Published as a conference paper at ICLR 2024
color, symbol
o, circle
1, square
2, diamond
0.8
3, triangle-up
4, star
star-square
Second t-SNE
0.6
6, circle
7, square
8, diamond
9, triangle-up
10, star
11, star-square
12, circle
0.2
13, square
14, diamond
0.2
0.4
0.6
0.8
First t-SNE
Figure 6: Visualization Results on z'
0.05
3 0.10
a 0.15
3 0.20
0.25
0.0
0.90
0.90
0.90
0.90
0.8
0.85
0.35
0.00
0.25
0.50
0.75 1.00
0.00
0.23
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.00 0.25
0.50
0.73 1.00
0.00
0.25 0.50
0.55 1.50
0.90
0.90
0.90
0,90
0.90
0.85
8.85
0.85
0.65
0.35
0.25
0.75 1.00
0.00
0775
0.50
0.75
0.50
0.73
0.00 0.5
0.60
0.90
0.90
0.90
0.90
0.904
0.85
0.85
0.85
0.85
0.85
0.80
0:15
0.80
0.85
0.95
1.00
0.85
0.85
0.83
0.85
0.35
0.50
0.80
0.90
0.50
0.50
1:00
0.75
0.55
Figure 7: Grid study on hyperparameter a and b in Module 3. The red line is b = 1.0, meaning no mixup data. The blue line portrays the performance of various a b settings. The light blue area symbolizes the range encompassing mean + std. This figure is the ablation results of Acc@ on with a fixed individually.
majority of the range. Most assignments provide effective mixup samples that better transfer the knowledge of the teacher model.
21

---

Published as a conference paper at ICLR 2024
- 0.00
b 0.05
b
b- 0.15
- 0.20
0.90
0.90
0.90
0.90
0.90
0.85
0.85
0.35
0.25 0.50
0.75 1.00
0.25 0.50
0.75 1.00
0.25 0.30
0.75 1.00
0.25 0.50 0.75 1,00
0.25 0.50
0.75
1.00
0.22
b
0.40
0.45
0.90
0.90
0.90
0.90
0.00
0.35
0.85
0.85
0.85
0.85
0.50 1.00
0.75
0.25 0.75
0.25 6.75
b-
b
0.65
0,70
0.90
00
0.00
0.90
0.35
0.85
0.85
0.55
0.85-
0.50
0.50 1.00
1.00
0.25 0.50
0.25
0.75
0.25 0.50
1 0.75
b 0.80
b 0.85
b 0.90
- 0.95
0.90
0.85
0.85
0.85
0.85
0.55
0.80
0.90
0,80
0.50
0.80
0.50 0.75 1:00
0:50
1.00
0.25 0.50 0.75
0.25
0.75
0.25
0.50 0.75 1.00
Figure 8: Acc@I on b, a is fixed individually.
0.15
0.01
975
0.955
0.975
80000
0.950
950
8.060
0,950
0:35 0.50 0:7 170
0,900
1100
0:35 0,50 075
0,900
075
0,900
0.00
0.900
0.55 1.00
0.30
1,000
a- 0.35
1,000
0.40
1,000
0.45
1,000
3 0.50
0.975
0.73
0.915
0,975
0.975
0.500
0.850
0.850
0.950
0.550-
0.
0:00 0:25 0.50 0.75 1.00
0.900
0:00
0.25
0.50 0:75 1:00
900
0.00
0:25
0.50 0.75
0.00
0:75
0
0,00 0.25 0350 1.00
a= 0.55
a= 0.60
== 0.65
a= 0.70
0.75
1000
1.000
1,000
1,000
1,000
0.73
0.975
0.975
0.975
0.955
0.925
0.900
1:00 0.35 0.50 0.5 1.00
0.900
0.00 0.35 0.50 0.5 1.00
0,900
0.00
0.25 0.50 0.75 1.00
0,900
0.00
0.25
0.50 0.55 1.00
0,500
0.00
0.25
0 0.35 1.00
a- 0.80
a- 0.8
8- 0,90
0.95
a 1.1000
1,000
1.000
1,000
1,000
1,000
0.975-
0.975.
0.950
0.23
8.15
0.925
0.925
0.925
0.900
1:00
0:35 0.50 075 120
0.900
100 0.58 0,30 075 1.00
0.900
0.00
0.25 0.50 a75 1.00
0.900
0.00
0.5
0:30 0.55 1.00
0.500
0.00
0.2 0300 0.35 1.00
Figure 9: Acc@3 on b, a is fixed individually.
b 0.00
b - 0.05
b - 0.10
- 0.15
b 0.20
1000
1.000
1,000
1,000
1.000
BOTS
99955
0.975
0.975
0.925
0.925
0.915
0.925-
0.925
0.900
0.25 0.50 0.75 1,50
0.900
0.25
0,50 0.75 1.00
0,900
0.25 0,30 0.75 1.00
0,900
0.25
0.50 0.75 1.00
0.500
0.25 0.50 0.75 1.00
b-
b 0.30
b 0.35
b 0.40
b- 0.45
L.00
1.000
1.000
1.000
1,000
0.925
B.925
0.925
0.525
0.925-
0.900
0.25 0.50 0.75 1.00
0.900
0.25 0.50 0.75 1.00
0.900
0.25 0.50 0.75 1.00
0,900
0.25
0.50 0.75 1.00
0.500
0.25 0.50 0.75 1.00
1,000
b 0.50
1.000
b- 0.55
1,000
- 0.60
1,000
- 0.65
1,000
b 0,70
0.840
0.000
950
0,950-
0.950-
89955
0.975
975
0.075.
0.925
0.900
0.25 0.50 0.75 1.00
0.25 0.50 0.75 1.00
0.900
0.25 0.50 077 1000
0,900
0.25 0.50 0.53 1.00
0.9
0.25 050 0.5 1.00
0.85
0.90
b 0.95
1.00)
1000
1,000
0.975
0.000
8.050
0,950
0.350
0.950
0.925
0,925-
0.925-
0.25 0.75 1.00
0.75
0.50 0.75
0.35 0.50
0.35 0.50 1.00
Figure 10: Acc@3 on a, b is fixed individually.
22

---

Published as a conference paper at ICLR 2024
0.955
0.955
0.925
0550
0.90
0.950
0.850
0.550
0925
0.95
0.925
0.925
0.925
0.35 0.50 0.75 170
0.35 0.50 075
0.50 0.55
0.00
0.55
1.00
0.35
1,000
0.40
1,000
0.45
0.50
0.950
0.925
0.25 0.50 0.75 1:00
0.925
0.25
0.50 0.75
0.925
0.50 0.75
0,925
0.75
0.925
0.25
0.35
1,00
8 0.55
0.60
0.65
a 0.70
a 0.75
1.000
1,000
1,000
1.000
0.755
0,975-
0.925
0.00 0.25 0.50 0.75 1.00
0.00 0.25 0.50 0.75 1.00
0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.00 0.25 0.50 0.75 1.00
a 0.80
a= 0.85
a= 0.90
a 0.95
a= 1.00
1.000
1.000
1,000
1,000
1.000
0.955
0.955
0.975
0.975-
0.975
0.950
0,950
0,950-
0.925
0.925
0.00 0.25 0.50 0.75 1.00
0.00 0.25 0.50 0.75 1.00
0.00 0.25 0.50 0.75 1.00
0.00 0.25 0.50 0.75 1,00
0.00
0.25 0:50
0.75 1.00
Figure A b, a fixed individually.
b
0.05
0.10
b- 0.15
b 0.20
0.975
0.955
0.90
0.50
0.950
0.950
0.950
0.25
0.955
0.925
0.925
0.925
0.75
b 0.25
b 0.30
1,000
0.35
1.000
5-00 0.40
b 0.45
0.950
0.25 0.75 1.00
0.925
0.25 0.75 1.00
0.925
0.25 0.30 0.7
0.925
0.25 0.50 0.55
0.925
0.25 0.50
0.55 1.00
1000
b 0.50
1.000
b 1 0.55
1,000
- 0.60
1,000
- 0.65
1.000
6-7.0
0.73
0.975
0.975
0.25 0.50 0.75 1.00
0.25
0.50
0.75 1.00
0.25 0.50 0.75 1.00
0.25 0.50
0.75 1.00
0.25 0.50
0.75 1.00
b 0.75
b - 0.90
- 0.85
b- 0.90
0.95
1.000
1.000
1,000
1,000
1.000
0.75
0.75
0.975
0.975
0.975
0.950
0.950
0.950-
0.925
0.25 0.50 0.75 1.00
0.25 0.50 0.75 1.00
0.25 0.30 0.75 1.00
0.25 0.50 0.75 1.00
0.55
030
0.5 1.00
Figure 12: Acc@5 on a, is fixed individually.
23

---

Published as a conference paper at ICLR 2024
C.3 COMBINATIONS WITH MORE METHODS
Although the mainstream of Data Free Knowledge Distillation lies in the generation methods, i.e.. they rely on generator generate teachers' training data for compensation (Li et al. 2023), there exist some sampling methods relying on the tremendous unlabeled data samples in the wild. For example, DFND (Chen et al. 2021) identifies images most relevant to the given teacher and tasks from a large unlabeled dataset, selects useful data samples and finally uses them to conduct super- vised learning to the student network with the labels the teacher give. ODSD (Wang et al., (023b) sample open- world data close to the original data's 's distribution by an adaptive sampling module, in- troduces a low-noise representation to alleviate the domain shifts and builds a structured relationship of multiple data examples to exploit data knowledge.
When discussing the problem of OOD-KD, some readers might come up with Source-Free Domain Adaptation (Huang et al. 2021 Pei et al., 2023 Ding al., 2022). specific setting in Domain Adaptation. Although it falls beyond the scope of our current work, for the sake of rigor, we now highlight the differences between OOD-KD and SFDA.
SFDA assumes the absence of training data for the source models, which is akin to the scenario of the teacher model in OOD-KD. However, SFDA does its adaptation the source model and assumes that the target model shares the same framework as the source model. The difference between source model (teacher model) and target model (student model) in the framework makes integrating teachers' knowledge into the SFDA framework remains an open problem. Moreover, some SFDA methods involve specific modifications to the backbone model, which violates the immutability of the teacher model in OOD-KD. For example, approaches like SHOT (Liang al. 2020) and SHOT++ (Liang al. 2022) divide the backbone model into feature extractor and classifier sharing the classifier across domains. SFDA methods like C-SFDA (Karim et al., 2023) utilize confident examples for better performance. Their performance is limited when deploying to resource-constrained edge devices. What's worse, some SFDA methods base their methodology only on ResNet series models (Yang et al. 2021 Kundu et al. 2022). which inapplicable to most lightweight neural networks.
Table 7: Results of SFDA, DFKD methods, and our proposed method. means additional distilla- tion progress is applied to this method.
Office-31: Amazon, Webcam DSLR
Method
Acc@3
Acc@5
DFQ+
80.445.7
93.3+44.1
96.442.11
CMI+
67.1+:3.5
86.6:t4.3
92.9+3.0
Deeplav+
65.911..3
84.7:14.9
90.61:3.88
w/o KD
63.5+77
84.7+1455
90.24:3.7
ZSKT+
33.3
55.3:1:1.88
65.9::11.5
PRE-DFKD+
68.3 19.5
14.8
13.3
DEND+
59.6-1..2
78.4-19.6
89.34-4.2
CASEDA*
62.7.1488
80.8.1..00
8.34.6.3
SFDA-DE*
19.6=11.7
83.9:14.55
89.7422.1
U-SED
61.6:6..
81.6:t:4.5
90.6::3388
Ours
84.3.13.1
94.9-12.66
97.6-100.8
We demonstrate the results of some splendid SFDA methods (C-SFDA Karim et al. 2023). SFDA- DE (Ding et al., al. ). Uncertainty-SFDD
(U-SFDA) (Roy et al.) 2022) under the setting Office-31 Amazon, Webcam - DSLR in Table 7 Since in OOD-KD, T remains immutable, adaptation is adopted to S directly. As SFDA methods do not make use of ground truth labels or teachers' knowl- edge, in order to align with our methods, we apply additional distillation progress after employing them. However, suffers great performance degradation confronted with domain shift, similar to that observation of T in main experiments. Consequently, they cannot be fully exploited, resulting in inferior performance compared to DFKD methods.
24


Reviews:
[{"review_id": "Upm4LGWbqV", "reviewer": "Reviewer_aobZ", "summary": "The study introduces a technique for knowledge distillation wherein a student model learns from a pre-trained teacher model, even when the domains differ. Assuming a consistent label space across both domains, the research addresses the inherent distribution shift. It suggests the development of an 'anchor network' designed to extract domain-neutral features within the latent space. These domain-agnostic images, generated via latent features, are then employed as samples, mixed with student domain data, to facilitate efficacious distillation.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript is articulated with clarity, ensuring effective comprehensibility to the readers.\n\n2. The authors masterfully contextualize their work in the introduction, clearly delineating their approach from conventional Generic KD and DFKD paradigms.\n\n3. Their introduction of the anchor-based mixup strategy not only showcases novelty but also delivers a marked improvement in knowledge distillation outcomes when benchmarked against prior DFKD techniques and selected source-free domain adaptation methods.\n\n4. The strategy to employ a mask for distinguishing between domain-specific and domain-neutral features is underpinned by a solid rationale, and its formulation and application are both commendably executed.", "weaknesses": "Major:\n\n1. The proposed problem definition appears to closely resemble source-free domain adaptation (SFDA) [1]. The only distinction being that the target domain network is more lightweight compared to the pre-trained teacher network. As a result, the problem setup seems to simply be a specific instance of the broader source-free domain adaptation scenario. This raises concerns about the novelty of the problem statement.\n\n2. Given the dataset $D_{s}$, which consists of OOD student domain data with labels, I wonder about the performance of the student when trained solely on $D_{s}$  in a supervised manner. Did the authors conduct any initial tests to gauge baseline performance? Without such a baseline, I question how the authors determined the severity of the problem.\n\nMinor:\n\n1. In Equation 2, the left-hand side (LHS) includes $z_{0}$  in its argument, but $z_{0}$ is absent from the right-hand side (RHS). Clarification is needed regarding the functional operations. This inconsistency is observed in Equations 2 and 3 as well.\n\n2. For Equation 4, while the expectation is based on $z_{0}$, $z_{0}$ is not reflected in the RHS's loss combination. A similar issue is present in Equation 6.\n\n3. In Equation 4, the loss function $L_{generator}$ is introduced. However, it's unclear whether this loss is optimizing the generator weights, the latent space, or both. This lack of clarity persists in Equations 5, 6, and 10.\n\n[1] Kurmi, Vinod K., Venkatesh K. Subramanian, and Vinay P. Namboodiri. \"Domain impression: A source data free domain adaptation method.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pVdp7CQTn4", "reviewer": "Reviewer_hjEr", "summary": "This paper explores a significant and practical problem, Out-of-Domain Knowledge Distillation (OOD-KD). The authors believe that adopting models derived from DFKD for real-world applications suffers significant performance degradation, due to the discrepancy between teachers’ training data and real-world scenarios (student domain). Therefore, teachers’ knowledge must be selectively transferred. So the authors proposed AuG-KD, which utilizes an uncertainty-guided and sample-specific anchor to align student-domain data with the teacher domain. Experiments illustrate the effectiveness of the method.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. It is valuable to explore the relationship between OOD and data-free distillation.\n\n2. Although experiments were conducted on multiple dataset settings, the scale was small.", "weaknesses": "1. The motivation behind the setting of this paper is confusing. Why do we need to use out-of-distribution teachers when we have student domain data? Wouldn’t it be better to directly train the teacher under the student domain?\n2. Lack of discussion with DFND[1], MosiacKD[2] and ODSD[3]. We think that since generation methods are known, sampling methods should also be compared.\n3. Why not conduct experiments with CIFAR10 and CIFAR100? This is the mainstream dataset in the DFKD. Furthermore, can this method generalize on ImageNet?\n\n[1] Learning Student Networks in the Wild, CVPR 2021\n\n[2] Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data, NIPS 2021\n\n[3] Sampling to Distill: Knowledge Transfer from Open-World Data, arxiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1yiIwf9K6u", "reviewer": "Reviewer_5Sg9", "summary": "This work studies on a new but practical problem: Out-of-Domain Knowledge Distillation (OOD-KD). OOD-KD resembles Data-Free Knowledge Distillation (DFKD), which assumes the unavailability of teacher model’s training data except for one significant difference: teacher models’ training data and student models’ test data are not IID distributed. To tackle OOD-KD, the authors propose AuG-KD to align student-domain data with the teacher domain and leverage a generative method to progressively evolve the learning process from OOD knowledge distillation to domain-specific information learning. Experimental results demonstrate its promise of the proposed methods.", "questions": "1.\tThe same to weakness 1. To be specific, could you provide more details about which scheduler function you use and what kind of scheduler function would be preferred?\n\n2.\tThe same to weakness 2. More specifically, the ablation result of w/o Anchor has a great performance improvement compared to that of w/o Mixup. Could you further explain it?\n\n3.\tThis work provides detailed visualization on the mixup sample generated in the main body. We could see a gradual change as f increases. However, are there any relationships between the imgs at the same position with different f?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe problem proposed is new but practical. With the development of ML techniques, most large-scale models are released in a black box or without access to their training data. Under this circumstance, OOD problems are unavoidable. This work focuses on this novel problem and offers a practical solution.\n\n2.\tThe writing is clear and sophisticated. To best clarify the problem and its solution, this work utilizes formulae, method framework, pseudocode, flow chart and visualization.\n\n3.\tThe experiments are extensive. This work conducts its experiments on 3 datasets and 8 settings in five times. Besides, 3 more ablation studies are designed to substantiate the effectiveness of each module of the method. In the appendix, 3 more baselines are taken into consideration. Quite a lot of DFKD methods pay little attention to the repeatability and stability, conducting each experiment with only one seed. This work considers more rigorously in the experiment settings.\n\n4.\tThe experiment analyses are detailed and convincing. Apart from analyzing the stability and superiority of their proposed method, this work steps further. They provide a clear explanation towards the high variance of each method in the data perspective, which lacks related analysis in previous studies. Besides, they provide visualization for the mixup samples generated by their methods.", "weaknesses": "1.\tThe detail of the scheduler function is not released. Although the important characteristics of scheduler functions are provided in the main body, I cannot figure out the specific scheduler function used in the experiments.\n2.\tThe ablation study of the effectiveness of each component is not clear. The setting seems ambiguous and in need of further explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejYOLSTLZX", "reviewer": "Reviewer_8qr9", "summary": "This work proposed a new task of Out-of-Domain Knowledge Distillation (OOD-KD) extended from data-free knowledge distillation, which focused on the distribution shift between teacher domain and student domain. The main challenges are (1) the absence of teacher domain's data, (2) how to selectively transfer teachers' knowledge due to the distribution shift, and (3) how to balance OOD KD and domain-specific information learning. To tackle these three challenges, the method consists of a data-free learning generator, anchor learning module, and mixup learning module. Experiments on three datasets verified the effectiveness of the proposed method.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ This work proposed a new problem for knowledge distillation, Out-of-Domain Knowledge Distillation (OOD-KD), which is challenging and practical to solve.\n\n+ The presentation and organization of this paper is good. It is easy to figure out the main challenges and the solutions.\n\n+ The experimental results on Office-31, Office-Home, and VisDA-2017 reached state-of-the-art performance.", "weaknesses": "- The class-specific mask is essential in anchor learning, but there lacks visualizations on masks to show whether they correctly captured class-specific information.\n\n- According to ablation study 5.3 (a), the contribution of anchor is not significant compared to mixup module, which doubts the effectiveness and necessity of anchor learning module. Here raises a question about why anchor learning is necessary and successfully selectively transfer teachers' knowledge.\n\n- A related work [1] shared the similar motivation on the invalid IID hypothesis and the gap between teacher domain and student domain. Although the research problem is different ([1] focused on conventional KD while this work proposed OOD-KD), it would be more comprehensive if the comparison between this work and [Niu et al., NeurIPS 2022] can be discussed, especially on the above mentioned similarities on motivations, and how to selectively transfer teachers' knowledge.\n\nReference\n[1] Respecting Transfer Gap in Knowledge Distillation. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)