Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data
Authors: Gongfan Fang
Yifan Bao
Jie Song
Xinchao Wang
Donglin Xie
Chengchao Shen
Mingli Song
Venue: NeurIPS 2021 Poster (2021)
Keywords: Knowledge distillation
Out-of-domain data
Patch learning

Abstract: Knowledge distillation~(KD) aims to craft a compact student model that imitates the behavior of a pre-trained teacher in a target domain. Prior KD approaches, despite their gratifying results, have largely relied on the premise that \emph{in-domain} data is available to carry out the knowledge transfer. Such an assumption, unfortunately, in many cases violates the practical setting, since the original training data or even the data domain is often unreachable due to privacy or copyright reasons. In this paper, we attempt to tackle an ambitious task, termed as \emph{out-of-domain} knowledge distillation~(OOD-KD), which allows us to conduct KD using only OOD data that can be readily obtained at a very low cost. Admittedly,  OOD-KD is by nature a highly challenging task due to the agnostic domain gap. To this end, we introduce a handy yet surprisingly efficacious approach, dubbed as~\textit{MosaicKD}. The key insight behind MosaicKD lies in that, samples from various domains share common local patterns, even though their global semantic may vary significantly; these shared local patterns, in turn, can be re-assembled analogous to mosaic tiling, to approximate the in-domain data and to further alleviating the domain discrepancy. In MosaicKD, this is achieved through a four-player min-max game, in which a generator, a discriminator, a student network,  are collectively trained in an adversarial manner, partially under the guidance of a pre-trained teacher. We validate MosaicKD over {classification and semantic segmentation tasks} across various benchmarks, and demonstrate that it yields results much superior to the state-of-the-art counterparts on OOD data. Our code is available at \url{https://github.com/zju-vipa/MosaicKD}.

Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data
Gongfan Fang 1.4, Yifan Bao', Jie Song', Xinchao Wang", Donglin Xie Chengchao Shen', Mingli Song
*Zhejiang University, "National University of Singapore, *Central South University
Aliiaaa--Zeejaang University Joint Institute of Frontier Technologies [fgf, yifanbaa,,,aaa. sjie, ,jii,dong] inxie, brooksong] jew.eeuuuuueuu......... edu......ee cn xinchao@nus edu sg, sce. cs@csu. edu. en
Abstract
Knowledge distillation (KD) aims to craft a compact student model that imitates the behavior of pre-trained teacher in target domain. Prior KD approaches, despite their gratifying results, have largely relied on the premise that in-domain data is available carry the knowledge transfer. Such assumption, unfortunately, in many cases violates the practical setting, since the original training data or even the data domain is often unreachable due privacy copyright reasons. In this paper, we attempt to tackle an ambitious task, termed as out-of-domain knowledge distillation (OOD-KD), which allows OOD data that can be readily obtained at very low cost. Admittedly, OOD- KD is by nature a highly challenging task due to the agnostic domain gap. To this end, we introduce handy yet surprisingly efficacious approach, dubbed MosaicKD. The key insight behind MosaicKD lies in that, samples from various domains share common local patterns, even though their global semantic may vary significantly; these shared local patterns, in turn, can be re-assembled analogous to mosaic tiling, to approximate the in-domain data and to further alleviating the domain discrepancy. In MosaicKD, this is achieved through a four-player min-max game, in which a generator, discriminator, student network, are collectively trained an adversarial manner, partially under the guidance teacher. We validate MosaicKD over classification and semantic segmentation tasks across various benchmarks, and demonstrate that yields results much superior to the state-of-the-an art counterparts OOD data. Our code is available at https: //githu. com/zju-
conduct KD using only
pre-trained
Introduction
Knowledge distillation (KD) has emerged as a popular paradigm for model compression and knowl- edge transfer, attracting attention from various research communities [18 41 47 15). The goal of KD to train a lightweight model, known as the student, by imitating a pre-trained but more cumbersome model, known the teacher, that the student masters the expertise of the teacher. In recent years, KD has demonstrated encouraging results over various machine learning applications, including but not limited computer vision 6 29
data mining 2 and natural language processing
[46 20
Nevertheless, the conventional setup for KD has largely relied on the premise that, data from least the same domain, if not the original training data, is available to train the student. This seemingly- mind assumption, paradoxically, imposes a major constraint for conventional KD approaches: in
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

many cases, the training data and even their domain for a pre-trained network are agnostic, due to for example confidential or copyright reasons. Hence, the in-domain prerequisite significantly limits the applicable scenarios of KD, and precludes taking advantage of the sheer number of publicly-available pre-trained models, many of which with unknown training domain 25 [9]] knowledge transfer.
to earry out massive
In this paper, we aim at the ambitious goal of conducting KD using only out-of-domain (OOD) data, which, in turn, enables us to greatly relax the conventional prerequisite and thereby largely strengthens applicability of KD. Unarguably, OOD-KD by nature a highly challenging task, since the domain discrepancy will inevitably impose a major obstacle towards the proper functioning of the pre-trained teacher. In fact, if we are conduct naive KD on the raw OOD data, the resulting student model, as will be demonstrated in our experiments, fails to provide any performance guarantee on the target domain. This phenomenon signifies the limited generalization capability learned from OOD) data, which is unsurprising.
To this end, we propose novel assembling-byy beaeniingbb-dismanlingg approach, termed as MosaicKD, that allows us to take advantage of OOD data to conduct KD. Our motivation stems from the fact that, even though data from different domains exhibit divergent global distributions, their local distributions, such as patches in images, may however resemble each other. This observation further inspires us to leverage the local patterns, shared by the OOD and target-domain data, to resolve the domain shift problem in OOD-KD ODD. As such, the core idea of MosaicKD is to synthesize in-domain data, of which the local patterns imitate those from real-world OOD data, while the global distribution, assembled from local ones, is expected to fool the pre-trained teacher. As shown in Figure [ the shared local patterns are extracted from OOD data and re-assembled into in-domain data. Intuitively, this process is analogous to mosaic tiling, where tesserae are utilized to compose the whole art piece.
Specifically, in MosaicKD, we frame OOD-KD problem as a novel four- player min-max game involving a gen- erator, a discriminator, a student, and a teacher, among which the former three are to be learned while the last one is pre-trained and hence fixed. The gen- erator, as those in prior GANs, takes as input a random noise vector and learns to mosaic synthetic in-domain samples with locally-authentic and globally-legitimate distributions, un- der the supervisions pnok-propagated from the other three players. The dis- criminator, on the other hand, learns to distinguish local patches extracted from the real-world OOD data and from the synthetic samples. The entire synthetic images are fed to both the pre-trained teacher and the to-be-trained student, based on which the teacher provides category knowledge for data synthesis and the student mimics the behavior of the teacher so as to carry out KD. The four players collaboratively reinforce one another in an adversarial fashion, and collectively accomplish the student training.
car
deer
frog
maple
plain
orange
Out-of- Domain Data
Shared Local Patterns
In-Domain Mosaic
Figure Natural images share common local patterns. In MosaicKD, these local patterns are first dissembled from OOD data and then assembled synthesize in-domain data, making OOD-KD feasible.
In short, our contribution the first dedicated attempt towards OOD-KD, highly practical yet largely overlooked problem, achieved through novel scheme that mosaics in-domain data. The synthetic samples, generated via a four- oour-player min-max game, enjoy realistic local structures and sensible global semanties, laying the ground for a dependable knowledge distillation from the pre- pre-trained teacher. We conduct experiments over classification and semantic segmentation tasks across various benchmarks, and demonstrate that MosaicKD yields truly encouraging results much superior to those derived by its state-of-the-aas competitors on OOD data.
2 Related Work
Knowledge distillation. Knowledge Distillation aims to craft lightweight student model from cumbersome teachers [18), by either transferring network outputs 18] or intermediate representa- tions 41 38). In the literature, knowledge distillation and its variants largely rely on the premise that

---

original training data is available during distillation, which is vulnerable in real-world applications due privacy or copyright reasons [33 58
45 2T 43]. Recently, data-free knowledge
distillation 30 33
59 has attracted attention from various research communities, which trains student model only with synthetic data. However, due to the difficulty in data synthesis without real-world samples, data-free KD usually leads to a degraded student on complicated tasks [6]] []] Another solution to relax the conventional prerequisite on training data is to use some OOD samples. Unfortunately, is found that naive knowledge distillation on OOD usually fails to learn comparable student model from teachers 32,27
10
61
Domain adaptation and generalization. Most learning algorithmss strongly rely on the premise that the source data for training and the target data for testing are independent and identically distributed [50], ignoring the OOD problem that is frequently encountered in real-world applications. In the literature, the OOD problem usually addressed by domain generalization (DG) or adaptation (DA) 3 64] Adaptation is a popular technique for aligning the source and target domain [42 11 48
which typically requires the target domain to be accessible during training. In recent years, Domain adaptation has been extended to open-set settings where the label space of training and testing data are different [37]. In comparison, domain generalization is similar to domain adaptation, but does not requires the information from the target domain F. Domain generalization trains model on
the source domain only once and directly applied the model on the target domain [28] 12 51, [6]. Despite the success of DA and DG in supervised learning, OOD problem is still under-studied in the context of knowledge distillation.
Generative adversarial networks (GAN), Generative adversarial network is initially introduced by Goodfellow et al. for image generation [3) where generator is trained to fool a discriminator in an adversarial minimax game. In recent years, several works have been proposed to improve the performance of GANs, from the perspective of image quality [22 223, data diversity [31 (56), and training stability [.. Besides, Some explorations have been taken to make GAN training more efficient with limited data via transfer learning [52] or augmentation (62] this work, we study data synthesis in OOD settings, where the original training data is unavailable and thus conventional GAN technique can not be diretly deployed.
3 Out-of-Domain Knowledge Distillation
Without loss of generality, study OOD problem in the context of image classification task. The domain underlying a dataset is defined as a triplet D = yx Pxxy); }, consisting of input space X' C Rexhxw label space y {1, 2.., K} and joint distribution Pxxy over X x V. Given a teacher model T(x; 0e) optimized on target domain D, vanilla KD trains a lightweight student model to imitate teacher's 's behaviour, by directly minimizing the empirical risk on the original domain:
0- = arg ming, B(ay))~Pxx (T<;; Szx 0.)) + (au(S B)),
(1)
where fRT. and lep refers to the KL divergence and the cross entropy loss. However, when the original training domain D is unavailable and some alternative data from another domain D' (X", )", Pxxy) is used for training, Equation 1 may be problematic if the domain gap is huge. In this work, we focus on the out-of-domain problem in knowledge distillation, described as follows:
Problem Definition (OOD-KD). Given teacher model T'x; 0z) obtained from training domain D = {x' Px the goal of OOD-KD to craft a student model S(x; 0.) only leveraging out-of-domain data from D {x', >" Pxixyn) where X' + X and y" y..
In OOD-KD, due to the domain divergence between OOD data and original training data, some important patterns may be the missing and the corresponding knowledge on these patterns might not be appropriately transferred from teachers to students. To address the OOD problem, we propose approach to craft in-domain samples from out-of-domain ones, which effectively alleviates the domain gap between the transfer set and unavailable training set, making KD applicable on out-of-domain data.
novel
4 Proposed Method
In the absence of original training data X, directly minimizing the risk on an OOD set X' would be problematic due to the diverged data domain. In this work, we introduce generative method for
3

---

OOD-KD, dubbed as MosaicKD, where generator G(2; fg) is trained to synthesize more helpful
distribution Pg for student learning. Specifically, MosaicKD is developed upon the distributionally robust optimization (DRO) framework which has been widely used to tackle domain shift 440
16 g
Given pre-defined distance metric .(::.) for distributions, the basic form of DRO framework can be formalized as the following:
min max PE [6kL (TT(TTGG))))))((()))) : P Sc 8 G
(2)
In equation 2. &xL denotes the KI. divergence for student learning, and d(Pc, Px.) denotes the distribution distance between the generated samples and OOD data. The hyper-parameter e specifies the radius of a ball space centered at Px. According this definition. DRO framework aims to find the worst-case distribution from the searching space, which establishes an upper bound for the empirical risk of other distributions covered by the searching space. Ideally, if the target distribution Px of original training data exactly lies in the searching space, its empirical risk can effectively be optimized by the DRO framework. However, we would like to argue that this premise may be problematic in OOD settings, where X' / X" and y o y" Note that if two distributions Px, and Px2 are close in the input space under certain metric .(: their label space Y and 3/2 should be also similar [49]. Based on this, distributions within the small e ball space, centered at OOD distribution Px,, are likely to share the same label space, i.e., X' R X' and y ~ y", which obviously conflicts with the OOD settings. To this end, the target domain of original training data might not be covered by the search space and can not be bounded by the DRO framework. A remedy for this issue is to use sufficiently large radius e. Unfortunately, it will only lead to intractable searching space, flooded with meaningless distributions.
4.1 Mosaicking to Distill
As mentioned above, the searching space built upon OOD data is insufficient for establishing an reliable upper bound for optimization. To address this problem, MosaicKD introduces a new way to construct the searching space based on local patches. Our motivation stems from the fact that, patterns of natural images are often organized hierarchically, where high-level patterns are assembled from low-level ones. Although the domain of original training data X and OOD data X' are diverged, their local patterns may still resemble each other. For example, the patterns of "fur" can be shared by different animal species from varied domains. Note that each images is assembled from local patches, we propose an aasembling-hy-dismamee synthesize in-domain data for training.
strategy to re-organizes shared local patches and
Patch Learning. The first step towards MosaicKD is to extract local patterns from OOD data X and estimate the patch distribution for generation. Given an OOD dataset X' = {=1, 22y .... x'xi x E R xW x3 }, we obtain patches through L x L cropping, which produces a patch dataset C =
{ci, 02, ...y CM; oj E LxL33.) j. The patch size L is an important hyper-parameter for MosaicKD. For example, if L = W- = H, each patch will cover a full image, which contains all high-level features of original images. When we decrease the patch size to
1, then each patch only contains lowelevel color information. Obviously, small size L can lead to more general patterns than large one, which are more likely to be shared by different domains. Besides, increasing the patch size will introduce more structural information, making the distribution of patches closer to the that of full images. In this work, we model local patch learning as a generative problem, where a generator G(2; 0g) is trained to approximate the patch distribution by fooling a discriminator network D(r: fa). Note that our goal is to synthesize full images instead of pieces of patches, we train the generator G(r; 0g))
to produce images of full resolutions and craft patches on the fly. Let C(-) refers to the cropping operation, the objective of patch learning can be formulated as follows:
min max Liocal (G, D) Ex'PP [log ((C(+'))] + E [log(1 - D(C(G()))))
(3)
where Px refers to the distribution of OOD data and P2 refers to the prior distribution of the latent variable z. C(x') and C(G(z)) refers to the cropped patches from OOD data and generated data. The main difference between Eqn. (3) and the objectives in vanilla GANs [3] lies in the peteh- discrimination, where MosaicKD only regularizes local patterns to be authentic, leaving the global structure unrestricted. As mentioned above, global patterns can be assembled from local ones, MosaicKD assembles these patches synthesize in-domain data through label space aligning.
Label Space Aligning. As no inter-patch restriction introduced in Eqn. 3 the generator may only produce images with meaningless global semantic, although their local patterns are plausible. In

---

Local
x'
OOD data
D
Ltocal Latiinn
Synthetic data
Global
T
Tugg:
Z
G
X
CooO
Worst
S
Domain
Figure 2: The framework of MosaicKD. We leverage the local patterns of OOD data and the category knowledge of teacher to synthesize Docally-authenti and globally-legtimate samples for KD.
this step, turn to the teacher model for more information for in-domain data synthesis. In KD, the teacher model is trained on the original training data X, whose output is a conditional probability T(x; 0.) p(y|ar, 0t), which corresponds the confidence that x belongs align the label space, a naive method is to maximize the confidence of teacher predictions, i.c., minimizing the entropy term H (p(y) y| (,)). However, prior works have shown that such a simple probability maximization may only lead to some "rubbish samples" [14] without too much useful visual information for student training. To address this problem, we propose a regularized objective to align label space, which is formalized as:
y-th category. To
min Latign (G, D, T) Rz-P, [log(1 D(C(G())) + H [(y] 0.)]
In Equation 4 the first term refers the discrimination loss as mentioned in Equation which regularizes the local patterns to be authentic. The second term refers to the entropy loss for confidence maximization, which works on full images and assembles local patterns synethsize desired categories. This objective will be simultaneously optimized with Equation 3 to keep the authenticity
of local patches.
DRO in MosaicKD. As aforementioned, Equation 3 regularizes the local patterns to be authentic and Equation 4 aligns the label space of synthetic data that of training data. They collaboratively construct a new searching space for DRO framework as follows:
min max DDRO D,S,T) T,, (E)),,(s) G where R(G, T) is a regularization term derived from Equation 3 and Note that optimizing generative adversarial networks as Equation 3 is equivalent to mimimizingg the Jensen-Shannon divergence of two patch distribution, i.e., fssD ( (paach PG pratch), the above regularization can be written as:
(T(()
R(G,D,T) < (5)
(6) Regularization RGG, D, T) force the generator to leverage local patterns OOD data for data syn- thesis, which leads to special searching space defined on all possible schemes of patch assembling. Different to the conventional DRO, MosaicKD uses small radius for robust optimization, where the target domain can be covered by the searching space. We relax the regularization of Equation 5 to obtain an optimizable DRO objective for training, formalized as follows:
R(G, D,T) (Ppech Pa ppatch) + EZ~P. [H(p()]G(3)) 0.))]
mín max LDno(G, S, T) Enrr, (<x (TGG( - XR(G, D,T))
G
4.2 Optimization
Patch Discriminator. For training efficiency, the discriminator in equation 3 can be implemented as Patch GAN [19] with carefully designed receptive fields and patch overlap. Specifically, we stack several convolutional layers to build a fully convolutional network, whose output a score map instead of a single rue-or-fake scalar. Each score unit accepts a L x L local patches for discrimination. We apply an additional stride downsampling with step size s on the score map to control the overlap between patches. A large step size will lead to more independent patches, which effectively reduce the structure restrictions in OOD images.
Full Algorithm. The full algorithm of MosaicKD is summarized in Alg. where generator G(2; 0g), a discriminator (xx; 0d), a fixed teacher model T(x; 0e) and student S(x; 0.) are collec- tively optimized under the guidance of Llocot, Latign and L
5

---

end for Algorithm MosaicKD for out-of-domain knowledge distillation
Input: Pretrained teacher [2; 0z), student S(x; 0,) and out-of-domain data X'.
Output: An optimized student ((;; 0,)
1: Initialize generator G(2; 0g) and discriminator (22; 0a)
2: repeat
3:
Patch Discrimination
4: Sample a mini-bated of OOD data x' from X' and synthetic data æ from G(z);
5:
update discriminator to distinguish fake patches from real ones using Liocat from Equ. Generation
6:
7:
Sample mini-batch of generated data from G(z); Update generator G to:
8:
9:
(a) fool the discriminator D using Clocat from Equ.
(b) align label space with teacher T using Calign from Equ.
11:
(c) fool the student S using LDo from Eqn.
12:
Knowledge Distillation
13:
for j steps do:
14:
Sample generated samples from G(z);
15:
Update student through knowledge distillation using LDRO from Eqn.
16:
17:
until converge
Experiments
5.1
Settings
Datasets.
The proposed method is evaluated on two mainstream vision tasks, i.e., image classification
and semantic segmentation. Four datasets are considered in our experiments as in-domain training
set, including CIFAR-100 (26), CUB-200 [53], Stanford Dogs (24] and NYUv2 (3 For OOD-
KD, substitute original training data with OOD data, including CIFAR- 10 26 Places365 663
ImageNet [9] and SVHN (36].
Evaluation metrics. For image classification, accuracy and Frechet Inception Distance (FID) are used to evaluate different methods. FID indicates the divergence of two datasets, which was originally used to assess the synthesis quality of GANs 17
FID = |l - 1211 + tr -
2(EE))))
(8)
where (y1. Σi) and (M2, Σ2) are the mean and covariance statistics of generated and original samples. For semantic segmentation, we use mean Intersection of Unions (mloU) as the performance metric. More details about datasets, training protocol, and metrics can be found in supplementary materials.
5.2 Results of Knowledge Distillation
CIFAR-100. reports the results of knowledge distillation on CIFAR- 1100 dataset. Here we use 10, ImageNet, Places365 and SVHN as OOD data evaluate MosaicKD for OOD settings. We compare the proposed MosaicKD to various baselines, including data-free KD methods (DAFL [7],
7SKT 33 DeepInv. [61. DFQ g and OOD-KD methods naively adapted from mrate-ofthhe-an
KD approaches (BKD 18 Balanced 35
FitNet 41], RKD [38], CRD [47) and SSKD (54)).
As shonw in Table I despite the mismatched distributions, conventional KD approaches still learn some useful but incomplete knowledge from OOD data (i.e., yielding significantly superior perfor- mance to random guessing), which indicates the existence of shared patterns between OOD data and training data. Further, some exploration was taken to evaluate importance of category balance and representation transfer for OOD-KD. KD. First, we balance the OOD data by re-sampling the scarce categories according teacher's predictions. However, results show that balancing the OOD data may not help students learn correct class information, because most samples in OOD data are just misclassified outliers. the context of OOD settings, the balance operation may lead to over-fitting on outliers, which may further degrade the student performance the test set. As mentioned before,

---

resnet-34
vgg-11
wrn40-2
wrn40-2
wrn40-2
Method
Data
Average
resnet-18
resnet-18
wrn16-1
wrn40-1
wrn16-2
Teacher
78.05
71.32
75.83
75.83
75.83
75.37
CIFAR-100
Student
(Original Data)
77.10
77.10
65.31
72.19
73.56
73.05
KD 188
77.87
75.07
64.06
68.58
70.79
72.27
DAFL
74.47
54.16
20.88
42.83
43.70
47.20
ZSKT
67.74
54.31
36.66
53.60
54.59
53.38
Data-Free Free
DeepInv. 66
61.32
54.13
53.77
61.33
61.34
58.38
DFQ g
77.01
66.21
51.27
54.43
64.79
62.74
KD 18
73.55
68.04
47.47
61.17
63.48
62.74
Balanced 35
68.54
64.14
50.50
56.50
57.33
59.40
FitNet 41
70.14
67.52
50.31
60.17
60.60
63.15
CIFAR- (OOD Data)
RKD 38
67.45
63.06
45.37
53.29
57.10
57.25
CRD 47
71.23
66.48
47.00
59.59
61.37
61.13
SSKD 54
73.81
68.72
49.57
60.71
64.61
63.48
Ours
77.01
71.56
61.01
69.14
69.41
69.55
KD [18
50.89
50.52
36.54
36.87
41.69
43,30
Balanced 35
41.74
47.04
31.61
29.57
35.65
37.12
ImageNett (OOD Subset)
FitNet 41
60.15
58.23
42.63
44.21
48.53
50.75
RKD 38
40.26
35.80
31.15
24.95
34.48
33.32
Ours
75.81
68.94
59.32
66.61
67.36
67.60
KD 18
43.49
46.24
33.28
31.39
36.37
38.15
Balanced 35
28.16
38.85
23.22
21.54
28.62
28.08
Places365t (OOD Subset)
FitNet 41
54.08
54.15
36.33
44.21
38.74
45.50
RKD 47
30,25
33.06
28.07
21.12
21.12
26.72
Ours
74.70
68.55
56.70
65.34
65,89
66.23
KD 18
31.55
34.00
19.77
23.07
24.75
26.63
Balanced 35
26.93
29.34
16.18
18.96
21.50
22.58
SVHN (OOD Data)
FitNet 41
33.69
36.22
20.02
23.72
25.41
27.81
RKD
26.83
27.31
18.09
22.55
24.29
23.81
Ours
47.18
37.63
31.87
45.84
44.40
41.38
Table 1: Test accuracy (%) of student networks trained with the following settings: conventional KD with original training data, data-free KD with synthetic data, and OOD-KD with OOD data. t: As Places365 and ImageNet contain some in-domain samples, craft OOD subsets with low teacher confidence (high entropy) from the original dataset, so as to match our OOD setting.
Method
Data
FLOPs
mloU
Method
CUB-200
Stanford Dogs
Teacher
41G
0.519
Teacher
49.41
56.65
NYUv2
Student
5.54G
0.375
Student
41.44
48.61
ZSKT
5.54G
0.364
KD
11.07
10.24
Data-Free
DAFL
5.54G
0.105
Balanced
4.56
6.42
5.54G
0.406
FitNet
18.12
19.13
ImageNet
Ours
5.54G
0.454
Ours
26.11
28.02
Table 2: Mcan Intersection over Union (mloU) of student models on NYUv2 data set.
Table Test accuracy of student networks on fine- fine-grained datasets.
OOD data and original data may share some local patterns, which can be extracted by shallow layers of networks. We apply four representation transfer approaches, i.e., FitNet, RKD, CRD and SSKD to study the their role in OOD-KD, Compared with RKD that focus instance relation, we found that response-bascl basond methods like fitnet can transfer more helpful information OOD settings, where the student directly imitates the teacher's 's intermediate outputs of teachers. In general, transferring low-level features sometimes can be helpful for OOD-KD. However, note that CRD works on the high-level representation extracted from the penultimate layer, transferring these knowledge may be inappropriate for OOD-KD KD because high-level features may be unrelated to target tasks.
In this work, we handles the OOD-KD problem as generative problem, instead of directly using OOD data for training. The proposed method leverage local patterns of OOD data for data synthesis, where some task-related patterns will be "assembled" from shared local patches. Results show that these re-assembled data can effectively transfer knowledge from teachers to students. Table we also extend method to different types of OOD sets. We found that the performance of MosaicKD

---

(a) Cifar-10 10
(b) SVHN
Pero-Ui
PID-Umi
122
Percenaag (90
PeroUUi
200
Pero-Syn
9ID-8yn
192
Pwrodyn
Percentage
2.00
100
FID
9.16
FID
3.01
2.20
3.05
3.00
3.00
130
Category Rank
Calegory Rank
Figure 3: Statistical information of OOD data and out generated data. Category percentage (the first
y-axis) and FID score (the second y-axis) original data (CIFAR-100) is reported.
Patch size
8
18
22
32
Acc.
52.22
55.58
58,94
61.01
61.03
58.94
$1.34
CIFAR-10
FID
2.43
4.47
8.78
16.82
22.81
26.07
28.30
Acc.
46.44
45,90
50.67
56.70
53.99
53.44
40.29
Places365
FID
12.32
14.77
21.32
30.09
35.54
38.64
41.41
Acc.
19.83
21.86
32.09
31.87
21.05
22.08
20.54
SVHN
FID
93.39
146.76
145.64
143.72
148.33
147.25
148.94
Table 4: Test accuracy (%) of students obtained with different patch sizes. The Patch FID score between OOD data and original data is also reported. Results show that our approach requires smaller patch sizes to handle severe domain discrepancies.
is related to the degree of domain divergence between OOD data and original data. For example, ImageNet is object recognition dataset while Places365 is scene classification dataset. Results show that, for the target data CIFAR- 100, MosaicKD can achieve better performance on ImageNet compared to Places365.
Fine-grained Classification, To further study the effectiveness of our approach, we conduct knowl- edge distillation on fine eme-irained datasets as shown in Table using Places365 as OOD data. OOD-KD on fine-grained data a challenging problem, because different categories is visually similar. Results show that our method achieves superior performance compared to baselines methods.
Semantic Segmentation. Semantic segmentation can also be viewed as classification task, where the network is trained to predict the category of each pixel. We apply our method to the NYUv2 dataset, following the protocol in [TO]. The teacher network a deeplab v3 network with resnet-50 model. Our method can
backbone. The student is freshly initialized effectively improve the knowledge transfer on OOD data and achieve competitive results even compared to vanilla KD settings.
5.3
Quantitative Analysis
Data balance and FID. Figure 3 provides some statistical information of OOD data and generated samples, including the category balance predicted by teachers and the per-class FID scores. The category is ranked according to their percentages. Note that the original CIFAR-10 dataset only contains 10 categories, which is very limited compared with the 100 categories of CIFAR-100. As illustrated in Figure 3 (a), some CIFAR- categories are missing in CIFAR- 10. Besides, the large FID between OOD data and original training data also indicates that, even though some samples are categorized to some classes by the teacher, they may still belong to outliers. by contrast, our method successfully balances different CIFAR-100 categories and alleviates the domain gap (lower class FID), especially for unbalanced categories.
The influence of patch size. Patch size plays an essential role in our method, which determines the flexibility of data synthesis. The optimal patch size actually depends on the divergence between OOD data and in-domain data. For OOD data with large domain discrepancy, small patches are usually required due to the limited local similarity. As shown in Table 4 we evaluate our method with different patch sizes and report the test accuracy of student models as well as the patch FID. According to the Table we find that for OOD data like CIFAR-10 and Places365, large patch size (e.g., 18) can be used for student learning. However, for SVHN dataset, a smaller patch size would be more appropriate, SVHN severely diverges from CIFAR-1000
8

---

Teacher prediction: tree True calogories car deer frog Figure 4: Visualization of synthetic data with and without patch learning. GANs without patch learning will be trapped by OOD data and fails to present correct semantic for different categories (highlighted in blue). In our method, the semantic can be correctly aligned with target domain.
Teacher prediction: apple Thue catergor:: car
Teacher prediction apple Thee calegory: apple
Teacher prediction: tree Thue category: tree
(a) W/O Patch Leaming
(b) Patch Leaming (Ours)
Cifar-10
Places365
wrn40-2
wrn40-2
wrn40-2
Method
wrn16-1
wrn40-1
wrn16-2
Acc. (80
61.01
69.14
69.41
patch
51.34
56.23
57.23
w/o disc.
44.05
58.11
59.73
DFF
Unaredst
DFO
w/o adv. 55.11
67.57
68.25
Epocha
Epocits
Table 5: Test accuracy (%) students in differ- ent ablation settings.
Figure 5: Training efficiency of our method compared with DFQ and vanilla KD.
Ablation study. In this section, we conduct ablation to understand the role of patch learning further. We consider the following settings: (a) MosaicKD (b) mosaicKD without patch learning (c) mosaicKD without discriminator (d) mosaicKD wihout adversarial training. As shown in Table 5 we find that full image discrimination sometimes even lead to worse results than MosaicKD without discrimination. Figure visualizes the synthetic data with or without patch learning. Results show that the generator without patch learning is trapped by the label space of OOD data, failing to synthesize in-domain categories like trees and apples.
Discussion
Relation unlabeled knowledge distillation. In the literature knowledge distillation, slice of works also use unlabeled data for student learning. However, they either make an i.i.d. assumption about unlabeled data and original training [60], or assume that there are sufficient in-domain samples inside unlabeled set [55] In this work, we allow the unlabeled set to be fully OOD, which is more challenging than existing unlabeled settings.
Relation to data-free knowledge distillation. The commonality between the data-free algorithm and MosaicKD lies in that they both solve the KD problem through data synthesis. However, data-free KD leverages some simple priors such as category confidence [7] and gaussian assumptions [61]. which ignores the structural details in natural images. By contrast, MosaicKD achieves data synthesis in an assembling-by baangg------mmmniiin manner, where natural patterns can be utilized to improve synthesis quality. The training curves of different methods can found Figure
Conclusion
In this work, propose novel approach, termed as MosaicKD, that enables to take advantage of only the OOD data for KD. MosaicKD follows a aasmmbing-by-ddimanne synthetic sample is generated by locally resembling the real-world OOD data while globally fooling the pre-trained teacher. This technically achieved through handy yet effective four-player min-max game, in which a generator, a discriminator, and student network are learned in the presence of pre-trained teacher. We validate MosaicKD over classification and semantic segmentation tasks across various benchmarks, and showcase that it yields results significantly superior the sate-of-ttheant techniques on OOD data.
scheme where

---

8 Acknowledgements and Disclosure of Funding
This work is supported by National Natural Science Foundation of China (U20B206661976 Key Research and Development Program of Zhejiang Province (2020C01023), the Major Scientific Research Project of Zhejiang Lab (No. 2019KD0AC01,, the Fundamental Research Funds for the Central Universities, Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, Start- Up Grant from National University Singapore FRC Research Grant 08-2233-000-F44-114
and MOE AcRF TIER
References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[2] Siddhartha Arora, Mitesh M Khapra, and Harish G Ramaswamy, On knowledge distillation from complex networks for response prediction. In Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3813-3822 2019.
[3] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to new unlabeled sample. Advances in neural information processing systems, 24:2178 1442178-2186, 2011.
[4] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[5] Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. Data-free knowledge distillation for object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3289 3298, 2021.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Leamingg efficient object detection models with knowledge distillation. Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 742-751, 2017.
[7] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3514-3522, 2019.
[8] Yoojin Choi, Jihwan Choi, Mostafa El- Khamy, and Jungwon Lee. Data-free network quantization with adversarial knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 710-711, (10-711,220 2020.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[10] Gongfan Fang. Jie Song. Chengehao Shen, Xinchao Wang, Da Chen, and Mingli Song. Data-free adversarial distillation. arXiv preprint arXiv: 1006, 2019.
[1]] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pages 1889. PMLR, 2015.
[12] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pages 2551-2559, 2015.
[13] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[14] Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1412 2014.
[15] Jianping Gou, Baosheng Yu. Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey. arXiv preprint 20XX:200006055255 2020.
10

---

[16] Tianyu Guo, Chang Xu, Boxin Shi, Chao Xu, and Dacheng Tao. Learning from bad data via generation. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. arXiv preprint arXiv:1706.08500 2017.
[18] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arriv:: 550.00233,, 2015.
[19] Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexci A Efros. Image-to to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1125-1133 134, 2017.
[20] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang. and Qun Liu. Tinybert: Distilling bert for natural language understanding, arXiv preprint (ete:::9999033300 2019.
[21] Yongcheng Jing, Yiding Yang. Xinchao Wang, Mingli Song, and Dacheng Tao, Amalgamating knowledge from heterogeneous graph neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2021.
[22] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710. 10196, 2017.
[23] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[24] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for finc- fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Puttern Recognition, Colorado Springs, CO, June 2011.
[25] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation lcarning, arXiv preprint 1912.:/911..11777 6(2):8, 2019.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. technical report. 2009.
[27] Mandar Kulkarni, Kalpesh Patil, and Shirish Karande. Knowledge distillation using unlabeled mismatched images. arXiv preprint vrXiv:1703.071333 2017.
[28] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018.
[29] Muyang Li, Lin, Yaoyao Ding, Zhijian Liu, Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5284-5294, 2020.
[30] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint 171.0037,, 2017.
[31] Qi Mao, Hsin- Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming Hsuan Yang. Mode seeking generative adversarial networks diverse image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1429-1434 1437, 2019.
[32] Gaurav Menghani and Sujith Ravi. Learning from teacher using unlabeled data. arXiv preprint arXiv:1911.0521 2019.
[33] Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial belief matching, arXiv preprint 1oXiv:1905.09700 2019.
[34] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Roh Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[35] Gaurav Kumar Nayak, Konda Reddy Mopuri, and Anirban Chakraborty. Effectiveness of arbitrary transfer sets for data- knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2021.

---

[36] Yuval Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[37] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEF Interna- tional Conference on Computer Vision, pages 754-763, 20177...117
[38] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 3967-3976, 2019.
[39] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl blog. 1(8)9, 2019.
[40] Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv: 05659, 2019.
[41] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fimeets: Hints for thin deep nets, arXiv preprint arXiv:14122 1412.5550, 2014.
[42] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3723-3732, 2018.
[43] Chengehao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Amalgamating knowledge towards comprehensive classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[44] Chengchao Shen, Xinehao Wang, Youtan Yin, Jic Song, Sihui Luo,, and Mingli Song. Progressive network grafting for few-shot knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[45] Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song. Li Sun, and Mingli Song. Customizing student networks from heterogencous teachers via adaptive knowledge amalgamation In Proceedings of the IEEF International Conference on Computer Vision, 2019.
[46] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint /908 190::::00999000 0935, 2019.
[47] Yonglong Tian, Dilip Krishnan, and Phillip Isola, Contrastive representation distillation. arXiv preprint arXiv: 1910. 10699, 2019.
[48] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7167-7176. 2017.
[49] Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learning, 109(2):373 440, 2020.
[50] Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances neural information processing systems, pages 831-838, 1992.
[51] Riccardo Volpi, Hongscok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. arXiv preprins arXiv:1805.1220 120188 2018
[52] Yaxing Wang, Chenshen Wa, Luis Herranz, Joost van de Weijer, Abel Conzalez-Garcia and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pages 218 224, 2018.
[53] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical Report CNS-TR-200 2-20000000, California Institute of Technology, 2010.
[54] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets supf-super In European Conference on Computer Vision, pages 588 604. Springer, 2020.
[55] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, XU Chunjing, Dacheng Tao, and Chang Xu. Positive- unlabeled compression on the cloud. In Advances in Neural Information Processing Systems, pages 2561 -2700, 2019.
[56] Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity-snnsitie conditional generative adversarial networks. arXiv preprint 4rXiv.1901.09024, 2019.
12

---

[57] Yiding Yang. Zunlei Feng. Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. In Conference Neural Information Processing Systems, 2020.
[58] Yiding Yang. Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[59] Jingwen Ye, Yixin Ji, Xinchao Wang, Xin Gao, and Mingli Song. Data-free knowledge amalgamation via group-stack dual-gan. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[60] Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, Mingli Song. Student becoming master: Knowledge amalgamation for joint scene parsing, depth estimation, and more. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[61] Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M Alvarez, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. arXiv preprint arXiv: 1912.08795 2019.
[62] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data- efficient gan training. arXiv preprint arXiv:2006.10738 2020.
[63] Bolei Zhou, Agata Lapedriza, Aditya Khosla. Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. JEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
[64] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint arXiv:2103.000 02500, 2021.
13


Reviews:
[{"review_id": "IeZBQdfapDz", "reviewer": "Reviewer_jgX3", "summary": "This paper studies knowledge distillation in out-of-domain settings and proposes a generative method called MosaicKD to handle the agnostic domain gap between OOD transfer set and original training data. The core idea of MosaicKD lies in the observation that different domain may still share some local patterns, which can be re-assembled for cross-domain synthesis. The author introduces a four-player adversarial training method to learn a locally-authentic and globally-legitimate distribution for knowledge distillation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**[Strengths]:**\n\nThe motivation of this work is well presented and the idea of patch learning is novel. In my opinion, knowledge distillation with DRO is reasonable and practical for OOD settings as the goal is to optimize the risk on an unknown distribution. And I agree with the statement that conventional DRO framework would be problematic if the target distribution is far away from the OOD data. To address this problem, the author further introduces a patch-restricted method to make DRO applicable for OOD settings and effectively improves the performance of learned student models on OOD data. The experiments in table 1 are very interesting. It seems that MosaicKD not only works on similar data like CIFAR10, but also works on extremely mismatched data like SVHN. Besides, extensive experiments are conducted in this work for evaluation. Table 1 benchmarks existing methods on several datasets, which would be helpful for future works. \n\n**[Weaknesses]:**\n\nHere are my questions: \n1) To my knowledge, some KD methods are also designed for large-scale unlabeled data, like PU Compression [50]. Why they are not used as baselines in this work? \n2) In some settings, data-free KD is also a strong competitor (table 1, resnet34 to resnet18). What is the advantage of the data-driven approach proposed in this paper compared to the data-free approach that does not require data? \n3) There is some room for improvement in the selection of patch size, such as adaptive patch partition or selective patch generation.\n\n**[Additional Feedback]:**\n\nIn addition to the above issues, there are some suggestions for improvement: \n1) Some equations should be presented in more appropriate ways, such as patch distribution in Eqn 3 and the entropy term in Eqn 4. \n2) A brief description of the algorithm can be included at the beginning of the Sec. 4.2 to make the whole algorithm clearer. \n3) In L205, the citation to the equivalence between GAN objective and JS divergence is missing.\n\n**Update: post rebuttal**\n\nI have read the authors' response and other reviewers' comments. Overall I think the rebuttal addresses most of my concerns and I will increase my rating to *7*.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Part of these have been discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "9RNkV8m1Cvs", "reviewer": "Reviewer_aTEX", "summary": "The authors propose a knowledge distillation algorithm for out-of-domain data, where local similarity between different domains is leveraged to transfer knowledge across models. In particular, the authors first present a basic method based on distributionally robust optimization and discuss the its problem in OOD settings. Then the authors improve the basic method by introducing a local loss and an aligning loss. The proposed method, named MosaicKD aims at synthesizing locally-authentic and globally-legitimate samples for KD. The authors demonstrate that MosaicKD can be applied to various OOD data and outperforms several data-free and data-driven methods (e.g. ZSKT, DFQ and CRD)", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros: \nIt is the first attempt to study knowledge distillation with out-of-domain data. The author utilizes the local similarity to alleviate the domain gap between OOD data and source training data. The idea of crafting in-domain data from low-level patches is novel and practical. The author also demonstrate that patch learning can help the generator escape from the OOD distribution and synthesize correct semantic, as the global structure is freed by the patch-level training.  \n\ncons: \nIt is not clear to me how the authors determine the patch size (the receptive fields of Patch Discriminator) and the overlap between patches (the stride applied on discriminator’s logits) for different OOD data. According to the code in supplementary materials, the stride is set to 1 by default. Please provide more details about the design of patch discrimination. Besides, some equations and derivations are not clearly presented in the main paper, such as the Equation 7 and its derivations. \n\n========= post rebuttal ===========\nI have read the rebuttal. The rebuttal addresses all my concerns. I will change my score to 8.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have some questions about the proposed method: \n1) what is the relation between MosaicKD and existing data augmentation methods, like mixup, or perturbation. In the literature of KD, some works adopts augmented data for training [1,2]. Please make a clarification for their difference.\n2) In table 4, a too small patch size leads to `best ` FID but worse student performance. Please provide a discussion about this phenomenon. In particular, what kind of information is missing in small patches which degrades student’s accuracy.\n[1] Heo, B., Lee, M., Yun, S., & Choi, J. Y. (2019, July). Knowledge distillation with adversarial samples supporting decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 3771-3778).\n[2] Fu, J., Geng, X., Duan, Z., Zhuang, B., Yuan, X., Trischler, A., ... & Dong, H. (2020). Role-wise data augmentation for knowledge distillation. arXiv preprint arXiv:2004.08861.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1CvU9Djh9_", "reviewer": "Reviewer_LcFs", "summary": "This paper proposes a method for knowledge distillation with out-of-domain data (OOD-KD). The key idea is to dismantle out-of-domain images into patches and assemble in-domain data by training a generative adversarial network. It is based on observation that different domain may still share common local patterns from which new (in-domain) categories can be re-assembled. The proposed method is developed undert the framework of the so called Distributionally Robust Optimization (DRO), restricted by a ball space centered at the OOD patch distribution.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like the idea of this paper. First, I think the proposed method is a clear and novel way for knowledge distillation from out-of-domain data. Theoretically, the method is reasonable, since it is an extension of DRO framework for distant distributions (OOD and ID data). The empirical evaluation sufficiently validates the authors' claim. I think the proposed method can be useful in practice.\n\nOn the other hand, the main weakness of this paper lies in the patch learning on local patterns. Specifically, the proposed method trains a generator on all patches directly and it is not very clear what kind of local patterns is useful for the target domain. Maybe the aligning loss L_{align} can help select task-related patterns for synthesis but is not clear enough in the method part. I think the author should take a further step on this point and provide more discussion on this topic as it is the core of the method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper conveys the motivation and idea nicely but the presentation needs to be improved. After going through the paper and appendix, I have some questions about the proposed MosaicKD: \n\n1) Adversarial attack adds a small perturbation to the image and is also able to modify the category of OOD data. What is the difference between samples obtained from MosaicKD and adversarial attacks? \n\n2) In Figure 3, why the FID of MosaicKD in some categories is a bit worse than the OOD data? \n\n\n3) some errors and typos need to be fixed: \n- Line 178: The symbol V(G) and V(X’) is a little confusing. \n- Line 148: diverged => divergent.\n- Line 167: V(X^prime) => V(X^\\prime)\n- Line 197: The first term refers to … => The second term refers to …\n- Line 207,: a redundant bracket for x ~ P_G in Eqn 6. \n- Line 221: Equation 4.1 => Equation 4. Besides, I think the derivation of equation 6 and equation 7 should be moved from appendix to the main paper for better presentation.\n- Line 298: We conduct ablation … => we conduct ablation study… \n- Line 1 in Algorithm 1: D(z; \\theta_d) => D(x; \\theta_d). \n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "-Cugtvpn3Ml", "reviewer": "Reviewer_yT4j", "summary": "This paper considers a new task, knowledge distillation using out-of-domain data, and proposed a novel method, MosaicKD. Leveraging the shared common local patterns, MoscaiKD is able to effectively learn from the OOD data.  The paper is well organized and presented, and experimental results demonstrated the effectiveness of the proposed solution. Even I have some questions about the experiments, the quality of the paper is high.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: In this submission, a novel problem is presented: Out-of-Domain Knowledge Distillation, which could be considered as a simple combination of two sub-problems: domain adaptation and knowledge distillation. To deal with this new problem, a new method MosaicKD is proposed, which also could be considered as a combination of KD and GAN methods (adapted to patches, local patterns). The main idea is presented cleverly, but the contribution is not significant. Combining two sub-problems (DA and KD) and proposing a plausible insight (local pattern)  cloud be a good choice for publication, but not an outstanding work. Related work is adequately cited.\n\nQuality: the submission is technically sound, and the claims are well supported. This work is a complete piece. \nQuestions about Equations (3) and (4): How can we evaluate the effectiveness of the first term (about the patch) in Equations (3) and (4)? Since the key insight behind this submission is the shared common local patterns, we should validate the effectiveness of the first term. Maybe this could be answered by results in Table 4.?\nI understand the insights behind Fig.1, but I could not get helpful information from Fig. 1 as there seems no overlapping local patterns between large images.\n\nClarity: The submission is clearly written and well organized. The writing is good. Some errors:\nLine 197: The first term refers to the entropy …    => should be “The second”?\n\nSignificance and  Experiments: Experimental results demonstrate the effectiveness of the proposed methods. However, the comparison is not fair. In Table 1, the authors mainly compared with KD [18] Balanced [35] FitNet [41] RKD [38] methods, which are not relatively new. \nIn Table 2, first, if the FLOPs indicate the FLOPs of the student model? If so, it is unnecessary to present the FLOPs here since all student models are the same. Also, why use mIoU on ImageNet?\nAn important experiment is about the patch size (Table 4). We see these datasets using a resolution of 32\\times 32 (cifar10 and SVHN, not sure about the Places365). But the best result is achieved given 18 * 18 patch (aka., 1/4 image area) which seems not reasonable. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. No critical points are missing.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies a new setting, called out-of-domain knowledge distillation, where the teacher network and the student network are trained on different datasets. The proposed new setting is interesting and more practical than existing ones. In terms of the solution, the authors propose a mosaic idea to synthesize in-domain data by imitating the local patterns from real world OOD data.  The key technical idea is to use minimax optimization to ensure the synthesized data could fool the discriminator. The solution is novel and technically sound. Reviewers all agreed that the proposed problem is interesting and the solution is novel.Thus, I am recommending acceptance. 

Of course, the authors need to carefully revise the manuscript according to reviewers’ comments. The answers to Q3 and Q4 of Reviewer yT4j must be included in the final version.

Decision: Accept (Poster)