Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

1 Introduction . Knowledge distillation ( KD ) has emerged as a popular paradigm for model compression and knowledge transfer , attracting attention from various research communities [ 18 , 41 , 47 , 15 ] . The goal of KD is to train a lightweight model , known as the student , by imitating a pre-trained but more cumbersome model , known as the teacher , so that the student masters the expertise of the teacher . In recent years , KD has demonstrated encouraging results over various machine learning applications , including but not limited to computer vision [ 6 , 29 ] , data mining [ 2 ] , and natural language processing [ 46 , 20 ] Nevertheless , the conventional setup for KD has largely relied on the premise that , data from at least the same domain , if not the original training data , is available to train the student . This seeminglymind assumption , paradoxically , imposes a major constraint for conventional KD approaches : in ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . many cases , the training data and even their domain for a pre-trained network are agnostic , due to for example confidential or copyright reasons . Hence , the in-domain prerequisite significantly limits the applicable scenarios of KD , and precludes taking advantage of the sheer number of publicly-available pre-trained models , many of which with unknown training domain [ 25 , 39 ] , to carry out massive knowledge transfer . In this paper , we aim at the ambitious goal of conducting KD using only out-of-domain ( OOD ) data , which , in turn , enables us to greatly relax the conventional prerequisite and thereby largely strengthens applicability of KD . Unarguably , OOD-KD is by nature a highly challenging task , since the domain discrepancy will inevitably impose a major obstacle towards the proper functioning of the pre-trained teacher . In fact , if we are to conduct naive KD on the raw OOD data , the resulting student model , as will be demonstrated in our experiments , fails to provide any performance guarantee on the target domain . This phenomenon signifies the limited generalization capability learned from OOD data , which is unsurprising . To this end , we propose a novel assembling-by-dismantling approach , termed as MosaicKD , that allows us to take advantage of OOD data to conduct KD . Our motivation stems from the fact that , even though data from different domains exhibit divergent global distributions , their local distributions , such as patches in images , may however resemble each other . This observation further inspires us to leverage the local patterns , shared by the OOD and target-domain data , to resolve the domain shift problem in OOD-KD . As such , the core idea of MosaicKD is to synthesize in-domain data , of which the local patterns imitate those from real-world OOD data , while the global distribution , assembled from local ones , is expected to fool the pre-trained teacher . As shown in Figure 1 , the shared local patterns are extracted from OOD data and re-assembled into in-domain data . Intuitively , this process is analogous to mosaic tiling , where tesserae are utilized to compose the whole art piece . In-Domain MosaicShared Local Patterns plainmaple orange Out-of-Domain Data deercar frog Figure 1 : Natural images share common local patterns . In MosaicKD , these local patterns are first dissembled from OOD data and then assembled to synthesize in-domain data , making OOD-KD feasible . Specifically , in MosaicKD , we frame OOD-KD problem as a novel fourplayer min-max game involving a generator , a discriminator , a student , and a teacher , among which the former three are to be learned while the last one is pre-trained and hence fixed . The generator , as those in prior GANs , takes as input a random noise vector and learns to mosaic synthetic in-domain samples with locally-authentic and globally-legitimate distributions , under the supervisions back-propagated from the other three players . The discriminator , on the other hand , learns to distinguish local patches extracted from the real-world OOD data and from the synthetic samples . The entire synthetic images are fed to both the pre-trained teacher and the to-be-trained student , based on which the teacher provides category knowledge for data synthesis and the student mimics the behavior of the teacher so as to carry out KD . The four players collaboratively reinforce one another in an adversarial fashion , and collectively accomplish the student training . In short , our contribution is the first dedicated attempt towards OOD-KD , a highly practical yet largely overlooked problem , achieved through a novel scheme that mosaics in-domain data . The synthetic samples , generated via a four-player min-max game , enjoy realistic local structures and sensible global semantics , laying the ground for a dependable knowledge distillation from the pre-trained teacher . We conduct experiments over classification and semantic segmentation tasks across various benchmarks , and demonstrate that MosaicKD yields truly encouraging results much superior to those derived by its state-of-the-art competitors on OOD data . 2 Related Work . Knowledge distillation . Knowledge Distillation aims to craft a lightweight student model from cumbersome teachers [ 18 ] , by either transferring network outputs [ 4 , 18 ] or intermediate representations [ 41 , 38 ] . In the literature , knowledge distillation and its variants largely rely on the premise that original training data is available during distillation , which is vulnerable in real-world applications due to privacy or copyright reasons [ 33 , 58 , 44 , 57 , 45 , 21 , 43 ] . Recently , data-free knowledge distillation [ 30 , 33 , 10 , 61 , 59 ] has attracted attention from various research communities , which trains student model only with synthetic data . However , due to the difficulty in data synthesis without real-world samples , data-free KD usually leads to a degraded student on complicated tasks [ 61 , 5 ] . Another solution to relax the conventional prerequisite on training data is to use some OOD samples . Unfortunately , it is found that naive knowledge distillation on OOD usually fails to learn a comparable student model from teachers [ 32 , 27 ] . Domain adaptation and generalization . Most learning algorithms strongly rely on the premise that the source data for training and the target data for testing are independent and identically distributed [ 50 ] , ignoring the OOD problem that is frequently encountered in real-world applications . In the literature , the OOD problem is usually addressed by domain generalization ( DG ) or adaptation ( DA ) [ 3 , 64 ] . Adaptation is a popular technique for aligning the source and target domain [ 42 , 11 , 48 ] , which typically requires the target domain to be accessible during training . In recent years , Domain adaptation has been extended to open-set settings where the label space of training and testing data are different [ 37 ] . In comparison , domain generalization is similar to domain adaptation , but does not requires the information from the target domain [ 3 ] . Domain generalization trains a model on the source domain only once and directly applied the model on the target domain [ 28 , 12 , 51 , 16 ] . Despite the success of DA and DG in supervised learning , OOD problem is still under-studied in the context of knowledge distillation . Generative adversarial networks ( GAN ) . Generative adversarial network is initially introduced by Goodfellow et al . for image generation [ 13 ] , where a generator is trained to fool a discriminator in an adversarial minimax game . In recent years , several works have been proposed to improve the performance of GANs , from the perspective of image quality [ 22 , 23 ] , data diversity [ 31 , 56 ] , and training stability [ 1 ] . Besides , Some explorations have been taken to make GAN training more efficient with limited data via transfer learning [ 52 ] or augmentation [ 62 ] . In this work , we study data synthesis in OOD settings , where the original training data is unavailable and thus conventional GAN technique can not be diretly deployed . 3 Out-of-Domain Knowledge Distillation . Without loss of generality , we study OOD problem in the context of image classification task . The domain underlying a dataset is defined as a triplet D = { X , Y , PX×Y } , consisting of input space X ⊂ Rc×h×w , label space Y = { 1 , 2 , ... , K } and joint distribution PX×Y over X × Y . Given a teacher model T ( x ; θt ) optimized on target domain D , vanilla KD trains a lightweight student model to imitate teacher ’ s behaviour , by directly minimizing the empirical risk on the original domain : θ∗s = arg minθs E ( x , y ) ∼PX×Y [ ` KL ( T ( x ; θt ) ‖S ( x ; θs ) ) + ` CE ( S ( x ; θs ) , y ) ] , ( 1 ) where ` KL and ` CE refers to the KL divergence and the cross entropy loss . However , when the original training domain D is unavailable and some alternative data from another domain D′ = { X ′ , Y ′ , PX×Y } is used for training , Equation 1 may be problematic if the domain gap is huge . In this work , we focus on the out-of-domain problem in knowledge distillation , described as follows : Problem Definition ( OOD-KD ) . Given a teacher model T ( x ; θt ) obtained from training domain D = { X , Y , PX×Y } , the goal of OOD-KD is to craft a student model S ( x ; θs ) only leveraging out-of-domain data from D′ = { X ′ , Y ′ , PX ′×Y′ ) } , where X ′ 6= X and Y ′ 6= Y . In OOD-KD , due to the domain divergence between OOD data and original training data , some important patterns may be the missing and the corresponding knowledge on these patterns might not be appropriately transferred from teachers to students . To address the OOD problem , we propose a novel assembling-by-dismantling approach to craft in-domain samples from out-of-domain ones , which effectively alleviates the domain gap between the transfer set and unavailable training set , making KD applicable on out-of-domain data . 4 Proposed Method . In the absence of original training data X , directly minimizing the risk on an OOD set X ′ would be problematic due to the diverged data domain . In this work , we introduce a generative method for OOD-KD , dubbed as MosaicKD , where a generator G ( z ; θg ) is trained to synthesize a more helpful distribution PG for student learning . Specifically , MosaicKD is developed upon the distributionally robust optimization ( DRO ) framework which has been widely used to tackle domain shift [ 40 , 16 , 8 ] . Given a pre-defined distance metric d ( · , · ) for distributions , the basic form of DRO framework can be formalized as the following : min S max G { Ex∼PG [ ` KL ( T ( G ( z ) ) ‖S ( G ( z ) ) ) ] : d ( PG , PX′ ) ≤ } ( 2 ) In equation 2 , ` KL denotes the KL divergence for student learning , and d ( PG , PX′ ) denotes the distribution distance between the generated samples and OOD data . The hyper-parameter specifies the radius of a ball space centered at PX′ . According to this definition , DRO framework aims to find the worst-case distribution from the searching space , which establishes an upper bound for the empirical risk of other distributions covered by the searching space . Ideally , if the target distribution PX of original training data exactly lies in the searching space , its empirical risk can effectively be optimized by the DRO framework . However , we would like to argue that this premise may be problematic in OOD settings , where X 6= X ′ and Y 6= Y ′ . Note that if two distributions PX1 and PX2 are close in the input space under certain metric d ( · ; · ) , their label space Y1 and Y2 should be also similar [ 49 ] . Based on this , distributions within the small ball space , centered at OOD distribution PX′ , are likely to share the same label space , i.e. , X ≈ X ′ and Y ≈ Y ′ , which obviously conflicts with the OOD settings . To this end , the target domain of original training data might not be covered by the search space and can not be bounded by the DRO framework . A remedy for this issue is to use a sufficiently large radius . Unfortunately , it will only lead to intractable searching space , flooded with meaningless distributions .

Summary: This paper studies knowledge distillation in out-of-domain settings and proposes a generative method called MosaicKD to handle the agnostic domain gap between OOD transfer set and original training data. The core idea of MosaicKD lies in the observation that different domain may still share some local patterns, which can be re-assembled for cross-domain synthesis. The author introduces a four-player adversarial training method to learn a locally-authentic and globally-legitimate distribution for knowledge distillation.