Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

1 Introduction . Knowledge distillation ( KD ) has emerged as a popular paradigm for model compression and knowledge transfer , attracting attention from various research communities [ 18 , 41 , 47 , 15 ] . The goal of KD is to train a lightweight model , known as the student , by imitating a pre-trained but more cumbersome model , known as the teacher , so that the student masters the expertise of the teacher . In recent years , KD has demonstrated encouraging results over various machine learning applications , including but not limited to computer vision [ 6 , 29 ] , data mining [ 2 ] , and natural language processing [ 46 , 20 ] Nevertheless , the conventional setup for KD has largely relied on the premise that , data from at least the same domain , if not the original training data , is available to train the student . This seeminglymind assumption , paradoxically , imposes a major constraint for conventional KD approaches : in ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . many cases , the training data and even their domain for a pre-trained network are agnostic , due to for example confidential or copyright reasons . Hence , the in-domain prerequisite significantly limits the applicable scenarios of KD , and precludes taking advantage of the sheer number of publicly-available pre-trained models , many of which with unknown training domain [ 25 , 39 ] , to carry out massive knowledge transfer . In this paper , we aim at the ambitious goal of conducting KD using only out-of-domain ( OOD ) data , which , in turn , enables us to greatly relax the conventional prerequisite and thereby largely strengthens applicability of KD . Unarguably , OOD-KD is by nature a highly challenging task , since the domain discrepancy will inevitably impose a major obstacle towards the proper functioning of the pre-trained teacher . In fact , if we are to conduct naive KD on the raw OOD data , the resulting student model , as will be demonstrated in our experiments , fails to provide any performance guarantee on the target domain . This phenomenon signifies the limited generalization capability learned from OOD data , which is unsurprising . To this end , we propose a novel assembling-by-dismantling approach , termed as MosaicKD , that allows us to take advantage of OOD data to conduct KD . Our motivation stems from the fact that , even though data from different domains exhibit divergent global distributions , their local distributions , such as patches in images , may however resemble each other . This observation further inspires us to leverage the local patterns , shared by the OOD and target-domain data , to resolve the domain shift problem in OOD-KD . As such , the core idea of MosaicKD is to synthesize in-domain data , of which the local patterns imitate those from real-world OOD data , while the global distribution , assembled from local ones , is expected to fool the pre-trained teacher . As shown in Figure 1 , the shared local patterns are extracted from OOD data and re-assembled into in-domain data . Intuitively , this process is analogous to mosaic tiling , where tesserae are utilized to compose the whole art piece . In-Domain MosaicShared Local Patterns plainmaple orange Out-of-Domain Data deercar frog Figure 1 : Natural images share common local patterns . In MosaicKD , these local patterns are first dissembled from OOD data and then assembled to synthesize in-domain data , making OOD-KD feasible . Specifically , in MosaicKD , we frame OOD-KD problem as a novel fourplayer min-max game involving a generator , a discriminator , a student , and a teacher , among which the former three are to be learned while the last one is pre-trained and hence fixed . The generator , as those in prior GANs , takes as input a random noise vector and learns to mosaic synthetic in-domain samples with locally-authentic and globally-legitimate distributions , under the supervisions back-propagated from the other three players . The discriminator , on the other hand , learns to distinguish local patches extracted from the real-world OOD data and from the synthetic samples . The entire synthetic images are fed to both the pre-trained teacher and the to-be-trained student , based on which the teacher provides category knowledge for data synthesis and the student mimics the behavior of the teacher so as to carry out KD . The four players collaboratively reinforce one another in an adversarial fashion , and collectively accomplish the student training . In short , our contribution is the first dedicated attempt towards OOD-KD , a highly practical yet largely overlooked problem , achieved through a novel scheme that mosaics in-domain data . The synthetic samples , generated via a four-player min-max game , enjoy realistic local structures and sensible global semantics , laying the ground for a dependable knowledge distillation from the pre-trained teacher . We conduct experiments over classification and semantic segmentation tasks across various benchmarks , and demonstrate that MosaicKD yields truly encouraging results much superior to those derived by its state-of-the-art competitors on OOD data . 2 Related Work . Knowledge distillation . Knowledge Distillation aims to craft a lightweight student model from cumbersome teachers [ 18 ] , by either transferring network outputs [ 4 , 18 ] or intermediate representations [ 41 , 38 ] . In the literature , knowledge distillation and its variants largely rely on the premise that original training data is available during distillation , which is vulnerable in real-world applications due to privacy or copyright reasons [ 33 , 58 , 44 , 57 , 45 , 21 , 43 ] . Recently , data-free knowledge distillation [ 30 , 33 , 10 , 61 , 59 ] has attracted attention from various research communities , which trains student model only with synthetic data . However , due to the difficulty in data synthesis without real-world samples , data-free KD usually leads to a degraded student on complicated tasks [ 61 , 5 ] . Another solution to relax the conventional prerequisite on training data is to use some OOD samples . Unfortunately , it is found that naive knowledge distillation on OOD usually fails to learn a comparable student model from teachers [ 32 , 27 ] . Domain adaptation and generalization . Most learning algorithms strongly rely on the premise that the source data for training and the target data for testing are independent and identically distributed [ 50 ] , ignoring the OOD problem that is frequently encountered in real-world applications . In the literature , the OOD problem is usually addressed by domain generalization ( DG ) or adaptation ( DA ) [ 3 , 64 ] . Adaptation is a popular technique for aligning the source and target domain [ 42 , 11 , 48 ] , which typically requires the target domain to be accessible during training . In recent years , Domain adaptation has been extended to open-set settings where the label space of training and testing data are different [ 37 ] . In comparison , domain generalization is similar to domain adaptation , but does not requires the information from the target domain [ 3 ] . Domain generalization trains a model on the source domain only once and directly applied the model on the target domain [ 28 , 12 , 51 , 16 ] . Despite the success of DA and DG in supervised learning , OOD problem is still under-studied in the context of knowledge distillation . Generative adversarial networks ( GAN ) . Generative adversarial network is initially introduced by Goodfellow et al . for image generation [ 13 ] , where a generator is trained to fool a discriminator in an adversarial minimax game . In recent years , several works have been proposed to improve the performance of GANs , from the perspective of image quality [ 22 , 23 ] , data diversity [ 31 , 56 ] , and training stability [ 1 ] . Besides , Some explorations have been taken to make GAN training more efficient with limited data via transfer learning [ 52 ] or augmentation [ 62 ] . In this work , we study data synthesis in OOD settings , where the original training data is unavailable and thus conventional GAN technique can not be diretly deployed . 3 Out-of-Domain Knowledge Distillation . Without loss of generality , we study OOD problem in the context of image classification task . The domain underlying a dataset is defined as a triplet D = { X , Y , PX×Y } , consisting of input space X ⊂ Rc×h×w , label space Y = { 1 , 2 , ... , K } and joint distribution PX×Y over X × Y . Given a teacher model T ( x ; θt ) optimized on target domain D , vanilla KD trains a lightweight student model to imitate teacher ’ s behaviour , by directly minimizing the empirical risk on the original domain : θ∗s = arg minθs E ( x , y ) ∼PX×Y [ ` KL ( T ( x ; θt ) ‖S ( x ; θs ) ) + ` CE ( S ( x ; θs ) , y ) ] , ( 1 ) where ` KL and ` CE refers to the KL divergence and the cross entropy loss . However , when the original training domain D is unavailable and some alternative data from another domain D′ = { X ′ , Y ′ , PX×Y } is used for training , Equation 1 may be problematic if the domain gap is huge . In this work , we focus on the out-of-domain problem in knowledge distillation , described as follows : Problem Definition ( OOD-KD ) . Given a teacher model T ( x ; θt ) obtained from training domain D = { X , Y , PX×Y } , the goal of OOD-KD is to craft a student model S ( x ; θs ) only leveraging out-of-domain data from D′ = { X ′ , Y ′ , PX ′×Y′ ) } , where X ′ 6= X and Y ′ 6= Y . In OOD-KD , due to the domain divergence between OOD data and original training data , some important patterns may be the missing and the corresponding knowledge on these patterns might not be appropriately transferred from teachers to students . To address the OOD problem , we propose a novel assembling-by-dismantling approach to craft in-domain samples from out-of-domain ones , which effectively alleviates the domain gap between the transfer set and unavailable training set , making KD applicable on out-of-domain data . 4 Proposed Method . In the absence of original training data X , directly minimizing the risk on an OOD set X ′ would be problematic due to the diverged data domain . In this work , we introduce a generative method for OOD-KD , dubbed as MosaicKD , where a generator G ( z ; θg ) is trained to synthesize a more helpful distribution PG for student learning . Specifically , MosaicKD is developed upon the distributionally robust optimization ( DRO ) framework which has been widely used to tackle domain shift [ 40 , 16 , 8 ] . Given a pre-defined distance metric d ( · , · ) for distributions , the basic form of DRO framework can be formalized as the following : min S max G { Ex∼PG [ ` KL ( T ( G ( z ) ) ‖S ( G ( z ) ) ) ] : d ( PG , PX′ ) ≤ } ( 2 ) In equation 2 , ` KL denotes the KL divergence for student learning , and d ( PG , PX′ ) denotes the distribution distance between the generated samples and OOD data . The hyper-parameter specifies the radius of a ball space centered at PX′ . According to this definition , DRO framework aims to find the worst-case distribution from the searching space , which establishes an upper bound for the empirical risk of other distributions covered by the searching space . Ideally , if the target distribution PX of original training data exactly lies in the searching space , its empirical risk can effectively be optimized by the DRO framework . However , we would like to argue that this premise may be problematic in OOD settings , where X 6= X ′ and Y 6= Y ′ . Note that if two distributions PX1 and PX2 are close in the input space under certain metric d ( · ; · ) , their label space Y1 and Y2 should be also similar [ 49 ] . Based on this , distributions within the small ball space , centered at OOD distribution PX′ , are likely to share the same label space , i.e. , X ≈ X ′ and Y ≈ Y ′ , which obviously conflicts with the OOD settings . To this end , the target domain of original training data might not be covered by the search space and can not be bounded by the DRO framework . A remedy for this issue is to use a sufficiently large radius . Unfortunately , it will only lead to intractable searching space , flooded with meaningless distributions .

Summary: This paper proposes a method for knowledge distillation with out-of-domain data (OOD-KD). The key idea is to dismantle out-of-domain images into patches and assemble in-domain data by training a generative adversarial network. It is based on observation that different domain may still share common local patterns from which new (in-domain) categories can be re-assembled. The proposed method is developed undert the framework of the so called Distributionally Robust Optimization (DRO), restricted by a ball space centered at the OOD patch distribution.