Sampling to Distill: Knowledge Transfer from Open-World Data

Authors: yuzheng  wang
zhaoyu  chen
dingkang  yang
zuhao  ge
yang  liu
siao  liu
yunquan  sun
wenqiang  zhang
lizhe  qi
jie  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 09:05:57

Abstract: Data-Free Knowledge Distillation (DFKD) is a novel task that aims to train high-performance student models using only the pre-trained teacher network without original training data. Most of the existing DFKD methods rely heavily on additional generation modules to synthesize the substitution data resulting in high computational costs and ignoring the massive amounts of easily accessible, low-cost, unlabeled open-world data. Meanwhile, existing methods ignore the domain shift issue between the substitution data and the original data, resulting in knowledge from teachers not always trustworthy and structured knowledge from data becoming a crucial supplement. To tackle the issue, we propose a novel Open-world Data Sampling Distillation (ODSD) method for the DFKD task without the redundant generation process. First, we try to sample open-world data close to the original data's distribution by an adaptive sampling module and introduce a low-noise representation to alleviate the domain shift issue. Then, we build structured relationships of multiple data examples to exploit data knowledge through the student model itself and the teacher's structured representation. Extensive experiments on CIFAR-10, CIFAR-100, NYUv2, and ImageNet show that our ODSD method achieves state-of-the-art performance with lower FLOPs and parameters. Especially, we improve 1.50\%-9.59\% accuracy on the ImageNet dataset and avoid training the separate generator for each class.

Main paper: 
 introduction: 
 Deep learning has made refreshing progress in computer vision and multimedia fields [14, 20, 28, 30, 32, 37, 40, 42, 52, 56, 57, 60, 61, 68] . Despite the great success, large-scale models [9, 12, 26, 29, 31, 33, 41, 47, 51, 58, 59, 62] and unavailable privacy data [3, 46, 50, 53, 54] often impede the application of advanced technology on mobile devices. Therefore, model compression and data-free technology have become the key to breaking the bottleneck. To this end, Lopes et al. [34] propose Data-Free Knowledge Distillation (DFKD). In this process, knowledge is transferred from the cumbersome model to a small model that is more suitable for deployment without using the original training dataset. As a result, this widely applicable technology has gained much attention.To replace unavailable private data, most existing data-free knowledge distillation methods rely on alternately training of the generator and student, called the generation-based method. However, these generation-based methods have many issues. First, their trained generators are abandoned after the students' training [6, 13, 16, 19, 36, 67] . The training of generators brings additional computational costs, especially for large datasets. For instance, a thousand generators are trained for the ImageNet dataset [11] , which introduces more computational waste [15, 35] . Then, a serious domain shift issue exists between the generated substitution data and the original training data. Because the substitute data are composed of random noise transformation without supervision information and are highly susceptible to teacher preferences [54] . As a result, the efficiency and effectiveness of the generation-based methods are constrained, affecting student performance [2, 13, 39] . Rather than relying on additional generation modules, Chen et al. [5] propose a sampling-based method to train the student network via unlabeled data without the generation calculations. Compared with generation-based methods, sampling-based methods can avoid the training cost of generators, thus improving algorithm efficiency. The comparison of the two methods is shown in Figure 1 . Meanwhile, they try to reduce label noise by updating the learnable noise matrix, but the noise matrix's computational costs are expensive. More restrictedly, their sampling method relies on the strict confidence ranking and does not consider the data domain similarity issue (We discuss the distribution similarity between sampled data and original data in detail in Section 4.4). In addition, the existing generation-based and sampling-based methods can be summarized as simple imitation learning, i.e., the student mimics the output of a particular data example represented by the teacher [37, 53, 64] . Therefore, these methods do not adequately utilize the implicit relationship among multiple data examples, which leads to the lack of effective knowledge expression in the distillation process.Based on the above observations, we construct a sampling-based method to sample helpful data from easily accessible, low-cost, unlabeled open-world data, avoiding unnecessary computational costs. In addition, we propose two aspects of customized optimization. (i) To cope with the domain shift issue between the open-world and original data, we preferentially try to sample data with a similar distribution to the original data domain to reduce the shifts and design a low-noise knowledge representation learning module to suppress the interference of label noise from the teacher model. (ii) To explore the data knowledge adequately, we set up a structured representation of unlabeled data to enable the student to learn the implicit knowledge among multiple data examples. As a result, the student can learn from carefully sampled unlabeled data instead of relying on the teacher. At the same time, to explore an effective distillation process, we introduce a contrastive structured relationship between the teacher and student. The student can make better progress through the structured prediction of the teacher network.In this paper, we consider a solution to the DFKD task that does not require additional generation costs. On the one hand, we look forward to the solution to data domain shifts from both data source and distillation methods. On the other hand, we try to explore an effectively structured knowledge representation method to deal with the missing supervision information and the training difficulties in the DFKD scenes. Therefore, we propose an Openworld Data Sampling Distillation (ODSD) method, which includes Adaptive Prototype Sampling (APS) and Denoising Contrastive Relational Distillation (DCRD) modules. Specifically, the primary contributions and experiments are summarized as follows:•  
related work 2.1 data-free knowledge distillation: 
 Data-free knowledge distillation aims to train lightweight models when the original data are unavailable. Therefore, the substitute data are indispensable to help transfer knowledge from the cumbersome teacher to the flexible student. According to the source of these data, existing methods are divided into generation-based and sampling-based methods.Generation-based Methods. The generation-based methods depend on the generation module to synthesize the substitute data. Lopes et al. [34] propose the first generation-based DFKD method, which uses the data means to fit the training data. Due to the weak generation ability, it can only be used on a simple dataset such as the MNIST dataset. The following methods combine the Generative Adversarial Networks (GANs) to generate more authentic and reliable data. Chen et al. [6] firstly put the idea into practice and define an information entropy loss to increase the diversity of data. However, this method relies on a long training time and a large batch size. Fang et al. [16] suggest forcing the generator to synthesize images that do not match between the two networks to enhance the training effect. Hao et al. [19] suggest using multiple pre-trained teachers to help the student, which leads to additional computational costs. Do et al. [13] propose a momentum adversarial distillation method to help the student recall past knowledge and prevent the student from adapting too quickly to new generator updates. The same domain typically shares some reusable patterns, so Fang et al. [15] introduce the sharing of local features of the generated graph, which speeds up the generation process. Since the generation quality is still not guaranteed, some methods spend extra computational costs on gradient inversion to synthesize more First, all open-world unlabeled data passes through adaptive prototype sampling so that the substitute dataset resembles the distribution of the original data. Then, based on these data, the student can make progress through low-noise information representation, data knowledge mining, and structured knowledge from the teacher.realistic data [17, 65] . In addition, Choi et al. [10] combine DFKD with other compression technologies and achieve encouraging performance. However, generation-based DFKD methods generate a large number of additional calculation costs in generation modules, while these modules will be discarded after students' training [5] .Sampling-based Methods. To train the student more exclusively, Chen et al. [5] propose to sample unlabeled data to replace the unavailable data without the generation module. Firstly, they use a strict confidence ranking to sample unlabeled data. Then, they propose a simple distillation method with a learnable adaptive matrix. Despite no additional generating costs and promoting encouraging results, their method ignores the intra-class relationships of multiple unlabeled data. Simultaneously, the simple strict confidence causes more data to be sampled for simple classes, leading to imbalanced data classes. In addition, their proposed distillation method is relatively simple and lacks structured relationship expression, which limits the student's performance.Contrastive learning makes the model's training efficient by learning the data differences [63] . The unsupervised training pipeline usually requires storing negative data by a memory bank [55] , large dictionaries [21] , or a large batch size [8] . Even it requires a lot of computation costs, e.g., additional normalization [18] , and network update operations [4] . The high storage and computing costs seriously reduce knowledge distillation efficiency. But at the same time, this idea of mining knowledge in unlabeled data may be helpful for the student's learning. Due to such technical conflicts, there are few methods to combine knowledge distillation and contrastive learning in the past perfectly. As a rare attempt, Tian et al. [48] propose a contrastive data-based distillation method by updating a large memory bank. However, data quality cannot be guaranteed for data-free knowledge distillation, and data domain shifts are intractable, making the above process challenging.In this work, we attempt to explore additional knowledge from both the data and the teacher. Therefore, we further stimulate students' learning ability by using the internal relationship of unlabeled data and constructing a structured contrastive relationship. To our best knowledge, this is the first combination of data-free knowledge distillation and contrastive learning at a low cost during the distillation process, which achieves an unexpected effect. 
methodology 3.1 overview: 
 Our pipeline includes two stages: (i) unlabeled data sampling and (ii) distillation training, as shown in Figure 2 . For the sampling stage, we sample unlabeled data by an adaptive sampling mechanism to obtain data closer to the original distribution. For the distillation stage, the student learns the knowledge representation after denoising through a spatial mapping denoise module. Further, we mine more profound knowledge of the unlabeled data and build the structured relational distillation to help the student gain better performance. The complete algorithm is shown in Algorithm 1.The unlabeled data and the original data are distributed differently in many cases. To obtain the substitution data with a more similar distribution to the original data from the specific unlabeled dataset, we propose an Adaptive Prototype Sampling (APS) module, which considers the teacher's familiarity with the data, excludes misclassified offset noisy data, and focuses on the class balance of the sampled data. Based on these, we design three score indicators to evaluate the effectiveness of the unlabeled data for student training corresponding to the above three aspects, including the data confidence score, the data outlier score, and the class density score.(a) Data Confidence Score. The teacher provides the prediction logits 𝑃 = [𝑝 1 , . . . , 𝑝 𝑛 ] ∈ R 𝑛×𝐶 on the unlabeled dataset {𝒙 0 , . . . , 𝒙 𝑛 }, where 𝑝 𝑖 denotes the prediction for the 𝑖-th sample satisfying 𝑝 𝑖 ∈ R 1×𝐶 . 𝑛 denotes the number of data, and 𝐶 denotes the number of classes. Then the prediction is converted into the probability of the unified scale as p𝑖 = 𝜎 (𝑝 𝑖 ), where 𝜎 denotes the softmax layer and p𝑖 denotes the confidence probability corresponding to the predicted result class. Therefore, p = [ p1 , . . . , p𝑛 ] represents the confidence of each data in the unlabeled dataset. We choose the largest max{ p} for normalization. The confidence score of 𝑖-th sample 𝒙 𝑖 can be calculated as:𝑠𝑐 𝑖 = p𝑖 |max{ p } | . (b) Data Outlier Score.The data distribution of the substitution data and the original data is different. Therefore, the confusing edge data should be excluded, i.e., the data with different distributions but also predicted as the same target class. For example, a tiger is predicted as the class of cat, as shown in the orange part of Stage 1 in Figure 2 . We first separate the teacher predictions according to the predicted classes as 𝜌 𝑖,𝑐 = 𝑝 𝑖 ∈ 𝑐. For each class, 𝜌 𝑖,𝑐 is clustered [25] to explore the intra-class relationships through 𝑘 layering as 𝜇 𝑐,𝑘 . Then the prediction features for the whole unlabeled dataset can be expressed as a group of 𝐶𝐾 prototypes as 𝜇 𝑐,𝑘 ∈ R 1×𝐶 𝐶,𝐾 𝑐,𝑘=1 , where 𝑐 denotes the 𝑐-th class, and 𝐾 denotes the hyperparameter representing the number of prototypes for each class. The prototype centers of the 𝑐-th class can be expressed as 𝜇 𝑐,𝑘 𝐾 𝑘=1 . The outlier of each unlabeled data 𝑥 𝑖 can be calculated with the sum of the prototype centers of its class as õ𝑖 = 𝐾 𝑘=1 cos(𝜌 𝑖,𝑐 , 𝜇 𝑐,𝑘 ), where cos denotes the cosine similarity metric. Similar to the confidence score, we select the maximum value max{ õ } for normalization. As a result, the outlier score can be calculated as:𝑠𝑜 𝑖 = õ𝑖 |max{ õ } | .(c) Class Density Score. To help the student learn various classes effectively, we calculate the class density for the class balance of the sampled data. As shown in Stage 1 of Figure 2 , we increase the sampling range for classes with sparse data (the blue part) while we reduce the sampling range for classes with redundant data (the orange part). Based on this, we first separate the above intra-class outliers õ𝑖 of all data by their predicted classes. The outliers mean value of each class can be calculated as 𝑢 𝑐 = 1 𝑛 𝑐 𝑝 𝑖 ∈𝑐 õ𝑖 , where 𝑛 𝑐 is the number of the data predicted as 𝑐-th class. Therefore, the Dcluster parameter 𝐷 𝑐 can be calculated as:𝐷 𝑐 = √ 𝑢 𝑐 log 𝑒 (𝑛 𝑐 +𝐶 ), which reflects the data density predicted to be 𝑐-th class. The introduction of a constant 𝐶 (the number of classes) helps the numerical stability when the available unlabeled data is small while having little effect on the results when the amount of data is sufficient (under normal conditions). After selecting the maximum value max{𝐷 } for normalization, the density score of each data can be calculated according to the predicted class as 𝑠𝑑 𝑖 = 𝐷 𝑐 |max{𝐷 } | , when arg max(𝑝 𝑖 ) = 𝑐. Finally, we calculate the total score as 𝑆 𝑡𝑜𝑡𝑎𝑙 = 𝑠𝑐 𝑖 -𝑠𝑜 𝑖 + 𝑠𝑑 𝑖 . Based on this, the data closer to the distribution of the original data domain are sampled, which can help the student learn better. The quantitative analysis is shown in Table 7 .After obtaining the high score data, the distillation process can be carried out. We denote 𝑓 𝑇 and 𝑓 𝑆 as the teacher and student networks and denote 𝒙 as the data in sampled set 𝑋 𝐴 . According to the definition [23] , the knowledge distillation loss is calculated as:EQUATIONwhere 𝐷 𝐾𝐿 is the Kullback-Leibler divergence, and 𝜏 𝑘𝑑 is the distillation temperature. L 𝐾𝐷 allows the student to imitate the teacher's output. However, the main challenge is the distribution differences between the substitute and original data domains, leading to label noise interference. Simultaneously, the ground-truth labels are unavailable, so correct information supervision is missing. Therefore, we propose a Denoising Contrastive Relational Distillation (DCRD) module, which includes a spatial mapping denoise component and a contrastive relationship representation component to help the student get better performance and mitigate label noise. Spatial Mapping Denoise. The distribution in the unlabeled data differs from the unavailable original data, which indicates the inevitable label noise. Inspired by manifold learning [44] , low dimensional information representation contains purer knowledge with less noise interference [1] . Here, we utilize a low-dimensional spatial mapping denoise component to help the student learn low-noise knowledge representation. Based on this, we perform eigendecomposition Φ on the teacher's prediction and its transposed product matrix [24] . According to the distance invariance, the autocorrelation matrix 𝑑 2 𝑖 𝑗 in a mini-batch as:EQUATIONwhere 𝑁 denotes the batch size, and 𝑡𝑟 (•) denotes the trace of a matrix. 𝑍 𝑡 is the low-dimensional spatial vector representation from the teacher calculated as Φ(𝑓 𝑇 (𝒙)• 𝑓 𝑇 𝑇 (𝒙)) = 𝑍 𝑡 = 𝑉 𝑡 Λ 1/2𝑡 , where 𝑉 𝑡 is the eigenvalue, and Λ 𝑡 is the eigenvector. Similarly, we can get the student predictions of low-dimensional representation as 𝑍 𝑠 . Then, we set up a distillation loss to correct the impact of label noise by the spatial mapping of the two networks. The spatial mapping denoise distillation loss is calculated as:EQUATIONwhere ℓ ℎ (•, •) denotes the Huber loss. Contrastive Relational Distillation. The missing supervision information limits the student's performance. It is indispensable to adequately mine the knowledge in unlabeled data to compensate for the lack of information. To avoid a single imitation of a particular data example, we build two kinds of structured relationships to mine knowledge from the data and the teacher. Firstly, the student can adequately explore the structured relation among multiple unlabeled data by learning the instance invariant. 𝒙 𝑖 , 𝒙 𝑗 are the different data in a mini-batch. We calculate the prediction difference between data as follows:EQUATIONwhere 𝜏 1 denotes contrastive temperature. Next, we can calculate the consistency instance discrimination loss as:EQUATIONwhere x 𝑗 denotes the strong data augmentation of 𝒙 𝑗 . This unsupervised method is especially effective when the teacher makes wrong results. Secondly, we construct a structured contrastive relationship between the teacher and student, which promotes consistent learning between the teacher and student. The structured knowledge transfer process is calculated as:EQUATIONwhere 𝒙 ′ = 𝒙 ∪ x denotes the set of the sampled data before and after strong data augmentation. And 𝒙 ′ contains 2𝑁 samples for each batch. We calculate the teacher-student consistency loss as:EQUATION)The student can obtain better learning performance through the mixed structured and consistent relationship learning between the two networks. Then, the contrastive relational distillation loss is L 𝑐 = L 𝑐1 + L 𝑐2 .Finally, we can get the total denoising contrastive relational distillation loss as:EQUATIONwhere 𝜆 1 , 𝜆 2 are the trade-off parameters for training losses. 
experiments 4.1 experimental settings: 
 Datasets. We evaluate the proposed ODSD method for the classification and semantic segmentation tasks. For classification, we evaluate it on widely used datasets: 32 × 32 CIFAR-10, CIFAR-100 [27] , and 224 × 224 ImageNet [11] . For semantic segmentation, we evaluate the proposed method on 128 × 128 NYUv2 dataset [45] . Besides, the corresponding open-world datasets are shown in Table 1 , which is the same as DFND [5] for a fair comparison. Implementation Details. The proposed model is implemented in PyTorch [38] and trained with RTX 3090 GPUs. For the CIFAR-10 and CIFAR-100 datasets, we conduct five sets of backbone combinations, set two groups of different numbers of sampled samples (150k or 600k), and train the students for 200 epochs. For the ImageNet dataset, we conduct three sets of backbone combinations and train the students for 200 epochs. The number of sampled samples is 600k. For the NYUv2 dataset, the DeeplabV3 [7] is used as the model architecture followed previous work. The teacher uses ResNet-50 [22] as the backbone, and the student uses mobilenetv2 [43] . We sample 200k unlabeled samples and train the student for 20 epochs.For the above datasets, we set 𝜏 𝑘𝑑 as 4 to be the same as other distillation methods and set 𝜏 as 0.5 to be the same as [8] . Besides, we set 𝜆 1 as 10 and 𝜆 2 as 0.5, use the SGD optimizer with momentum as 0.9, weight decay as 5 × 10 -4 , the batch size 𝑁 as 64, and cosine annealing learning rate with an initial value of 0.025.Baselines. We compare generation-based and sampling-based DFKD methods, including DeepInv [65] , CMI [17] , DAFL [6] , ZSKT [36] , DFED [19] , DFQ [10] , Fast [15] , MAD [13] , DFD [35] , KAKR [39] , SpaceshipNet [66] , DFAD [16] , and DFND [5] .To evaluate the effectiveness of our ODSD, we compare it with SOTA DFKD methods regarding the student's performance, the effectiveness of the sampling method, and training costs.Experiments on CIFAR-10 and CIFAR-100. We first verify the proposed method on the CIFAR-10 and CIFAR-100 [27] reproduce the DFND using the unified teacher models, and the result is slightly higher than the original paper.As shown in Table 2 , our ODSD has achieved the best results on each baseline. Under most baseline settings, ODSD brings gains of 1% or even higher than the SOTA methods, even though students' accuracy is very close to their teachers. In particular, the students of our ODSD outperform the teachers on some baselines. As far as we know, it is the first DFKD method to achieve such performance. The main reasons for its breakthrough in analyzing the algorithm's performance come from three aspects. First, our data sampling method comprehensively analyzes the intra-class relationships in the unlabeled data, excluding the difficult edge data and significant distribution differences data. At the same time, the number of data in each class is relatively more balanced, which is conducive to all kinds of balanced learning compared with other sampling methods. Second, our knowledge distillation method considers the representation of low-dimensional and low-noise information and expands the representation of knowledge through data augmentation. The structured relationship distillation method helps the student effectively learn knowledge from both multiple data and its teacher. Finally, the knowledge of our ODSD does not entirely come from the teacher but also the consistency and differentiated representation learning of unlabeled data, which is helpful when the teacher makes mistakes. The previous methods ignore the in-depth mining of data knowledge, decreasing students' performance. Experiments on ImageNet. We conduct experiments on a largescale ImageNet dataset to further verify the effectiveness. Due to the larger image size, it is challenging to effectively synthesize training data for most generation-based methods. Generation-based methods train 1,000 generators (one generator for one class), resulting in a large amount of computational costs. In this case, our sampling method reduces the computational costs more significantly. We set up three baselines to compare the performance of our method with the SOTA methods. Comparison of Data Sampling Efficiency. To verify the sampling mechanism's effectiveness, we compare our APS method with the current SOTA unlabeled data sampling method DFND [5] . Three data sampling methods (random sampling, DFND sampling, and our proposed APS) are set on three different distillation algorithms, including: KD [23] , DFND [5] , and our ODSD method. Table 5 reports the results. For KD, we use the sampled data instead of the original generated data with L 𝐾𝐷 distillation loss. From the result, this setting is competitive, even better than the distillation loss of DFND.For DFND, we reproduce it with open-source codes and keep the original training strategy unchanged. We find the performance of the DFND sampling method is unstable, which causes it to be lower than random sometimes. For ODSD, we use the distillation loss in Equation (8) . Our proposed sampling method achieves the best performance in all three benchmarks and significantly improves performance. By comprehensively considering the data confidence, the data outliers, and the class density, our ODSD can more fully mine intra-class relationships of the unlabeled data. As a result, the sampled data are more helpful for subsequent student learning. Experiments about Semantic Segmentation. In addition to image classification tasks, our algorithm can also effectively solve the problem of DFKD in image semantic segmentation on the NYUv2 dataset. Mean Intersection over Union (mIoU) is set as the segmentation evaluation metric. No generation module is defined for our method, and other settings are the same as DFAD [16] . Table 6 shows segmentation results on the NYUv2 dataset. Our ODSD also achieves the best performance. Besides, we visualize the segmentation results of different networks to get more convincing results as shown in Figure 3 . "Input" and "Ground Truth" represent the input test data and their corresponding real labels. Most data-free distillation algorithms hide the code of the segmentation part, so it is not easy to make a visual comparison. Here, we choose DFAD as the baseline algorithm of visualization. Our proposed ODSD algorithm achieves better segmentation results than DFAD, especially for object contour segmentation. The slight noise around the contour is effectively suppressed. Further, through in-depth mining the knowledge from the data and teacher, our student have gained better understanding ability.We conduct the diagnostic studies on the CIFAR-100 dataset and use ResNet-34 and ResNet-18 as the teacher's and student's backbones with 50k and 150k sampled data. Distillation Training Objective. We first investigate our overall training objective (cf. Equation ( 8)). As shown in the experiments (1-4) of Table 7 , the model with L 𝐾𝐷 alone achieves accuracy scores of 74.39% and 77.27% on 50k and 150k data sampling settings. Adding L 𝑛 or L 𝑐 individually brings gains (i.e., 0.32%, 0.31%/ 0.43%, 0.44%), indicating the effectiveness of our proposed distillation method. Our method performs better withGround Truth Teacher ODSD (ours) DFAD To verify the distribution similarity between the sampled data and the original data of our APS sampling method and the DFND sampling method, we use t-SNE [49] to visualize the data feature distribution. The teacher uses ResNet-34 on the CIFAR-100 and ResNet-50 on the ImageNet. For both datasets, we reserve 100 classes from validation data. In addition, we also visualize the distribution of data obtained by random sampling as a baseline reference. Figure 4 shows the data distribution results of different sampling methods.Our clustering results are closer to the extracted features of the original data. For the more complex ImageNet, this advantage is further amplified. Reducing the distribution difference between sampled and original data helps reduce data label noise, which is the key for the student to perform well. 
conclusion: 
 Most existing data-free knowledge distillation methods rely heavily on additional generation modules, bringing additional computational costs. Meanwhile, these methods disregard the domain shifts issue and ignore the data knowledge. This paper proposes an Openworld Data Sampling Distillation method. We sample unlabeled data with a similar distribution to the original data and introduce low-noise representation learning to cope with domain shifts. To explore the data knowledge adequately, we design a structured knowledge representation. Comprehensive experiments illustrate the effectiveness of the proposed method, which achieves state-ofthe-art performance on various benchmarks. 
 
 Appendix:  


Decision: Thanks for authors's rebuttal and reviewers' contributions. All the reviews are positive and the work is accepted by MM.

Reviews: review: paper_summary: paper_summary: The paper "Sampling to Distill: Knowledge Transfer from Open-World Data" introduces a novel method called Open-world Data Sampling Distillation (ODSD) aimed at enhancing Data-Free Knowledge Distillation (DFKD). The motivation behind this work is to eliminate the reliance on computationally expensive generation modules and to leverage the vast amounts of readily available, low-cost, unlabeled open-world data while addressing the domain shift issues inherent in existing methods. Key contributions include the Adaptive Prototype Sampling (APS) module, which samples data closely resembling the original dataset to mitigate domain shifts, and the Denoising Contrastive Relational Distillation (DCRD) module, which builds structured relationships among multiple data examples to improve knowledge transfer. The ODSD method improves upon Chen et al.'s DFND by reducing computational costs and enhancing accuracy, achieving significant performance gains across various datasets such as CIFAR-100 and ImageNet, demonstrating improvements in accuracy on ImageNet.
strength_weakness: strength_weakness: 1. Theoretical Approach: The Adaptive Prototype Sampling (APS) and Denoising Contrastive Relational Distillation (DCRD) modules are theoretically robust. APS mitigates domain shift by sampling data that closely matches the original distribution, while DCRD enhances knowledge transfer by building structured relationships among multiple data samples.

2. Technical Correctness: The methodology is sound, with detailed explanations and justifications for APS and DCRD. The use of cosine similarity for outlier detection and structured relational distillation is well-founded, ensuring logical consistency and technical rigor.

3. Adequate Improvement and Evaluation: The paper provides extensive experiments on benchmark datasets. These evaluations compare ODSD against existing methods, showing significant improvements in accuracy and efficiency, with up to 9.59% accuracy gains on ImageNet.

4. Clarity: The paper is well-organized and clearly written, with explicit motivations, methodologies, and contributions. Detailed diagrams and tables enhance understanding, and the logical structure ensures accessibility for readers.
limitations: limitations: 1. Novelty: While the proposed ODSD method introduces a sampling-based approach to DFKD, the concept of using open-world data for knowledge distillation is not entirely new. Previous works, such as Chen et al.'s DFND, have explored sampling-based methods. The primary novelty lies in the specific implementation of APS and DCRD, but the overall approach may be seen as an incremental improvement rather than a groundbreaking innovation.

2. Limited Exploration of Different Domains: Although the method is tested on several datasets, its effectiveness across a wider range of domains and tasks is not fully explored. Additional experiments on more diverse datasets would strengthen the generalizability claims.
paper_summary: paper_summary: The paper proposes a novel approach to Data-Free Knowledge Distillation (DFKD) using an Open-world Data Sampling Distillation (ODSD) method. The approach aims to train student models effectively without the original training data or costly generative models. This method includes Adaptive Prototype Sampling (APS) to sample unlabeled data similar to the original data distribution and Denoising Contrastive Relational Distillation (DCRD) to reduce noise and capture data relationships. Experiments on CIFAR-10, CIFAR-100, NYUv2, and ImageNet datasets show state-of-the-art performance with reduced computational costs.
strength_weakness: strength_weakness: * The methodology is clearly explained,and the inclusion of figures helps in understanding the proposed method.
* The paper provides comprehensive experimental results on multiple datasets (CIFAR-10, CIFAR-100, NYUv2, and ImageNet). The results show significant improvements in accuracy and efficiency, supporting the claims of the proposed method's effectiveness.
limitations: limitations: Potentially Unfair Performance Comparison: The performance comparisons in the paper might not be entirely fair, as the time required for the teacher model to make predictions on the entire unlabeled open-world dataset is not accounted for. Additionally, the method involves sampling 150k or more examples for CIFAR-10/100, which far exceeds the original dataset sizes. The authors could provide a comparison of training hours to address this concern.
Generalization to Rare Image Categories: The datasets used, such as CIFAR and ImageNet, consist of common image categories, making it easier to sample relevant training data from an open-world dataset. It is uncertain whether the proposed method would perform as well on tasks involving rare image categories.
paper_summary: paper_summary: The paper introduces a framework named ODSD for Data-Free Knowledge Distillation, dedicated to addressing the issue that existing methods mostly rely on designing additional generative modules to synthesize substitute data, leading to high computational costs. It also points out the neglect of the abundant, easily accessible, low-cost, unlabelled open-world data. Through an adaptive sampling module, ODSD samples data from the open world close to the original data distribution and introduces low-noise representations to mitigate domain shift problems. It then constructs the structured relationships of data examples to utilize the knowledge in the data through the student model itself and the structured representations of the teacher. The method has shown notable improvements over various comparative baselines.
strength_weakness: strength_weakness: 1. The comparative experiments in the article employ numerous baselines, making the comparisons comprehensive.
2. The article demonstrates the computational efficiency of the method, significantly reducing the computational cost compared to generative approaches.
3. The visual results presented in the article are intuitive and demonstrate the effectiveness of the method in semantic segmentation tasks.
limitations: limitations: 1. The notation and formulaic expressions in the article could be improved. For example, the cosine similarity is directly denoted as cos(), which is misunderstood as a common trigonometric function and does not follow the conventional notation in the field of contrastive learning, such as using 'cos_sim' or 'sim'. Despite 'C' and 'K' being constants, the lowercase 'c' and 'k' are used to denote variables, and these are also employed as pronouns for prototype names. Additionally, the initial generation of prototypes mentions "is clustered [24]", leaving the reader uncertain about the actual process of obtaining prototypes.
2. The motivation of the paper is not well substantiated. The premise is that current methods overlook the massive amounts of easily accessible, low-cost, unlabeled open-world data. However, the distributional differences between datasets like ImageNet, CIFAR, and Flickr1M are minimal, or the paper should have provided statistics to display the distribution before and after selection. There is a lack of experiments with more diverse unlabeled open-world data to support this motivation. Furthermore, closely related distributions do not validate the effectiveness of the proposed DCRD module, a point not demonstrated by the ablation experiments.
3. The referencing style could be more appropriate. Expressions such as "the definition [22]," "are clustered [24]," and "with less noise interference [1]" are seldom used. Specific work names should be employed to provide clearer context. The expression of task definition in the summary is also lacking.
4. The structured relation repeatedly mentioned in the paper is not well defined, and it is also confusing why the relationship between the organizational structure of data can be easily learned through comparison learning.
paper_summary: paper_summary: This paper proposes a novel Open-world Data Sampling Distillation (ODSD) method for the DFKD task, which samples open-world unlabeled data to replace the fake data via an adaptive sampling module.  Extensive experiments demonstrate the effectiveness of ODSD.
strength_weakness: strength_weakness: 1) ODSD utilizes the open-world unlabeled data to perform the distillation process, requiring no additional generation costs.
2) An adaptive Prototype Sampling (APS) module is proposed to alleviate the domain shift issue between the unlabeled data and the original data.
limitations: limitations: Some concerns are listed below:
1) The adaptive prototype sampling module is proposed to obtain the substitution data with a more similar distribution to the original data. It is unclear why three score indicators (i.e., sc_i, so_i and sd_i) contribute equally to the total score (S_total), which should be further discussed. If not, which of them is more critical to the sampling process?
2) Missing some crucial literatures, such as
-- Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation, CVPR 2023.
-- Unpacking the Gap Box Against Data-Free Knowledge Distillation, T-PAMI 2024.