Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation

Authors: muquan  li
dongyang  zhang
yuan-fang  li
ke  qin
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 18:57:18

Abstract: Data-free knowledge distillation (DFKD) has emerged as a pivotal technique in the domain of model compression, substantially reducing the dependency on the original training data. Nonetheless, conventional DFKD methods that employ synthesized training data are prone to the limitations of inadequate diversity and discrepancies in distribution between the synthesized and original datasets. To address these challenges, this paper introduces an innovative approach to DFKD through diverse diffusion augmentation (DDA). Specifically, we revise the paradigm of common data synthesis in DFKD to a composite process through leveraging diffusion models subsequent to data synthesis for self-supervised augmentation, which generates a spectrum of data samples with similar distributions while retaining controlled variations. Furthermore, to mitigate excessive deviation in the embedding space, we introduce an image filtering technique grounded in cosine similarity to maintain fidelity during the knowledge distillation process. Comprehensive experiments conducted on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets showcase the superior performance of our method across various teacher-student network configurations, outperforming the contemporary state-of-the-art DFKD methods.

Main paper: 
 introduction: 
 Model compression is an essential task that seeks to reduce the size and complexity of deep models while maintaining their performance and functionality [42] . This becomes particularly significant in an era where large models are extensively used across a variety of platforms while computational resources are frequently limited. In the current wide variety of model compression technologies, knowledge distillation [17, 19, 52] stands out as a pivotal approach, which facilitates the transfer of knowledge from a complex and resource-intensive model, commonly known as the teacher model, to a more lightweight model referred to as the student model.Despite knowledge distillation has demonstrated substantial success across multiple domains, the conventional approach typically necessitates access to the original data used to train the teacher model [7, 17] . Acquiring such training data can be exceptionally challenging due to its high cost and privacy concerns [3] , which highlights the urgent demand for alternative approaches to training data acquisition. In response, recent literature has endeavored to address the issue of data scarcity by utilizing synthetic data in place of the original training data, a method referred to as data-free knowledge distillation (DFKD) [2, 4, 5] .Specifically, DFKD represents a fundamental paradigm within knowledge distillation, comprising two interrelated steps: synthesizing data that emulates the distribution of the original training data and distilling knowledge from teacher model to student model, as shown in Fig. 1 . The first aspect, data synthesis, is typically divided into two distinct strategies: noise optimization [2, 29, 45] and generative reconstruction [4, 5, 9, 46] . Noise optimization involves using optimization algorithms to modify the noise of the input data, whereas generative reconstruction harnesses a generator network to establish a mapping from low-dimensional noise to the intricate data manifold. Compared to the former, generative reconstruction based methods alternately conduct data synthesis in each round of adversarial iteration, avoiding substantial time consumption and guarantees the production of high-quality data [23] .Nevertheless, current synthetic methods based on generative reconstruction are still inadequate in fulfilling several critical needs.To enable the student model to exhibit robust generalization across a multitude of tasks, the synthetic data must possess a richness and diversity that goes beyond mere replication of the original training data [9, 32] . Techniques such as image rotation for data augmentation in CSD [27] and channel-wise feature exchange (CFE) in SpaceshipNet [47] are employed to generate varied images. However, neither approaches can fully achieve the balance between the diversity and fidelity of the synthetic data inherent in adversarial network, where the number of synthesis must be constrained to maintain reliable domain prior [23] , resulting in merely limited image generation while preserving the original image features. In addition, another concern is the necessity for the synthetic data to closely match the distribution of the original training data. Methods such as CutMix [48] , Mixup [50] , and SpaceshipNet [47] may inadvertently amplify unwanted noise in the synthesised image, consequently leading to potential distributional shifts in the final image set, as shown in Fig. 2 . Therefore, adopting an updated approach to achieve alignment between synthetic and authentic training data is crucial for accurately capturing underlying patterns and complexities.Recent researches have made it possible to generate richer visual representations with remarkable realism through the usage of diffusion models [1, 6, 43] . Specifically, these models excel at incorporating data with consistent semantics but varying information [43] , thereby meeting the data augmentation needs in DFKD. However, the diffusion model may occasionally incur bias in a small number of cases during augmentation, causing spurious augmentation to certain extend. To mitigate this concern, continued investigation is essential to refine the augmentation process, effectively minimizing the occurrence of low-quality augmented images.In this paper, we propose a novel DFKD method termed Diverse Diffusion Augmentation (DDA), which leverages diffusion models to enrich the diversity of the generated data and employs cosine similarity-based filtering technique to ensure the fidelity of the augmentation process. Acknowledging the limitations of traditional generative reconstruction methods in low-quality data synthesis, we tackle the issue of noise amplification in synthetic images by deliberately reducing the impact of generative adversarial networks through model inversion [9, 45] . Instead of relying solely on single data synthesis, such process is expanded by incorporating a diffusion augmentation step, as shown in Fig. 1 . In this setting, diffusion models are allowed to adaptively augment the images based on the semantic information understanding of student model within the images, thereby achieving a self-supervised data augmentation that enhances data diversity and constrain distribution bias simultaneously. To ensure augmentation fidelity, we further introduce cosine similarity to filter spurious augmentation. Referring to Fig. 3 , through the integration of diffusion augmentation and cosine similarity-based filtering technique, our proposed method overcomes the inherent limitations of generative reconstruction, striking a balance between data diversity and fidelity, both preand post-augmentation. To comprehensively evaluate the generalizability and robustness of our proposed method, we conducted experiments on CIFAR-10, CIFAR-100 [6] , and Tiny-ImageNet [22] . Extensive experiments provide concrete evidence that our method significantly outperforms state-of-the-art DFKD methods.Our key contributions can be summarized as follows:• We propose a novel DFKD method called DDA, which innovatively extends the conventional data synthesis in DFKD with data augmentation to further enhance data diversity, thus establishing a new paradigm for DFKD. • We are the first to introduce diffusion models as a means of data augmentation in DFKD, which enriches the semantics and mitigates the distributional bias in synthetic data. • To ensure the fidelity of the augmented images, we propose to use the cosine similarity method to filter out spurious augmentations. • Experimental results confirm the effectiveness of our proposed DDA, showcasing superior performance when compared to the contemporary state-of-the-art DFKD methods. 
related work 2.1 data-free knowledge distillation: 
 The objective of DFKD is to transfer knowledge from a pre-trained teacher model to a student model without direct access to the original training data. Early approaches, exemplified by Lopes et al. [25] , initially explore the utilizing of metadata from teacher model to reconstruct training data for knowledge distillation. Subsequent researches has largely moved away from metadata reliance and instead emphasizes alternative methodologies for knowledge transfer. One category of methods involves updating randomly initialized noise images by using optimisation algorithms to synthesize data. Nayak et al. [29] models the output of the teacher network as a Dirichlet distribution, while Yin et al. [45] regulates the distribution of synthesised images based on batch normalization statistics. Besides, CMI [9] asserts that data diversity enhances distillation performance and enables comparative learning to enhance instance [45] , CMI [9] and SpaceshipNet [47] are chosen to compared with our method. Obviously, our DDA is capable of achieving stronger instance distinguishability.discrimination. An alternative category of methods employs generative networks to synthesise training data [4, 5, 8, 9, 26, 28, 46] . These methods can be classified into non-conditional generative network-based methods [4, 5, 9, 26, 28] and conditional generative network-based methods [26, 46] , depending on whether or not they combine conditional vectors when sampling random noise. Notably, Spaceshipnet [47] utilizes features from previous synthetic images to carry out channel-wise feature exchange (CFE) and employs multi-scale spatial activation region consistency (mSARC) as the constraint of similar network regions. Additionally, CDFKD-MFS [13] explores distilling knowledge from multiple teacher networks without access to original data, which uses a student network with additional parameters and multi-level feature-sharing to learn from multiple teachers. However, conventional DFKD methods are incapable of essentially achieving the balance of data diversity and distributional consistency inherent in generative reconstruction methods [23] , causing less effective DFKD. In this paper, we propose a novel DDA method, which shifts the focus of synthetic data from adversarial networks to further data augmentation, overcoming inherent limitation in DFKD.Data augmentation plays a crucial role in improving the robustness and generalisation capabilities of machine learning models across various domains. Traditional data augmentation techniques, as outlined in [39] , typically involve fundamental transformations such as random flipping, cropping, and colour shifting, aimed at generating diverse versions of original images. Recent advancements in data augmentation have introduced mixup-based techniques [15, 20, 24, 30, 48, 50] , which enhance diversity by blending patches of two input images through convex combinations and adaptive sample mixing policies. Furthermore, generative models have emerged as a significant area of research in data augmentation, particularly in domains such as medical image augmentation [11, 36] , domain adaptation [18] , and bias mitigation [38] . These methods leverage trained or pre-trained Generative Adversarial Networks (GANs) [33] to generate images that adhere to desired distributions, which also facilitate dense visual alignment supervision and pixel-level annotation generation from limited labels. Notably, diffusion models [30, 31, 35, 51, 53] have demonstrated promise in generating training data in zero or few-shot settings, as well as producing challenging training examples. As research advances [34] , it is anticipated that further exploration of diffusion models will enhance the effectiveness of data augmentation techniques. In this work, we re-visit the DFKD paradigm from another perspective, where the ability of data augmentation is utilized to enhance data diversity.Diffusion models represent a significant advancement in generative models, offering remarkable capabilities in authentic image generation. While earlier methods like Variational Autoencoders (VAEs) [21] and GANs [33] laid the groundwork for realistic image synthesis, recent breakthroughs in this domain have predominantly been attributed to diffusion models. Demonstrated by [30, 31, 35, 51, 53] , diffusion models have exhibited superior sample quality compared to traditional GAN-based approaches. Moreover, the evolution of diffusion models has paved the way for advancements in highresolution image synthesis [34] and text-to-image generation [53] , facilitated by innovations such as classifier-free guidance [41] . Diffusion models trained on large-scale datasets like LAION-5B [37] Student model  x x x  x Contrastive Learningsyn k k F G M cos  ) (M   k ) (M  ＜ k self L ,e ,f θ ini w L KL L total L   ) , ( S S   I) (0, ñ N w syn f  ,EQUATIONThe BN regularization [45] process utilizes statistical information stored in the batch normalization layer of the teacher network as prior knowledge regarding the data. The regularization technique utilizes the running mean 𝜇 𝑙 and running variance 𝜎 2 𝑙 of the 𝑙-th BN layer, which encapsulate the feature statistics of the original training data. The BN regularization is computed based on the disparity between the feature statistics of the synthetic data and those of the original training data. Mathematically, the BN regularization can be represented as the distance between the feature statistics and the batch normalized statistics:EQUATIONwhere 𝜇 𝑙 (𝑥) and 𝜎 2 𝑙 (𝑥) denote the mean and variance of the feature maps at the 𝑙-th BN layer, respectively.Utilizing these two techniques, we introduce a unified inversion framework for data-free knowledge distillation:EQUATIONwhere 𝛼 and 𝛽 are parameters to balance L 𝑐𝑙𝑠 and L 𝑏𝑛 , respectively. However, the aforementioned model inversion framework lacks consideration on data diversity and distributional consistency, which can result in the redundant generation of duplicate or irrelevant samples. To remedy this problem, we propose to improve the existing framework by incorporating contrastive learning and expand the data synthesis into a composite technique based on the diffusion model, as shown in Fig. 3 . 
data synthesis: 
 Data diversity refers to the variability and distinctiveness within a training dataset [9] . A large amount of existing researches [9, 47] indicate that increased data diversity results in more robust instance discrimination and more effective knowledge distillation, even when the amount of data remains constant. Additionally, contrastive learning [44] represents a self-supervised technique that enables a neural network to learn the distinction between different instances, thereby serving as a appropriate metric for assessing data diversity.𝑑𝑖𝑠𝑐 is integrated into the contrastive learning framework, which is a multi-layer perception that takes the representation from the penultimate layer and the global pooling of intermediate features as input. A memory bank is introduced to store both historical and newly synthesized samples. For each image, a positive view 𝑥 + is constructed using random augmentation, while other images in memory bank are considered as negative views 𝑥 - 𝑝 . The correct pairing of images from these positive and negative views necessitates a formulation to define the contrastive loss L 𝑐 :EQUATIONwhere 𝑡𝑝 signifies the temperature parameter of knowledge distillation to soften the distribution and 𝑐𝑜𝑠 denotes the cosine similarity between individual data point 𝑥 in the new embedding space, which is mathematically expressed as the cosine of the angle between two vectors. The discriminator serves to assess data diversity by distinguishing between different views, thus pulling positive views closer together and pushing negative views further apart.The initial data synthesis task can be optimized through contrastive learning [44] , seamlessly integrated into the foundational framework of model inversion.A generator 𝑔 and a memory bank B are introduced into the enhanced model inversion framework and merely one batch of data are synthesized by the generator 𝑔 in each timestamp 𝑇 . At the beginning of timestamp 𝑇 , the teacher model is reinitialized for model inversion by generator 𝑔 and its initial potential features 𝑓 𝑖𝑛𝑖 along with the shared weights 𝜃 𝑤 are iteratively optimized. Hence, the model inversion framework is structured as follows:EQUATIONwhere L 𝑖𝑛 and L 𝑐 refer to the regular inversion framework in Eqn.3 and the data diversity contrastive loss in Eqn. 4. Parameters 𝛼 ′ and 𝛽 ′ are updated parameters utilized to balance these two losses, respectively. However, it is imperative to note that this inversion technique is constrained in achieving the balance of data diversity and distributional consistency [23] . To address this issue, the selfsupervised diverse diffusion augmentation strategy is employed to further augment the available data.This section introduces the diverse diffusion augmentation technique for self-supervised data augmentation. For the first time, diffusion models are incorporated into DFKD methods not only to generate diverse and informative augmented images but also to maintain semantic consistency across various distribution. The Stable Diffusion-V2 [1] is selected as the decoder 𝐷𝑖 𝑓 𝑓 with sampled noise 𝑛 ∼ N (0, 𝐼 ), which is capable of generating the images G from the potential features 𝑓 𝑠𝑦𝑛 converted by a image encoder [1] . After model inversion, the obtained potential feature 𝑓 𝑠𝑦𝑛 of a single synthetic image is optimized and then passed through the diffusion model to generate the augmented images. Consequently, the augmented images can be represented as:EQUATIONwhere G 𝑘 denotes the 𝑘-th augmented image. However, employing a single augmented data for knowledge distillation may prompt the student model to repetitively synthesize samples, which may not be conducive to optimal student performance. Therefore, we self-supervise this augmentation process, involving adaptively data augmentation guided by the comprehension abilities of students model, evaluated through its acquired semantic information from images in each training iteration. Specifically, the student model, typically a CNN model, comprises a feature extractor Φ with feature dimension 𝑑 and a classifier 𝜓 . The augmented image is classified by the classifier, and the resulting classification is employed to compute the classification cross-entropy loss L 𝑠𝑒𝑙 𝑓 based on the categories of the pre-augmented image:EQUATIONwhere 𝜓 (Φ(G 𝑘 )) denotes the predicted output of student model. The cross-entropy loss of this self-supervised task effectively reflects the ability of student model to comprehend the semantic information of the image, which can also contribute to enhancing the efficiency of data augmentation. Additionally, in comparison to other augmentation methods, our images generated using the diffusion model exhibit higher resolution and diversity. Some examples of these images are illustrated in Fig. 2 . Nevertheless, images generated solely from the potential features of the images may exhibit a certain degree of anisotropy, causing bite-sized portion of images to deviate excessively from the original images and diminishing the learning efficacy of the student EQUATIONwhere 𝐹 𝑠𝑦𝑛 represents the previous synthetic image by model inversion corresponding to the potential feature 𝑓 𝑠𝑦𝑛 . This filtering process eliminates spurious augmentations generated by the diffusion model, effectively striking a balance between the diversity and fidelity of the augmented data. The outcomes of the filtering are visualized in Fig. 4 , showcasing the improved quality of the augmented dataset.In the stage of knowledge distillation, a final prediction loss is synthesised to serve as a constraint for the student model to emulate the output of the teacher model. The Kullback-Leibler (KL) divergence is commonly employed to minimize the logarithm of the outputs of both the teacher and student models:EQUATIONAdditionally, feature maps are frequently employed to evaluate the loss of feature mapping between the two models. Furthermore, our self-supervised diffusion augmentation method integrates the image recognition ability of the student model into image augmentation, implementing a form of adversarial learning. Consequently, the total knowledge distillation loss can be mathematically expressed as:EQUATIONwhere 𝜂 𝐾𝐿 , 𝜂 𝜃 𝑤 ,𝑓 ,𝑒 and 𝜂 𝑠𝑒𝑙 𝑓 are the parameters to balance three loss terms. The complete process and algorithm of DDA are outlined in Fig. 3 and Alg. 1, respectively. In summary, DDA enables the provision of more knowledge from the teacher model, thus enhancing distillation efficiency. 
experiments: 
 In this section, we present a comprehensive set of experiments for DFKD, aiming to validate the efficacy of our proposed DDA method. We start by outlining the experimental setup, including the specific tools and configurations employed throughout the experimentation process. Next, we benchmark the performance of DDA against state-of-the-art DFKD methods to assess its potential as a promising paradigm in the field. Additionally, we conduct ablation studies to analyze the significance and effectiveness of the different components within our method.4.1.1 Models and Datasets. We implement knowledge distillation on several different models, including resnet [14] , vgg [40] , and wide resnet (wrn) [49] . Three popular classification datasets, namely CIFAR-10, CIFAR-100 [6] , and Tiny-ImageNet [22] are utilized as benchmarks to evaluate several existing DFKD methods. CIFAR-10 and CIFAR-100 each consist of 60,000 images with 32×32 resolution, of which 50,000 are used for training and the remaining 10,000 for testing. Alternatively, Tiny-ImageNet has 64×64 resolution images, encompasssing 100,000 training images and 10,000 test images. These three types of datasets also contain 10, 100 and 200 classes, respectively.4.1.2 Diffusion models. The data augmentation step occurs after generating data through model inversion. We employed Stable Diffusion-V2 [1] to augment the data obtained from the synthetic images, expanding three new images for each synthetic image. The guidance scale of the Stable Diffusion-V2 and the number of diffusion steps are set to 0.5 and 50 respectively. During model inversion, we update the generator using the Adam optimizer with a learning rate of 1e-3, which in turn synthesises 200 images per step, with 500 repetition steps. Subsequently, the student model is trained using a SGD optimizer with a learning rate of 0.1 and a momentum of 0.9, with cosine annealing decay of 1e-4.Table 1 elaborately presents the results of DFKD for state-of-theart (SOTA) methods in recent years, including DAFL [4] , ZSKT [28] , ADI [45] , DFQ [5] , CMI [9] and SpaceshipNet [47] , as well as our DDA. According to the classification of DFKD methods, DAFK, ZSKT, DFQ, CMI, SpaceshipNet, and DDA belong to the generative reconstruction type of DFKD, while ADI belongs to the noise optimization type. The main difference between our method and the baselines methods is the usage of a diffusion model to extend the data synthesis. For a fair comparison, we utilize the same teacher-student networks as visual backbones for all experiments. The results in Tab. 1 evidence that DFKD method guided by DDA quantitatively outperforms the current SOTA methods on all three commonly used datasets by a large margin.In addition, Fig. 2 further visualizes the data utilized in knowledge distillation training process of four methods: ADI [45] , CMI [9] , SpaceshipNet [47] and DDA. It is evident that our augmented images exhibit considerably clearer and more restorative characteristics with better diversity. The visualization also distinct character evidently that DeepInversion produces images with similar colour and texture, while the CMI method improves highly image diversity but neglects sufficient clarity and resolution. Additionally, Space-shipNet uses CFE network to enhance image content but still falls short of enabling recognition with the naked eye. In contrast, the images produced by DDA are readily recognizable even without magnification, facilitating an effective capture of the features and objects present in each image.The influence of cosine similarity. Before investigating the effects of other parameters on the distillation results, it is imperative to rigorously establish the positive correlation of cosine similarity with the final outcomes. An experiment was conducted on the CIFAR-10 dataset, employing wrn-40-2 to wrn-16-1 and resnet-34 to resnet-18 as the two teacher-student networks. Fig. 5 illustrates that the growth of cosine similarity improves test accuracy, indicating that a augmented image dataset with higher cosine similarity yields superior results.The effect of guidance scale and diffusion steps. This section investigates the influence of two critical parameters in the diffusion model, diffusion steps and guidance scale [1] , which are related to the cosine similarity of the augmented dataset. In a stable diffusion model, each diffusion step predicts noise from a Gaussian distribution, and the prediction of model relies more on the source condition as the guidance scale increases. Fig. 6 illustrates that the diffusion model, when applied for augmenting our synthetic data approach, produces highest quality images with 50 diffusion steps and guidance scale of 0.5.The effect of threshold parameter 𝝎. As stated in Sec 3.3, 𝜔 is utilized to filter augmented images below a specific cosine similarity threshold. We evaluated the results of using different values of 𝜔 on the CIFAR-10 and CIFAR-100 datasets using wrn-40-2 to wrn-16-1 as the teacher-student model network. Tab. 2 reveals that threshold parameter of 0.75 enhances knowledge distillation performance. A reasonable explanation is that a small threshold may result in low-quality knowledge being transferred, while a large threshold may decrease the diversity of data and constrain the generalisation of the student model. Such value of 0.75 is exactly moderate for the threshold, striking a balance that results in the best performance.The effect of the scale of the augmented dataset. We further explore the impact of augmented dataset scale on the results, building upon the experiments in Sec 4.3.3. As depicted in Fig. 7 , it is apparent that the testing accuracy exhibits remarkable growth with the increase in augmented data scale, while the growth in accuracy becomes marginal during the transition from 3 × to 4 × augmented images, and the incorporation of more augmented images undoubtedly diminishes the extent of model compression.To strike a balance between accuracy and model size, our work augments each synthesized image into three images. Moreover, we estimate the average of Fréchet Inception Distance (FID) [16] score for tripled number of augmented images, where a lower score indicates higher quality of images. Referring to the Tab. 3, our augmented images achieve the lowest score across all three layers, demonstrating their superior diversity from low-level features to high-level semantics.The effect of diffusion model and cosine similarity. The effects of two core modules in DDA: the diffusion model and the cosine similarity are investigated, where we disable each module both in conjunction and individually to evaluate its influence. As shown in Tab. 4, the test accuracy decreases by an average of 1.245% when the diffusion model is deactivated, and by an average of 0.225% when the cosine similarity is disabled. This indicates that the diffusion model has a more significant impact on the results, with its deactivation leading to a more substantial degradation in accuracy. Moreover, Fig. 8 shows the effect of parameter 𝜂 𝑠𝑒𝑙 𝑓 in Eqn. 10 where the term L 𝑠𝑒𝑙 𝑓 is proposed by ourselves. It can be observed that the model's accuracy nearly consistently improves with the increase of the 𝜂 𝑠𝑒𝑙 𝑓 , which demonstrates the effectiveness of our diffusion model and cosine similarity. 
conclusion: 
 In this paper, we pioneer the application of diffusion models in the domain of DFKD, achieving enhanced data diversity with reduced distributional bias. Additionally, to mitigate redundancy in data augmentation, we introduce a image filtering technique based on cosine similarity, which eliminates augmented images exhibiting significant deviations pre-and post-augmentation, resulting in improved performance. Through extensive experimentation on three widely-used datasets and various teacher-student model pairs, we have achieved state-of-the-art results, highlighting the high effectiveness of our DDA method. As diffusion models and large-scale models continue to evolve rapidly, investigating alternative diffusion models for data augmentation in knowledge distillation offers a promising direction for future research. 
 
 Appendix:  


Decision: This paper received two borderline accept and one borderline reject, making it a borderline paper. After the rebuttal process, many of the concerns have been addressed, and the authors have promised further modifications. Therefore, I lean towards accepting this paper. Despite some remaining concerns, the proposed method shows strong potential and achieves excellent performance in data-free knowledge distillation, contributing to the field.

Reviews: review: paper_summary: paper_summary: This paper introduces a Data-Free Knowledge Distillation (DFKD) method named DDA. DDA aims to address the diversity-fidelity dilemma in data synthesis in DFKD, with proposed diffusion augmentation and cosine similarity filtering. Experiments show the effect of DDA in generating high-fidelity samples in DFKD, which results in a better distillation performance and beats previous SOTA.
strength_weakness: strength_weakness: The proposed method is a novel application of diffusion models in DFKD. The idea is attractive, and intuitively, the idea makes sense and has potential.

Proven by the experiments, the performance is attractive.

The quality of the manuscript is above average.
limitations: limitations: Flaws in presentation. 

1) While the majority of the paper is easy to follow, some important parts of the proposed method are confusing. For example, in section 3.2.1, how the positive pairs and negative pairs are generated is not clear. Figure 3 indicates $x^-$ comes from a memory buffer, while there are not any descriptions in section 3.2.1. Also, in line 479, the "random transformations" are not clear. It is recommended to be specific.

Flaws in equations. 

1) The equations use the same symbols for different variables/values, and it can be misleading. For example, $x_j$ is used as labels in line 419 and Eqn. (1), while it is used as negative images in Eqn. (4). Another example is that $\tau$ is used as the temperature parameter of contrastive learning in Eqn. (4), while it is used as the temperature of KL divergence in Eqn. (9).

2) It might be a misunderstanding, but in line 547, $x_j$ should be the label, not $\uppsi(\cdot)$.


More information about experiments is suggested.

1) Since the author claims DDA is better than other methods in generating high-resolution images (in line 777). It is suggested to include experimental results on high-resolution datasets, for example, ImageNet or its subset. Such experiments would be valuable to the paper and could better ground the claim.

2) It is suggested that more information about the experiment be included to help reviewers assess the quality of the experiments. For example, the number of runs of the experiment conducted for a given setting and the error bars are important information. Also, it is suggested to include the training time for each method.

Summary:
The idea is great, and the quality of the paper is above average. However, I do have some questions/concerns. I'm glad to update the score if the concerns are properly addressed.
paper_summary: paper_summary: This paper introduces a diffusion model for data synthesis in DFKD to increase the diversity and fidelity of synthetic data.

DFKD is divided into two steps: data synthesis to emulate the original data (training data for a teacher model) and knowledge distillation from the teacher model to the student model. In particular, this paper focuses on the former.

Experiments show that the proposed DDA is superior to the state-of-the-art DFKD method.
strength_weakness: strength_weakness: 1. This is the first study to incorporate a diffusion model into DFKD data synthesis.

2. Experiments have shown the effectiveness of the proposed method and an analysis of the influence of hyperparameters. In particular, the effectiveness of the new ideas ($\mathcal{L}_{\text{self}}$ and the filtering by cosine similarity) are also analysed.
limitations: limitations: 1. Most of the framework appears to rely on CMI [9]. For example, the procedures in Sec. 3.1 and 3.2 would be already described in the literature [9], although with some variations.

2. Some lack of explanation of the proposed method. If a part of the method follows CMI [9], it should be clearly stated to enhance reproducibility. The following questions also remain:  
* What kind of network is $\textit{disc}$ in Sec. 3.2.1 (line 477)?  
* How is $f_{\text{syn}}$ in Sec. 3.3 (line 525) is obtained? It is stated that “the potential feature $f_{\text{syn}}$ converted by a image encoder [43]”, but there is no mention of "image encoder" or "potential features" in the reference [43].  
* What is $\mathcal{L}_{\theta{w}, f}$ in Eq. (10) (line 631)?  

3. About novelty. The most important point is how much the idea of incorporating a diffusion model into data synthesis in DFKD contributes to this field. Although not in the field of DFKD, several studies have already attempted to synthesize training data using the diffusion model [1*][2*].

[1*] He, Ruifei, et al. "IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?." Proceedings of the International Conference on Learning Representations. 2022.  

[2*] Shipard, Jordan, et al. "Diversity is definitely needed: Improving model-agnostic zero-shot classification via stable diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
paper_summary: paper_summary: This work focuses on DFKD (Data-Free Knowledge Distillation) and tries to solve the limitations of conventional DFKD methods: synthesized training data are prone to the limitations of inadequate diversity and discrepancies in distribution between the synthesized and original datasets. This work tries to use diffusion models to solve the above challenges through diverse diffusion augmentation (DDA). To be specific, they revise the paradigm of common data synthesis in DFKD to a composite process through leveraging diffusion models subsequent to data synthesis for self-supervised augmentation and introduce an image filtering technique grounded in cosine similarity to maintain fidelity during the knowledge distillation process.
strength_weakness: strength_weakness: 1.	The proposed method is the first to empower DFKD with diffusion models.
2.	The proposed method achieves the best performance across 3 datasets compared with the baselines.
limitations: limitations: 1.	The proposed method uses cosine similarity to maintain the fidelity of diffusion-generated images during the process of DFKD and uses an ablation study in Table 2 to obtain the optimal threshold. Would it be better if the authors could show us some examples (image pairs) with different cosine similarities? In that case, it could be more vivid for us to have a general idea about what kind of generated images are suitable for the distillation.
2.	Generally, as the authors stated in this work, the limitations of DFKD lie in the inadequate diversity and discrepancies in distribution between the synthesized and original datasets, which could be attributed to the inaccessibility of the training data. I totally agree with this statement. However, the proposed method directly adopts Stable-Diffusion-V2, which is trained with mountains of data collected from the Internet (maybe ImageNET included). As a result, if the diffusion model was trained with the datasets the authors use in their experiments, I am afraid that it violates the fundamental assumption of DFKD problems – the inaccessibility of training data, making the process non-DFKD. Consequently, I could hardly agree that it is fair to compare the proposed method with DFKD methods.
3.	The results of Figure 5 and Table 2 seem confusing. Figure 5 offers a 3D view of the influence of the cosine similarity on two teacher-student networks, whereas I would prefer a 2D chart with the accuracy annotated. Besides, from it, we can conclude that the larger the cosine similarity is, the better the test acc is. This conclusion seems to contradict the results in Table 2, which suggests that a too-high threshold harms the final accuracy. Could the author further clarify these two results?