Diversified Semantic Distribution Matching for Dataset Distillation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-23 09:48:43

Abstract: Dataset distillation, also known as dataset condensation, offers a possibility for compressing a large-scale dataset into a small-scale one (i.e., distilled dataset) while achieving similar performance during model training. This method effectively tackles the challenges of training efficiency and storage cost posed by the large-scale dataset. Existing dataset distillation methods can be categorized into Optimization-Oriented (OO)-based and Distribution-Matching (DM)-based methods. Since OO-based methods require bi-level optimization to alternately optimize the model and the distilled data, they face challenges due to high computational overhead in practical applications. Thus, DM-based methods have emerged as an alternative by aligning the prototypes of the distilled data to those of the original data. Although efficient, these methods overlook the diversity of the distilled data, which will limit the performance of evaluation tasks. In this paper, we propose a novel Diversified Semantic Distribution Matching (DSDM) approach for dataset distillation. To accurately capture semantic features, we first pre-train models for dataset distillation. Subsequently, we estimate the distribution of each category by calculating its prototype and covariance matrix, where the covariance matrix indicates the direction of semantic feature transformations for each category. Then, in addition to the prototypes, the covariance matrices are also matched to obtain more diversity for the distilled data. However, since the distilled data are optimized by multiple pre-trained models, the training process will fluctuate severely. Therefore, we match the distilled data of the current pre-trained model with the historical integrated prototypes. Experimental results demonstrate that our DSDM achieves state-of-the-art results on both image and speech datasets. Codes will be released soon.

Main paper: 
 related work 2.1 coreset selection: 
 The classical approach to compressing the original dataset is coreset selection or instance selection. In addition to random sampling from the original dataset, most existing strategies progressively select important data points based on some heuristic selection criteria. For example, Herding [5, 32] minimizes the distance between the feature space's center of the chosen subset and that of the original dataset by greedily incorporating one instance at a time during each iteration. K-Center [9, 25] selects data points that minimize the maximum distance between a data point and its nearest center. Gradient-Greedy [1] employs gradients as features to maximize the diversity of instances in the replay. However, these heuristic selection criteria do not guarantee that the selected subset is optimal for training the model, particularly in the deep neural networks.Existing Dataset Distillation methods can be classified into two types: Optimization-Oriented (OO)-based and Distribution Matching (DM)-based methods [36, 42] .Optimization-Oriented (OO)-based methods integrate information learned from the dataset into the synthesized dataset during model training, necessitating iterative bi-level optimization of both the model and distilled dataset. Wang et al. are the first to introduce the concept of dataset distillation from an optimization perspective, updating synthesized instances through meta-learning [30] . DC utilizes this bi-level meta-learning optimization approach to match gradients of the distilled dataset and the original dataset to simulate the training process [41] . DSA further improves data efficiency by introducing differentiable Siamese augmentation, enabling effective training of neural networks with augmented data [39] . IDC injects a differentiable multi-formation function into synthesized instances within the constraint of fixed distilled dataset storage [13] . Along these lines, various approaches have been proposed to generate distilled instances, including matching the training trajectories [2] , using data hallucination networks to construct sample features [26] , and selecting original sample features via clustering [18] . Although these methods have improved performance of the distilled data, they still follow the bi-level optimization paradigm to update the model and the distilled data alternately, which is difficult to be optimized and time-consuming.Distribution-Matching (DM)-based methods directly align the distribution of distilled dataset with that of the real dataset in the feature space [28, 40] . They avoid the bi-level optimization by leaving out the updating of the model. DM [40] is proposed to match the average feature representations of the original dataset 𝐹 by randomly selecting a model from the pre-trained models. Then, DSDM utilizes three key components to optimize the distilled data: prototype alignment (L 𝑝𝑟𝑜𝑡𝑜 ), semantic diversity alignment (L 𝑆𝐷 ), and historical model alignment (L ℎ ). After the 𝑡-th iteration of optimization, DSDM updates and stores the historically optimized prototypes 𝒉 (𝑡 ) in the memory bank.with those of the distilled dataset. However, this approach ignores the discriminative nature of the classes. To address this limitation, CAFE [28] employs a discriminative loss that utilizes the prototypes of the distilled instances as classifiers for the real instances. However, the embeddings to calculate the prototypes are extracted with a randomly initialized model, which are usually inaccurate and will affect the performance of distillation. To address this issue, IDM [42] introduces a queue of pre-trained models to find more accurate class centers. In addition, M3D [36] embeds the class centers into the Reproducing Kernel Hilbert space, which aligns the distributions more efficiently. We recognize that only aligning class centers while overlooking the semantic diversity in the original dataset will produce oversimplified distilled data, which are similar to the class centers. Therefore, our method considers semantic information to enhance the diversity of distilled data. 
preliminary: 
 Dataset distillation is to condense a large-scale training dataset T = {(𝒙 𝑖 , 𝑦 𝑖 )} | T | 𝑖=1 into a small-scale distilled dataset S = (𝒔 𝑗 , 𝑦 𝑗 ) | S | 𝑗=1 ,where models trained on S resemble those trained on T . Typically, the distilled dataset S is initialized with randomly selected original instances from T or random noise. Then, S is optimized by minimizing the information loss between the distilled and the original examples, which can be formulated as:EQUATIONwhere 𝐷 represents a distance metric such as Mean Square Error (MSE), and 𝜙 denotes the matching objective. As mentioned above, based on whether to perform a costly bi-level optimization, existing methods can be mainly divided into Optimization-Oriented (OO)based and Distribution-Matching (DM)-based methods [35, 36] .In OO-based methods, 𝜙 typically represents the gradients. In each iteration, to simulate different backbones, the parameters of a model are initialized to 𝜽 (0) . Then, in the outer loop, 𝜽 (0) is updated for N steps using T to generate gradients through a training procedure. Next, in the inner loop, S is optimized by minimizing the gradient matching losses between the original and distilled data. The loss function can be defined as follows:EQUATIONwhere 𝑃 𝜽 is the distribution for the initialization of network parameters, 𝑙 is the empirical cross-entropy loss and 𝜂 is the learning rate for updating the network.Although OO-based methods achieve superior performance, these methods incur significant time overhead due to their bi-level loops. To mitigate this challenge, DM-based methods aim to generate distilled data to approximate the original data by directly matching the class centers of the distilled dataset with those of the original dataset [40] . This objective can be formulated as:EQUATIONwhere 𝜽 𝐹 denotes the feature sub-network parameterized by 𝜽 , instantiated by the model without the output layer and the function 𝑔 represents the embedding function parameterized with 𝜽 𝐹 . 
method 4.1 overview: 
 We illustrate the framework of our proposed Diversified Semantic Distribution Matching (DSDM) in Figure 2 . To obtain accurate instance embeddings, we train multiple models using the original dataset. For the 𝑡-th iteration, we initialize a feature sub-network𝜽 (𝑡 )𝐹 instantiated by randomly selecting one of these pre-trained models. Then, we derive instance embeddings from both T and S. The extracted features are then used to compute prototypes 𝝁 S and 𝝁 T , as well as covariance matries 𝑪 𝒐𝒗 S and 𝑪 𝒐𝒗 T for each category in both T and S. The prototypes encapsulate inherent features, while the covariance matrices capture the semantic diversity of all features within each category. We then match the prototypes and covariance matries for each category. However, the feature distributions of S from different pre-trained models can exhibit significant discrepancies during the iteration process, as elaborated in Section 4.3. To mitigate the instability stemming from these differences, we align the historical prototypes of distilled data 𝒉 (𝑡 -1) with the current distilled data 𝝁 S . To preserve the historical prototypes for each pre-trained model, we establish a memory bank to memory historical integrated prototypes. Finally, we summarize the total loss to optimize distilled instances.In the deep neural networks, disparate instances belonging to the same category often undergo nuanced semantic transformations within their deeper feature representations [3, 27, 37] . Given that covariance matrices encapsulate both the interrelationships and variances among features, they provide a rich understanding of semantic characteristics. We argue that leveraging category-specific covariance matrices enables the representation of features with distinctive semantic transformations. Thus, we derive semantic directions from estimated covariance matrices, assuming a class-centered normal distribution. Our objective simplifies to matching covariance matrices of features generated for each category S to closely approximate those of T (i.e., N (𝝁 S , 𝑪 𝒐𝒗 S ) ∼ N (𝝁 T , 𝑪 𝒐𝒗 T )).Previous DM [40] utilizes initialized neural networks to measure the Maximum Mean Discrepancy (MMD) between the original dataset T and the distilled dataset S. However, aligning the embeddings with randomly initialized neural networks will lead to inaccurate embeddings, as neural networks are ignorant to the current task. Only accurate embeddings can capture effective semantic features. To mitigate this issue, we adopt a strategy of pre-training several models using T . At each iteration 𝑡, we randomly select one of these pre-trained models and utilize its last layer (excluding the output layer) as the feature extractor 𝜽 𝐹 ; 𝒔 𝑗 ), where 𝒇 𝑖 , f𝑗 ∈ 𝑅 𝑑 . Given the inherent differences in the direction of semantic features across classes, we should first align the inherent semantic features of each class between T and S. Thus, we utilize the euclidean distance to align the prototype of T , denoted as 𝝁 T,𝑐 , with the prototype of S, denoted as 𝝁 S,𝑐 . The loss function for this prototype alignment is defined as follows:EQUATIONwhere 𝐵 T 𝑐 , 𝐵 S 𝑐 represent a batch of data sampled from the same category 𝑙 𝑐 in T and S.When aligning the prototypes, we let the covariance matrices align to capture semantic diversity. The objective of semantic diversity alignment can be expressed as follows:EQUATIONEQUATIONwhere (𝑚, 𝑛) denotes the position of the covariance matrix.To enhance model generalization of distilled data across diverse evaluation tasks, we adopt multiple pre-trained models to capture various semantic features. However, notable differences emerge in the feature distributions of distilled data for the same class across different pre-trained models. In Figure 3 (a), we visualize the feature distributions of the distilled data from a category across six pretraining models. It is obvious that data distilled by different pretrained models are distributed differently. Figure 3 (b) demonstrates the matching loss between the distilled data and the original data under these six pre-trained models. We can see that the distance between the same distilled data and the same original data varies greatly among different pre-trained models, resulting that training loss for data distillation fluctuations severely when the model for feature extraction changes. To enhance the stability of distillation performance over multiple pre-trained models, we introduce a memory bank to store the integrated prototypes of distilled data optimized by previous pre-trained models. Then, we align the distilled data of the current model with the corresponding integrated prototype 𝒉 (𝑡 -1) 𝑐 . Thus, we compute the historical prototype alignment loss as:EQUATIONTo store and update the prototypes used for historical prototype alignment, we first compute prototypes 𝒉 where S𝑐 represents the distilled data of category 𝑙 𝑐 after the 𝑡th optimization. Next, we update the integrated prototypes in the memory bank:EQUATIONwhere 𝛼 is a smoothing factor, typically set to be 0.99.In a nutshell, we minimize the sum of the above losses, including the prototype alignment loss L 𝑝𝑟𝑜𝑡𝑜 , semantic diversity alignment loss L 𝑆𝐷 , and the historical prototype alignment loss L ℎ . Combining these loss terms together, our diversified semantic distribution matching loss can be formulated as:EQUATIONwhere 𝜆 1 and 𝜆 2 are hyperparameters for balancing these terms.The pseudocode for DSDM is provided in Algorithm 1. In addition, we adopt partitioning and expansion augmentation for the initialization of S, as proposed in IDM [42] , aims to increase the number of representations extracted from S without incurring additional storage costs. Specifically, employing a factor parameter 𝑙, each distilled instance is factorized into 𝑙 × 𝑙 mini-examples, which are then up-sampled to their original size during training. This Compute semantic diversity matching loss L 𝑆𝐷 with Eq.( 5).8Compute historical prototype alignment loss L ℎ with Eq.( 6).Compute the total loss L 𝑡𝑜𝑡𝑎𝑙 with Eq.( 9). approach effectively maximizes the utilization of storage space in S. We integrate the same augmentation technique into our work, thereby further leveraging its advantages in aligning distributions for enhanced diversity. 
experiments 5.1 experimental setup: 
 Datasets. To evaluate the effectiveness of our dataset distillation method, we conduct experiments on five image datasets and one speech dataset. The five image datasets are MNIST [15] , Fashion-MNIST (FMNIST) [34] , SVHN [19] , CIFAR-10 [14] , and CIFAR-100 [14] . Specifically, MNIST is a black and white dataset consisting of 60,000 training images and 10,000 test images from 10 different classes. SVHN is a color dataset and consists of 73,257 digits and 26,032 digits for training and testing respectively. CIFAR-10/100 consists of 50,000 training images and 10,000 test images from 10 and 100 different classes respectively. For the speech dataset, we use the Mini-Speech-Commands dataset [7] , consisting of 8000 one-second audio clips representing eight command categories.Implementation Details. We follow the configuration of DM [40] to implement our method. Specifically, we use a 3-layer convolutional network (ConvNet-3) and a 4-layer convolutional network (ConvNet-4) [24] for the image and speech datasets, respectively. We initialize 𝑀 = 10 pre-trained models, each trained for 𝐸 = 20 epochs. We execute 𝑇 = 10𝐾 matching optimization iterations. For the weight parameters, we set 𝜆 1 = 50 and 𝜆 2 = 0.2 (see section 5.5 for further analysis). Moreover, we set the optimization learning rate for distilled instances 𝜂 𝐷𝑆𝐷𝑀 = 0.1. Following IDM [42] , we fix the factor parameter 𝑙 to 2 across all datasets. We use Top-1 test accuracy on the test set of the original dataset as the evaluation metric to assess the performance of a network trained on the distilled dataset. The evaluation task is trained for 1500 epochs. We perform 5 experiments and report the mean and standard deviation of the results. Compared Methods. We compare our DSDM with several state-of-the-art baselines across three categories: Coreset Selection, Optimization-Oriented (OO)-based methods, and Distribution Matching (DM)-based methods. For coreset selection methods, we consider the following: Random [4, 23] , Herding [32] and K-Center [9, 25] . For OO-based methods, we include: Dataset Distillation (DD) [30] , Dataset Condensation (DC) [41] and DSA (adding differentiable Siamese augmentation for DC) [39] . Although recent OO-based methods [18, 20, 21] achieve better performance, we do not compare them due to the remarkable difference in training scheme and cost. For DM-based methods, we include: CAFE [29] , its variant CAFE+DSA [29] , DM [40] , IDM [42] , and M3D [36] .7UDLQLQJ6WHSV 7HVW$FFXUDF\ &,)$5,3& '0 '6'0ZR)DF '6'0 7UDLQLQJ6WHSV 7HVW$FFXUDF\ &,)$5,3& '0 '6'0ZR)DF '6'0 7UDLQLQJ6WHSV 7HVW$FFXUDF\ &,)$5,3& '0 '6'0ZR)DF '6'0 7UDLQLQJ6WHSV 7HVW$FFXUDF\ &,)$5,3& '0 '6'0ZR)DF '6'0The performance of our DSDM and that of our competitors on the five image datasets is shown in Table 1 . From these results we can draw the following conclusions: Firstly, our DSDM consistently outperforms current state-of-the-art techniques, especially when the IPC in the synthetic dataset is set to 10 and 50. In particular, our method shows significant improvements over the suboptimal method, M3D, by 3.0% and 5.9% on CIFAR-10, respectively. Secondly, using the original dataset (Whole) for training represents the upper bound performance for dataset distillation. Encouragingly, we observe a marginal degradation of only 4.1% at a condensation ratio of 0.7% on SVHN, and a slight decrease of 2.2% at a condensation ratio of 10% on CIFAR-100. These results suggest that our method has the potential to achieve lossless compression by maintaining semantic diversity. Thirdly, the improvements of Our DSDM are limited when IPC=1. This is because the covariance matrix of only one distilled instance cannot be estimated. However, our DSDM Following the setting in previous work [13] , each speech data is preprocessed to be a magnitude spectrogram with the size of 64×64. For the factor parameter 𝑙, each spectrogram is factorized into 2 miniexamples along the time axis. Table 2 shows the test accuracy for mini-speech commands. It can be seen that our DSDM also greatly outperforms the baseline method, confirming its effectiveness in the speech domain. This means that our approach can be useful in both the image and the speech domain.A more important issue for distilled data is whether they can generalize well to different networks in the evaluation tasks. In Table 3 we show the performance of our distilled dataset (optimized on ConvNet-3) on the evaluation task trained with ConvNet-3, ResNet-10 [10] and DenseNet-121 [11] . Notably, our DSDM not only outperforms other methods on specific distillation model (ConvNet-3) but also shows significant performance gains on other unseen network architectures (ResNet-10 and DenseNet-121). This robust crossarchitecture generalization suggests that our distilled dataset with enhanced semantic diversity and integrated information from different pre-models can generalize well to different backbones in the evaluation tasks. (3) Historical prototype alignment emphasizes the capacity to maintain the semantic diversity of feature distributions across various models, it can further boost performance. In summary, the three designed loss functions are complementary and together improve the overall performance.Effect of Pre-trained Epochs. The epochs 𝐸 for pre-training directly affect the accuracy of instance embeddings (further affect the prototypes and semantic diversity for distillation) extracted by the pre-trained models. To show their effects, we pre-train Con-vNets on CIFAR-10 by 𝐸 in {1, 5, 10, 15, 20, 25, 30} and utilize these pre-trained models to perform our DSDM for IPC=50. As shown in the left of Figure 5 , there is a gradual improvement in performance with increasing epochs until reaching optimal performance at 𝐸 = 20. After that, increasing epochs do not significantly affect the performance, indicating that our DSDM only needs a few additional pre-training epochs to acquire better performance.Effect of Number of Pre-trained Models. More pre-trained models mean the distilled data generalize better for the evaluation task. However, more pre-trained models also occupy more training time. Therefore, we perform our DSDM with different numbers of models 𝑀 ∈ {1, 5, 10, 20} with the pre-training epochs 𝐸 = 20. As illustrated in the right of Figure 5 , we show the distillation performance. When M is set to 1 and 5, the performance is slightly worse. This indicates that the number of models has a significant effect on the distillation performance. Despite this, our DSDM achieves optimal performance with only 10 pre-trained models with little storage and computation consumption.Effect of 𝜆 1 . The parameter 𝜆 1 is a weighting parameter for the degree of improved semantic diversity. effect of 𝜆 1 on the distillation performance for IPC=50 on CIFAR-10. We find that performance improves as the range of 𝜆 1 goes from 10 to 50, while the effect of increasing the range of 𝜆 1 from 50 to 100 is almost negligible. So we set 𝜆 1 to 50. Furthermore, we observe that our DSDM improves by 2.3% over the suboptimal M3D result despite 𝜆 1 = 0.01, which further illustrates the importance of semantic diversity matching in dataset distillation. Effect of 𝜆 2 . Similarly, we observe the effect of the weighting parameter 𝜆 2 on the performance of the historical prototype. When 𝜆 2 = 0.2, our DSDM performance reaches the optimal value. As 𝜆 2 continues to increase, the performance decreases, which may be because the integration of the historical model is quite different from the current model, causing the current model to deviate from the class center of the original data.Distribution Visualization. To qualitatively illustrate the effectiveness of our DSDM in modeling the distribution of the original dataset, we use t-SNE to visualize the distilled data of the same category on CIFAR-10 for IPC=10/50 obtained by DM and DSDM, respectively. The results are depicted in Figure 7 , where the "green dots" and the "red stars" stand for the real data and distilled data. It is obvious that the distilled data obtained by DSDM distribute more dispersedly than those of DM, especially for the distilled instances in the blue circles.Distilled Images. To evaluate whether our method captures the semantic diversity of the original dataset, we visualize the distilled data of DM and our DSDM on CIFAR-10 in Figure 8 . Each class contains 10 images. Our observations lead to two key conclusions: (1) The distilled data of our DSDM are more different compared to those of DM, proving that our covariance matrix alignment captures the semantic diversity of the original dataset. (2) The distilled data of our DSDM is more discriminative between each class, indicating that the class center of the pre-trained model effectively preserves the inherent features of the classes. 
conclusion: 
 In this work, we propose a novel approach called Diversified Semantic Distribution Matching (DSDM) for Dataset Distillation, which explicitly aims to capture the semantic diversity features of the original dataset. DSDM consists of three carefully designed losses: prototype alignment, semantic diversity alignment, and historical prototype alignment. The prototype alignment and semantic diversity alignment modules are employed to capture the distribution of the original dataset. The historical prototype alignment enables DSDM to perform more stably during the training process. Experimental results across five image datasets and one speech dataset demonstrate that our DSDM outperforms existing techniques and generalizes well across different model architectures. In the future, we will explore how to better capture semantic diversity in more challenging scenarios of a low condensation ratio (e.g., IPC=1). 
original data deep networks: 
 Perspective Covariance Matrix Prototype Matching Diversity Matchingon an impressive 400 million data, consuming thousands of GPU days [17, 22, 31] . This poses an interesting issue: how to summarize the large-scale multimedia data into a compact but insightful subset to facilitate the training procedure while maintaining similar performance. To achieve this purpose, an intuitive approach is to select representative instances from the original dataset, known as coreset selection [4, 9, 23, 25, 32] . However, sample selection involves directly discarding a significant portion of instances from the original dataset, leading to the loss of crucial underlying information essential for model training. Fortunately, Dataset Distillation (DD) has been proposed as a solution to condense the original largescale dataset into smaller ones (i.e., distilled dataset) while ensuring that models trained on the distilled dataset exhibit performance comparable to those trained on the original dataset [6, 16, 30] .Existing methods for dataset distillation can generally be categorized into two main groups: Optimization-Oriented (OO)-based methods and Distribution-Matching (DM)-based methods [36, 42] . OO-based methods employ a bi-level optimization process [2, 8, 30, 41] , where the model is first updated with the original dataset, and then, the distilled dataset is optimized by minimizing the distance between the original and distilled data. Although OO-based methods have demonstrated good performance, they are often criticized for their significant time consumption during training. In contrast, DM-based methods are proposed to improve the training efficiency by eliminating model updates [28, 38] . Specifically, DM [40] optimizes the the distilled data by matching corresponding prototypes calculated with a random initialized model on the distilled and original datasets. However, the embeddings generated by a randomly initialized model tend to be inaccurate, which can impact the performance of the distilled dataset. To tackle this issue, IDM [42] utilizes a sequence of pre-trained models to obtain accurate prototypes. Although these prototypes are more accurate, they ignore the diversity of the distilled dataset, which will reduce the generalization of the models trained on it.As illustrated in Figure 1 , when only prototypes of the distilled data and the original data are aligned to optimize the distilled data, the algorithm may lazily minimize the alignment loss by distilling instances near the center of each class into the distilled data. As a result, all the distilled instances contain similar information, which overlooks the diverse and meaningful semantic information in the original data. Besides the prototype, the covariance matrix is another important statistic to depict the variations of a category, where each item in the covariance matrix reflects the transformation from one-dimensional semantic feature to another. As shown in Figure 1 , suppose the "purple, orange, blue" in the covariance matrix represent the transformation from standing to "color transformation, posture transformation, and perspective transformation", applying these transformations to the a white, frontal and standing horse can obtain horses of "brown, lying, and running". Therefore, the covariance matrix carries rich semantic diversity information. Inspired by this, we can improve the diversity of the distilled data by aligning their covariance matrix to that of the original data to make the distilled and original data distribute similarly.In this paper, we propose a simple yet efficient Diversified Semantic Distribution Matching (DSDM) method for dataset distillation, aiming to obtain semantic diversified distilled data. To acquire accurate semantic information from the original data, we first pre-train a set of models on the original dataset. Then, each category in the original dataset is modeled into a Gaussian distribution with the semantic embedding extracted by a pre-trained model, where the prototype represents the intrinsic feature of a category, and the covariance matrix represents the semantic variations of instances in this category. Subsequently, we align the prototype and covariance matrix of the corresponding distilled data to make the distribution of the distilled data similar to that of the original data. With the additional alignment of the covariance matrix, our distilled data will be distributed dispersedly in the space of the original data rather than gathered around the class centers, ensuring the performance of the distilled data in the evaluation tasks. In addition, since the distilled data are optimized by multiple pre-trained models to make them generalize well to different models, the training process will fluctuate severely. Therefore, we introduce a memory bank to store the integrated prototypes of distilled data optimized by previous pre-trained models. Finally, the distilled data of the current pretrained model are aligned with the historical integrated prototypes to ensure that the distilled data generalize well across various models. Empirical evaluations conducted on five image datasets and a speech dataset demonstrate that our method not only preserves the efficiency of DM-based approaches but also exhibits significant performance improvements.The contributions of this paper are summarized as follows:• We introduce a novel approach called Diversified Semantic Distribution Matching (DSDM) for dataset distillation. To the best of our knowledge, this is the first work to align features from a semantic distribution perspective, thereby enhancing the semantic diversity of distilled dataset.• We analyze the disparities in feature space distributions among different models. To ensure stable optimization of distillation, we align the prototype of the current distilled data with the historically optimized distilled data. • We conduct a series of experiments to thoroughly validate the effectiveness of DSDM on both image and speech datasets. Through rigorous experimentation, our DSDM has demonstrated state-of-the-art performance with robust generalization capabilities across various scenarios. 
 
 Appendix:  


Decision: The paper proposes a novel dataset distillation method named Diversified Semantic Distribution Matching (DSDM). This method aims to enhance the semantic diversity and generalization capability of distilled datasets by aligning the semantic distributions of the original and distilled datasets. The experimental results across multiple image and speech datasets demonstrate the method's effectiveness, achieving state-of-the-art results.

Pros:
+ The paper is well-organized and easy to follow.
+ The use of covariance matrix matching to enhance the semantic diversity of distilled datasets is a novel and effective contribution.
+ The method achieves state-of-the-art results on multiple benchmarks, demonstrating its potential for widespread adoption in different domains.

Cons:
+ Some details have not been well solved according to the reviewers.

Given the 4 positive reviews and it may be the first dataset distillation work in ACM MM, I recommend accepting this paper for an Oral presentation.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel dataset distillation method named Diversified Semantic Distribution Matching (DSDM). This method aims to enhance the semantic diversity and generalization capability of distilled datasets by aligning the semantic distributions of the original and distilled datasets. The experimental results across multiple image and speech datasets are great.
strength_weakness: strength_weakness: 1. DSDM employs pre-trained models to extract semantic embeddings and models each class as a Gaussian distribution, which is novel in DM methods.
2. This matching strategy circumvents the complexity of bi-level optimization and ensures the distilled data maintains a high degree of semantic richness.
2. The experimental results across multiple image and speech datasets demonstrate superior generalization capabilities in various scenarios, highlighting its potential for widespread adoption in different domains​​.
limitations: limitations: 1. The motivation and key component of this paper should be capturing the semantic features. However, the factor technology plays a greater role in improving performance. This undermines the paper's motivation and the method's validity.
2. I wonder how to set the value of the two hyperparameters 𝜆1 and 𝜆2.
3. The reliance on multiple pre-trained models to extract semantic embeddings adds to the complexity of the implementation. 
4. This paper does not compare the time complexity with other DM methods. I would like to know how much additional time the DSDM method requires compared to DM, (e.g., calculating the covariance matrix).
5. If all the pre-trained models used are ConvNet-3? Have different architectures been tried to compose the model pool?
paper_summary: paper_summary: This paper upgrades the classic distribution matching method in Dataset Distillation by matching the prototype and covariance matrix. For better stability of training, the authors introduce the memory bank to save history synthetic data. The results on multiple datasets show that the new method outperforms several competitors. Ablation study proves that the upgraded distribution matching suppresses the classic one.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow. 
2. The proposed method is well motivated. The memory bank is well-designed and can improve the training stability.
3. Experimental results are competitive to similar methods. Ablation study on the proposed modules has been provided and well justified. Hyper-parameters have been examed.
limitations: limitations: 1. The paper lacks justification why no comparison to some popular methods, such as MTT. 
2. The setting and details of the experiments on speech data are unclear.
paper_summary: paper_summary: In this paper, authors propose a Diversified Semantic Distribution Matching framework for efficient dataset condensation. Results on benchmarks prove the effectiveness or proposed method.
strength_weakness: strength_weakness: 1.The written is clear and easy to follow. 

2.The expansion to semi-supervised setting is reasonable and results seem promising. 

3.high completion quality.
limitations: limitations: 1. The method and model performance rely heavily on a large number of pretrained models, which significantly hinders the method's effectiveness. The feasibility of the current method is based on small-scale datasets like CIFAR-10/100. If the scale of data is increased to a very high level, such as millions of images, the cost of pre-training models becomes non-negligible. IDM also has this problem. Could the authors provide some theoretical or empirical insights and discussions on the relationship between the number of pre-trained models and the scale of the dataset? The results in Figure 5 are not solid.


2. Have the authors considered the scalability limit of the dataset distillation task? Assuming, it is unreasonable to compress an image dataset with 10,000 classes, millions of images, and a resolution of 3 \* 224 \* 224 into IPC=1, 10, 50. Then, if we suppose each class is to be compressed into 1000 synthetic samples, our optimization target becomes a huge 5D matrix of 10,000 \* 1000 \* 3 \* 224 \* 224, which will significantly increase resource consumption and computational costs, even if compression is done class by class. Could the authors further discuss the advantages and disadvantages between dataset distillation and dataset selection, and whether the designed method could be applied to dataset selection? Of course, this point does not seriously constitute a limitation of this article, but I would like to see the authors' discussion and thoughts on this issue.
paper_summary: paper_summary: This paper proposes a diversified semantic distribution matching method for dataset distillation. Specifically, the distribution matching is conducted on both prototype embeddings and covariance matrix reflecting the sample relationship. The proposed method achieves state-of-the-art performance on multiple dataset distillation benchmarks. The covariance matrix matching significantly improves the validation performance, supporting its effectiveness.
strength_weakness: strength_weakness: 1. The idea of matching covariance matrix further enhances the potential of distribution matching for DD. 
2. The writing is generally easy to follow. 
3. The effectiveness of the proposed method is well supported by the experimental results.
limitations: limitations: 1. In terms of diversity matching, a more intuitive idea would be matching the feature variance. Can authors explain why covariance is used here instead of variance? Are there theoretical or empirical analysis supporting the choice?
2. Authors claim that when only matching prototypes, "the algorithm may lazily minimize the alignment loss by distilling instances near the center of each class into the distilled data". However, in Fig. 7, the naive DM actually yields a distilled dataset that well covers the whole distribution instead of clustered at the class center. 
3. The instability caused by different model architectures is not necessarily harmful. And aligning the historical prototypes doesn't bring obvious performance improvement in Tab. 4. The contribution of this part requires further evidence. 
4. Minor:
- Line 364. Why "sub-network"?
- Line 368. "matries" -> "matrices"
- Reference not sufficient. More recent works should be investigated.