Title: Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dataset Distillation

Synthetic Data

Diversity

Generatlization
Abstract: The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataset distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough to replace the original dataset in training a neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, we provide a thorough theoretical and empirical analysis of diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, we introduce a novel method that employs dynamic and directed weight adjustment techniques to modulate the synthesis process, thereby maximizing the representativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of a large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available at https://github.com/AngusDujw/Diversity-Driven-Synthesis.

Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment

Abstract

The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataset distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough to replace the original dataset in training a neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, we provide a thorough theoretical and empirical analysis of diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, we introduce a novel method that employs dynamic and directed weight adjustment techniques to modulate the synthesis process, thereby maximizing the representativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of a large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available at https://github.com/AngusDujw/Diversity-Driven-Synthesis.

1 Introduction

With the rapid growth in dataset size and the need for efficient data storage and processing [8, 17, 14, 13], how to condense datasets while preserving their key characteristics becomes a significant challenge in machine learning community [12, 38]. Unlike previous research [29, 39, 50, 44] that focuses on constructing a representative subset through selecting from the original data, Dataset Distillation [43, 31, 20] aims to synthesize a small and compact dataset that retains informative features from the original dataset. A model trained on the synthetic dataset is thus supposed to achieve comparable performance as one trained on the original dataset. The development of dataset distillation reduces data-related costs [7, 34, 49] and helps us better understand how Deep Neural Networks (DNNs) extract knowledge from large-scale datasets.

Numerous studies dedicate significant effort to synthesizing distilled datasets more effectively. For example, Zhao et al. employ a gradient-matching approach [52, 54] to guide the synthesis process. Trajectory-matching methods [1, 2, 5, 6] further align gradient trajectories to optimize the synthetic data. Additionally, distribution matching [42, 53, 55] and kernel inducing points methods [28, 25, 23, 24] also contribute to synthesizing representative data. Despite the great progress achieved by these methods on datasets like CIFAR [16], their extensive computational overhead (both GPU memory and GPU time) hinders the extension of these methods to large-scale datasets like ImageNet-1K [3].

t-SNE visualization on CIFAR-100

SRe2L

DWA (ours)

Tiny-lmageNet

lmageNet-1K

CIFAR100

30

35

40

45

50

55

60

65

Accuracy (%)

41.1

46.8

47.9

52.8

55.2

60.3

SRe2L

DWA (ours)

Several recent works [2, 46, 22, 51, 57] have attempted to address the efficiency issues of dataset distillation. In particular, Yin et al. [46] propose a lightweight distillation method, SRe2L, which successfully condenses the large-scale dataset ImageNet-1K. Unlike previous methods [1, 53, 15] that treat the synthetic set as a unified entity to utilize the mutual influences among synthetic instances, SRe2L synthesizes each synthetic data instance individually. As such, SRe2L significantly reduces both GPU memory costs and computational overhead.

Individually synthesizing each data instance can efficiently parallelize opti- mization tasks, thereby flexibly managing GPU memory usage and computational overhead. However, this approach may present challenges in ensuring the representativeness and diversity of each instance. If each instance is synthesized in isolation, there may be a risk of missing the holistic view of the data characteristics, which is crucial for the training of generalized neural networks. Intuitively, SRe2L might expect that random initialization of synthetic data would provide sufficient diversity to prevent homogeneity in the synthetic dataset. Nevertheless, our analysis, as demonstrated in Figure 1, reveals that this initialization contributes only marginally to diversity. Conversely, the Batch Normalization (BN) loss [45] in SRe2L plays the practical role in enhancing diversity of the distilled dataset.

Motivated by these findings, we further investigate the factors that enhance the diversity of synthetic datasets from a theoretical perspective. We reveal that the variance regularizer in the BN loss is the key factor ensuring diversity. Conversely, the mean regularizer within the same BN loss unexpectedly constrains diversity. To resolve this contradiction, we suggest a decoupled coefficient to specifically strengthen the variance regularizer's role in promoting diversity. Experimental results validate our hypothesis. We further propose a dynamic mechanism to adjust the weight parameters of the teacher model. Serving as the sole source of supervision from the original dataset, the teacher model guides the synthesis comprehensively. Our meticulously designed weight perturbation mechanism injects randomness without compromising the informative supervision, thereby improving overall performance. Importantly, our method incurs negligible additional computations ( < 0 . 1% ). Intuitively, our method perturbs the weight in a direction that reflects the characteristics of a large subset, varying with each batch of synthesized data.

We conduct extensive experiments across various datasets, including CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet-1K, to verify the effectiveness of our proposed method. The superior performance of our method not only validates our hypothesis but also demonstrates its ability to enhance the diversity of synthetic datasets. This success guides further investigations into searching for representative synthetic datasets for lossless dataset distillation. Our contribution can be summarized as follows:

We analyze the diversity of the synthetic dataset in dataset distillation both theoretically and empirically, identifying the importance of ensuring diversity in isolated synthesizing approaches.

We propose a dynamic adjustment mechanism to enhance the diversity of the synthesized dataset, incurring negligible additional computations while significantly improving overall performance. Extensive experiments on various datasets verify the remarkable performance of our method.

2 Preliminaries

Notation and Objective. Given a real and large dataset T = { (˜ x i , y i ) } |T | i =1 , Dataset Distillation aims to synthesize a tiny and compact dataset S = { (˜ s i , y i ) } |S| i =1 . The samples in T are drawn i.i.d from a natural distribution D , while the samples in S are optimized from scratch. We use θ T and

θ S to represent the converged weight trained on T and S , respectively. We define a neural network h = g ◦ f , where g acts as the feature extractor and f as the classifier. The feature extractor and the classifier loaded with the corresponding weight parameters from θ are denoted by g θ and f θ .

Throughout the paper, we explore the properties of synthesized datasets within the latent space. We transform both ˜ x , ˜ s ∈ R C × H × W from the pixel space, to the latent space, x , s ∈ R d , for better formulation. This transformation is given by x = g θ T (˜ x ) and s = g θ T (˜ s ) . The objective of Dataset Distillation is to ensure that a model h trained on the synthetic dataset S is able to achieve a comparable test performance as the model trained with T , which can be formulated as,

where ℓ can be an arbitrary loss function. The expression ℓ ( h θ T , x ) should be interpreted as ℓ ( h θ T , x , y ) , where y is the ground truth label.

Synthesizing S . A series of previous works mentioned in Section 5 have introduced various methods to synthesize S . Specifically, SRe2L [46] proposes an efficient and effective synthesizing method, which optimizes each synthetic instance s i by solving the following minimization problem:

where L BN denotes the BN loss, and λ is the coefficient of L BN . The detailed definition of L BN can be found in Equation 3. Minimizing the BN loss L BN significantly enhances the performance of SRe2L, which is designed to ensure that S aligns with the same normalization distribution as T . However, we argue that another essential but overlooked aspect of the BN loss L BN is its role in introducing diversity to S , which also greatly benefits the final performance. In the following section, we will analyze this issue in greater detail.

3 Methodology

Diversity in the synthetic dataset S is essential for effective use of the limited distillation budget. This section reveals that the BN loss, referenced in Equation 2, enhances S 's diversity. However, the suboptimal setting of BN loss limits this diversity. To overcome this, we propose a dynamic adjustment mechanism for the weight parameters of f θ T , enhancing diversity during synthesis. Finally, we detail our algorithm and theoretically demonstrate its effectiveness. The pseudocode of our proposed DWA can be found in Algorithm 1.

Algorithm 1 Directed Weight Adjustment (DWA)

,

3.1 Batch Normalization Loss Enhances Diversity of S

The BN loss L BN comprises mean ( L mean ) and variance ( L var ) components, defined as follows:

where µ l and σ 2 l refer to the channel mean and variance in the l -th layer, respectively. s i is optimized within a mini-batch S , where s i ∈ S and S ⊂ S . Each component of L BN operates from its own perspective to enhance dataset distillation. First, the mean component L mean regularizes the synthetic data s , ensuring its values align closely with those of the representative centroid of T in latent space. Second, the variance component L var encourages the synthetic data in S to differ from each other, thereby maintaining the variance σ 2 l ( S ) . Thus, this BN loss-driven synthesis can be decoupled as

where X c can be regarded as an optimal solution to Equation 2 when the variance regularization term L var is not considered, i.e. ,

where both α 1 , α 2 > 0 and α 1 , α 2 → 0 . ξ i represents a small perturbation and ξ i ∼ N ( 0 , σ 2 ξ ( λ L var ) ) . Therefore, the variance of the synthetic dataset S is,

We have Var ( X c ( λ L mean , θ T ) ) = 0 as X c is deterministic. Unlike other approaches that consider the mutual influences among synthetic data instances and optimize the dataset collectively, SRe2L [46] optimizes each synthetic data instance individually. Therefore, the diversity of the synthetic dataset S is solely determined by λ L var .

However, simply increasing λ contributes marginally to enhancing the diversity of S . This is because a greater λ will also emphasize the regularization term λ L mean , which contradicts the emphasis on λ L var . We provide a detailed analysis in the Appendix A.1. As a result, we propose using a decoupled coefficient, λ var , to enhance the diversity of S .

Additionally, the synthetic data instances are optimized individually to approximate the representative data instance X c . However, the gaussian initialization N (0 , 1) in pixel space does not distribute uniformly around X c in latent space, making the converged synthetic data instances to cluster in a crowed area in latent space, as dedicated in Figure 1. To address this, we propose initializing with real instances from T inspired by MTT [1], ensuring a uniform projection when synthesizing S .

3.2 Random Perturbation on θ T Helps Improve Diversity

In the previous section, we highlighted the often overlooked aspect of the BN loss in introducing diversity to S , which was also verified through experiments in Section 4.2. Building upon this, we propose to introduce randomness into θ T to further enhance S 's diversity, as it is the only remaining factor affecting Var( S ) , as shown in Equation 6.

Let x ∗ c = X c ( λ L mean , θ T ) to be the original optimal solution to Equation 2. We aim to solve the adjusted optimal solution x c = X c ( λ L mean , θ T +∆ θ ) = x ∗ c +∆ x , where θ T is randomly perturbed by ∆ θ , and ∆ θ ∼ N (0 , σ 2 θ ) . Consequently, we have:

To solve for ∆ x , we can apply a first-order bivariate Taylor series approximation because ∇ θ ℓ ( f θ T , X c ) ≤ α 1 , where α 1 → 0 , and both ∆ θ and ∆ x are small. Thus,

To satisfy Equation 7, we have:

Intuitively, ∆ x must compensate for the ∇ θ incurred by introducing the random perturbation ∆ θ ∼ N (0 , σ 2 θ ) on θ T . By Equation 9, Var(∆ x ) ∝ Var(∆ θ ) = σ 2 θ , then:

where β is determined by -∇ x [ ∇ θ ℓ ( f θ T , x c )] -1 ∇ 2 θ ℓ ( f θ T , x c ) , as shown in Equation 9. Therefore, the variance of the new synthetic dataset S ′ is greater than that of S without perturbing θ T .

3.3 Directed Weight Adjustment on θ T

Although perturbing θ T could significantly increase the variance of the synthetic dataset S , undirected random perturbation ∆ θ can also introduce noise, which in turn degrades the performance. We aim to address this limitation by directing the random perturbation ∆ θ without introducing noise into S . We propose to obtain directed ∆ θ by solving the following maximization problem:

where B ⊂ T represents a randomly selected subset of T , and | B | ≪ |T | . As such, ˜ ∆ θ will not introduce unanticipated noise when synthesizing S . The randomly selected B ensures that the randomness of ˜ ∆ θ continues to benefit the diversity of S . Next, we will demonstrate this theoretically.

Effective dataset distillation should provide concise and critical guidance from the original dataset T when synthesizing the distilled dataset. Here, this guidance is introduced primarily through the converged weight parameters θ T , i.e. ,

where θ T contains informative features of T because it achieves minimized training loss over T . We demonstrate that ˜ ∆ θ , obtained from Equation 11, decreases the training loss computed over T \ B , which, in fact, highlights the features of T \ B . By applying a first-order Taylor expansion, we obtain:

Since θ T is optimized until reaching a local minimum with respect to the loss function computed over the training set T , we have:

where 0 is the tensor of zeros with the same dimension as θ T . Substitute it back into Equation 13, we have:

L B ( f θ T + ˜ ∆ θ ) will clearly be greater than L B ( f θ T ) , as indicated by Equation 11. Thus, we demonstrate that the directed ˜ ∆ θ results in less noise and improved performance. In summary, after resolving ˜ ∆ θ as in Equation 11, our proposed method synthesizes data instance s i by solving:

4 Experiments

To evaluate the effectiveness of the proposed method, we have conducted extensive comparison experiments with SOTA methods on various datasets including CIFAR-10/100 ( 32 × 32 , 10/100 classes) [16], Tiny-ImageNet ( 64 × 64 , 200 classes) [18], and ImageNet-1K ( 224 × 224 , 1000 classes) [3] using diverse network architectures like ResNet-(18, 50, 101) [11], MobileNetV2 [33], ShuffleNetV2 [26], EfficientNet-B0 [37], and VGGNet-16 [35]. We conduct our experiments on the server with one Nvidia Tesla A100 40GB GPU.

Solving ˜ ∆ θ . Before we conduct our experiments, we propose to use a gradient descent approach to solve ˜ ∆ θ in Equation 11. There are two coefficients, K and ρ , used in the gradient descent approach. K represents the number of steps, and ρ normalizes the magnitude of the directed weight adjustment. The details for solving ˜ ∆ θ can be found in Line 7 of Algorithm 1.

Experiment Setting. Unless otherwise specified, we default to using ResNet-18 as the backbone for distillation. For ImageNet-1K, we use the pre-trained model provided by Torchvision while for CIFAR-10/100 and Tiny-ImageNet, we modify the original architecture under the suggestion in [10]. More detailed hyper-parameter settings can be found in Appendix A.2.1.

Baselines and Metrics. We conduct comparison with seven Dataset Distillation methods including DC [54], DM [53], CAFE [42], MTT [1], TESLA [2], SRe2L [46], and DataDAM [32]. For all the considered comparison methods, we assess the quality of the distilled dataset by measuring the Top-1 classification accuracy on the original validation set using models trained on them from scratch. Blue cells in all tables highlight the highest performance.

4.1 Results & Discussions

CIFAR-10/100. As shown in Table 1, our DWA exhibits superior performance compared to conventional dataset distillation methods, particularly evident on CIFAR-100 with a larger distillation budget. For instance, our DWA yields over a 10% performance enhancement compared to MTT [1] with ipc = 50 . Leveraging a more robust distillation backbone like ResNet-18, our approach surpasses the SOTA method SRe2L [46] across all considered settings. Specifically, we achieve more than 5% and 8% accuracy improvement on CIFAR-10 and CIFAR-100, respectively.

(a)

(b)

(a)Backbone:SRe2L

(b)Backbone:ourDwA

57.0

61.5

decoupledvar

decoupledvar

56.0

coupledvar

60.5

coupledvar

6

55.0

59.5

AcC.

Acc.

54.0

Test

Test

58.5

53.0

57.5

52.0

56.5

1

3

5

7

9

1113 15 17 1921 23

1357911131517192123

Avar (x0.01)

Avar (x0.01)

Tiny-ImageNet & ImageNet-1K. Compared with CIFAR-10/100, ImageNet datasets are more closely reflective of real-world scenarios. Table 2 lists the related results. Due to the limited scalability capacity of conventional distillation paradigm, only a few methods have conducted evaluation on ImageNet datasets. Here we provide a comprehensive comparison with SRe2L [46], which has been validated as the most effective one for distilling large-scale dataset. It is obvious that our method significantly outperforms SRe2L on all ipc settings and validation models. For instance, our DWA surpasses SRe2L by 16.6% when ipc = 10 on ImageNet-1K using ResNet-18. Figure 2 further provides the visualization results, the enhanced diversity is the key driver behind the substantial performance improvement.

4.2 Ablation Study

Decoupled L var Coefficient. We first test our hypothesis, as outlined in Section 3.1, positing that strengthening L mean conflicts with the emphasis on L var , which is critical for ensuring diversity in synthetic datasets. Therefore, we compare the synthetic dataset distilled with an emphasis on L BN (which strengthens both L mean and L var ) against one that emphasizes L var alone. As depicted in Figure 3, focusing solely on L var outperforms the combined emphasis on λ BN in both SRe2L [46] and our proposed Directed Weight Adjustment (DWA). These experimental results verify our hypothesis in Section 3.1, indicating the optimal value of the decoupled coefficient L var is 0.11. We also employ

1.2

decoupledvar

ance

1.0

coupledvar

distar

0.8

feature

0.6

nalized

0.4

0.2

0.0

1

2

3

4

56

6

7

8

9

10

Classindex

the normalized feature distance as a metric to comprehensively evaluate our emphasis. This metric measures the mutual feature distances between instances, as defined in Appendix A.2.2. By randomly selecting 10 classes from CIFAR-100, we calculate the normalized feature distances between synthetic datasets emphasized by the decoupled L var and the coupled L BN . The findings, illustrated in Figure 4, validate our hypothesis from a different perspective.

Directed Weight Adjustment. We clarify the necessity of restricting the direction of weight adjustment in Section 3.3. To test its effectiveness, we apply a random ∆ θ , sampled from a Gaussian Distribution, to θ T . As shown in Table 3, we assess synthetic datasets derived from three scenarios: no weight adjustment, random weight adjustment, and our directed weight adjustment (DWA) method, using the CIFAR-100 dataset. The results, examined across various architectures, underscore the importance of directing weight adjustments in distillation processes. Notably, we observe performance degradation in the synthetic dataset optimized with random weight adjustment at ipc = 10 compared to those without weight adjustment. This decline occurs because, at smaller ipc values, the noise introduced by random weight adjustment outweighs the benefits of diversity. However, as the number of synthetic instances increases, diversity becomes more effective in capturing a broader range of features, leading to improved performance, as reflected at ipc = 50 .

Parameters Study on K and ρ . Apart from direction, the number of steps K and magnitude ρ of perturbation also influence the distillation process. Figure 5 illustrates the grid search for these two hyper-parameters and demonstrates the positive impact of perturbation, which is achieved effortlessly, requiring no meticulous manual parameter tuning. In our experiments, we set K = 12 and ρ = 15 e -3 for all the datasets. Readers can adjust these hyper-parameters according to their specific circumstances (different datasets and networks) to obtain better results.

Cross-Architecture Generalization. The generalizability across different architectures is a

key feature for assessing the effectiveness of the distilled dataset. In this section, we evaluate the surrogate dataset condensed by different backbones (ResNet-18 and ConvNet-128) on various archi-

tectures including MobileNetV2 [33], ShuffleNetV2 [26], EfficientNet-B0 [37], and VGGNet-16 [35]. The experimental results are reported in Table 4 and Table 5. It is evident that our DWA-synthesized dataset can effectively generalize across various architectures. Notably, for ipc = 50 on CIFAR-100 with ShuffleNetV2, EfficientNet-B0, and ConvNet-128-three architectures not involved in the data synthesis phase-our method achieves impressive classification performance, with accuracies of 41.7%, 40.7%, and 37.0%, respectively, outperforming the latest SOTA method, SRe2L [46], by 14.2%, 15.8%, and 17.6%. In Appendix A.2.3, we further extend the proposed method to a vision transformer-based model, DeiT-Tiny [40].

5 Related Works

Dataset Distillation [43] emerges as a derivative of Knowledge Distillation (KD) [9], emphasizing data-centric efficiency over traditional model-centric one. Previous studies have explored various strategies to condense datasets, including performance matching, gradient matching [54, 52, 19] distribution matching [42, 53, 55, 48, 4], and trajectory matching [1, 2, 5, 6, 21, 41].

What distinguishes DD from KD is the bi-level optimization, which considers both model parameters and image pixels. The consequent complexity and computational burden intricate optimization significantly diminish the effectiveness of the aforementioned methods. To address this issue, SRe2L [46] introduced a three-step paradigm known as Squeeze-Recover-Relabel . This approach relies on the highly encoded distribution prior, i.e. , the running mean and running variance in the BN layer, to circumvent supervision provided by model training. With this decoupled optimization, SRe2L is able to extend DD to high-resolution and large-scale datasets like ImageNet-1K.

Another critical challenge in dataset compression, not limited to distillation, is how to represent the original dataset distribution with a scarcity of synthetic data samples [36]. Previous research claims that the diversity of a dataset can be evaluated by spatial distribution [27], the maximum dispersion or convex hull volume [47], and coverage [56]. Conventional dataset distillation [49, 15] treats the synthetic compact dataset as an integrated optimizable tensor without specialized guarantees for diversity and relies entirely on the matching objectives mentioned above. Recognizing this limitation, Dream [23] proposed using cluster centers to induce synthesis and ensure adequate diversity. Besides, SRe2L resorts to the second-order statistics, i.e. , variance of representations in pre-trained weights to provide diversity.

6 Conclusion

In this work, we hypothesize that ensuring diversity is crucial for effective dataset distillation. Our findings indicate that the random initialization of synthetic data instances contributes minimally to ensuring that each instance captures unique knowledge from the original dataset. We validate our hypothesis through both theoretical and empirical approaches, demonstrating that enhancing diversity significantly benefits dataset distillation. To this end, we propose a novel method, Directed Weight Adjustment (DWA), which introduces diversity in synthesis by customizing weight adjustments for each mini-batch of synthetic data. This approach ensures that each mini-batch condenses a variety of knowledge. Extensive experiments, particularly on the large-scale ImageNet-1K dataset, confirm the superior performance of our proposed DWA method.

Limitations and Future work. While DWA provides a straightforward and efficient approach to introducing diversity in dataset distillation, its reliance on the sampling of a random distribution to adjust weight parameters presents limitations. Increasing the variance of the random distribution can introduce unexpected noise, thereby bottlenecking overall performance. Future investigations could explore synthesizing data instances in a sequential manner, encouraging later instances to consciously distinguish themselves from earlier ones, thereby further enhancing diversity.

References

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 10708-10717, 2022.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet1k with constant memory. In Proc. Int. Conf. Mach. Learn. (ICML) , pages 6565-6590, 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 248-255, 2009.

Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. arXiv preprint arXiv:2404.00563 , 2024.

Jiawei Du, Yidi Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 3749-3758, 2023.

Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. In Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023.

Yunzhen Feng, Shanmukha Ramakrishna Vedantam, and Julia Kempe. Embarrassingly simple dataset distillation. In Adv. Neural Inf. Process. Syst. Workshop (NeurIPS Workshop) , 2023.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2021.

Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. Int. J. Comput. Vis. , 129(6):1789-1819, 2021.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 9726-9735, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 770-778, 2016.

Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530 , 2024.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Proc. Int. Conf. Mach. Learn. (ICML) , pages 11102-11118, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi PontTuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. Int. J. Comput. Vis. (IJCV) , 128(7):1956-1981, 2020.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In Proc. Int. Conf. Mach. Learn. (ICML) , pages 1235212364, 2022.

Shiye Lei and Dacheng Tao. A comprehensive survey of dataset distillation. IEEE Trans. Pattern Anal. Mach. Intell. , 46(1):17-32, 2024.

Dai Liu, Jindong Gu, Hu Cao, Carsten Trinitis, and Martin Schulz. Dataset distillation by automatic training trajectories. arXiv preprint arXiv:2407.14245 , 2024.

Songhua Liu and Xinchao Wang. MGDD: A meta generator for fast dataset distillation. In Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023.

Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. DREAM: efficient dataset distillation by representative matching. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV) , pages 17268-17278. IEEE, 2023.

Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In International Conference on Machine Learning , pages 22649-22674. PMLR, 2023.

Noel Loo, Ramin M. Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In Adv. Neural Inf. Process. Syst. (NeurIPS) , 2022.

Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Proc. Eur. Conf. Comput. Vis. (ECCV) , pages 122-138, 2018.

Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2024.

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2021.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Adv. Neural Inf. Process. Syst. (NeurIPS) , pages 20596-20607, 2021.

Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Proc. Eur. Conf. Comput. Vis. (ECCV) , pages 524-540. Springer, 2020.

Noveen Sachdeva and Julian J. McAuley. Data distillation: A survey. Trans. Mach. Learn. Res. , 2023.

Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstantinos N. Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV) , pages 17051-17061, 2023.

Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 4510-4520, 2018.

Yuzhang Shang, Zhihang Yuan, and Yan Yan. MIM4DD: mutual information maximization for dataset distillation. In Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2015.

Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. arXiv preprint arXiv:2312.03526 , 2023.

Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. Int. Conf. Mach. Learn. (ICML) , pages 6105-6114, 2019.

Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: improving LLM pretraining via document de-duplication and diversification. In Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023.

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2019.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Kai Wang, Zekai Li, Zhi-Qi Cheng, Samir Khaki, Ahmad Sajedi, Ramakrishna Vedantam, Konstantinos N Plataniotis, Alexander Hauptmann, and Yang You. Emphasizing discriminative features for dataset distillation in complex scenarios. arXiv preprint arXiv:2410.17193 , 2024.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. CAFE: learning to condense dataset by aligning features. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 12186-12195, 2022.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan S. Kankanhalli. Efficient adversarial contrastive learning via robustness-aware coreset selection. 2024.

Hongxu Yin, Pavlo Molchanov, José M. Álvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 8712-8721, 2020.

Zeyuan Yin, Eric P. Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from A new perspective. In Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023.

Yu Yu, Shahram Khadivi, and Jia Xu. Can data diversity enhance learning generalization? In Proc. Int. Conf. Comput. Linguistics (COLING) , pages 4933-4945, 2022.

Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. Echo: Efficient dataset condensation by higher-order distribution alignment. arXiv preprint arXiv:2312.15927 , 2023.

Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 11950-11959, 2023.

Xin Zhang, Jiawei Du, Yunsong Li, Weiying Xie, and Joey Tianyi Zhou. Spanning training progress: Temporal dual-depth scoring (TDDS) for enhanced dataset pruning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024.

Xin Zhang, Jiawei Du, Ping Liu, and Joey Tianyi Zhou. Breaking class barriers: Efficient dataset distillation via inter-class feature compensator. arXiv preprint arXiv:2408.06927 , 2024.

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Proc. Int. Conf. Mach. Learn. (ICML) , pages 12674-12685, 2021.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV) , pages 6503-6512, 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2021.

Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 7856-7865, 2023.

Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2023.

Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation: A good compression is all you need. arXiv preprint arXiv:2404.07976 , 2024.

A Appendix

A.1 Minimizing L mean and L var can be contradictory

To prove that minimizing L mean and L var can result in contradictory objectives for some existing instances, we will demonstrate that the gradients required to minimize L mean and L var , respectively, may point in opposite directions. Specifically, for any arbitrary instance s i ∈ S , our goal is to establish:

For ∂ L mean ∂ s i , we have

̸

because µ ( S ) = 1 |S| s i + ∑ j = i 1 |S| s j , thus ∂µ ( S ) ∂ s i = 1 |S| . For ∂ L var ∂ s i , we have

̸

Substitute Equation 17 and Equation 18 back into Equation 16,

Let R = [ µ ( S ) -µ ( T )] · [ σ 2 ( S ) -σ 2 ( T )] , where R is a constant that can be either positive or negative, depending on the values of µ ( S ) , µ ( T ) , σ 2 ( S ) , and σ 2 ( T ) . Suppose R > 0 . In this scenario, instances for which ( s i -µ ( S )) < 0 will encounter contradictory objectives in optimization. Conversely, if R < 0 , instances where ( s i -µ ( S )) > 0 will face similar contradictions.

A.2 Experiments

A.2.1 Hyper-parameter Settings

Table 6, Table 7, and Table 8 list the hyper-parameter settings of our method on experimental datasets. We maintain consistency with SRe2L for a fair comparison.

A.2.2 Feature Distance Calculation

In Figure 4, we use feature distance D fea to measure the diversity of distilled dataset. The following is how the class-wise feature distance is calculated,

where g θ T (˜ s c i ) and g θ T (˜ s c j ) are the latent representations of i -th and j -th synthetic instances of class c , specifically the outputs from the last convolutional layer.

A.2.3 Generalization to Vision Transformer-based Models

We acknowledge that our proposed approach cannot be directly applied to models without BN layers, such as Vision Transformers (ViTs). Our baseline solution, SRe2L, involves developing a ViT-BN model that replaces all LayerNorm layers with BN layers and adds additional BN layers

between the two linear layers of the feed-forward network. We followed their solution and conducted cross-architecture experiments with DeiT-Tiny [40] on the ImageNet-1K dataset. The results are listed in Table 9. The results demonstrate that our approach can be applied to ViT-BN with superior performance compared to the baseline.

A.2.4 Application to Downstream Tasks

We evaluate our proposed DWA on a continual learning task, based on an effective continual learning method GDumb [30]. Class-incremental learning was performed under strict memory constraints on the CIFAR-100 dataset, with 20 images per class ( ipc = 20 ). CIFAR-100 was divided into five tasks, and a ConvNet was trained on our distilled dataset, with accuracy measured as new classes were incrementally introduced. As shown in Table 10, DWA significantly outperforms SRe2L across all class-incremental stages, demonstrating superior retention of knowledge throughout the learning process.

A.2.5 Computational Overhead of Distillation

We compare the average time required to generate one ipc using ResNet-18 on CIFAR-100. As shown in Table 11, our proposed DWA incurs only a 7.32% increase in computational overhead while significantly enhancing the diversity of the synthetic dataset. This additional overhead arises from the K -step directed weight perturbation applied before generating each ipc , as detailed in lines 6-7 of Algorithm 1,

Since each ipc requires 1000 iterations of forward-backward propagation for generation, the additional K = 12 forward-backward propagations required by DWA are negligible in the overall distillation process.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope. The abstract and introduction summarize the key contributions and findings, which are consistently supported by detailed methodologies, experiments, and results in the main body. The claims are accurately presented without exaggeration.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper does discuss its limitations, adhering to guidelines regarding assumptions, scope of claims, performance factors, computational efficiency, privacy, fairness, and honesty.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper ensures that for each theoretical result, it provides the full set of assumptions and a complete (and correct) proof. This adheres to the guidelines by clearly stating or referencing all assumptions in the statement of theorems, numbering and crossreferencing all theorems, formulas, and proofs, and providing formal proofs either in the main paper or supplemental material. Additionally, the paper appropriately references any external theorems or lemmas relied upon in the proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results, ensuring transparency and reproducibility. This includes detailed descriptions of experimental setups, methodologies, and any necessary parameters or configurations. The paper offers clear instructions and explanations on how to replicate the results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: This paper provides implementation details and algorithm descriptions for reproduction. We also release our codes for reproduction in camera-ready version.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details necessary to reproduce the results, including datasets, hyperparameters, optimizer type, and how they were chosen.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experiments were rigorously conducted with five repetitions each, and we meticulously reported both the mean values and standard deviations for each experimental trial.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper comprehensively details compute resources in both the experiments section and supplementary materials, covering GPU type, memory, and storage specifics.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research aligns with the NeurIPS Code of Ethics, ensuring ethical standards are upheld throughout the study.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is foundational research, and therefore, it does not have direct societal impacts to discuss.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not involve the release of data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper provides proper credit to asset creators, citing relevant papers and explicitly mentioning license and terms of use. URLs are included where possible, and all licenses are respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets introduced in the paper are well-documented, providing comprehensive details alongside the assets, including training procedures, licenses, limitations, and consent processes, ensuring transparency and reproducibility.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dynamic adjustment mechanism to enhance the variation component in batch normalization loss so that increase the diversity of synthetic images. By adjusting the weights, the synthesized images achieve sufficient diversity to cover the widespread distribution of the real dataset.\n\nSTRENGTHS:\na. Originality: The author innovatively analyzes the different components of the loss function. This work differs from previous contributions by proposing a directed weight adjustment algorithm to increase the diversity of synthetic images. Related work is adequately cited.\nb. Quality: The submission is technically sound. The claims are well supported by theoretical analysis and experimental results. It is a complete piece of work. The authors are careful and honest in evaluating both the strengths (increased diversity and minimal computational expense) and weaknesses (unexpected noise) of their work.\nc. Clarity: The submission is clearly written and well organized.\nd. Significance: The results are important. The submission addresses the diversity problem more effectively than previous work. Most results advance the state of the art with a large margin, providing a unique theoretical approach that is also demonstrated in experiments.\n\nWEAKNESSES:\nc. Clarity: A few equations, such as (14), need more explanation. The current text does not adequately inform the reader on how to reproduce the results, as there is insufficient connection to the implementation details.\nd. Significance: The results on CIFAR10 using ConvNet are significantly worse than the current state of the art. Additionally, the authors did not mention whether or how the method can be integrated as a plug-in.\n\nQUESTIONS:\n1. In equation 14, can the authors explain why it is <=0? In line 176, is the first item clearly greater than the second item because equation 14 is <=0, or is the causal relation inverse, with the first item being greater than the second item concluding that equation 14 is <=0?.\n2. In Table 1, Can the author explain why the results of ConvNet on CIFAR10 are obviously worse than state of the art?\n3. Why there is no results for ipc=1 in all result tables.\n\nLIMITATIONS:\nThe authors address the limitation about the unexpected noise.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method to enhance dataset distillation by improving the diversity of synthesized datasets. The authors introduce a dynamic and directed weight adjustment technique to maximize the representativeness and diversity of synthetic instances. Theoretical and empirical analyses demonstrate that increased diversity enhances the performance of neural networks trained on these synthetic datasets. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, show that the proposed method outperforms existing approaches while incurring minimal computational expense.\n\nSTRENGTHS:\n- The paper is well-written and structured.\n- The analysis of the decoupled variance component is interesting.\n- The proposed dynamic adjustment mechanism is novel and can enhance the diversity of the synthesized dataset.\n- The paper provides a solid theoretical foundation and extensive empirical validation across multiple datasets and architectures.\n- The findings have important implications for the efficiency of dataset distillation, particularly for large-scale datasets.\n\nWEAKNESSES:\n- The effectiveness of the diversity in generated datasets has been discussed in many prior works [1, 2]. \n- The authors should compare the proposed method with recent SOTA dataset distillation methods that extended from SRe2L [2, 3]. Especially for [3], the relation to it needs to be clarified.\n- A comparison with recent diffusion-based generative dataset distillation methods [4, 5] is also needed because they are free of optimization.\n- How long does it typically take to search for the optimal values of $K$ and $\\rho$ for ResNet-18 and other different datasets and network architectures? Can the authors provide insights into the computational cost associated with this parameter-tuning process?\n- Are there any scenarios where the directed weight adjustment might negatively impact performance or diversity? If so, how can these be mitigated?\n- Have the authors considered some downstream applications to further verify the effectiveness of the proposed method, such as continual learning and privacy protection?\n\n[1] Liu, Yanqing, et al. \"Dream: Efficient dataset distillation by representative matching.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Shao, Shitong, et al. \"Generalized large-scale data condensation via various backbone and statistical matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Gu, Jianyang, et al. \"Efficient dataset distillation via minimax diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[5] Su, Duo, et al. \"D4M: Dataset Distillation via Disentangled Diffusion Model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nSee the weaknesses section. I am happy to raise my score if the above concerns can be addressed.\n\nLIMITATIONS:\nThe authors discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a random directed perturbation strategy to the teacher models for further strengthening the diversity of the existing sample-wise data synthesis method. Through intuitive empirical and theoretical analysis of the diversity factors, the authors proposed a simple yet overlooked weight adjustment strategy, showing promising results in various vision models and benchmarks including ImageNet.\n\n========== Post rebuttal ==========\n\nThank you for the detailed responses and additional analysis. Having read the responses, my major concerns were addressed. Therefore, I raise the rating.\n\nSTRENGTHS:\n- Intuitive motivational analysis of how the existing methods, SRe2L, lack of diversity, while providing empirical and theoretical backgrounds for the factors that can affect diversity during the synthesis process.\n- Proposed simple yet overlooked perspectives of existing data synthesis loss via 1) perturbing the teacher model in a random directed way, 2) decoupled variance matching, both of which collaboratively enhances sample diversity.\n- These simple yet intuitive methods exhibit largely improved performance on various vision models and benchmarks including large-scale ImageNet dataset. \n- Thorough ablation studies with the decoupled variance coefficient, weight adjustment, and cross-architecture additionally provide an intuition of how the proposed method enhances the actual performance.\n\nWEAKNESSES:\n- The proposed method has a heavy reliance on BN statistics matching from CNN-based models. Considering that the recent State-of-the-art models such as ViT are not leveraging BN, the proposed approach might not be directly applicable to these vision transformer-based models.\n- Failure cases of DWA should be more clarified. For example, In Table 1, when the model and the number of classes in the target dataset is small (ConvNet / CIFAR-10), DWA underperforms compared to the other methods.\n- Comparison with baseline (SRe2L) in terms of computational overhead is required. Since this paper introduces add-on procedure, weight adjustment, to the baseline, the authors should clarify if the proposed method does not induce significant computation overhead during synthesis.\n- In Figure 5, the test performance varies depending on the perturbation parameters. Although the authors deterministically set one set of parameters for all the different datasets, it should be clarified if these hyper-parameters are not sensitive and generalizable to all the datasets, to reduce time-consuming trial and error for searching these parameters on the other general datasets.\n\nQUESTIONS:\n- Is the proposed synthesis method applicable to the ViT-based models that leverage LayerNorm rather than BatchNorm layers? Further experiments (directly synthesizing on ViT-based models or cross-architecture experiments by synthesizing with CNN models and testing on ViT-based models) would be required for the generalization of the proposed method to the recent models.\n- Disregarding the computational overhead, can searching for the optimal perturbation parameters per sample rather than using one set of predefined universal perturbation parameters help improve the performance?\n- Can you provide comparison of computation overhead with SRe2L on ImageNet dataset?\n\nLIMITATIONS:\nSee above weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method for dataset distillation (DD): DD through Directed Weight Adjustment (DWA). The authors first identify that the current dataset distillation methods often fail to ensure diversity in the distilled data, which leads to suboptimal performances. To address this issue, they propose a method that uses dynamic and directed weight adjustment techniques to increase the diversity of synthetic data. The propose method DWA introduces three modifications based on SOTA method SRe2L: 1). Decoupled coefficients for batch norm loss 2). Initialize with real images 3). Random perturbation on model weights. Extensive experiments on datasets like CIFAR, Tiny-ImageNet, and ImageNet-1K demonstrate the superior performance of this approach, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense.\n\nSTRENGTHS:\n1. Increasing diversity in distilled datasets is a key challenge posed by existing methods, and this paper addresses it effectively. The authors conducted a detailed analysis showing that batch normalization plays a crucial role in enhancing diversity in dataset distillation. By leveraging this observation, they proposed innovative techniques to incorporate batch normalization more effectively within the distillation process. The result is a significant improvement over state-of-the-art (SOTA) dataset distillation methods.\n2. The authors also conduct detailed ablation studies that demonstrate the effectiveness of each component of their proposed method. The contribution of each element is clearly shown, providing a comprehensive understanding of the method's strengths. Additionally, they show that the method's effectiveness is robust across different hyperparameter settings and datasets, which makes the method have wide practical applicability.\n\nWEAKNESSES:\n1. In the ablation study, the authors analyzed and justified the design choices for two key components: the decoupled $L_var$ coefficient and the Directed Weight Adjustment (DWA) scheme. Additionally, they proposed using real initialization. However, the study lacks an ablation analysis that separately evaluates the contribution of these three components to the overall performance of DWA. Such an ablation would provide a clearer understanding of the individual and combined effects of the decoupled $L_var$ coefficient, the Directed Weight Adjustment scheme, and real initialization on the method's efficacy.\n2. The paper primarily presents results using moderate to large data budgets (10 and 50 IPCs) for most datasets and 100 IPC for ImageNet-1K. Since the notion of “diversity” only becomes significant when each class has multiple prototypes, the proposed method’s benefits are limited to situations when the distillation budget is high. Overall, the proposed method brings sizable performance improvement but only for one SOTA method with specific data budget constraints.\n\nQUESTIONS:\n1. The paper considered using a directed perturbation to increase the diversity of the model. Is it equivalent or similar to using model trained on different random seeds? \n2. In section 2, the authors pointed out $\\vec{x}$ and $\\vec{s}$ are in latent space, mapped by $g_{\\theta}$, following the same notation, Eqn. 2 is operating on latent space and the trained classifier $f_{\\theta}$. Is the random weight perturbation method proposed in Section 3.2 only on the classifier part then (Alg 1 only refers to $f_{\\theta}$ or the entire network $g$ is perturbed? \n3. In section 3.2, how is min/max taken over $L$, if I am not mistaken, $L$ is a gradient of the loss function on $\\theta$?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}