Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
Authors: Jiawei Du
Xin Zhang
Juncheng Hu
Wenxin Huang
Joey Tianyi Zhou
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Dataset Distillation
Synthetic Data
Diversity
Generatlization
Primary area: optimization

Abstract: The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataset distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough to replace the original dataset in training a neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, we provide a thorough theoretical and empirical analysis of diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, we introduce a novel method that employs dynamic and directed weight adjustment techniques to modulate the synthesis process, thereby maximizing the representativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of a large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available at https://github.com/AngusDujw/Diversity-Driven-Synthesis.

Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
Jiawei Du
Xin Zhang 123
Juncheng Hu* Wenxing Huang Joey Tianyi Zhou 1.28
Centre for Frontier Al Research (CFAR), Agency for Science, Technology and Research (AAS)))), Singapore
2 Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR). Singapore
3 XiiDian University, Xi'an, China "Nationa University of Singapore, Singapore Hubee University, WuHan, China
Abstract
The sharp increase in data-related expenses has motivated research into condensing datasets while retaining the most informative features. Dataser distillation has thus recently come to the fore. This paradigm generates synthetic datasets that are representative enough replace the original dataset in training neural network. To avoid redundancy in these synthetic datasets, it is crucial that each element contains unique features and remains diverse from others during the synthesis stage. In this paper, provide thorough theoretical and empirical analysis diversity within synthesized datasets. We argue that enhancing diversity can improve the parallelizable yet isolated synthesizing approach. Specifically, introduce a novel method that employs dynamic and directed weight adjustment techniques modulate the synthesis process, thereby maximizing the represen- tativeness and diversity of each synthetic instance. Our method ensures that each batch of synthetic data mirrors the characteristics of large, varying subset of the original dataset. Extensive experiments across multiple datasets, including CI- FAR, Finy-ImageNie and ImageNet-IK. demonstrate the superior performance of our method, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense. Our code is available http:ggiiiiuioooogguuuuuuuae
Driven- ServennSynthesis..
Introduction
With the rapid growth in dataset size and the need for efficient data storage and processing [8. 17, 14, 13], how to condense datasets while preserving their key characteristies becomes a significant challenge in machine learning community [12, 38]. Unlike previous research [29, 39, 50, 44] that focuses on constructing representative subset through selecting from the original data, Dataser Distillation [43, 31, 20] aims synthesize small and compact dataset that retains informative features from the original dataset. A model trained on the synthetic dataset is thus supposed to achieve comparable performance as one trained on the original dataset. The development of dataset distillation reduces data-related costs [7, 34, 49] and helps us better understand how Deep Neural Networks (DNNs) extract knowledge from large-scale datasets.
Numerous studies dedicate significant effort to synthesizing distilled datasets more effectively. For example, Zhao et al. employ gradient-matchin approach [52, 54] guide the synthesis process. Trajectory-matchinn the synthetic data. Additionally, distribution matching [42, 53, 55] and kernel inducing points methods [28, 25, 23, 24] also contribute to synthesizing representative data. Despite the great progress achieved by these methods on datasets like CIFAR [16], their extensive computational overhead (both GPU memory and GPU time) hinders the extension of these methods to large-scale datasets like ImageNet- [3].
methods [1, 2, 5, 6] further align gradient trajectories to optimize
Email: dujiawei w.u.u..duu. joey.tianyi.zhon @ gmmail.com. represents corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

bSNE visualization CFAR1100
Performance Comparison
Sfizt
Several recent works [2, 46, 22, 51, 57] have attempted to address the efficiency issues of dataset distilla- tion. particular, Yin et al. [46] pro- pose lightweight distillation method, SRe2L, which successfully condenses the large-scale dataset ImageNet-IK. Unlike previous methods [1, 53, 15] that treat the synthetic set a unified entity to utilize the mutual influences among synthetic instances, SRe2L synthesizes each synthetic data in- stance individually. As such, SRe2L. significantly reduces both GPU mem- ory costs and computational overhead.
DMW (Hun))
55.2
48,0
47A
41.1
Praipshak-1K
Figure 1: Left: t-SNE visualization logit embeddings on CIFAR- 100 [16] dataset. The seatter plot illustrates the distribution of syn- thetic data instances distilled by SRe2L (blue dots) and our DWA method (red stars). The blue density contours represent the distri- bution of natural data instances. Our DWA method demonstrates more diverse and widespread distribution compared SRe2L. [46], indicating better generalization and coverage of the feature space. Right: The consequent performance improvement of DWA in vari- datasets. Experiments are conducted with 50 images per class.
Individually synthesizing each data in- stance can efficiently parallelize opti- mization tasks, thereby flexibly managing GPU memory usage and computational overhead. However, this approach may present challenges in ensuring the representativeness and diversity of each instance. each instance is synthesized in isolation, there may be a risk of missing the holistic view of the data characteristics, which is crucial for the training of generalized neural networks. Intuitively, SRe2L might expect that random initialization of synthetic data would provide sufficient diversity to prevent homogeneity in the synthetic dataset. Nevertheless, our analysis, as demonstrated in Figure 1, reveals that this initialization contributes only marginally to diversity. Conversely, the Batch Normalization (BN) loss [45] in SRe2L plays the practical role in enhancing diversity of the distilled dataset.
Motivated by these findings, we further investigate the factors that enhance the diversity of synthetic datasets from theoretical perspective. We reveal that the variance regularizer in the BN loss the key factor ensuring diversity. Conversely, the mean regularizer within the same BN loss unexpectedly constrains diversity. To resolve this contradiction, to specifically strengthen the variance regularizer's s role in promoting diversity. Experimental results validate our hypothesis. We further propose dynamic mechanism to adjust the weight parameters of the teacher model. Serving as the sole source of supervision from the original dataset, the teacher model guides the synthesis comprehensively. Our meticulously designed weight perturbation mechanism injects randomness without compromising the informative supervision, thereby improving overall performance. Importantly, our method incurs negligible additional computations (< 0.1%). Intuitively, our method perturbs the weight in a direction that reflects the characteristics of a large subset, varying with each batch of synthesized data. We conduct extensive experiments across various datasets, including CIFAR-10, CIFAR- 100, Tiny- ImageNet, and ImageNet-IK, to verify the effectiveness of our proposed method. The superior performance of our method not only validates our hypothesis but also demonstrates its ability to en- hance the diversity of synthetic datasets. This success guides further investigations into searching for representative synthetic datasets for lossless dataset distillation. Our contribution can be summarized as follows:
suggest decoupled coefficient
We analyze the diversity of the synthetic dataset in dataset distillation both theoretically and empirically, identifying the importance of ensuring diversity in isolated synthesizing approaches.
We propose a dynamic adjustment mechanism to enhance the diversity of the synthesized dataset, incurring negligible additional computations while significantly improving overall performance. Extensive experiments on various datasets verify the remarkable performance of our method.
2 Preliminaries
Notation and Objective. Given a real and large dataset T ((x;; 9:)) Dataset Distillation aims to synthesize a tiny and compact dataset = {(,, |51 The samples in T are drawn i.i.d from natural distribution D, while the samples in S are optimized from scratch. We use 07 and
2

---

0s to represent the converged weight trained on T and S, respectively. We define neural network go o f. where g acts as the feature extractor and f as the classifier. The feature extractor and the classifier loaded with the corresponding weight parameters from 0 are denoted by go and fo-
Throughout the paper, explore the properties of synthesized datasets within the latent space. We transform both ã, E RCxHxW from the pixel space, to the latent space, 2, E R for better formulation. This transformation
given by ge- and Dataset Distillation is to ensure that a model á trained on the synthetic dataset S is able to achieve a comparable test performance as the model trained with T, which can be formulated as,
ge- (8). The objective of
[e (hoT, m)) 20D
IE [e (hag; [)]
where l can be an arbitrary loss function. The expression E(hor, x) should be interpreted as [her, x. v) where y is the ground truth label.
Synthesizing S. A series of previous works mentioned in Section 5 have introduced various methods to synthesize S. Specifically, SRe2L. [46] proposes an efficient and effective synthesizing method, which optimizes each synthetic instance 8; by solving the following minimization problem:
arg min [e (f s;) + XCBN (fer ss)].
(2)
#:ERd
where LaN denotes the BN loss, and À is the coefficient of CBN. The detailed definition of Lbn can be found in Equation 3. Minimizing the BN loss LBN significantly enhances the performance of SRe2L, which designed to ensure that S aligns with the same normalization distribution T. However, we argue that another essential but overlooked aspect of the BN loss LBN is its role in introducing diversity to S, which also greatly benefits the final performance. In the following section, we will analyze this issue in greater detail.
3 Methodology
Diversity in the synthetic dataset S is essential for effective use of the limited distillation budget. This section reveals that the BN loss, referenced in Equation enhances S's diversity. However, the suboptimal setting of BN loss limits this diversity. To overcome this, we propose dynamic adjustment mechanism for the weight parameters of fe.- enhancing diversity during synthesis. Finally, we detail our algorithm and theoretically demonstrate its effectiveness. The pseudocode of our proposed DWA can be found in Algorithm
Algorithm Directed Weight Adjustment (DWA)
Input: Original dataset T; Number of iterations T; Image per class ipc; Number of steps K, magnitude solve the weight adjustment A0; Learning rate m7; A network for with weight parameter 0r, for is well trained on T.
1: Initialize S (). A0o Odim(Pr)
2: for i= to ipc do
3: Randomly select instance for each class from T, to initialize S i.e.,
4: Sj = =( y:) (æ;, y:) E T and each y; is unique}
5: Compute the adjustment of weights AO by solving Equation 11
6: for = 1 to K do
7:
A0₃ = Ab 100-11 + + V (fe
8:
AO -A0
Directed Weight Adjustment
9:
Optimize S
for = 1 T do
11:
S = S v/V e t2 S1-1)
L is defined in Equation 15
12:
= S
Output: Synthetic dataset S
In the actual optimization process, operations occur within the pixel space using the entire network her However, as we discuss the optimization in the latent space, only consider solutions within this space. Then, transform the solution in latent space back into pixel space á 9 (s).

---

3.1 Batch Normalization Loss Enhances Diversity of S
The BN loss LB comprises mean (Emean) and variance (Lvar) components, defined as follows:
LBN femean + fovar where Lmman (J07: 8:) = Σ, I| (S) nu (7)
and Lvar (fer 8,) = Σ, |oo
of (2)
(3)
where 14 and of refer to the channel mean and variance in the I-th layer, respectively. s; is optimized within mini-batch S, where 8; S and & C S. Each component of LAN operates from its own perspective to enhance dataset distillation. First, the mean component Lmaan regularizes the synthetic data 8, ensuring its values align closely with those of the representative centroid of T in latent space. Second, the variance component Lvar encourages the synthetic data in S to differ from each other, thereby maintaining the variance (S). Thus, this BN loss-driven synthesis can be decoupled as 8; X (Xhreaan; 0T) Ei;
(4)
where X, can be regarded as an optimal solution to Equation 2 when the variance regularization term fivar is not considered, i.e.,
VVoel (JoT, Xx)122 an and Armaa (Jo7: X.) = Σ, || (X.) - lu (T) S 02 where both on, 02 0 and a1 02 + 0. &, represents small perturbation and &, N (0, (A(vvaa)) Therefore, the variance of the synthetic dataset S is,
(5)
Var(S) = Var X (eCCma 0T)) + Var (E) = =o (ACvar).
(6)
We have Var( (e..XXo ④T)) 0 X, is deterministic. Unlike other approaches that consider the mutual influences among synthetic data instances and optimize the dataset collectively, SRe2L [46] optimizes each synthetic data instance individually. Therefore, the diversity of the synthetic dataset S is solely determined by ALvar-
However, simply increasing À contributes marginally to enhancing the diversity of S. This is because a greater À will also emphasize the regularization term XLmeann which contradicts the emphasis on XLvar- We provide a detailed analysis in the Appendix A.1. As result, we propose using decoupled coefficient, Avars to enhance the diversity of S.
Additionally, the synthetic data instances are optimized individually to approximate the representative data instance X.. However, the gaussian initialization ((0 1) in pixel space does not distribute uniformly around X. in latent space, making the converged synthetic data instances to cluster in crowed area in latent space, as dedicated in Figure To address this, we propose initializing with real instances from T inspired by MTT [1], ensuring a uniform projection when synthesizing S.
3.2 Random Perturbation on 0f Helps Improve Diversity
In the previous section, we highlighted the often overlooked aspect of the BN loss in introducing diversity to S, which was also verified through experiments in Section 4.2. Building upon this, we propose introduce randomness into 0 to further enhance S's diversity, it is the only remaining factor affecting Var(S), as shown in Equation 6.
Let x = X Xe(ACmean f to be the original optimal solution to Equation 2. We aim to solve the adjusted optimal solution x. Xe(XLmennn 0T + A0) x. + Aæ. where 07 is randomly perturbed by A0, and AO ~ N'0 oz). Consequently, we have:
oo (0,+,,0;))ll = oo (Jo x. + Az) S on-
To solve for Ax. we can apply a first-order bivariate Taylor series approximation because Vol(Jor, X.)
< an, where an + 0. and both AO and Ar are small. Thus,
Vol (fer+a0; x. + Ax)
Vol (Jor; xx)) + Vz (Sor xx) AB V. Vol (Sor x;)) Am
< || V (fer, 2.2 + V (fer> x.) A0 + V Vol ( (for x;)] Aæll2
Son + V| (foy, x AA + Va V of (for x Ax
(8)
We disregard the class differences in the following analysis since they are identical across all classes.

---

To satisfy Equation 7, we have:
Vgf (Jo,; x A0 V Vol (So, x A 0,
then
Az - v Vol (Jo,: 2;) 2 , AA.
(9)
Intuitively, A must compensate the Va incurred by introducing the random perturbation A0 NO, oz) on ry. By Equation 9, Var(Az) Var(Ad) oy then:
Var (5") Var Xx (AC fy A0) ) + Var (E)
Var (x² + Ax) + Var (E)
Bo + 2 E (X 2 og (ALvar),
(10)
where / is determined by Va Voe(for (0 - xx), ) as shown in Equation 9. Therefore, the variance of the new synthetic dataset S' is greater than that of S without perturbing 0y.
3.3 Directed Weight Adjustment on 0T
Although perturbing 0 could significantly increase the variance of the synthetic dataset S, undirected random perturbation Á can also introduce noise, which turn degrades the performance. We aim address this limitation by directing the random perturbation Ae without introducing noise into S. We propose to obtain directed AO by solving the following maximization problem:
A
arg max LB (fo++00) where LB (for+ -E (fer+se.2)))
(11)
where represents randomly selected subset of 'T, and [B|| < [T]. As such, A will not introduce unanticipated noise when synthesizing S. The randomly selected B ensures that the randomness of Á continues benefit the diversity of S. Next, we will demonstrate this theoretically,
Effective dataset distillation should provide concise and critical guidance from the original dataset T when synthesizing the distilled dataset. Here, this guidance introduced primarily through the converged weight parameters 0y, i.e.,
f arg min LeT (for) where LT (Sot)
EE e(for a.),
(12)
#:ET
where 07 contains informative features of because achieves minimized training loss over T. We demonstrate that Áo, obtained from Equation 11, decreases the training loss computed over T I, which, in fact, highlights the features of T B. By applying first-order Taylor expansion, obtain:
LTV (for+30) Lox (fe,) VaLrY (fer) Ao.
(13)
Since OT is optimized until reaching a local minimum with respect to the loss function computed over the training T, we have:
VOLT (fer) VoLB (fer) + VOLT (fe,) = thus VOLTAB (fer) = VoLB (fer),
where the tensor of zeros with the same dimension as 07. Substitute it back into Equation 13, we have:
TT (So++50) ( ~VoT)=> (Jo7) Ao
= VeLa (fo,) Ao
= LB (For La (for) S 0,
(14)
La La (o)) will clearly be greater than L3 (f( ). as indicated by Equation 11. Thus, we demonstrate that the directed Ao results in less noise and improved performance. In summary, after resolving Ae as in Equation 11, our proposed method synthesizes data instance s; by solving:
arg min L where L L= (f + XLmean (Jor 8.) + Avarhwor (Jor, si) (15)
5

---

4 Experiments
To evaluate the effectiveness of the proposed method, we have conducted extensive comparison
experiments with SOTA methods on various datasets including CIFAR-10/100 (32 x 32, 10/100
classes) [16], Tiny-ImageNet (64 x 64, 200 classes) [18], and ImageNet-IK IK (224 x 224, 1000 classes) [3] using diverse network architectures like ResNet-(18, 50, 101) [11], MobileNet [33],
ShuffleNetV2 [26], EfficientNet-B BO [37], and VGGNet- [35]. We conduct our experiments on the server with one Nvidia Tesla A100 40GB GPU.
Solving 20. Before we conduct our experiments, we propose to use a gradient descent approach to solve 2 in Equation 11. There are two coefficients, K and used in the gradient descent approach. K' represents the number of steps, and p normalizes the magnitude of the directed weight adjustment. The details for solving A can be found in Line of Algorithm I.
Experiment Setting, Unless otherwise specified, we default to using ResNet-18 18 as the backbone for distillation. For ImageNet- IK, we use the pre-trained model provided by Torchvision while for CIFAR- 10/100 and Tiny-ImageNNe we modify the original architecture under the suggestion in [10]. More detailed hyper- prper-parameter settings can be found Appendix 1.2.1.
Baselines and Metrics. We conduct comparison with seven Dataset Distillation methods including
DC [54], DM [53], CAFE [42], MTT [1], TESLA [2]. SRe21. [46], and DataDAM [32]. For all
the considered comparison methods, we assess the quality of the distilled dataset by measuring the Top-1 classification accuracy on the original validation set using models trained on them from scratch.
Blue cells in all tables highlight the highest performance.
4.1 Results & Discussions
CIFAR-10/100. As shown in Table I, our DWA exhibits superior performance compared to conven- tional dataset distillation methods, particularly evident on CIFAR-100 with a larger distillation budget. For instance, our DWA yields over 10% performance enhancement compared to MTT [1] with ipc = 50. Leveraging a more robust distillation backbone like ResNet-18, 18, our approach surpasses the SOTA method SRe21. [46] across all considered settings. Specifically, achieve more than 5% and 8% accuracy improvement on CIFAR-10 and CIFAR- 100, respectively.
Table 1:
Comparison with SOTA dataset distillation haselines on CIFAR-10/100, Unless otherwise specified, we use the same network architecture for distillation and validation. Following the settings in their original papers, DC [54], DM [53], CAFE [42], MTT [1], and TESLA [2] ConyNet-122 128 (small model). For SRe21. [46], ResNet-18 (large model) is used synthesis and validation.
ConvNet
ResNet-18
Dataset
ipc
DC [54]
DM [53]
CAFE [42]
MTT [i]
TESLA [2]
DWA (ours)
SR:2L [46]
DWA (ours)
10
44.920.5
4
46.3
65.4
66.410.
45.0a04
27.2.0.4
32.6 620.4
CIFAR-10
50
53.99.s
63.0104
55.51a0
71.6107
72.6197
63.3
47.5_0.5
53.1118
CIFAR-IOD
25.2sn.
29.7+003
27. 8e9.
40. 10.4
41.7
47.680.4
31.6=0.5
39. 6aa.
50
43.66..4
37 37.91
47.7
47.9103
59.0.oo.
52.2....
60.9 Lo.s
Table 2: Comparison with SOTA dataset distillation basclines ImageNet and ImageNet- 1K. Unless otherwise specified, we use the same network architecture for distillation and validation. Following the settings their original papers, MTT [1], and TESLA [2] use ConvNett 128 (small model). For SRe2L [46], ResNet-18 (large model) is used for synthesis, and the distilled dataset is evaluated on ResNet-18. 50, and 101. f indicates MTT is performed on a 10-class subset of the full ImageNet- 1K dataset.
CouvNet
ResNet-18
ResNet-50
ReaNet-101
Dataset
1pc
MTT [1]
DaraDAM [32] TESLA [2]
SRc2L [46]
DWA (ours)
SRe2L
DWA (ours)
SRe2L
DWA (ours)
Tmny-emageNet
50
28.0
28.710a
11.1
52.8102
42.2
53.7_m2
42.5.002
54.719a
49.7
56.020.
51.240.4
56.9-m44
51.5ae.
57.4
10
64.0asa!
6.3gua
17.8
21.31c.
37. 37.9102
28. 18.1
43.0_mm
30.91u.1
46.91a4
ImageNet- IK
27.9
16.
25.2
55.6103
62.3_a.1
60.8.0.5
63.31a7
100
52.8.0.a
59. 29.2.
61.0#0.4
65.7=0.4
62.8=0.2
66.7+0.

---

(8)
(b)
Figure 2: Visualization of distilled images for the goldfish class. Panels (a) and (b) show the synthesized results by SRe2L [46] and DWA. respectively. The synthetic data instances generated by DWA method exhibit significantly greater diversity compared those produced by SRe2L, highlighting the effectivencss of our approach capturing a broader range of features.
(a) Backbone: SRe2L SRe2L
Backbone: DWA
57.0
61.5
decoupled
decoupled
56.0
coupled var
60.5
coupled var
Test Acc. (%)
55.0
59.5
58.5
53.0
57.5
52.0
56.5
1
3
5
13
15
17
19
21
11 13 15 17
21 23
Avar (4001)
Avar (5001)
Figure 3: Analysis of decoupled Lvar coefficient. We vary Avar across wide range of (0.01 0.23). "decoupled var' indicates Aver changing individually with a fixed mean component whose weight defaults to 0.01. 'coupled var' represents the weight of the mean and Avar change in tandem. (a) and (b) illustrate the performance of the original SRe2L [46] and our DWA in these two scenarios, respectively. This analysis is conducted CIFAR-100 using ResNet-18. Each Avar undergoes five independent experiments, with variance indicated by lighter color shades.
1.2
Tiny-emageNet & ImageNet-IK. Compared with CIFAR-10/100 ImageNet datasets are more closely reflective of real-world scenar- ios. Table 2 lists the related results. Due to the limited scalability capacity of conven- tional distillation paradigm, only few methods have conducted evaluation on ImageNet datasets. Here we provide comprehensive comparison with SRe2I. [46], which has been validated as the most effective one for distilling large-scale dataset. obvious that our method signifi- cantly outperforms SRe2L all ipc settings and validation models. For instance, our DWA surpasses SRe2L by 16.6% when ipc 10 ImageNet-IX IK using ResNet-18 18. Figure 2 further provides the visualization results, the enhanced diversity is the key driver behind the substantial performance improvement.
decouptes
Normllzzd feature distarce
1.0
0.8
0.6
0.4
0.2
0.0
2
3
4
5
7
8
9 10
Class index
Figure 4: Normalized feature distance of decoupled variance component with Avar 0.11 (the weight of mean component defaults 0.01) and coupled variance con- volutional layer outputs are used for feature distance calculation (see Appendix A.2.2). Ten classes ran- domly chosen from CIFAR- 100 distilled dataset.
component with ABN 0.11. ResNet-18's
4.2 Ablation Study
Decoupled Cyar Coefficient. We first test our hypothesis. outlined in Section 3.1, positing that strengthening Lmean conflicts with the emphasis on Lvar, which critical for ensuring diversity in synthetic datasets. Therefore, we compare the synthetic dataset distilled with an emphasis on LBN (which strengthens both Lmean and Lvar) against one that emphasizes Evar alone. As depicted in Figure 3, focusing solely on Lvar outperforms the combined emphasis on ABN in both SRe2L [46] and our proposed Directed Weight Adjustment (DWA). These experimental results verify our hypothesis in Section 3.1, indicating the optimal value of the decoupled coefficient Lvar is 0.1 We also employ
7

---

Table 3: An ablation study of DWA was conducted using various network architectures. The synthetic dataset distilled by ResNet- from the CIFAR-ION 100 dataset. We use X denote the distilled dataset without weight adjustment, O to denote the distilled dataset with random weight adjustment. and V represent Directed Weight Adjustment (DWA).
ipc
ipc
Perturbation
x
X
RosNet-18
30.6
14.9e0.1
39. 69.62
56. 1=0.4
56. 36.2+00
60.3=0.5
ResNet-50
26.5.111
15.01oa
35.2
55.7_oa
57.1..0
60.6_ux
MobileNetV2
18.2_.o5
14.4113
27.8.007
46.9_..
50.7..6
53.6_.2
ShuffleNet
10.3
10.7+00
19.1s0,9
30.9=111
39.1+0u
41. 11.7
EflicientNeet
11.8
11.1+97
20.2s0.4
28.6=10
38.8+10
40. 70..
the normalized feature distance as a metric to comprehensively evaluate our emphasis. This metric measures the mutual feature distances between instances, defined in Appendix A.2.2. By randomly selecting 10 classes from CIFAR-1OW we calculate the normalized feature distances between synthetic datasets emphasized by the decoupled Lvar and the coupled LBN. The findings, illustrated Figure validate our hypothesis from a different perspective.
Directed Weight Adjustment. We clarify the necessity of restricting the direction of weight adjustment Section 3.3. To test its effectiveness, we apply random A6, sampled from a Gaussian Distribution, to 0T. As shown in Table no weight adjustment, random weight adjustment, and our directed weight adjustment (DWA) method, using the CIFAR-100 dataset. The results, examined across various architectures, underscore the importance of directing weight adjustmentss in distillation processes. Notably, we observe performance degradation in the synthetic dataset optimized with random weight adjustment at ipc 10 compared to those without weight adjustment. This decline occurs because, at smaller ipc values, the noise introduced by random weight adjustment outweighs the benefits of diversity. However, as the number of synthetic instances increases, diversity becomes more effective in capturing a broader range of features, leading to improved performance, as reflected at ipc 50.
assess synthetic datasets derived from three scenarios:
Table 4: Crost-acchitctture performance of distilled dataset of CIFAR- using ResNet- 18 and ConvNet-122
1pc
Methods
MobileNetv2
ShuffleNer
EfficientNet
VGG-16
ResNet-50
ConvNet-128
SRe21.
16.1
11.8107
11.1
19.2.aa
22.4.1.
19.4_n.
10
DWA (ours)
27.8
19.4109
20.2
30.0aaa
35.220.7
27.3
18
50
SRelI.
43.21ma
27.5111
24.9
40.4
52.81u7
19.4_n.
DWA (ours)
53.61n.
41.7208
40.7a0a
51.6294
60.620.s
37.0=0.3
SRc2L
28.7+1.3
25 15.3
0.0.
21.5e14
41.8203
10
DWA (ours)
37.3tn.
25.359
24.5
29. 18.6
47.1
47.6mn.4
ComvNet-128
SRc2L
48.8+0.
49
45.
38
53.4=0.5
DWA (ours)
53.3+n0
44.3740.
7500
98.9#0.
56.3=n.
39.0=n.1
Parameters Study on and p. Apart from di- rection, the number of steps K and magnitude of perturbation also influence the distillation pro- cess. Figure 5 illustrates the grid search for these two hyper-parameters and demonstrates the pos- itive impact of perturbation, which is achieved effortlessly, requiring no meticulous manual pa- rameter tuning. In experiments, set
16-
58.4
5166
12
56.8
K
6
K = 12 and p = 15e for the datasets. Readers can adjust these syper-parameters ac- cording to their specific circumstances (different datasets and networks) to obtain better results.
51.6
52.8
0
10
25
30
52.0
(x00001)
Figure 5: Performance grid of ResNet-18 18 with changes
Cross-Architectune Generalization. The gen- eralizability across different architectures is key feature for assessing the effectiveness of the distilled dataset. In this section, we evaluate the surrogate dataset condensed by different backbones (ResNet-18 and Manweet-128 on various archi-
in perturbation steps K and magnitude p.

---

tectures including MobileNetV2 [33], ShuffleNetV2 [26], EfficientNet-BO [37], and VGGNet- [35]. The experimental results are reported in Table and Table It evident that our DWA-synthesized dataset can effectively generalize across various architectures. Notably, for ipc 50 on CIFAR-100 with ShuffleNet22, EfficientNet-Bll and ConvNNe- 1ee-Nee--28-theee three architectures not involved in the data synthesis phase our method achieves impressive classification performance, with accuracies of 41.7%, 40.7%, and 37, 37.0%, respectively, outperforming the latest SOTA method, SRe2L [46], by 14.2%, 15.8%, and 17.6%. In Appendix A.2.3, we further extend the proposed method to vision transformer-based model, DeiT-Tiny [40].
Related Works
Dataset Distillation [43] emerges deriva- tive of Knowledge Distillation (KD) [9], em- phasizing data-centric efficiency over tradi- tional model-centric one. Previous studies have explored various strategies condense datasets, including performance matching, gra- dient matching [54, 52, 19] distribution match- ing [42, 53, 55, 48, 4], and trajectory match- ing [1, 2, 5, 6, 21, 41].
Table 5: Cross architectee performance of distilled dataset ImageNet-1 IK using R8sNet-18.
ipc
Methods
MobileNetv2
ShuffleNet
EfficientNet
SRc2L
15.
9.0=a.7
11.7+0.2
10
DWA (ours)
29.1
11.4
37.4sus
SRc2L
48.3+06
9.0ea.
53 640.
50
DWA (ours)
51.0a0.5
28. 28.5g9.
56.3ao
What distinguishes DD from KD is the bi-level
optimization, which considers both model parameters and image pixels. The consequent complexity and computational burden intricate optimization significantly diminish the effectiveness of the aforementioned methods. To address this issue, SRe2L [46] introduced three-step paradigm known as Squeeze- Reseee-Reoove--eeeee Relabel. This approach relies on the highly encoded distribution prior, i.e., the running mean and rumning variance in the BN layer, eircumvent supervision provided by model training. With this decoupled optimization, SRe2L is able to extend DD to high-resolution and large-scale datasets like ImageNet-IK.
Another critical challenge in dataset compression, not limited to distillation, is how to represent the original dataset distribution with scarcity of synthetic data samples [36]. Previous research claims that the diversity of a dataset can be evaluated by spatial distribution [27]. the maximum dispersion or convex hull volume [47], and coverage [56]. Conventional dataset distillation [49, 15] treats the synthetic compact dataset as an integrated optimizable tensor without specialized guarantees for diversity and relies entirely on the matching objectives mentioned above. Recognizing this limitation, Dream [23] proposed using cluster centers to induce synthesis and ensure adequate diversity. Besides, SRe2L resorts the second-order statisties, i.e., variance of representations pre-trained weights to provide diversity.
6 Conclusion
In this work, we hypothesize that ensuring diversity crucial for effective dataset distillation. Our findings indicate that the random initialization of synthetic data instances contributes minimally to ensuring that each instance captures unique knowledge from the original dataset. We validate our hypothesis through both theoretical and empirical approaches, demonstrating that enhancing diversity significantly benefits dataset distillation. To this end, we propose a novel method, Directed Weight Adjustment (DWA), which introduces diversity in synthesis by customizing weight adjustments for each mini-batch of synthetic data. This approach ensures that each mini-bate condenses a variety of knowledge. Extensive experiments, particularly the large-scale ImageNet- K dataset, confirm the superior performance of our proposed DWA method.
Limitations and Future work. While DWA provides a straightforward and efficient approach to introducing diversity dataset distillation, its reliance on the sampling of a random distribution to adjust weight parameters presents limitations. Increasing the variance of the random distribution can introduce unexpected noise, thereby bottlenecking overall performance. Future investigations could explore synthesizing data instances in sequential manner, encouraging later instances consciously distinguish themselves from earlier ones, thereby further enhancing diversity.

---

Acknowledgements
This research supported by Jiawei Du's STAR Career Development Fund (CDF) C233312004 and Joey Tianyi Zhou's S*SAR SERC Central Research Fund (Use-inspired Basic Research). This research is also supported by National Natural Science Foundation of China under Grant 62301213.
References
[1] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE/CVF Conf. Comput. Vis. Pattem Recognit. (CVPR), pages 10708-10717, 2022.
[2] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation imagenet- 1K with constant memory. In Proc. Int. Conf. Mach. Learn. (ICML), pages 6565-6590, 2023.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 248-255 2009.
[4] Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. arXiv preprint arXiv:2404.00550 000663,00000,,, 2024.
[5] Jiawei Du, Yidi Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proc. IEEE/CVE Conf. Comput. Vis. Puttern Recognit. (CVPR), pages 7749-3758 2023.
[6] Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2023.
[7] Yunzhen Feng. Shanmukha Ramakrishna Vedantam, and Julia Kempe. Emharrassingly simple dataset distillation. In Adv. Neural Inf. Process. Syst. Workshop (NeurIPS Workshop), 2023.
[8] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2021.
[9] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. Int. J. Comput. Vis., (79(6))7779101 1819, 2021.
[10] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 9726-9735, 2020.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 770-778, 2016.
[12] Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint 2tt::2402.11330 2024.
[13] Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[14] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXh::001..03611, 2020.
10

---

[15] Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjuun Song, Joonhyun Jeong, Jung- lung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteri- zation. In Proc. Inl. Conf. Mach. Learn. (ICML., pages 11 11102-11118, 2022.
[16] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[17] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont- Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. Int. J. Comput. Vis. (IJCV), 128(7): 1956-1981, 2020.
[18] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[19] Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In Proc. Int. Conf. Mach. Learn. (ICML), pages 12352- 12364. 2022.
[20] Shiye Lei and Dacheng Tao. A comprehensive survey of dataset distillation. IEEE Trans. Pattern Anal. Mach. Intell., 46(1):17-32, 2024.
[21] Dai Liu, Jindong Gu, Hu Cao, Carsten Trinitis, and Martin Schulz. Dataset distillation by automatic training trajectories. arXiv preprint 1xXiv:2407.1424444 2024.
[22] Songhua Liu and Xinehao Wang. MGDD: meta generator for fast dataset distillation. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2023.
[23] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. DREAM: efficient dataset distillation by representative matching. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 17268-17278. IEEE, 2023.
[24] Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexi- fied implicit gradients. In International Conference on Machine Learning, pages 22649-22674. PMLR, 2023.
[25] Noel Loo, Ramin M. Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Adv. Neural Inf. Process. Syst. (NeurIPS), 2022.
[26] Ningning Ma, Xiangyu Zhang, Hai-Tao Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 122-138. 2018.
[27] Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D22 pruning: Message passing for balancing diversity and difficulty in data pruning. In Proc. Int. Conf. Learn. Represent. (ICLR), 2024.
[28] Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-lcarning from kernel ridge-regression. In Proc. Int. Conf. Learn. Represent. (ICLR), 2021.
[29] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. In Adv. Neural Inf. Process. Syst. (NeurIPS), pages 20596-20607 20607, 2021.
[30] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 524-540. Springer, 2020.
[31] Noveen Sachdeva and Julian J. McAuley. Data distillation: A survey. Trans. Mach. Learn. Res., 2023.
[32] Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstanti- nos Plataniotis. Datadam: Efficient dataset distillation with attention matching. Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 17051-17061, 2023.

---

[33] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Laing-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. IEEE/CVF Conf. Comput. Vis. Pultern Recognit. (CVPR), pages 4510-4520, 2018 8.
[34] Yuzhang Shang, Zhihang Yuan, and Yan Yan. MIM4DD: mutual information maximization for dataset distillation. In Adv. Neural Inf. Process. Syst. (NewrIPS), 2023.
[35] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proc. Int. Conf. Learn. Represent. (ICLR), 2015.
[36] Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset:
An efficient dataset distillation paradigm. arXiv preprint arXiv:2312.03552 2023.
[37] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. Int. Conf. Mach. Learn. (ICML), pages 6105-6114, 2019.
[38] Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: improving LLM pretraining via document de-duplication and diversification. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2023.
[39] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In Proc. Int. Conf. Learn. Represent. (ICLR), 2019.
[40] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[41] Kai Wang, Zekai Li, Zhi-Qi Cheng, Samir Khaki, Ahmad Sajedi, Ramakrishna Vedantam, Konstantinos N Plataniotis, Alexander Hauptmann, and Yang You. Emphasizing discriminative features for dataset distillation in complex scenarios. arXiv preprint arXiv:2410. 17193, 2024.
[42] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. CAFE: learning to condense dataset by aligning features. 2022. In Proc. IEEF/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 121
[43] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. arXiv preprint arXiv; 10959, 2018.
[44] Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan S. Kankanhalli. Efficient adversarial contrastive learning via aobustness-wware coreset selection. 2024.
[45] Hongxu Yin, Pavlo Molchanov, José M. Álvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 8712-8721, 2020.
[46] Zeyuan Yin, Eric P. Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensa- tion imagenet scale from A new perspective. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2023.
[47] Yu Yu, Shahram Khadivi, and Jia Xu. Can data diversity enhance learning generalization? In Proc. Int. Conf. Comput. Linguistics (COLING), pages 4933-4945, 2022.
[48] Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. Echo: Efficient dataset condensation by higher-order distribution alignment. arXiv preprint arXiv:2312.15927, 2023.
[49] Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proc. IEEE/VV Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 11950-11999 11999, 2023.
[50] Xin Zhang, Jiawei Du, Yunsong Li, Weiying Xie, and Joey Tianyi Zhou. Spanning train- ing progress: Temporal dual-depth scoring (TDDS) for enhanced dataset pruning. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit, (CVPR), 2024.
12

---

[51] Xin Zhang, Jiawei Du, Ping Liu, and Joey Tianyi Zhou. Breaking class barriers: Efficient dataset distillation via inter-class feature compensator. arXiv preprint arXiv:2408.0699 2024.
[52] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Proc. Int. Conf. Mach. Learn. (ICML), pages 12674-12685, 2021.
[53] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proc. JEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), pages 6503-6512, 2023.
[54] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Proc. Int. Conf. Learn. Represent. (ICLR), 2021.
[55] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit, (CVPR), pages 7856-7865, 2023.
[56] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In Proc. Int. Conf. Learn. Represent. (ICLR), 2023.
[57] Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self supervied dataset distillation: good compression is all you need. arXiv preprint arXiv:2404.07976, 2024.
13

---

A Appendix
A.1
Minimizing Lmaan and L can be contradictory
To prove that minimizing Lmean and v can result in contradictory objectives for some existing instances, will demonstrate that the gradients required to minimize Luean and Va respectively, may point in opposite directions. Specifically, for any arbitrary instance S; E S, our goal to establish:
MLmaan varar 0s.
<0, 0,
(16)
0s:
For
Mornn Ds.
we have
mL
0 0l((S)) - 1e (T))
alu (S) - 1a (T)I²
(p(()
Üs:
Øs:
(p)(S)
Øs:
= 2 =2(x(S) - μ (7)
(17)
[S]
because (SS)
[ 8; + Σ j+t [3] 8, thus (1(S)
1
For Dham we have
S
dh
0 002 (S) 0² (T)]
0 0 (S) 0² (T)
0 (S)
0s;
0s;
do² (S)
Øs:
0o² (S)
=2 - 0² (T)
0s;
(8: 11 (5))² + +E
jæi [S] (8) / (S))²
=2 - d² (7)
S
8s:
1
0 (s: (S))
= 2 - o² (T)
S
ds:
d 0 p (S))
= 2 (8) o² (T)
2 (8: (S))
[S]
Øs:
2 (s, 1 (S)) (1
= 2 (S) - 0² (T)
(18)
[S]
S
Substitute Equation 17 and Equation 18 back into Equation 16,
DLmean dLvar
os: 0s;
1
=2 [M (S) (T) [S] ([0 02))
(7) 2(s; - 2(s;-M(5)) (1- [S| S (18) 1) [8) - - (T) [0²
[S]
2 3
(T) oz ( (8: - (S))).
(19)
Let R [u(S) - p(T) - oT) where R is constant that be either positive or negative, depending on the values of (S)), (()),,(())))))))))))) (T) and (T). Suppose R > 0. In this scenario, instances for which (s; -(()) < 0 will encounter contradictory objectives in optimization. Conversely, if R < O, instances where (s; p(S)) > 0 will face similar contradictions.
A.2
Experiments
A.2.1
Hyper-parameter Settings
Table 6, Table 7, and Table list the hyper-parameter settings of our method on experimental datasets. We maintain consistency with SRe2L for fair comparison.
14

---

Table 6: Hyper-parameter settings for CIFAR-10/100
Distillation
Validation
#lteration
1000
#Epoch Batch Size
400
Batch Size
100
128
Optimizer
Adam with (Bt, = (0.5,
Optimizer
Adam with weight decay 0.01
Learning Rate
0.25 using cosine decay
Learning Rate
0.001 using cosine decay
RandomCrop Randomforrizott alFlip
Augmentation
Augmentation
Avaz
11
Tempreture
30
K
15e -3 12
Table 7: Hyper-paraneee settings for ImageNe..
Distillation
Validation
#lteration Batch Size
2000
#Epoch
200
100
Batch Size
128
Optimizer
Adam with f } = (0.5,
Optimizer
SGD with weight decay of 0.9
Learning Rate
0.1 using cosine decay RandomBesizedCrop
Learning Rate
0.2 using cosine decay RandomResizedCrog op mandonHorizontaln Fiip
Augmentation
Augmentation
Randomloorizonnal1Fii
Avar
11
Tempreture
20
P, K
15e ,, 12
Table 8: Hyper-parameter settings for ImageNct- I.
Distillation
Validation
#lteration
2000
#Epoch
300
Batch Size
100
Batch Size
128
Optimizer
Adam with (2) (0.5,0.9)
Optimizer
AdamW with weight decay 0.01
Learning Rate
0.25 using cosine decay RandomResizedCrop RandontHorizonaalFiig
Learning Rate
0.001 using cosine decay
RandomResizzedCrop
Augmentation
Augmentation
BandonHorizontall77
Avar
2
Tempreture
20
K
15c 3 --,1 .22
A.2.2 Feature Distanee Calculation
In Figure 4, we use feature distance Dea measure the diversity of distilled dataset. The following how the class-wise feature distance calculated,
ipc
ipc
Dies
1190 (x.) - gor (8
(20)
1=1
1=1
where go, (B)) and ge, (89) are the latent representations of i-th and j-th synthetic instances of class e, specifically the outputs from the last convolutional layer.
A.2.3 Generalization to Vision Transformer-based Models
We acknowledge that our proposed approach cannot be directly applied models without BN layers, such as Vision Transformers (ViTs). Our baseline solution, SRe2I., involves developing ViT-BN model that replaces all LayerNorm layers with BN layers and adds additional BN layers
15

---

between the two linear layers of the feed-forward network. We followed their solution and conducted cross-architecture experiments with DeiT-Tiny [40] on the ImageNet- dataset. The results are listed in Table 9. The results demonstrate that our approach can be applied to ViT-BN with superior performance compared the baseline.
Table Generalization to vision transformer- based model DeiT-Tiny.
Methods
DeiT-Ting Tiny
ResNet-18
ResNet-50
ResNet- 101
SRe2L
15.41
46.80
55.60
60.81
ResNet-18
DWA (ours)
22.72
55.20
62.30
63.3
SRe2L
25.36
24.69
31.15
33.16
DeiT-Tiny-BN
DWA (ours)
37.0
32.64
40.77
43.15
A.2.4 Application to Downstream Tasks
We evaluate our proposed DWA on continual learning task, based on effective continual learning method GDumb [30]. inasemincremettll learning was performed under strict memory constraints on the CIFAR- 100 dataset, with 20 images per class (ipc 20). CIFAR-100 was divided into five tasks, and a ConvNet was trained on our distilled dataset, with accuracy measured as new classes were incrementally introduced. As shown Table 10, DWA significantly outperforms SRe2L across all maas-inncremenall stages, demonstrating superior retention of knowledge throughout the learning process.
Table 10: Application continual learning task.
Class
20
40
60
80
100
SRe2L
15.7
10.6
9.0
7.9
6.9
DWA (ours)
34.6
25.7
22.5
20.2
18.1
A.2.5 Computational Overhead of Distillation
We compare the average time required to generate one ipc using ResNet-18 on CIFAR-100. As shown in Table 11, our proposed DWA incurs only 7.32% increase in computational overhead while significantly enhancing the diversity of the synthetic dataset. This additional overhead arises from the K -step directed weight perturbation applied before generating each ipc, as detailed in lines 6-7 of Algorithm I,
For k to do
Ad A0
P LLs (For+A0 K
Since each ipc requires 1000 iterations of forward-backward backward propagation for generation, the ad- ditional K = 12 forward-backward propagations required by DWA are negligible in the overall distillation process.
Table 11: Computational overhead of distillation on CIFAR-100 with ResNet-
Methods
Avg. time for generating one ipc
SRe2L
1 116.58 (100%)
DWA (ours)
125. (107.32%)
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's s contributions and scope. The abstract and introduction summarize the key contributions and findings, which are consistently supported by detailed methodologies, experiments, and results in the main body. The claims are accurately presented without exaggeration.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper does discuss its limitations, adhering to guidelines regarding as- sumptions, scope of claims, performance factors, computational efficiency, privacy, fairness, and honesty.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper ensures that for each theoretical result, it provides the full set of assumptions and complete (and correct) proof. This adheres to the guidelines by clearly stating or referencing assumptions in the statement of theorems, numbering and cross- referencing all theorems, formulas, and proofs, and providing formal proofs either in the main paper or supplemental material. Additionally, the paper appropriately references any external theorems or lemmas relied upon in the proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results, ensuring transparency and reproducibility. This includes detailed descriptions of experimental setups, methodologies, and any necessary parameters or con- figurations. The paper offers clear instructions and explanations how to replicate the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
18

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper provides implementation details and algorithm descriptions for reproduction. We also release our codes for reproduction in camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guided Soaddes/CooeSuumissi...
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. //ips.cc//ppiii wttpppccc//ccccccccccccc.cc.cclt /tt.g/uddeeeooedduiiiieeeeeeeee SooodeSubmisssooool....
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all the training and test details necessary to reproduce the results, including datasets, hyperparameters, optimizer type, and how they chosen.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
19

---

Answer: [Yes]
Justification: Our experiments were rigorously conducted with five repetitions each, and we meticulously reported both the mean values and standard deviations for each experimental trial.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard crror of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper comprehensively details compute resources in both the experiments
section and supplementary materials, covering GPU type, memory, and storage specifics,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the lines?
NeurIPS Code of Ethics https: h/tp:://weuu./eeeeeeeeeee
Answer: [Yes]
Justification: The research aligns with the NeurIPS Code of Ethics, ensuring ethical standards are upheld throughout the study.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
20

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational research, and therefore, it does not have direct societal impacts to discuss.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
21

---

Answer: [Yes]
Justification: The paper provides proper credit to asset creators, citing relevant papers and explicitly mentioning license and terms of use. URLs are included where possible, and all licenses are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcom thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced the paper are well-documented, providing com- prehensive details alongside the assets, including training procedures, licenses, limitations, and consent processes, ensuring transparency and reproducibility.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
22

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "Rv6pJmW2MP", "reviewer": "Reviewer_Uwti", "summary": "This paper introduces a novel method for dataset distillation (DD): DD through Directed Weight Adjustment (DWA). The authors first identify that the current dataset distillation methods often fail to ensure diversity in the distilled data, which leads to suboptimal performances. To address this issue, they propose a method that uses dynamic and directed weight adjustment techniques to increase the diversity of synthetic data. The propose method DWA introduces three modifications based on SOTA method SRe2L: 1). Decoupled coefficients for batch norm loss 2). Initialize with real images 3). Random perturbation on model weights. Extensive experiments on datasets like CIFAR, Tiny-ImageNet, and ImageNet-1K demonstrate the superior performance of this approach, highlighting its effectiveness in producing diverse and representative synthetic datasets with minimal computational expense.", "questions": "1. The paper considered using a directed perturbation to increase the diversity of the model. Is it equivalent or similar to using model trained on different random seeds? \n2. In section 2, the authors pointed out $\\vec{x}$ and $\\vec{s}$ are in latent space, mapped by $g_{\\theta}$, following the same notation, Eqn. 2 is operating on latent space and the trained classifier $f_{\\theta}$. Is the random weight perturbation method proposed in Section 3.2 only on the classifier part then (Alg 1 only refers to $f_{\\theta}$ or the entire network $g$ is perturbed? \n3. In section 3.2, how is min/max taken over $L$, if I am not mistaken, $L$ is a gradient of the loss function on $\\theta$?", "limitations": "Yes, the authors adequately addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Increasing diversity in distilled datasets is a key challenge posed by existing methods, and this paper addresses it effectively. The authors conducted a detailed analysis showing that batch normalization plays a crucial role in enhancing diversity in dataset distillation. By leveraging this observation, they proposed innovative techniques to incorporate batch normalization more effectively within the distillation process. The result is a significant improvement over state-of-the-art (SOTA) dataset distillation methods.\n2. The authors also conduct detailed ablation studies that demonstrate the effectiveness of each component of their proposed method. The contribution of each element is clearly shown, providing a comprehensive understanding of the method's strengths. Additionally, they show that the method's effectiveness is robust across different hyperparameter settings and datasets, which makes the method have wide practical applicability.", "weaknesses": "1. In the ablation study, the authors analyzed and justified the design choices for two key components: the decoupled $L_var$ coefficient and the Directed Weight Adjustment (DWA) scheme. Additionally, they proposed using real initialization. However, the study lacks an ablation analysis that separately evaluates the contribution of these three components to the overall performance of DWA. Such an ablation would provide a clearer understanding of the individual and combined effects of the decoupled $L_var$ coefficient, the Directed Weight Adjustment scheme, and real initialization on the method's efficacy.\n2. The paper primarily presents results using moderate to large data budgets (10 and 50 IPCs) for most datasets and 100 IPC for ImageNet-1K. Since the notion of “diversity” only becomes significant when each class has multiple prototypes, the proposed method’s benefits are limited to situations when the distillation budget is high. Overall, the proposed method brings sizable performance improvement but only for one SOTA method with specific data budget constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! We give point-to-point replies to your questions in the following.\n\n**Q1**: an ablation analysis. \n\n**A1**: We conducted an ablation study on these three components using the CIFAR-100 dataset under the ipc=50 setting, as you suggested. The experimental results are listed in the following table:\n|Real initialization|Directed weight adjustment|Decoupled $L_{var}$| ACC.(%)|\n|:-:|:-:|:-:|:-:|\n||||47.9|\n|$\\checkmark$|||52.1|\n|$\\checkmark$|$\\checkmark$||57.0|\n|$\\checkmark$||$\\checkmark$|56.6|\n|$\\checkmark$|$\\checkmark$|$\\checkmark$|60.3|\n\nApplying directed weight adjustment without real initialization is not suitable here because the solved $\\tilde{\\Delta\\theta}$ is equivalent to random noise if $\\mathbb{B}$ is initialized from a random distribution (Equation 11), as studied in Table 3 of our submission.\n\n**The results illustrate that both direted weight adjustment and decoupled $L_{var}$ contributes to the performance enhancement.** Additionally, they work in an orthogonal manner to improve diversity without mutual interference.\n\n\n**Q2**: the proposed method’s benefits are limited to situations.\n\n**A2**: We would like to thank you for your insightful critique. We acknowledge that our proposed method performs worse in scenarios with a smaller distillation budget. However, the original intention of our work is to move further towards lossless distillation with a large distillation budget first.\n\nThe existing distillation methods are still inferior to traditional coreset selection methods in scenarios with a large distillation budget (when the size of the set is greater than 30% of the size of the original dataset). This is because existing distillation methods usually overfit to small distillation budget scenarios. For example, the kernel-ridge-regression method (KRR) [1] and the data factorization-based method (RTP) [2] achieve superior performance with ipc = 1 but underperform with ipc = 50. Simply increasing the distillation budget brings limited performance gain in most distillation methods.\n\nTherefore, we chose SRE2L as our baseline, which underperforms with a small distillation budget (ipc = 1) but achieves SOTA performance with a large distillation budget and pioneers dataset distillation in ImageNet-1K. We provide additional results under various ipc settings using ImageNet-1k with ResNet-18, verifying that our method shows a clear advantage over SRE2L, regardless of the ipc setting. \n\n|ipc|1|5|10|20|30|40|50|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|SRe2L|0.1|15.2|21.6|33.6|37.5|42.4|46.8|\n|DWA (ours)|3.4|20.5|37.9|47.4|51.4|53.6|55.2|\n\nWe will incorporate your advice to explore the adaptability of our DWA method to more distillation methods for wider application.\n\n**Q3**:Is it equivalent to using model trained on different random seeds?\n\n**A3**: Thank you for suggesting this interesting investigation. We conducted experiments to compare our directed weight adjustment with models trained on different random seeds. We trained five ResNet models with different random seeds on the CIFAR-100 dataset. We used the setting of ipc = 50, meaning each model generated 10 images per class. Both approaches use the real initialization, as the directed weight adjustment is based on real initialization for directed perturbation. The results are listed below.\n\n| Real initialization | Directed weight adjustment |  5 models with different random seeds| ACC.(%)|Avg. time used for generating 1 ipc|\n|:-:|:-:|:-:|:-:|:-:|\n|$\\checkmark$|||52.1|116.58 s(100%)|\n|$\\checkmark$||$\\checkmark$|54.6|596.58 s (511%)|\n|$\\checkmark$|$\\checkmark$||57.0|125.12 s (107%)|\n\nTo some extent, the models trained with different random seeds **are similar to our proposed directed perturbation**, as they can enhance performance. However, **training models with different random seeds is computationally expensive**. Our approach incurs only about 7% additional computational overhead compared to the 411% additional computational overhead required for models trained with different random seeds.\n\n**Q4**: Is the random weight perturbation method proposed in Section 3.2 only on the classifier part or the entire network\n\n**A4**: The weight perturbation is actually applied to the entire network. However, $f_\\theta$ refers only to the classifier. We clarified this in the footnote below Algorithm 1 in our original submission.\n\nAs stated in line 95 of our submission, \"Throughout the paper, we explore the properties of synthesized datasets within the latent space\". We made this assumption for theoretically proving the enhanced variance in latent space because comparing the variance at the pixel level is meaningless (the variance of each pixel is very large). Therefore, the feature extractor $g$ is supposed to be fixed under our assumption and only the classifier $f$ is perturbed in Algorithm 1. We use the footnote to indicate that the operation is actually conducted over the entire network $h$ in the actual optimization process.\n\nThank you for pointing out this misunderstanding, we will revise the footnote and our assumptions to avoid such misunderstanding in our paper. \n\n**Q5**:$L$ definition \n\n**A5**: We sincerely apologize for the typo. $L$ is the sum of instance-wise losses, instead of the gradients. We incorrectly added  $\\nabla\\theta$ in the Equations 11 and 12 in Section 3.3. The corrected definition of $L$ should be \n\n$L_{\\mathbb{B}} (f_{\\theta_{\\mathcal{T}} + \\Delta\\theta} ) = \\sum_\\limits{x_i \\in \\mathbb{B}}  \\ell (f_{\\theta_{\\mathcal{T}} + \\Delta\\theta}, x_i)$  in Equation 11 \n$L_{\\mathcal{T}} (f_{\\theta_{\\mathcal{T}}} ) = \\sum_\\limits{x_i \\in \\mathcal{T}}  \\ell (f_{\\theta_\\mathcal{T}}, x_i ),$ in Equation 12. \n\n\n[1] Timothy Nguyen, et al. Dataset meta-learning from kernel ridge-regression. In Proc. Int. Conf. Learn. Represent. (ICLR), 2021.\n\n[2] Zhiwei Deng, et al. Remember the past: Distilling datasets into addressablememories for neural networks. (NeurIPS), 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ntfMwScHWV", "reviewer": "Reviewer_NmcN", "summary": "This paper introduces a random directed perturbation strategy to the teacher models for further strengthening the diversity of the existing sample-wise data synthesis method. Through intuitive empirical and theoretical analysis of the diversity factors, the authors proposed a simple yet overlooked weight adjustment strategy, showing promising results in various vision models and benchmarks including ImageNet.\n\n========== Post rebuttal ==========\n\nThank you for the detailed responses and additional analysis. Having read the responses, my major concerns were addressed. Therefore, I raise the rating.", "questions": "- Is the proposed synthesis method applicable to the ViT-based models that leverage LayerNorm rather than BatchNorm layers? Further experiments (directly synthesizing on ViT-based models or cross-architecture experiments by synthesizing with CNN models and testing on ViT-based models) would be required for the generalization of the proposed method to the recent models.\n- Disregarding the computational overhead, can searching for the optimal perturbation parameters per sample rather than using one set of predefined universal perturbation parameters help improve the performance?\n- Can you provide comparison of computation overhead with SRe2L on ImageNet dataset?", "limitations": "See above weaknesses and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Intuitive motivational analysis of how the existing methods, SRe2L, lack of diversity, while providing empirical and theoretical backgrounds for the factors that can affect diversity during the synthesis process.\n- Proposed simple yet overlooked perspectives of existing data synthesis loss via 1) perturbing the teacher model in a random directed way, 2) decoupled variance matching, both of which collaboratively enhances sample diversity.\n- These simple yet intuitive methods exhibit largely improved performance on various vision models and benchmarks including large-scale ImageNet dataset. \n- Thorough ablation studies with the decoupled variance coefficient, weight adjustment, and cross-architecture additionally provide an intuition of how the proposed method enhances the actual performance.", "weaknesses": "- The proposed method has a heavy reliance on BN statistics matching from CNN-based models. Considering that the recent State-of-the-art models such as ViT are not leveraging BN, the proposed approach might not be directly applicable to these vision transformer-based models.\n- Failure cases of DWA should be more clarified. For example, In Table 1, when the model and the number of classes in the target dataset is small (ConvNet / CIFAR-10), DWA underperforms compared to the other methods.\n- Comparison with baseline (SRe2L) in terms of computational overhead is required. Since this paper introduces add-on procedure, weight adjustment, to the baseline, the authors should clarify if the proposed method does not induce significant computation overhead during synthesis.\n- In Figure 5, the test performance varies depending on the perturbation parameters. Although the authors deterministically set one set of parameters for all the different datasets, it should be clarified if these hyper-parameters are not sensitive and generalizable to all the datasets, to reduce time-consuming trial and error for searching these parameters on the other general datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and answer your questions in order.\n\n**Q1 (Weakness 1 & Question 1)**: Considering that the recent State-of-the-art models such as ViT are not leveraging BN, the proposed approach might not be directly applicable to these vision transformer-based models.\n\n**A1**: We acknowledge that our proposed approach cannot be directly applied to models without BN layers, such as ViT. Our baseline solution, SRE2L, involves developing a ViT-BN model that replaces all LayerNorm layers with BN layers and adds additional BN layers between the two linear layers of the feed-forward network. We followed their solution and conducted cross-architecture experiments with DeiT-Tiny on the ImageNet-1K dataset. The results are listed below.\n\n- Squeezed Model: DeiT-Tiny-BN\n\n| Evaluation Model | DeiT-Tiny | ResNet-18 | ResNet-50 | ResNet-101 |\n|:----------------:|:---------:|:---------:|:---------:|:----------:|\n|      SRe2L       |   25.36   |   24.69   |   31.15   |   33.16    |\n|    DWA (ours)    |  **37.0**         | **32.64**         |  **40.77**         |  **43.15**          |\n\n- Squeezed Model: ResNet-18\n\n| Evaluation Model | DeiT-Tiny | ResNet-18 | ResNet-50 | ResNet-101 |\n|:----------------:|:---------:|:---------:|:---------:|:----------:|\n|      SRe2L       |   15.41   |   46.71   |   55.29   |   60.81    |\n|    DWA (ours)    |  **22.72**        |  **55.2**         |       **62.3**    |      **63.3**      |\n\n*\\*DeiT-Tiny and DeiT-Tiny-BN refer to the vanilla architecture and the BN-integrated variant, respectively.*\n\nThe results demonstrate that our approach can be applied to ViT-BN with superior performance compared to the baseline. Theoretically, our approach can be combined with other generation-based distillation methods to avoid the BN limitation. This can be our future exploration.\n\n**Q2 (Weakness 2)**: When the model and the number of classes in the target dataset is small (ConvNet / CIFAR-10), DWA underperforms compared to the other methods.\n\n**A2**: Our proposed DWA underperforms compared to other methods only with ConvNet on CIFAR-10, although it outperforms them on CIFAR-100. We believe the poor performance on CIFAR-10 is inherited from our baseline, SRE2L, which also underperforms with ConvNet on this dataset. However, our DWA still significantly outperforms SRE2L in this scenario. \n\nThe reasons for the poor performance of SRE2L and DWA in using ConvNet on CIFAR-10 are as below: \n\nFirst, ConvNet is a very small and simple architecture, consisting of only three convolutional layers. The typical bi-level optimization approach used in previous methods has advantages in these simplified architectures. In contrast, SRE2L and our DWA leverage a coarser metric (BN parameters) for distillation, which is more adaptive to complex architectures with more BN layers.\n\nSecond, the synthetic datasets in CIFAR-10 are only 1/10 the size of those in CIFAR-100. Our proposed DWA improves distillation performance through enhanced diversity of the synthetic datasets. As the reviewer Uwti stated, “since the notion of ‘diversity’ only becomes significant when each class has multiple prototypes,” our method is expected to achieve greater performance gains with larger synthetic datasets. This is because our DWA can prevent duplicated synthetic instances.\n\n**Q3 (Weakness 3 & Question 3)**: Since this paper introduces add-on procedure, weight adjustment, to the baseline, the authors should clarify if the proposed method does not induce significant computation overhead during synthesis.\n\n**A3**: Thank you for the constructive comments, we append the computational overload as below. \n\n| Methods |Avg. time used for generating 1 ipc|\n|:----:|:----:|\n|SRE2L |116.58 s (100%)|\n|DWA (ours) |125.12 s (107.32%)|\n\nWe compare the averaged time used for generating 1 image for each class using ResNet-18 on CIFAR-100. It can be seen that our proposed DWA only requires 7.32% additional computational overhead to improve the diversity of the synthetic dataset. \n\nThat is attributed to the K steps directed weights perturbation before the generation of each ipc as stated in lines6-7 in Algorithm 1:\n\nFor $k = 1$ to $K$ do\n    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\\Delta\\theta_k = \\Delta\\theta_{k-1} + \\rho/K \\nabla L_{S_{k0}} \\left( f_{\\theta_T + \\Delta\\theta_{k-1}} \\right)$.\n    \nSince each ipc requires 1000 iterations of forward-backward propagation for generation, our DWA’s requirement of only $K=12$ additional forward-backward propagations is negligible in the overall optimization process. We will add this analysis to our revised paper.\n \n\n**Q4 (Weakness 4 & Question 2)**: It should be clarified if these hyper-parameters are not sensitive and generalizable to all the datasets, to reduce time-consuming trial and error for searching these parameters on the other general datasets. And can searching for the optimal perturbation parameters per sample rather than using one set of predefined universal perturbation parameters help improve the performance?\n\n\n**A4**: We demonstrated the results of hyperparameters search in **Figure 5** of our submission. It can be seen that a significant performance gain can be achieved as long as $K>4 $ and $\\rho>0.001$, indicating the effectiveness of our DWA. Our method only involves two hyperparameters, $K$ and $\\rho$, and we use consistent values of $K=12$, $\\rho=0.0015$ across the four datasets: CIFAR-10/100, Tiny-ImageNet, ImageNet-1k.  Thus, the cost of searching for hyperparameters is very low for our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kxYPY53zlI", "reviewer": "Reviewer_e5no", "summary": "This paper proposes a new method to enhance dataset distillation by improving the diversity of synthesized datasets. The authors introduce a dynamic and directed weight adjustment technique to maximize the representativeness and diversity of synthetic instances. Theoretical and empirical analyses demonstrate that increased diversity enhances the performance of neural networks trained on these synthetic datasets. Extensive experiments across multiple datasets, including CIFAR, Tiny-ImageNet, and ImageNet-1K, show that the proposed method outperforms existing approaches while incurring minimal computational expense.", "questions": "See the weaknesses section. I am happy to raise my score if the above concerns can be addressed.", "limitations": "The authors discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and structured.\n- The analysis of the decoupled variance component is interesting.\n- The proposed dynamic adjustment mechanism is novel and can enhance the diversity of the synthesized dataset.\n- The paper provides a solid theoretical foundation and extensive empirical validation across multiple datasets and architectures.\n- The findings have important implications for the efficiency of dataset distillation, particularly for large-scale datasets.", "weaknesses": "- The effectiveness of the diversity in generated datasets has been discussed in many prior works [1, 2]. \n- The authors should compare the proposed method with recent SOTA dataset distillation methods that extended from SRe2L [2, 3]. Especially for [3], the relation to it needs to be clarified.\n- A comparison with recent diffusion-based generative dataset distillation methods [4, 5] is also needed because they are free of optimization.\n- How long does it typically take to search for the optimal values of $K$ and $\\rho$ for ResNet-18 and other different datasets and network architectures? Can the authors provide insights into the computational cost associated with this parameter-tuning process?\n- Are there any scenarios where the directed weight adjustment might negatively impact performance or diversity? If so, how can these be mitigated?\n- Have the authors considered some downstream applications to further verify the effectiveness of the proposed method, such as continual learning and privacy protection?\n\n[1] Liu, Yanqing, et al. \"Dream: Efficient dataset distillation by representative matching.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Shao, Shitong, et al. \"Generalized large-scale data condensation via various backbone and statistical matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Gu, Jianyang, et al. \"Efficient dataset distillation via minimax diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[5] Su, Duo, et al. \"D4M: Dataset Distillation via Disentangled Diffusion Model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! We sincerely appreciate the time and effort you dedicated to reviewing our work. We will incorporate the mentioned state-of-the-art methods [2,3,4,5] in the related work section and include the downstream experiments in our revised paper. Below are our responses to your questions.\n\n**Q1 (Weakness 1)**: prior works [1, 2]. \n\n**A1**: Yes, we will incorporate those prior works in the Related Works of our revised paper. In fact, Dream [1] is already cited in line 298 in our original submission. We will also include RDED [2] in our revised paper, which enhances diversity by compressing multiple key patches into a single image. These key patches are selected from the original dataset based on a specific metric.\n\nUnlike these prior works, we explore a novel perspective by adjusting weight parameters to introduce greater variance in generating synthetic instances. Our DWA is theoretically complementary to these prior works.\n\n\n**Q2 (Weakness 2 & 3)**: compare with recent SOTA [2,3,4,5]\n\n**A2**: Yes, we will compare our proposed DWA with the recent SOTA methods [2,3,4,5] in our revised paper. We first append the comparison results in the below tables. \n\n**Tiny-ImageNet**\n- ResNet-18\n\n|ipc|DWA (Ours)|RDED[2]|G-VBSM[3]|D4M[5]\n|-|-|-|-|-|\n|50|52.8|**58.2**|47.6|46.8\n|100|**56.0**|-|51.0|53.3\n- ResNet-101\n\n|ipc|DWA (Ours)|RDED[2]|G-VBSM[3]|D4M[5]\n|-|-|-|-|-|\n|50 |**54.7**|41.2|48.8|53.2\n|100|**57.4**|-|52.3|55.3\n\n\n**ImageNet-1K**\n- ResNet-18\n\n|ipc|DWA (Ours)|RDED [2]|G-VBSM[3]|Minimax[4]|D4M[5]\n|-|-|-|-|-|-|\n|50|55.2|56.5|51.8|**58.6**|55.2\n|100|59.2|-|55.7|-|**59.3**\n- ResNet-101\n\n|ipc|DWA (Ours)|RDED[2]|G-VBSM[3]|D4M[5]\n|-|-|-|-|-|\n|50|63.3|61.2|61.0|**63.4**\n|100|**66.7**|-|63.7|66.5\n\nOur proposed DWA outperforms the four state-of-the-art methods in the following scenarios: Tiny ImageNet with ResNet-18 and ipc 100, and ResNet-101 with ipc 50 and 100, as well as ImageNet-1K with ResNet-101 and ipc 100. While diffusion-based methods have advantages on the ImageNet-1K dataset using ResNet-18, our DWA performs very close to their level.\n\nRDED[2] enhances diversity by compressing multiple key patches into a single image using a dataset pruning approach, selecting patches from the original dataset based on specific metrics instead of synthesizing distilled datasets. G-VBSM[3] ensures data densification by performing matching based on diverse backbones, layers, and statistics, sampling a random backbone for distillation in each iteration. In contrast, our method introduces perturbations on a single model, whereas G-VBSM requires extra computation and memory for soft labels generated by different architectures, indicating the efficiency of our approach.\n\n\n**Q3 (Weakness 4)**: computational cost on parameter-tuning?\n\n**A3**: Our method, DWA, involves only two hyperparameters: $K$ and $\\rho$. We use consistent values of $K=12$ and $\\rho=0.0015$ across the four datasets: CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1k. The generalized parameters of our DWA make the parameter-tuning process straightforward to implement.\n\nWe demonstrated the results of the hyperparameter search in Figure 5 in our original submission. Figure 5 illustrates that a significant performance gain can be achieved as long as $K>4$ and $\\rho>0.001$.\n\nWe searched the parameters $K$ and $\\rho$ using binary search for 12 times (6 times for each). Each search requires 1.5 hours using ResNet-18 on CIFAR-100.The computational cost associated with this parameter-tuning process is low in our DWA method.\n\n\n**Q4 (Weakness 5)**: negatively impact performance\n\n**A4**: We would like to thank you for the insightful question. According to our experiments across four datasets and seven architectures, we did not observe any negative impact on performance from using our DWA method.\n\nHowever, based on our theoretical analysis in Section 3, there can be negative impacts of using our DWA. As we stated in line 160, perturbing converged weight parameters will introduce noise. This issue is verified by our experiments in the table of random perturbation, which also motivated us to investigate the directed perturbation to avoid introducing noise. Therefore, there are two scenarios where DWA can have a negative impact:\n\nFirst, **if the model $f_{\\theta_{\\mathcal{T}}}$ is not well-optimized or converged**. The theoretical guarantee of the directed perturbation is proved in Equation 14. Intuitively, this means that the directed perturbation $\\widetilde{\\Delta\\theta}$ will decrease the integrated loss $L_{\\mathcal{T} \\setminus \\mathbb{B}}$. Equation 14 will not hold if the weight parameters $\\theta_{\\mathcal{T}}$ have not converged; that is, the model used to generate the synthetic dataset is not well-optimized.\n\nSecond, **if we use a very large value of $\\rho$**. A larger $\\rho$ will also make Equation 14 invalid, as our assumption is that $\\rho$ is small enough for the first-order Taylor expansion.\n\nHowever, both scenarios can be easily avoided by using a well-optimized $f_{\\theta_{\\mathcal{T}}}$ and a smaller $\\rho$. Your suggestion inspires us to design a mechanism to test Equation 14 in our code to avoid these two scenarios and improve our DWA.\n\n**Q5 (Weakness 6)**:downstream applications\n\n**A5**: Thank you for the advices, we first examine the continual learning task to verify our proposed DWA. Our implementation is based on the effective continual learning method GDumb [1]. Class-incremental learning was performed under strict memory constraints on the CIFAR-100 dataset, using 20 images per class (ipc=20). CIFAR-100 was divided into five tasks, and a ConvNet was trained using our distilled dataset, with accuracy reported as new classes were incrementally introduced.\n|class|20|40|60|80|100|\n|-|-|-|-|-|-|\n|SRe2L|15.7|10.6|9.0|7.9|6.9|\n|DWA|34.6|25.7|22.5|20.2|18.1|\n\nAs you suggested, we will also incorporate downstream applications such as Neural Architecture Search(NAS) and privacy protection in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8WUtp2v73n", "reviewer": "Reviewer_LurV", "summary": "This paper proposes a dynamic adjustment mechanism to enhance the variation component in batch normalization loss so that increase the diversity of synthetic images. By adjusting the weights, the synthesized images achieve sufficient diversity to cover the widespread distribution of the real dataset.", "questions": "1. In equation 14, can the authors explain why it is <=0? In line 176, is the first item clearly greater than the second item because equation 14 is <=0, or is the causal relation inverse, with the first item being greater than the second item concluding that equation 14 is <=0?.\n2. In Table 1, Can the author explain why the results of ConvNet on CIFAR10 are obviously worse than state of the art?\n3. Why there is no results for ipc=1 in all result tables.", "limitations": "The authors address the limitation about the unexpected noise.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a. Originality: The author innovatively analyzes the different components of the loss function. This work differs from previous contributions by proposing a directed weight adjustment algorithm to increase the diversity of synthetic images. Related work is adequately cited.\nb. Quality: The submission is technically sound. The claims are well supported by theoretical analysis and experimental results. It is a complete piece of work. The authors are careful and honest in evaluating both the strengths (increased diversity and minimal computational expense) and weaknesses (unexpected noise) of their work.\nc. Clarity: The submission is clearly written and well organized.\nd. Significance: The results are important. The submission addresses the diversity problem more effectively than previous work. Most results advance the state of the art with a large margin, providing a unique theoretical approach that is also demonstrated in experiments.", "weaknesses": "c. Clarity: A few equations, such as (14), need more explanation. The current text does not adequately inform the reader on how to reproduce the results, as there is insufficient connection to the implementation details.\nd. Significance: The results on CIFAR10 using ConvNet are significantly worse than the current state of the art. Additionally, the authors did not mention whether or how the method can be integrated as a plug-in.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments! We answer your questions as below. \n\n**Q1 (Weakness 1 & Question 1)**: A few equations, such as (14), need more explanation. The current text does not adequately inform the reader on how to reproduce the results, as there is insufficient connection to the implementation details.\n\n**A1**: Thank you for your advice on improving the presentation of our paper. We will revise these equations and release our code for reproducibility once our paper is accepted.\n\nThe solutions to Equation 11, referenced in Algorithm 1, are obtained using a multi-step first-order gradient descent method as stated in lines 6-7 of Algorithm 1. \n\nEquation 14 aims to prove that directed perturbation will not introduce unanticipated noise while enhancing diversity. This provides a theoretical explanation for using Equation 11 to solve for the optimal perturbation. Additionally, it offers behavioral guidance for the \"Directed Weight Adjustment\" step in line 7 of Algorithm 1.\n\nDerivation of Equation 14： According to Equation 11, $\\Delta \\theta$ can achieve max $L_B$, so we have \n\n$$ L_B\\left(f_{\\theta +  \\widetilde{\\Delta\\theta}}\\right) - L_B\\left(f_{\\theta}\\right) \\leq 0. $$\n\nand then we can get Equation 14 $\\leq 0$.\n\n**Q2 (Weakness 2 & Question 2)**: The results on CIFAR10 using ConvNet are significantly worse than the current state of the art. \n\n**A2**: We believe the poor performance on CIFAR-10 is inherited from our baseline, SRE2L, which also underperforms with ConvNet on this dataset. However, our DWA still outperforms SRE2L in this scenario. \n\nThe reasons for the poor performance of SRE2L and DWA in using ConvNet on CIFAR-10 are as below: \n\nFirst, ConvNet is a very small and simple architecture, consisting of only three convolutional layers. The typical bi-level optimization approach used in previous methods has advantages in these simplified architectures. In contrast, SRE2L and our DWA leverage a coarser metric (BN parameters) for distillation, which is more adaptive to complex architectures with more BN layers.\n\nSecond, the synthetic datasets in CIFAR-10 are only 1/10 the size of those in CIFAR-100. Our proposed DWA improves distillation performance through enhanced diversity of the synthetic datasets. As the reviewer Uwti stated, “since the notion of ‘diversity’ only becomes significant when each class has multiple prototypes,” our method is expected to achieve greater performance gains with larger synthetic datasets. This is because our DWA can prevent duplicated synthetic instances.\n\n\n**Q3 (Weakness 3)**: Whether or how the method can be integrated as a plug-in.\n\n**A3**: Yes, our method is designed as a plug-in approach for generation-based distillation methods. Consequently, our DWA can be integrated with baselines such as [1] and [2], as our theoretical analysis remains valid for these methods. Additionally, our code is easy to implement, requiring only minor adjustments to the weights before synthesis.\n\nHowever, our DWA is not adaptable to trajectory-matching methods like [3], as these methods rely on stable and consistent gradient trajectories during training to guide the synthesis of distilled data. Using our DWA would disrupt the consistency of these trajectories, making it challenging to condense sufficient knowledge.\n\nWe plan to explore the integration of our DWA with more baselines in future work.\n\n\n**Q4 (Question 3)**: Why there is no results for ipc=1 in all result tables?\n\n**A4**: We do not present the results for ipc=1 for the following reasons:\n\nFirst, our proposed DWA is designed to introduce diversity in the dimension of ipc, specifically by introducing intra-class diversity to prevent duplicated synthetic instances within the same class. Therefore, our DWA is effective only when ipc > 1.\n\nSecond, the diversity issue becomes a performance bottleneck only when there is a sufficient distillation budget. In the scenario with ipc=1, the bottleneck of performance is the distillation budget rather than diversity.\n\nWe also conducted experiments to compare our DWA method to the baseline SRE2L in scenarios with different ipc settings. These experiments were conducted using ImageNet-1k with ResNet-18.\n\n|    ipc     |  1   |  5   |  10  |  20  |  30  |  40  |  50  |\n|:----------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|   SRe2L    | 0.1 | 15.2 |21.6  | 33.6 | 37.5 | 42.4 | 46.8 |\n| DWA (ours) | 3.4 | 20.5 | 37.9 | 47.4 | 51.4 | 53.6 | 55.2 |\n\nThe results verify effectiveness of our DWA across different ipc settings. \n\n[1] Shao, Shitong, et al. \"Generalized large-scale data condensation via various backbone and statistical matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Gu, Jianyang, et al. \"Efficient dataset distillation via minimax diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 10708–10717, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel directed weight adjustment (DWA) mechanism to enhance diversity in dataset distillation. The reviewers collectively recognised the significance of this paper: 
- Addressing the critical challenge of diversifying synthetic datasets
- Theoretical foundation
- Empirical validation across multiple datasets and architectures
- Detailed analysis of the method’s components
- Minimal computational overhead and simplicity
- Clear presentation

Most of the initial concerns were effectively addressed by the rebuttal, as confirmed by the reviewers.

The paper makes a substantial contribution to the field and merits consideration for a spotlight presentation.
Decision: Accept (spotlight)