Sequential Subset Matching for Dataset Distillation
Authors: Jiawei Du
Qin Shi
Joey Tianyi Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Dataset distillation
gradients matching

Abstract: Dataset distillation is a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields a similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current state-of-the-art methods treat the entire synthetic dataset as a unified entity and optimize each synthetic instance equally . This static optimization approach may lead to performance degradation in dataset distillation. 
Specifically, we argue that static optimization can give rise to a coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads to the failure of the distilled dataset to extract the high-level features learned by the deep neural network (DNN) in the latter epochs.
In this study, we propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data to encourage sequential acquisition of knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms state-of-the-art methods in various datasets, including SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet.

Sequential Subset Matching for Dataset Distillation
Jiawei Du, Qin , Joey Tianyi Zhou X
Centre for Frontier Al Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore Institute of High Performance Computing (IHPC), Ageney for Science, Technology and Research (A (A*TA))) Singapore {duju, Jeenj,,,ooe_-hhouhee 2hou) a-star.eduu edu.sg, (atpiin922444@mmillll
Abstract
Dataset distillation a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current ofate-of-the-art methods treat the entire synthetic dataset as unified entity and optimize each synthetic instance equally. This static optimization approach may lead to performance degradation dataset distillation. Specifically, we argue that static optimization can give rise to coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads the failure of the distilled dataset extract the high-level features learned by the deep neural network (DNN) in the latter epochs. In this study. we propose new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data encourage sequential acquisition knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms sate-of-the-and methods in various datasets, including SVNH, CIFAR-10 10, CIFAR- 100, and Tiny ImageNet. Our code is available at https///ithub..om/shqiii lisss//mtth
Introduction
Recent advancements in Deep Neural Networks (DNNs) have demonstrated their remarkable ability to extract knowledge from large-scale real-world data, as exemplified by the impressive performance of the large language model GPT-3, which was trained on staggering 45 terabytes text data [4]. However, the use of such massive datasets comes at significant cost in terms of data storage, model training, and hyperparameter tuning.
The challenges associated with the use of large-scale datasets have motivated the development of various techniques aimed reducing datasets size while preserving their essential characteristics. One such technique is dataset distillation [5, 6, 13, 22, 29, 35, 42, 48, 50, 51, 52, 53], which involves synthesizing smaller dataset that effeetively captures the knowledge contained within the original dataset. Models trained on these synthetic datasets have been shown to achieve comparable performance those trained on the full dataset. recent years, dataset distillation has garnered increasing attention from the deep learning community and has been leveraged in various practical applications, including continual learning [41, 52, 53], neural architecture search [21, 39, 40, 51, 53), and privacy- tasks [12, 15, 31], among others.
Conresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Existing methods for dataset distillation, as proposed in [5, 11, 13, 33, 37, 38, 47, 51, 531, have improved the distillation performance through enhanced optimization methods. These approaches have achieved commendable improvements in consolidating knowledge from the original dataset and generating superior synthetic datasets. However, the knowledge condensed by these existing methods primarily originates from the easy instances, which exhibit a rapid reduction in training loss during the early stages of training. These easy instances constitute the majority of the dataset and typically encompass low-level, yet commonly encountered visual features (c.g. edges and textures [49]) acquired in the initial epochs of training. In contrast, the remaining, less frequent, but more challenging instances encapsulate high-level features (e.g. shapes and contours) that are extracted in the subsequent epochs and significantly impact the generalization capability of deep neural networks (DNNs). The findings depicted in Figure 1 reveal that an overemphasis on low-level features hinders the extraction and condensation of high-level features from hard instances, thereby resulting in decline in performance.
n Syach
Pefomeres Compel
In this paper, we investigate the factors that hinder the efficient condensation of high-level features in dataset distillation. Firstly, we re- veal that DNNs are optimized through pro- cess of learning from low-level visual features and gradually adapting higher-level features. The condensation of high-level features deter- mines the effectiveness of dataset distillation. Secondly, we argue that existing dataset distil- lation methods fail extract high-level features because they treat the synthetic data as a unified entity and optimize each synthetic instance un- varyingly. Such static optimization makes the synthetic instances become coupled with each other easier in cases where more synthetic in- stances are optimized. As a result, increasing the size of synthetic dataset will over-condense the low-level features but fail condense addi- tional knowledge from the real dataset, let alone the higher-level features.
VIT
NTT SeplaaoATT
Had
Easy
Spoch
CI-AKT G
Figure 1: Left: MTT [5] fails to extract adequate high- level features. The loss drop rate between easy and hard instances is employed as the metric to evaluate the condensation efficacy of low-level and high-level fea- tures. The upper solid lines represent the loss change of hard instances, while the lower dashed lines depict the loss change of easy instances. The inability to decrease the loss of hard instances indicates MTT's ' inadequacy in capturing high-level features. In contrast, our pro- posed SeqMatch successfully minimizes the loss for both hard and casy instances. Right: The consequent performance improvement of SeqMatch in CIFAR [25] datasets. Experiments are conducted with 50 images per class (ipc 50).
Building upon the insights derived from our anal- ysis, we present a novel dataset distillation strat- egy, termed Sequential Subset Matching (Seq- Match), which is designed to extract both low-level and high-level features from the real dataset, thereby improving dataset distillation. Our approach adopts simple yet effective strategy for reorga- nizing the synthesized dataset S during the distillation and evaluation phases. Specifically, we divide the synthetic dataset into multiple subsets and encourage each subset to acquire knowledge in the order that DNNs learn from the real dataset. Our approach can be seamlessly integrated into existing dataset distillation methods. The experiments, as shown in Figure 1, demonstrate that SeqMatch effectively enables the latter subsets to capture high-level features. This, in turn, leads to substantial improvement in performance compared to the baseline method MTT, which struggles to compress higher-level features from the real dataset. Extensive experiments demonstrate that SeqMatch outper- forms state-of-the-art methods, particularly in high compression ratio² scenarios, across a range of datasets including CIFAR- 10, CIFAR- 100, TinylmageNet, and subsets of the ImageNet.
In nutshell, our contribution can be summarized as follows.
we examine the inefficacy of current dataset distillation condensing hard instances from the original dataset. We present insightful analyses regarding the plausible factors contributing to this inefficacy and reveal the inherent preference of dataset distillation in condensing knowledge.
We thereby propose a novel dataset distillation strategy called Sequential Subset Matching (SeqMatch) to targetedly encourage the condensing of higher-level features. SeqMatch seamlessly integrates with existing dataset distillation methods, offering easy implementa-
ccompression ratio compressed dataset size/ / full dataset size [8]
2

---

tion. Experiments on diverse datasets demonstrate the effectiveness of SeqMatch, achieving Late-of-the-ar art performance.
2
Related work
Coreset selection is the traditional dataset reduction approach by selecting representative prototypes from the original dataset[2. 17, 43, 45]. However, the non-editable nature the coreset limits its performance potential. The idea of synthesizing the "coreset" can be traced back to Wang et al. [47]. Compared to coreset selection, dataset distillation has demonstrated greatly superior performance. Based on the approach of optimizing the synthetic data, dataset distillation can be taxonomized into two types: mata-matching methods and meta-learning methods [30].
Data-matching methods encourage the synthetic data to imitate the influence of the target data, involv- ing the gradients, trajectories, and distributions. Zhao and Bilen [52] proposed distribution matching to update synthetic data. Zhao et al. [53] matched the gradients of the target and synthetic data in each iteration for optimization. This approach led to the development of several advanced gradient- matching methods[5, 20, 22, 51]. Trajectory trajectory-matching methods [5, 9, 13] further matched multi-step gradients to optimize the synthetic data, achieving sfate-of-the-at art performance. Factorization-baat methods [11, 29, 33] distilled the synthetic data into oowdimensional manifold and used a decoder recover the source instances from the factorized features.
Meta-learning methods treat the synthetic data as the parameters to be optimized by a meta (or outer) algorithm [3, 11, 32, 34, 37, 38, 54]. A base (or inner) algorithm solves the supervised learning problem and is nested inside the meta (or outer) algorithm with respect to the synthetic data. The synthetic data can be directly updated minimize the empirical risk of the network. Kernel ridge regression (KRR) based methods [34, 37, 38] have achieved remarkable performance among meta-learning methods.
Both data-mat matching and meta weta-learning methods optimize each synthetic instance equally. The absence of variation in converged synthetic instances may lead to the extraction of similar knowledge and result in over-representation of low-level features.
Preliminaries
Background Throughout this paper, we denote the target dataset as T = { 01) v1) Each pair of data sample is drawn i.i.d. from a natural distribution D, and E Rdd, E y = 1, -3 where is the dimension of input data and C' is the number of classes. We denote the synthetic dataset as S = {s;, y:) 1
[S] where s; E Rd Vi E V. Each class of S contains ipc (images per class)
data pairs. Thus, |S| = ipc x C and ipe is typically set to make (S] < [T].
We employ fe to denote a deep neural network f with weights 6. An ideal training progress is to search for an optimal weight parameter è that minimizes the expected risk over the natural distribution ) - [E( [e( (((f((z), 3)]. However, as we can only access the training T sampled from the natural distribution D. the practical training approach of the network f is to minimizing the empirical risk LT(fe) minimization (ERM) on the training set T, which is defined as
D. which is defined
as (D(e)
i -=aal alg(T, = arg min (foo) where LT(S0)
Σ ([o(I:), IEET
(I)
T
where É can be any training loss function; alg the given training algorithm that optimizes the initialized weights parameters 6g over the training set T; On is initialized by sampling from distribution Pa..
Dataset distillation aims to condense the knowledge of T into the synthetic dataset so that training over the synthetic dataset S achieve comparable performance as training the target dataset T. The objective dataset distillation can be formulated as,
E (2.9)~D,Bo-BB)) Poo
[e[fagg[T ..) (x), w)]
E (.0)D.Oo~PP.
(((fagg(s.oo)(0 v)].
(2)
Gradient Matching Methods We take gradient matching methods as the backbone method to present our distillation strategy. Matching the gradients introduced by T and S helps to solve S
3

---

in Equation 2. By doing so, gradient matching methods achieve advanced performance in dataset distillation. Specifically, gradient matching methods introduce distance metric (-.-)) measure the distance between gradients. A widely-used distance metric [53] is defined D(X,Y, Y) =
ΣL (1 [XI
where X,Y E IR and X; Yi E R are the hth columns of and Y respectively. With the defined distance metric ((,,)) gradient matching methods consider solving
(X,Y))
M
S= arg min E sca Po-Poo ip=i
C(S, CmS) m==1
Σ
where ((S,0) = D(VoLs(fo), VelT(fo)). (3)
where 8, is the intermediate weights which is continuously updated by training the network fe, over the target dataset T. The methods employ M as the hyperparameter to control the length of teacher trajectories be matched starting from the initialized weights On Pe.. C(S.0) is the matching loss. The teacher trajectory {0o, f..... 8M] is equivalent to series of gradients {gi 92, ... 9M J. To ensure the robustness the synthetic dataset S to different weights initializations, 0 will sampled from Po. for many times. As consequence, the distributions of the gradients for training can be represented as {P,) Paa:"" Pau
Algorithm Training with Megha in Distillation Phase.
Input: Target dataset T; Number of subsets K; Iterations N in updating each subset; A base distillation method A.
1: Initialize the synthetic dataset San1
2: Divide Sa11 into К subsets of equal size | i.e., San = Si US U...UU
3: for each Sy do
4: Optimize each subset Sy sequentially:
5: repeat
6:
kl then
7:
Initialize network weights Og ~ Pao
8:
else
9:
Load network weights Of Poy. saved in optimizing last subset Sk-
10:
for i = 1 to N do
b Update Network weights by subset Sy:
0f alg(S& US(-1), 0f
13:
> Update Sn by the base distillation method:
14:
Sk A(T, 0+)
15:
Record and save updated network weights -
16:
until Converge
Output:
Distilled synthetic dataset San1
Method
Increasing the size of a synthetic dataset is a straightforward approach to incorporating additional high-level features. However, our findings reveal that simply optimizing more synthetic data leads to an excessive focus on knowledge learned from easy instances. In this section, we first introduce the concept of sequential knowledge acquisition a standard training procedure (refer to subsection 4.1). Subsequently, we argue that the varying rate of convergence causes certain portions of the synthetic data to abandon the extraction of further knowledge in the later stages (as discussed Figure 4.2). Finally, present proposed strategy, Sequential Subset Matching (referred to SeqMatch), which is outlined Algorithm subsection 4.3.
4.1 Features Are Represented Sequentially
Many studies have observed the sequential acquisition of knowledge in training DNNs. Zeiler et al. [49] revealed that DNNs are optimized to extract low-level visual features, such as edges and textures, in the lower layers, while higher-level features, such as object parts and shapes, were

---

represented in the higher layers. Han et al. [16] leverages the observation that DNNs learn the knowledge from easy instances first, and gradually adapt to hard instances[1] propose noisy learning methods. The sequential acquisition of knowledge is a critical aspect of DNN.
However, effectively condensing knowledge throughout the entire training process presents significant challenges for existing dataset distillation methods. While the synthetic dataset S is employed to learn from extensive teacher trajectories, extending the length of these trajectories during distillation can exacerbate the issue of domain shifting in gradient distributions, thereby resulting in performance degradation. This is primarily due to the fact that the knowledge extracted from the target dataset T varies across different epochs, leading to corresponding shifts in the domains of gradient distributions. Consequently, the synthetic dataset S may struggle to adequately capture and consolidate knowledge from prolonged teacher trajectories.
To enhance distillation performance, common approach is to match a shorter teacher trajectory while disregarding knowledge extracted from the latter epochs of T. For instance, in the case of the CIFAR-I0 dataset, optimal hyperparameters for M (measured in epochs) in the MTT [5] method were found to be 2, 20, 40 for ipc = 1, 10, 50 settings, respectively. The compromise made in matching shorter teacher trajectory unexpectedly resulted in performance gain, thereby confirming the presence of excessive condensation on easy instances.
Taking into account the sequential acquisition of knowledge during deep neural network (DNN) training is crucial for improving the generalization ability of synthetic data. Involving more synthetic data is the most straightforward approach condense additional knowledge from longer teacher trajectory. However, our experimental findings, as illustrated in Figure 1, indicate that current gradient matching methods tend to prioritize the consolidation of knowledge derived from easy instances in the early epochs. Consequently, we conducted further investigations into the excessive emphasis on low-level features in existing dataset distillation methods.
4.2 Coupled Synthetic Dataset
The coupling issue within the synthetic dataset impedes its effectiveness in condensing addi- tional high-level features. Existing dataset dis- tillation methods optimize the synthetic dataset S as unified entity, resulting in the backprop- agated gradients used to update S being applied globally. The gradients on each instance only differ across different initializations and pre- assigned labels, implying that instances sharing a similar initialization within the same class will converge similarly. Consequently, portion of the synthetic data only serves the purpose of alleviating the gradient matching error for the pre-existing synthetic data.
Distillation discrepancy between S and S
10.0%
MTT
Acc (S*) (Sc
8.0%
SeqMatch
6.0%
4.0%
2.0%
0.0%
-2.0%
0.000
0.005
0.010
0.015 E [[(
0.020
0.025
E [[s fbh
.,5
Figure 2: The accuracy discrepancy between the net- works trained using S and S separately. The discrep- ancy will increase with the magnitude of R(st, fe.. These results verified the coupling issue between S and S and our proposed method SeqMatch success- fully mitigates the coupling issue. More experimental details can he found in subsection 5.3.
Consider synthetic dataset S is newly initial- ized to be distilled from a target dataset T. The distributions of the gradients for distillation are (Pa, Poz.".. ... )ax)), and the sampled gradi- ents for training (9), '2.... gMv) ). Suppose that G is the integrated gradients calculated by S. by minimizing the loss function as stated in Equation 3, the gradients used for updating s; when 0 = O. would be
0L
8G
2V om e(Jom, ((Jo,((s)))))
Vs, ((8,
0G
No (fom (s:),
Os;
0L
where fo.) 2 2V ((So (81). yi)
R(s,, femd)
(4)
0G
Os;
5

---

we have 8V (ifmm (Bi). = 1, because G' is accumulated by the gradients of each synthetic data, i.e. G= Ven Ls (fem)
DG
El Ve.. (fe (st). Here we define the amplification function R[si, fom. C Rd. Then, the gradients on updating synthetic instance Vs, 0m) shares the same #C aG and only varies in R(ss: Jo..). The amplification function R(s;, So..) is only affected by the pre-assigned label and initialization of s,.
More importantly, the magnitude of R(st, fe...) determines the rate of convergence of cach synthetic instance s,. Sorted by the I noom of amplification function [R(s, fo... )ll can be divided into two subsets S and S S contains the synthetic instances with greater values of R(ss, so... than those S That implies that instances in S converge faster to minimize D(Vo... Ls+ (fo.,); Im). and S is optimized to imitate 9m. On the other hand, the instances in S converge slower and optimized to minimize D(Vam. Ls- [. e). where e represents the gradients matching error of S i.e., 6 9m Ve,, S,.. L (Jo..) Therefore, S is optimized to imitate and its effectiveness is achieved by compensating for the gradients matching error of S+ S is coupled with S and unable to capture the higher-level features in the latter epochs.
We conducted experiments to investigate whether S solely compensates for the gradient matching error of S and is unable to extract knowledge independently. To achieve this, we sorted S and S- by the nn-rrrm of the amplification function [ss,, fo... )11 and trained separate networks with S+ and S As depicted in Figure 2, we observed significant discrepancy in accuracy, which increased with the difference in magnitude of R(s....... Further details and discussion are provided in subsection 5.3. These experiments verify the coupling issue wherein S compensates for the matching error of St, thereby reducing its effectiveness in condensing additional knowledge.
4.3 Sequential Subset Matching
We can use standard deep learning task as an analogy for the dataset distillation problem, then the synthetic dataset S can be thought of as the weight parameters that need to be optimized. However, simply increasing the size of the synthetic dataset is comparable to multiplying the parameters of a model in an exact layer without architecting the newly added parameters, and the resulting performance improvement is marginal. We thereby propose SeqMatch to reorganize the synthetic dataset S to utilize the newly added synthetic data.
We incorporate additional variability into the optimization process of synthetic data encourage the capture of higher-level feature extracted in the latter training progress. To do this, SeqMatch divides the synthetic dataset S into K subsets equally, i..., S = S US2 U.... USK. Sul | Seq optimizes each Sg by solving
lon
3x
arg min
Σ m=(k-1)n
L(S US(K-1),0m))
(5)
S. S Go- 18 [
where g(k-1) S US U.... S which represents the union set of the subsets in the former. g(k-1) are fixed and only Sg will be updated. The subset S₁ is encouraged to match the corresponding t segment of the teacher trajectory condense the knowledge in the latter epoch. Let n. = K denote the length of trajectory segment to be matched by each subset Se in the proposed framework. To strike balance between providing adequate capacity for distillation and avoiding coupled synthetic data, the size of each subset S₁ is well-controleed by K.
M
In the distillation phase, each subset is arranged in ascending order to be optimized sequentially. We reveal that the first subset S₁ with I K size of the original synthetic dataset S is sufficient to condense adequate knowledge in the former epoch. For the subsequent subset Sy. we encourage the tthh subset Sş condense the knowledge different from those condensed in the previous subsets. This is achieved by minimizing the matching loss L(S, (S(k-1), ,,,) while only S₁ will be updated.
During the evaluation phase, the subsets of the synthetic dataset are used sequentially to train the neural network fo. with the weight parameters 0 being iteratively updated by 0₂ = alg(Se, 0a-1). This training process emulates the sequential feature extraction of real dataset T during training. Further details regarding SeqMatch and the optimization of 0" can be found in Algorithm.

---

Experiment
In this section, we provide implementation details for our proposed method, along with instructions for reproducibility. We compare the performance of SeqMatch against state-of-the-art dataset distillation methods on a variety datasets. To ensure a fair and comprehensive comparison, we follow up the experimental setup as stated in [8, 30j. We provide more experiments to verify the effectiveness of SeqMatch including the results on ImageNet subsets and analysis experiments in Appendix A. due to page constraints.
5.1 Experimental Setup
We evaluate the performance of dataset distillation methods on several widely-used datasets across various resolutions. MNIST [28], which is fundamental classification dataset, is included with resolution of 28 x 28. SVNH [36] is also considered. which is composed of RGB images of house numbers ewith a resolution of 32 32. CIFARIO and CIFAR100 [25]. two datasets frequently used in dataset distillation, are evaluated in this study. These datasets consist of 50, 000 training images and 10, 000 test images from 10 and 100 different categories, respectively. Additionally, our proposed method is evaluated on the Tiny ImageNet [27] dataset with resolution of 64 x 64 and on the ImageNet [24] subsets with a resolution of 128 x 128.
Datasets:
The evaluation metric involves distillation phase and evaluation phase. In the former, the synthetic dataset is optimized with distillation budget that typically restricts the number of images per class (ipc). We evaluate the performance of our method and baseline methods under the settings ipc = {10, 50). We do not evaluate the setting with ipc since our approach requires ipc 2 2. To facilitate clear comparison, we mark the factorization- based baselines with an asterisk (*) since they often employ an additional decoder, following the suggestion in [30]. We employ 4aayer ConvNet [14] in Tiny ImageNet dataset whereas for the other datasets we use 3-layer ConvNet [14].
Evaluation Metric:
In the evaluation phase, we utilize the optimized synthetic dataset to train neural networks using a standard training procedure. Specifically, we use each synthetic dataset train five networks with random initializations for 1, 000 iterations and report the mean accuracy and its standard deviation of the results.
Implementation Details. To ensure the reproducibility of SeqMatch, we provide detailed implemen- tation specifications. Our method relies single hyperparameter, denoted by K, which determines the number of subsets. In order to balance the inclusion of sufficient knowledge in each segment with the capture of high-level features in the later stages, set K (2 for the scenarios where ipc = {10, 50), respectively. Notably, our evaluation results demonstrate that the choice of K remains consistent across the various datasets.
As plug-in strategy, SeqMatch requires a backbone method for dataset synthesis. Each synthetic subset is optimized using standard training procedure, specific to the chosen backbone method. The only hyperparameters that require adjustment in the backbone method are those that control the segments of the teacher trajectory to be learned by the synthetic dataset, whereas the remaining hyperparameters emain consistent without adjustment. Such adjustment is to ensure each synthetic subset effectively condenses the knowledge into stages. The precise hyperparameters of the backbone methods are presented in Appendix A.3. We conduct our experiments on the server with four Tesla V100 GPUs.
5.2 Results
Our proposed SeqMatch is plugged into the methods MTT [5] and IDC [23]. which are denoted as SeqMatch-MTT and SeqMatch-IDC, respectively. As shown in Table 1, the classification accuracies of ConvNet [14] trained using each dataset distillation method are summarized. The results indicate that SeqMatch significantly outperforms the backbone method across various datasets, and even surpasses state-of-the-art baseline methods in different settings. Our method is demonstrated to out- perform the state-of-the-a -tt baseline methods in different settings among different datasets. Notably, MeqMatch achieves a greater performance improvement in scenarios with a high compression ratio (i.e., ipc = 50). For instance, we observe 3.5% boost in the performance of MTT [5], achieving 51.2% accuracy on CIFAR-1OM Similarly, we observe 1.9% performance enhancement in IDC [23].
7

---

Table 1: Performance comparison of dataset distillation methods across variety of datasets. Abbreviations of
GM. TM, DM,META stand for gradient matching, trajectory matching, distribution matching, and meta-learning respectively. We reproduce the results of MTT [S] and IDC [23] and cite the results of the other baselines [30]. The best results of non con-factorized methods (without decoders) are highlighted in orange font. The best results of
factorization-bazt based methods highlighted in blue font.
MNIST
SVHN
CIFAR-10
CIFAR-100
Tiny ImageNet
Methods
Schemes
10
50
DD [47]
META
79.5
36.8
[C]
GM
94.7
98.8
76.1
82.3
44.9
53.9
32.3
42.8
#0.2
+002
DSA [51]
GM
97.8
99.2
79.2
84.4
52.1
60.6
32.3
42.8
10.1
1.011
1.0.5
10.44
0..5
1.0.5
1.0.3
1.0.4
DM [52]
DM
97.3
94.8
48.9
63.0
29.7
43.6
12.9
CAFE [46]
DM
97.5
98.9
77.9
82.3
50.9
62.3
31.5
42.9
10.1
+:0.2
+:0.6
2-0-4
KIP [37,38]
KRR
97.5
98.3
75.0
85.0
62.7
68.6
28.3
20.0
10.11
100
10.
20.8
2/9
3:0.1
FTD [13]
TM
66.6
73.8
43.4 4
50.7
24.5
MTT [S]
TM
97.3
98.5
79.9
87.7
65.3
71.6
40.1
47.7
23.2
10-1
1.0.1
1...3
1.0.
1.0.4
1.0.2
11.3
SeqMatch-MITT
TM
97.6
99.0
80.2
88.5
66.2
74.4
41.9
23.8
20.2
30.0
20.2
200
$:0.5
2:10 8
$9.8
IDC' [23] 3
GM
98.4
99.1
87.3
90.2
67.5
74.5
44.8
51.4
SeqMatch-IDC*
GM
98.6
99.2
87.2 2
92.1
68.3
75.3
45.1
51.9
10.11
-.000
1.0.2
1.0
200
1.0
20.3
20.3
RTP" [1]]
METА
99.3
99.4
89.1
89.5
71.2 .2
73.6
42.9
+0.8
+0.4
+0.2
+-0
+0004
+-0.4
+:0.7
HaBa" [33]
TM
83.2
88.3
69.9
74.0
40.2
47.0
99.6
95.4
84.8
56.2
37.6
Whole
20.0
29.11
250.35
20.4
achieving 92.1% accuracy SVNH, which approaches the 95.4% accuracy obtained using the real dataset. These results suggest that our method is effective in mitigating the adverse effects of coupling and effectively condenses high-level features in high compression ratio scenarios.
Cross-Architecturee Generalization: We also conducted experiments to evaluate cross-architectie generalization, as illustrated in Table 2. The ability generalize effectively across different archi- tectures is crucial for the practical application of dataset distillation. We evaluated our proposed SeqMatch the CIFAR-10 dataset with ipc = 50. Following the evaluation metric established in [13, 46], three additional neural network architectures were utilized for evaluation: ResNet [19], VGG [44], and AlexNet [26]. Our Match approach demonstrated significant improvement in performance during cross-architecture evaluation, highlighting its superior generalization ability.
5.3 Discussions
Table 2: Pross-Architecture Results trained with Con-
vNet on CIFAR-IO 10 with ipc = 50. We cite the results
Sequential Knowledge Acquisition: We con- ducted experiments on CIFAR-10 with ipc
reported in Du et al. [13].
50, presented in Figure 1, to investigate the in-
Evaluation Model
ability of existing baseline methods to capture the knowledge learned in the latter epoch, as
Method
ConvNet
ResNer18
VGG11
AlexNet
DC [53]
53.9
20.8
38.8
28.7
discussed in subsection 4.1. Inspired by [16],
we utilized the change in instance-wise loss on
CAFE [46]
55.5
+0.4
25.3
$2.3
40.5
$0.8
31.0
$0.5
real dataset to measure the effectiveness of con-
MIT [5]
71.6
61.9
48.2
10.2
13.7
2:0
11.0
densing high-level features. Specifically, we
FTD [13]
73.8
65.7
58.4
53.8
recorded the loss of each instance from the real dataset T at every epoch, where the network was trained with synthetic dataset for only 20
SeqMatch-MTT
74.4
68.4
64.2
50.7
20.5
20.7
$1.0
SoqMatch-IDC
75.3
69.7
73.4
72.0
10.2
20.0
10.11
10.2
Although IDC [23] is not categorized as factorization based method, employs data parameterization to better improve the performance of synthetic dataset. Therefore, we compare IDC to the factorization baased based method as factorization be treated a kind of special data parameterization.
8

---

iterations in each epoch. To distinguish hard instances from easy ones, we employed k-means algorithm [18] to cluster all instances the real dataset into two clusters based on the recorded instance-wise loss. The distribution of instances in terms of difficulty is as follows: 77% are considered easy instances, while 23% are classified as hard instances.
We evaluated MTT[5] and SeqMatch as mentioned above. Our results show that MTT[5] over- condenses the knowledge learned in the former epoch. In contrast, SeqMatch able to successfully capture the knowledge learned in the latter epoch.
Coupled Synthetic Subsets: In order validate hypothesis that the synthetic subset S is ineffective at condensing knowledge independently and results in over-condensation on the knowledge learned in the former epoch, subsets S+ and S of the same size by the 4ı-nrrm of the amplification function R[ss, Jo,,,) ),. as explained in Figure 4.2. We then recorded the accuracy discrepancies between the separate networks trained by S+ and S with respect to the mean 1ı-norm difference, i.e., Es, [[(s, So, [1] E- [[ f1.)
conducted experiments as shown in Figure 2. We sorted the
As shown in Figure 2, the accuracy discrepancies increased linearly with the -norm difference, which verifies our hypothesis that S is coupled with S and this coupling leads to the excessive condensation on low-level features. However, our proposed method, SeqMatch, is able to alleviate the coupling issue by encouraging S to condense knowledge more efficiently.
Baseline
1dt Subset
nd Subset
3rd Subset
Figure 3: Visualization example of "car" synthetic images distilled by MTT [5] and SeqMatch from 32 32 CIFAR-IO (ipc = 50).
Synthetic Image Visualization: In order to demonstrate the distinction between MTT [5] and Meqhatch, we visualized synthetic images within the "car" class from CIFAR-10 [25] and visually compared them. As depicted in Figure 3, the synthetic images produced by MTT exhibit more concrete features and closely resemble actual "car" images. Conversely, the synthetic images generated by SeqMatch in the 2nd and 3rd subsets possess more abstract attributes and contain complex car shapes. We provide more visualizations of the synthetic images in Appendix A.2.
5.4 Limitations and Future Work
We acknowledge the limitations our work from two perspectives. Firstly, our proposed sequential optimization of synthetic subsets increases the overall training time, potentially doubling or tripling it. To address this, future research could investigate optimization methods that allow for parallel optimization of each synthetic subset. Secondly, as the performance of subsequent synthetic subsets builds upon the performance of previous subsets, a strategy is required to adaptively distribute the distillation budget of each subset. Further research could explore strategies to address this limitation and effectively enhance the performance of dataset distillation, particularly in high compression ratio scenarios.

---

Conclusion
In this study, we provide empirical evidence of the failure in condensing high-level features dataset distillation attributed to the sequential acquisition of knowledge in training DNNs. We reveal that the static optimization of synthetic data leads bias over-condensing the low-level features, predominantly extracted from the majority during the initial stages of training. To address this issue targeted manner, we introduce an adaptive and plug-in distillation strategy called SeqMatch. Our proposed strategy involves the division of synthetic data into multiple subsets, which are sequentially optimized, thereby promoting the effective condensation of high-level features learned in the later epochs. Through comprchensive experimentation diverse datasets, validate the effectiveness of our analysis and proposed strategy, achieving state-of-the performance.
Acknowledgements
This work support by Joey Tianyi Zhou's A*STAR SERC Central Research Fund (Use-inspired Basic Research) and the Singapore Government's Research, Innovation and Enterprise 2020 Plan (Advanced Manufacturing and Engineering domain) under Grant A18AIb0044
10

---

References
[1] Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning,
pages 233-242. PMLR, 2017. 5
[2] Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv:/703.066 06476, 2017.
[3] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint 2XXi:2000608857 2020. 3
[4] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu, Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pages 4750-4759, 2022. 1, 5,
14, 15
[6] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3739-3748, 2023. 1
[7] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kemel herding, arXiv preprint arXiv: 2012. 3
[8] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. De-bench: Dataset condensation bench- mark. arXiv preprint arXiv:2207.09633 2022. 2,
[9] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- 1k with constant memory. arXiv preprint arXiv:2211. 1.1266, 2022. 3
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image datahase. In 2009 IEEE conference computer vision and pattern 14
recognition, pages 248-255. Ieee, 2009.
[11] Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint 22X.::2006002166,, 2022. 2, 8
[12] Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? arXiv preprint arXiv:2206.0022 2022.
[13] Jiawei Du, Yidi Jiang, Vincent TF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. arXiv preprint arXiv:22 1004, 2022. 1, 2, 3, 8, 14
[14] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4367-4375, 2018. 7, 14
[15] Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint arXiv:2008.0445. 2020.
[16] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu. Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels.
Advances in neural information processing systems, 31, 2018. 5, 8
[17] Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. In Proceedings of the twenty-first annual symposium on Computational geometry, pages 126-134, 2005. 3
11

---

[18] John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series с (applied statistics), 28(1):100-108, 1979. 9
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016. 8
[20] Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Pan. Delving into effective gradient matching for dataset condensation. arXiv preprini urXiv:2208.0031, 2022. 3
[21] Haifeng Jin, Qingquan Song, and Xia Hu, Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1956,1956, 2019.
[22] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteri- zation. International Conference on Machine Learning, pages 11102-11118. PMLR, 2022. 1,3
[23] Hang--yun&o Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteri- zation. arXiv preprint arXiv:2205. 14959, 2022. 7,8
[24] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491-507. Springer, 2020. 7
[25] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 and CIFAR-100 datasets. URI: htlps://www. cs. toronto. edu/kriz/cffar himl, 6(1):1, 2009. 2, 7, 14
[26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[27] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 7,
14
[28] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-23) 1998. 7
[29] Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208. 10494, 2022. 1,3
[30] Shiye Lei and Dacheng Tao. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301.05603 2023. 3, 7,8
[31] Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Soft-label anonymous gastric x-ray image distillation. In 2020 IEEE International Conference on Image Processing (ICIP), pages 305-309, IEEE, 2020.
[32] Ping Liu, Xin Yu, and Joey Tianyi Zhou. Meta knowledge condensation for federated learning. arXiv preprint arXiv: 14851, 2022.
[33] Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization, arXiv preprint w.Xiv:2210.167.. 2022. 2, 3, 8, 14
[34] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint arXiv:2210. 12067, 2022. 3
[35] Noel Loo, Ramin Hasani, Mathias Lechner, and Danicla Rus. Dataset distillation with convexi- fied implicit gradients. arXiv preprint 25X.02230..06755 2023. 1
[36] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011. 7
12

---

[37] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression arXiv preprint arXiv:2011.00050, 2020. 2, 3,
[38] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks, Advances in Neural Information Processing Systems, 115188--519, 2021. 8,
[39] Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference machine learning, pages 4095- 4104. PMLR, 2018. 1
[40] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1-34, 2021.
[41] Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bac- ciu. Distilled replay: Overcoming forgetting through synthetic samples. arXiv preprint
arXiv:2103.15851, 2021. 1
[42] Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint
arXiv:2331.04272, 2023. 1
[43] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint
2017. 3
[44] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014. 8
[45] Ivor W Tsang, James T Kwok, Pak-Ming Cheung, and Nello Cristianini. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research, 6(4), 2005.
[46] Kai Wang. Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang. Shuo Wang. Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22196-12209 2022. 8
[47] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 10959, 2018. 2, 3, 8
[48] Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review.
arXiv preprint arXiv:2301.000.1 00014,, 2023. 1
[49] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 818-833. Springer, 2014. 2,
[50] Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11950-11959, 2023.
[51] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685. PMLR, 2021. 1, 8
[52] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision pages 6514-6523, 2023. 1. 8
[53] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 1(2):3, 2021. 1, 2, 3, 4, 8
[54] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719, 2022. 3
13

---

A More Experiements
A.1 ImageNet Subsets
To assess the efficacy of our approach, we conducted experiments on subsets of the ImageNet dataset [10]. These subsets were constructed by selecting ten pertinent categories from the ImageNet- Ik dataset [10], with resolution of 128 x 128. Consequently, the ImageNet subsets pose greater challenges compared to the CIFAR-10/100 [25] and Tiny ImageNet [27] datasets. We adhered the configuration of the ImageNet subsets suggested by previous studies [5, 13, 33], encompassing subsets such as ImageNette (diverse objects), ImageWoof (dog breeds), ImageFruits (various fruits), and ImageMeow (cats).
To synthesize the dataset, we employed 5-layer ConvNet [14] with a parameter setting ipc = 10. The evaluation of the synthetic dataset involved performing five trials with randomly initialized networks. We compared the outcomes of our proposed method, referred to as SeqMatch, with the baseline approach MTT [5], well as the plug-in strategies FTD [13] and HaBa [33] which build upon MTT.
The comprehensive results are presented in Table 3. Our proposed SeqMatch consistently outper- formed the baseline MTT across all subsets. Notably, we achieved performance improvement of 4.3% on the ImageFruit subset. Additionally, SeqMatch demonstrated superior performance compared to HaBa and achieved comparable results FTD.
Table 3:
The performance comparison trained with 5-layer ConvNet on the ImageNet subsets with resolution of 128 128. We cite the results as reported in MTT [5], FTD [13] and HaBa [33]. The latter two methods, FTD and HaBa, are plug-in strategies that build upon the foundation of MTT. Our proposed approach, SeqMatch, exhibits superior performance compared both MTT and HaBa, demonstrating significant improvement in results.
ImageNette
Image Woof
ImageFruit
ImageMeow
87.4
67.0
63.9
66.7
Real dataset
#1.0
#1.3
=2.0
#1.1
63.0
35.8
40.3
40.4
MTT [5]
+1.8
#1.8
#1.8
+2.2
67.7
38.8
44.9
43.3
FTD [13]
+-1.3
#1.4
#1.8
10.6
64.7
38.6
42.5
42.9
HaBa* [33]
1:1.6
$1.2
21.6
20.9
66.9
38.7
44.6
44.8
SeqMatch-MTT
+1.7
+1.1
+1.7
+1.2
70.6
41.1
46.5
45.4
MeqMatch-FTD
#1.5
41.2
41.2
A.2 More Visualizations
Instance-wise loss change: We have presented the average loss change of easy and hard instances in Figure 1, revealing that MTT failed to effectively condense the knowledge learned from the hard instances. To avoid the bias introduced by averaging, we have meticulously recorded and visualized the precise loss change of each individual instance. This is accomplished by employing a heatmap representation, as demonstrated in Figure 6. Each instance is depicted as a horizontal line exhibiting varying colors, with deeper shades of blue indicating higher loss values. Following the same clustering approach depicted in Figure 1, proceed to visualize the hard instances the top and the easy instances at the bottom.
The individual loss changes of MTT, depicted in Figure 6, remain static across epochs. The losses of easy instances decrease to a small value during the initial stages, while the losses of hard instances persist at high value until the end of training. These results confirm that MTT excessively focuses on low-level features. In contrast, the visualization SeqMatch clearly exhibits the effect of color gradient, indicating a decrease in loss for the majority of instances. Notably, the losses of hard instances experience a significant decrease when a subsequent subset is introduced. These results validate that SeqMatch effectively consolidates knowledge in sequential manner.
14

---

MTT
MetMatch
Instance loss from target dataset
Instance-wise Loss
0.2
15
40
10
15
20
25
35
40
0.0
Epoch
Epoch
Figure The heatmap illustrates the loss change of each instance in real dataset across epochs. row the heatmap represents an instance, while the deeper blue color denotes higher instance-wise loss. The network is trained with the synthetic datasets distilled by MTT and SeqMatch. Left: MTT fails to reduce the loss of hard instances while excessively reducing the loss of easy instances. Right: SeqMatch minimizes the loss of both the hard easy instances.
IPC2 MTT
Subset
Subset
3rd Subset
Visualization example synthetic images distilled by MTT [5] SeqMatch from 32 CIFAR 10 (ipc = (2,3)). McqMMacchip==2)) is seamlessly embedded as first two rows within the hatcSeqMatch(iper visualization.
Figure
Synthetic Dataset Visualization: We compare the synthetic images with ipc = {2, 3} from the CIFARIO dataset highlight the differences between subsets of Seqmatch in Figure We provide more visualizations of the synthetic datasets for ipc 10 from the 128 128 resolution ImageNet dataset: Image Woof subset Figure 7 and ImageMeow subset Figure In addition, parts of the visualizations of synthetic images from the 32 x 32 resolution CIFAR- 100 dataset are showed Figure 6. We observe that the synthetic images generated by Seq in the subsequent subset. contains more abstract features than the previous subset.
A.3 Hyperparameter Details
The hyperparameters K SeqMatch-MTT set with {2,3) 3} for the settings ipc = (10, 50), respectively. The optimal value of hyperparameter K is obtained via grid searches within the set (2,3, 4, 5, 6) in a validation set within the CIFAR- dataset. We find that the subset with small size will fail to condense the adequate knowledge from the corresponding segment of teacher trajectories,
15

---

Figure 6: Visualization the first 10 classes of synthetic images distilled by SeqMatch from x CIFAR- 100 (ipc 10). The initial image rows and the final 5 image rows match the first and second subsets, respectively.
resulting in performance degradation in the subsequent subsets. For the rest of the hyperparamters, report them in Table 4.
'ImageFruit has different setting of Max Start Epoch from other ImageNet subesets: (10,10)
16

---

Figure 7: Visualization of the synthetic images distilled by SeqMatch from 32 x 32 Image Woof (ipc = 10).
The initial 5 image rows and the final 5 image rows match the first and second subsets, respectively.
Table Hyperparameter values we used for MeqMatch-MTT in the main result table. Most of the hyperparameters "Max Start Epoch" and "Synthetic Step" are various with the subsets, we use sequential numbers to denote the parameters used in the corresponding subsets. "Img." denotes the abbreviation of ImageNet.
CIFAR-10
CIFAR-100
Tiny Img.
Img. Subsets
ipc
10
50
50
10
10
2
3
2
2
Max Start Epoch
(20,10)
(20.20, 10)
(20,40)
(40,20.20)
(20,10)
(10,5)*
Synthetic Step
(30,80)
30
30
80
20
20
Expert Epoch
[2,3}
2
2
2
Synthetic Batch Size
125
100
20
Learning Rate (Pixels)
100
100
1000
1000
10000
100000
Learning Rate (Step Size)
le-5
le-5
le-5
1e-5
1e-4
le-6
Learning Rate (Teacher)
0.001
0.01
0.01
0.01
0.01
0.01
17

---

Figure 8: Visualization of the synthetic images distilled by SeqMatch from 32 x 32 ImageMeow (ipc 10).
The initial 5 image rows and the final 5 image rows match the first and second subsets, respectively.
18


Reviews:
[{"review_id": "w1wLd9EfjO", "reviewer": "Reviewer_u5kd", "summary": "This paper investigates an issue with dataset distillation, where synthesized datasets tend to overly condense low-level features but fail to efficiently incorporate high-level ones. The authors argue that this is due to existing methods treating the synthetic dataset as a unified entity and equally optimizing each instance, leading to a coupling issue as the size of the synthetic dataset increases. To address this problem, they propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch). SeqMatch divides the synthetic dataset into multiple subsets and optimizes them in sequence, mimicking the learning process from low-level to high-level features. This approach allows each subset of the synthetic dataset to progressively capture more complex, high-level features, reducing the coupling issue and enhancing overall performance.", "questions": "1. How do you account for the randomness in the initialization of the synthetic dataset, and how does it affect the rate of convergence of each synthetic instance?\n2. The authors mention that instances sharing a similar initialization within the same class will converge similarly. Could this lead to any form of bias in your findings?\n3. The authors identified that current gradient matching methods prioritize easy instances during early epochs. What is an effective mechanism for identifying or quantifying the 'difficult' instances to create a more balanced emphasis?", "limitations": "In section 5.4, the authors discussed two limitations of their work. Their openness in acknowledging these limitations adds to the credibility of their research and provides useful guidance for follow-up studies in this field.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is well-written, well-motivated, and well-organized. The authors provide comprehensive experiments on various datasets such as CIFAR-10, CIFAR-100, TinyImageNet, and subsets of the ImageNet. They provide insightful analysis of the experimental results, discussing the impact of different factors. The authors provide a detailed algorithm description that translates their theoretical insights into practical application.", "weaknesses": "1. It's not clear how generalizable these results are to other tasks and datasets. Their experiments are based on specific DNN architectures and datasets, and the proposed method's effectiveness might vary under different conditions.\n2. Since the method divides the synthetic dataset into subsets and optimizes them sequentially, there might be a risk of overfitting, especially when the number of subsets is high or when subsets are small.\n3. The method proposed requires a series of sequential optimization processes, which could potentially increase the computational cost and time required for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** It's not clear how generalizable these results are to other tasks and datasets.\n\n**A1:** Our chosen experimental framework adheres rigorously to the configurations delineated by a suite of established dataset distillation benchmarks, specifically MTT[5], DM[46], CAFE[42], KIP[34,45], IDC[21], FTD[11], and Haba[31]. This standardized experimental setup encompasses datasets, network architectures, and evaluation metrics, thereby ensuring an equitable and unbiased comparison among various dataset distillation methodologies.\n\nFurthermore, we extended our experimental endeavors to encompass additional cross-architecture generalization assessments, as presented in Table 2. These results verified the **superior cross-architecture generalization** inherent to our proposed SeqMatch method. It's important to highlight that conducting experiments under outlier dataset or architecture settings would yield outcomes devoid of substantive relevance or meaningful insights.\n\n**Q2:** There might be a risk of overfitting.\n\n**A2:** The size of each subset bears an impact on the efficacy of SeqMatch. Our parameter study on the number of subsets ($K$) reveals a significant correlation: subsets with insufficient instances inevitably falter in their ability to encapsulate essential features inherent to the original dataset. Our parameter study on SeqMatch-MTT in CIFAR10 dataset is reported as below,\n\n| K |1|2|3|4|5| \n|:----------- | :-----------: |:-----------: |:-----------:|:-----------: |:-----------:|\n|ipc=10|65.3|66.2|65.6|65.0|63.5|\n|ipc=50|71.6|73.2|74.4|74.1|74.3|\n\nThe outcomes distinctly indicate a decline in performance as K escalates from 2 to 5 when ipc=10. A plausible explanation for this trend is rooted in the inherent challenge faced by subsets characterized by an insufficient number of images (ipc < 5) to effectively encapsulate comprehensive knowledge from the target dataset. An evidense to support this explaintion is that the subset with ipc >=10 will no longer exhibit a decline in performance (results of ipc=50).  \n\n**Q3:** which could potentially increase the computational cost and time\n\n**A3:** We acknowledge the escalation in computational demands and have addressed this limitation in lines 360-367 of our paper. However, it's important to note that the augment in computations is not directly proportional to the number of subsets. This is attributed to the reduction in training iterations for each synthetic subset, wherein only a segment of teacher trajectories is condensed.\n\nOf greater significance, the computational expenditure does not constitute the principal concern within the dataset distillation endeavor. Highlighting this, the most recent survey paper on dataset distillation [49] underscores the discernible gap in performance between synthetic and original datasets, alongside the substantial memory overhead inherent to dataset distillation, thereby impeding its broader adoption in real-world scenarios. In addition to augmenting performance, our proposed SeqMatch offers an ancillary advantage – a reduction in memory expenses associated with dataset distillation. We report the memory usage of MTT and SeqMatch-MTT on CIFAR10 as below, \n| IPC |10|50| \n|:----------- | :-----------: |:-----------: \n|MTT|10,108 MiB (100%)|34,540 MiB (100%)|\n|SeqMatch-MTT|7,164 MiB (70.8%)|15,252 MiB (44.2%)|\n\nThe memory used by SeqMatch is **reduced** significantly in particular in the settings with huge memory usage (ipc=50). \n\n\n\n**Q4:** How do you account for the randomness in the initialization of the synthetic dataset, and how does it affect the rate of convergence of each synthetic instance?\n\n**A4:** \nAs **emphasized** within lines 196-202, it is evident that discrepancies in initialization and pre-assigned labels engender divergence in the convergence patterns of synthetic data. To address this, we introduce a novel metric termed the **\"amplification function,\"** as presented in line 212, to quantitatively measure these disparities. Our empirical investigations, demonstrated in Figure 2, illustrate that such discrepancies give rise to a coupling issue that obstructs the effective condensation of features from instances within the $\\mathcal{S}^-$ subset (comprising instances with small amplification values). This investigation and the associated formulation is one of the **major contribution** of our work.\n\nThe existing dataset distillation methods such as MTT[5], IDC[21], and CAFE[42] tend to clone a random instance from the original dataset as the initialization of the synthetic dataset, which aggravates the divergence in the convergence patterns of synthetic data.\n\n**Q5:** The authors mention that instances sharing a similar initialization within the same class will converge similarly. \n\n**A5:** We find the logic behind this inquiry somewhat perplexing. Is the reviewer implying that our claim might not be entirely accurate and could introduce a bias into our findings? In response, we assert that this is not the case. Our claim doesn't serve as either an assumption or a theoretical foundation for our findings. Rather, it represents a reasoned explanation, inspired by references [1], [14], and [44], to interpret the experimental outcomes showcased in Figures 1 and 2 of our paper.\n\n**Q6:** What is an effective mechanism for identifying or quantifying the 'difficult' instances?\n\n**A6:** The establishment of an effective approach for identifying or quantifying instances deemed 'difficult' presents a prospective avenue for future exploration. Currently, our approach involves employing the average loss reduction observed during standard training as a metric to discern these 'difficult' instances, in alignment with the methodology expounded in [14]. An adapted mechanism within the context of dataset distillation, capable of quantifying and recalibrating the intrinsic \"hardness\" of instances, offers a promising direction for intensified and enhanced dataset distillation strategies.\n"}, {"review_id": "g8BFaq4Gfn", "reviewer": "Reviewer_Qidh", "summary": "This paper propose a new method called sequential subset matching (SeqMatch) for dataset distillation. The proposed method is designed to continuously generate synthetic images at different training (distillation) iteration. This strategy is inspired by the general mechanism of optimization, which captures characteristics (low-level feature) of easy instances in an early stage, but takes characteristics (i.e., high-level feature) from increasingly difficult instances. SeqMatch was applied to various dataset distillation methods and showed marginal but better performance than the method in which SeqMatch was applied or other baseline methods in various four datasets.", "questions": "1.  In \"Gradient Matching Methods\" in Section 3, are the gradients from g_1 to g_M the gradients from different layers of a network? Or gradients from another training iteration?", "limitations": "This paper adequately addressed the limitation and promised to solve it in future work.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "1. The analysis about the general mechanism of optimization is so insightful that it deserves a lot of attention in other studies.\n\n2. The plots of several figures provide good support for the arguments in this paper. e.g., Figures 1 and 2 well illustrate the effect of the general mechanism of optimization on dataset distillation and the coupling issue, respectively.", "weaknesses": "First of all, I don't understand the motivation behind designing some SeqMatch.\n\n1. How was the claim for the optimization mechanism that captures low-level features in the early stages and high-level features in the later stages verified? Figure 1 seems to have been used to verify this claim, but can hard instances and easy instances represent high-level and low-level features, respectively?\n\n2. I don't understand how the analysis of the general mechanism of optimization was used to design the SeqMatch method. In particular, what is the motivation for applying the SeqMatch method when training f_theta in the evaluation phase?\n\n\nIn addition, in Table 1, most of the performance improvements acquired via SeqMatch are very marginal (~0.2). Performance improvement is not an absolute determining factor in judging the superiority of the proposed method, but it can be used to verify that the method works as claimed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** How was the claim for the optimization mechanism that captures low-level features in the early stages and high-level features in the later stages verified?\n\n**A1:** The derivation of the claim has been explicitly presented in lines 161-166 of our paper. In support of this claim, it is crucial to highlight that relevant citations [1], [14], and [44] emphasize that Deep Neural Networks (DNNs) exhibit an optimization pattern where they initially prioritize learning from simpler instances and gradually adapt to more complex ones. It is noteworthy that the term \"low-level\" features, referring to features primarily extracted by the lower layers, is established in [44].\n\nOur contribution lies not merely in the exposition of this claim, but rather in the extension and validation of these findings within the specific context of dataset distillation. By delving into the task of dataset distillation, we broaden the scope of these existing claims, exploring their applicability and implications in a distinct domain.\n\nIn the context of dataset distillation, these designated training instances are repurposed as a validation set to assess the efficacy of knowledge encapsulated within the synthetic dataset. An effective synthetic dataset should manifest an equivalent loss reduction across both easy and hard instances, mirroring the behavior of a standard vanilla training set.\n\nHowever, as vividly depicted in Figure 1, the current distillation method falls short in achieving this equilibrium, resulting in disparate loss reductions for easy and hard instances. Conversely, our proposed method showcases demonstrable improvements in mitigating this discrepancy, effectively bridging the gap between the two instance categories.\n\n**Q2:** how the analysis of the general mechanism of optimization was used to design the SeqMatch method. In particular, what is the motivation for applying the SeqMatch method when training f_theta in the evaluation phase?\n\n**A2:** We have succinctly expounded upon the impetus driving the conception of the SeqMatch methodology within the delineated sections of our manuscript, specifically in lines 65-73 and lines 233-238.\n\nIn essence, our motivation is inspited by the findings that simply increasing the size of synthetic dataset bears an analogous semblance to the amplification of weights in a singular-layer perceptron configuration, in which only a limited performance improvement could be gained. Recognizing the necessity for a more discerning approach, we advocate for a paradigmatic transition from the unidimensional architecture of a single-layer perceptron to the multi-layered intricacies of a Multilayer Perceptron (MLP) and thus encourage the distilled data to be optimized in a sequential manner. Therefore, SeqMatch requires the distilled data to be learned in a sequential manner as well in evaluation phase. \n\n**Q3:** Performance improvements acquired via SeqMatch are very marginal (~0.2)\n\n**A3:** SeqMatch has acquired an **averaged** performance improvement of **1.28%** over the baseline MTT[5] and **0.58%** over the **SOTA** IDC[21]. In particular, SeqMatch improves the performance significantly in the setting of **ipc=50** with an improvement of **1.9%** over the baseline MTT[5] and and **0.93%** over the **SOTA** IDC[21]. Furthermore, our method does not only improve specific algorithms. Our sequential learning framework can be widely applied to existing data distillation models, enhancing their performance. We also hope the versatility of our approach can be taken into consideration as a contribution.\n\nIt would be **unjust** to characterize these performance enhancements as merely marginal, approaching the 0.2% range. \n\n**Q4:** In \"Gradient Matching Methods\" in Section 3, are the gradients from g_1 to g_M the gradients from different layers of a network? Or gradients from another training iteration?\n\n\n**A4:** This represents the trajectory of gradients from the initial state $\\theta_0$ to the converged weights $\\theta_M$\". Consequently, $g_1$ through $g_m$ denote the gradients targeted for optimization in each iteration of a conventional optimization process. \n\n\n---\n\n[1] Devansh Arpit, Stanisław Jastrz˛ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233–242. PMLR, 2017. 4\n\n[14] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018. 4, 8\n\n[44] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer, 2014. 2, 4\n"}, {"review_id": "Aml4OSVsbX", "reviewer": "Reviewer_QS3g", "summary": "This work proposes a change to existing dataset distillation methods by sequentially optimizing different subsets at a time. At each iteration, the existing subset is frozen and a new subset of data is *added* to it and optimized. This method allows different subsets of the synthetic data to capture different levels of features required by a network to learn during training time. This method boosts the state of the art for subsets with >1 IPC (since this method does not make sense with 1 IPC).", "questions": "I would like for the authors to make Algorithm 1 more clear as described above.\n\nIt would also be nice to see visuals when there is just 1 sample per subset (i.e., K=IPC) even with just IPC 2 or 3 since this case would likely have the largest differences between the subsets.\n\nLastly, I recognize his surname is quite long, but Figure 1 should probably cite \"Cazenavette et al.\" rather than \"George et al.\" :)", "limitations": "yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I like this paper a lot. The authors addressed an obvious problem with existing dataset algorithms (all samples capture the same level of features) in an elegant way that is adaptable to all existing and future backbone methods. The authors both empirically and theoretically show that jointly optimizing the entire set couples the gradients in a way that prevents the synthetic set from learning the necessary variety of features.\n\nThe visuals are all very nice, clearly illustrating the authors' points.", "weaknesses": "Algorithm 1 is a bit confusing to read. According to line 4, it seems that a single network initialization is used to optimize the entire subset (but this can't be true since it would catastrophically overfit). It is also unclear what the $n$ parameter is. As the algorithm is currently presented, I cannot see how MTT can be slotted into it.\n\nMaybe it would be clearer if an additional inner loop was included along with a generic distillation loss and doing away with the sum over $m$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments! We sincerely appreciate the time and effort you dedicated to reviewing our work and answer your questions as below.\n\n---\n**Q1:** Algorithm 1 is a bit confusing to read.\n\n**A1:** We have made revisions to our Algorithm 1, and have appended the revised version in the response PDF. Regarding your concern about clarity in the optimization process, we have now included an inner loop in Algorithm 1, which explicitly illustrates how each subset is optimized in SeqMatch. We hope that this addition clarifies the approach and enhances the understanding for readers. We have added a conditional statement (lines 6-9) in Algorithm 1 to provide a clear indication of this step.\n\nIn addition, we have revised the section on input parameters to better explain the notation used. Specifically, we have explicitly defined $N$ as the number of iterations in optimizing each subset. We have introduced a new input parameter, \"Base Distillation Method\" $\\mathcal{A}$, which indicates for the embedding of other distillation methods with SeqMatch. \n\n**Q2:**  visuals with ipc = 2 and 3.\n\n**A2:** We have conducted experiments for $K=2$ and $K=3$ subsets with $\\texttt{ipc}=1$ in each subsets by our proposed SeqMatch. The experiments is based on the MTT baseline in CIFAR10 dataset. The evaluation accuracies are listed as below,\n\n| IPC|2|3| \n|:----------- | :-----------: |:-----------: |\n|MTT|51.6|54.5|\n|SeqMatch|52.9|57.0|\n\nSeqMatch outperforms MTT with performance enhancemanents of $\\\\{1.3\\\\%,2.5\\\\%\\\\}$ under the settings ipc=$\\{2,3\\}$. For SeqMatch, we designated parameter values of max start epoch as ${2, 4, 6}$ while retaining the remaining parameters consistent with the ipc=$10$ configuration. It is noteworthy that SeqMatch(ipc=2) consists of the first two subsets of SeqMatch(ipc=3). As a result,  the visualization corresponding to SeqMatch(ipc=2) is seamlessly embedded as the first two rows within the SeqMatch(ipc=3) visualization. \n\nThe visualization is demonstrated as the **Figure 4** in the response PDF.  However, it is imperative to acknowledge that the disparity between MTT and SeqMatch-MTT is comparatively less pronounced than that illustrated in Figure 3 of our paper. This variance can be attributed to the smaller max start epoch configuration (${2, 4, 6}$) for SeqMatch(ipc=3), a parameter designed to ensure subsequent subsets contribute to enhancing the performance of the initial subset.\n\n\n**Q3:** improper citation. \n\n**A3:** We sincerely apologize for the errors in the citation of MTT[5]. MTT[5] is a crucial baseline method that has greatly influenced our current work, serving as a significant milestone in the dataset distillation task. We have made the necessary revisions in the citation, accurately referencing \"Cazenavette et al.\" in the revised version of our paper. We will keep it in mind in our following work to correctly cite the research papers. We are grateful for pointing out the overlooked mistakes of our submission.\n"}, {"review_id": "4uxzDyRzwj", "reviewer": "Reviewer_xCSw", "summary": "This paper proposes a novel dataset distillation method called SeqMatch which focuses on extracting high-level features from later training trajectories. The authors highlight a limitation in state-of-the-art data distillation methods, which tend to condense low-level information from easy data while overlooking the high-level information contained in hard data. In response to this issue, the paper introduces a novel optimization technique that generates multiple small sets of synthetic data. Each of these sets distills distinct knowledge from various stages of the training trajectories. By addressing the inherent problem observed in previous dataset distillation methods, the authors conduct experiments to showcase the efficacy of SeqMatch.", "questions": "How is the repeated evaluation conducted to calculate the error bar?\n\nThe motivation of the paper is that SOTA methods fail to distill high-level information from hard data. From my perspective, a more natural and straightforward solution would be to divide the original training dataset into different subsets based on data difficulty and then distill distinct datasets based on these subsets. I am wondering if the authors explored this approach or considered it as an alternative solution in their research.\n", "limitations": "Yes, the authors have discussed the limitations.\n\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of condensing different sub-datasets for different stages is novel and interesting.\n2.\tThe proposed method does not require extra computation cost.\n3.\tThe writing is good and easy to follow.\n", "weaknesses": "-\tThe paper lacks evaluation numbers for certain settings, such as SeqMatch-IDC, on datasets like CFAIR100 50IPC, Tiny-ImageNet, and ImageNet subset. (SeqMatch-IDC seems to be the most favorable setting) This omission makes it difficult to determine the performance of SeqMatch in comparison to other methods in these specific scenarios.\n\n-\tSeqMatch underperforms the baseline method FTD on Tiny-ImageNet and ImageNet subset, suggesting that SeqMatch may not scale well to larger datasets. This raises the question of why SeqMatch was chosen over FTD in these cases. \n\n-\tThe paper lacks an ablation study, which would provide valuable insights into the impact of the number of subsets, K, on the performance of SeqMatch. Including such an evaluation would enhance our understanding of how SeqMatch operates and how different parameter settings influence its performance. \n\n-\tThe caption of Table 1 is somewhat misleading. Although IDC is not categorized as a factorization-based method, it does employ data parameterization (factorization can be treated as a special data parameterization). Therefore, it would be more appropriate to compare IDC with RTP and HaBa rather than other methods distilling information into a single image.\n\n-\tMinor: There are citation inconsistencies in the appendix. For example, FTD is referred to as [12] in the appendix but as [11] in the main text. Additionally, the appendix lacks a reference section, making it difficult to trace the sources of the cited works accurately.\n\n- The reproductivity checklist is chosen as Yes, but no code is provided.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions. We answer your questions in order. \n\n**Q1:** The paper lacks evaluation numbers for certain settings.\n\n**A1:** This is attributed to the notably sluggish training speed observed in the IDC[21] framework. Our experimental setup aligns with the configurations outlined in IDC[21], and DREAM[51], all of which exclusively present results of IDC under ipc=10, CIFAR 100.\n\nUnlike other gradient-matching baselines, IDC[21] employs an additional class-wise loop asthe complex three-level nested loop structure. This nesting, results in slow training speed, which takes up tp 14 days on a single Nvidia V100 GPU (ipc=50 CIFAR 100).\n\nAddressing your suggestion, we have initiated the training of SeqMatch-IDC with ipc=50, CIFAR100, and we are committed to incorporating the updated results.\n\n**Q2:** underperforms the FTD on Tiny-ImageNet and ImageNet subset.\n\n**A2:**  As asserted in lines 77-78, our SeqMatch serves as a training strategy seamlessly integrable into widely used dataset distillation frameworks. The central thrust of our contribution lies in uncovering a general, yet often unnoticed, limitation within existing distillation methods, wherein each synthetic dataset tends to encapsulate homogeneous features. Concomitantly, we propose an innovative and effective technique to mitigate this particular challenge.\n\nFTD[11] outperforms our base distillation method MTT[5] significantly on Tiny-ImageNet and ImageNet subsets, thereby diminishing the performance improvement attributed to SeqMatch-MTT. In light of this observation, we have initiated experiments utilizing FTD[11] as our foundational distillation approach and are committed to incorporating the updated results into the forthcoming version of our paper.\n\n**Q3:** The paper lacks an ablation study.\n\n**A3:** We appeciate your feedback and have conducted the experiments on the ablation study of K on the number of subsets K on CIFAR10 with SeqMatch-MTT. The results are listed below, \n\n| K |1|2|3|4|5| \n|:----------- | :-----------: |:-----------: |:-----------:|:-----------: |:-----------:|\n|ipc=10|65.3|66.2|65.6|65.0|63.5|\n|ipc=50|71.6|73.2|74.4|74.1|74.3|\n\nAdditionally, we present the scatter plot, denoted as **Figure 5** in the provided response PDF, which illustrates the findings from the ablation study conducted on the parameter K. The outcomes distinctly indicate a decline in performance as K escalates from 2 to 5 when ipc=10. A plausible rationale for this phenomenon lies in the fact that subsets with insufficient images (ipc < 5) struggle to effectively distill comprehensive knowledge from the target dataset.\n\nConversely, the degradation in performance as K increases remains marginal when ipc=50. This can be attributed to the subset size surpassing the threshold required to adequately capture essential features (ipc > 10), thereby substantiating the observed trend.\n\n**Q4:** The caption of Table 1, citation inconsistencies, and the reproductivity checklist. \n\n**A4:** We extend our gratitude for the meticulous review and extend our apologies for the errors identified. Acknowledging the oversight, we concede that the baseline IDC[21] should indeed be classified as a factorization-based method. Consequently, we will diligently revise the presentation in Table 1 within our manuscript.\n\nWe are committed to modifying the citation indexing and ensuring proper reference inclusion within both the present and forthcoming iterations of our work. With regard to the availability of our code, we intend to make it publicly accessible upon acceptance of our submission.\n\n**Q5:** How to calculate the error bar?\n\n**A5:** As asserted in lines 283, each network is initialized **5 times** with different random seed to evaluate the synthetic datasets. We follow the experimental setup outlined in the baselines MTT[5] and IDC[21]. We report the mean and variation of the accuracies. \n\n**Q6:** divide the original training dataset  based on data difficulty. \n\n**A6:** We conducted experiments wherein the original dataset was partitioned into three equal subsets termed \"Easy,\" \"Medium,\" and \"Hard,\" respectively. The sorting criterion is based on the instance-wise average loss reduction observed during standard training, following[14]. Subsequently, the synthetic dataset underwent division into three subsets, each earmarked for distillation from the corresponding \"Easy,\" \"Medium,\" and \"Hard\" target subsets. Post distillation, we proceeded to assess the synthetic dataset through two distinct approaches:\n\n(a) The neural network was trained using the synthetic dataset in the order of \"Easy,\" \"Medium,\" and \"Hard\", which is marked as \"Sequential\" in the following table. \n\n(b) The neural network was trained utilizing the entirety of the synthetic dataset, which is marked as \"Mixed\" in the following table. \n\nWe report the results with the reference of our SeqMatch below:\n\n||Easy |Medium |Hard |Entire| \n|:-:| :-: |:-: |:-:|:-: |\n|Sequential|61.7|65.3|65.9|65.9|\n|Mixed|-|-|-|65.5|\n|SeqMatch|67.9|70.8|74.4|74.4|\n\nThe results highlight a significant performance gap between the Divide-and-Conquer approach and our proposed SeqMatch. This discrepancy arises because feature extraction in standard training originates from mini-batches drawn from the entire dataset. The gradients, considered as the teacher trajectories in dataset distillation, constitute a mixture derived from both \"Easy\" and \"Hard\" instances.\n\nDespite \"Easy\" instances with marginal training loss contributing a lower proportion of gradients during subsequent training, these instances play a pivotal role in stabilizing the optimization direction once the network becomes overfitted to the \"Hard\" instances.\n\nConsequently, dividing the original training dataset along the instance dimension is not a more favorable choice compared to division within the \"Epoch\" dimension. The latter approach is the methodology implemented in SeqMatch.\n"}]

Author rebuttal:
We would like to express our gratitude to all the reviewers and the Associate Chair for dedicating their time and effort to the review of our work. We appreciate the constructive questions and suggestions that have contributed to the enhancement of SeqMatch. Based on the feedback received, we have made revisions to our work as outlined below:

1. We have revised Algorithm 1 as per the suggestion made by reviewer QS3g in order to enhance its clarity. We introduced an inner loop to provide explicit illustration of the optimization process for each subset within SeqMatch. Additionally, we incorporated an input parameter labeled "Base Distillation Method" to signify the potential integration of other distillation methods with SeqMatch. The updated version of Algorithm 1 has been appended to the response PDF for reference.

2. We have appended the visulizations of SeqMatch (**Figure 4** in the response PDF)  under the setting ipc=$\{2,3\}$, alongside SeqMatch-MTT, for the CIFAR 10 dataset. The corresponding evaluation accuracies are presented below:

    | IPC|2|3| 
    |:----------- | :-----------: |:-----------: |
    |MTT|51.6|54.5|
    |SeqMatch|52.9|57.0|


    SeqMatch outperforms MTT with performance enhancemanents of $\\{1.3\\%,2.5\\%\\}$ under the settings ipc=$\{2,3\}$.
    
3. We have presented the abation study over number of subsets $K$ as suggested by the reviewer xCSw. The results is visulized as **Figure 5** in the response PDF. The outcomes distinctly indicate a decline in performance as K escalates from 2 to 5 when ipc=10. A plausible rationale for this phenomenon lies in the fact that subsets with insufficient images (ipc < 5) struggle to effectively distill comprehensive knowledge from the target dataset.

    Conversely, the degradation in performance as K increases remains marginal when ipc=50. This is potentially attributed to the subset size surpassing the threshold required to adequately capture essential features (ipc > 10), thereby substantiating the observed trend.

4. We have studies the memory usage of SeqMatch. The high memory overhead is one of the critial bottolneck of dataset distillation [8][49]. In addition to augmenting performance, our proposed SeqMatch offers an ancillary advantage – a reduction (up to **56% reduction**) in memory expenses associated with dataset distillation. We report the memory usage of MTT and SeqMatch-MTT as **Table 3** in the response PDF.

5. We have modified the improper citation and the consistence issue of references. We also append more reference related to our work as below.

[49] Yu, Ruonan, Songhua Liu, and Xinchao Wang. "Dataset distillation: A comprehensive review." arXiv preprint arXiv:2301.07014 (2023).

[50] Cazenavette, George, et al. "Generalizing Dataset Distillation via Deep Generative Prior." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[51] Liu, Yanqing, et al. "DREAM: Efficient Dataset Distillation by Representative Matching." arXiv preprint arXiv:2302.14416 (2023).

[52] Sachdeva, Noveen, and Julian McAuley. "Data distillation: A survey." arXiv preprint arXiv:2301.04272 (2023).

[53] Zhang, Lei, et al. "Accelerating dataset distillation via model augmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[54] Loo, Noel, et al. "Dataset Distillation with Convexified Implicit Gradients." (2023).

Meta-review: The paper presents a plausible approach to dataset distillation. However, the proposed approach seems to introduce rather marginal improvements upon baselines. While this is a potentially critical issue, we do acknowledge that at least under certain settings the method enables certain gains (and reduction in memory consumption). The paper is definitely borderline, but we decided that the merits (namely the insights and inspirations for future work) outweigh the demerits. We recommend accepting the paper. We strongly urge the authors to include the additional discussion and results in the final version.
Decision: Accept (poster)