Title: Sequential Subset Matching for Dataset Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dataset distillation

gradients matching
Abstract: Dataset distillation is a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields a similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current state-of-the-art methods treat the entire synthetic dataset as a unified entity and optimize each synthetic instance equally . This static optimization approach may lead to performance degradation in dataset distillation. 
Specifically, we argue that static optimization can give rise to a coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads to the failure of the distilled dataset to extract the high-level features learned by the deep neural network (DNN) in the latter epochs.
In this study, we propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data to encourage sequential acquisition of knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms state-of-the-art methods in various datasets, including SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet.

Sequential Subset Matching for Dataset Distillation

Abstract

Dataset distillation is a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields a similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current state-of-the-art methods treat the entire synthetic dataset as a unified entity and optimize each synthetic instance equally. This static optimization approach may lead to performance degradation in dataset distillation. Specifically, we argue that static optimization can give rise to a coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads to the failure of the distilled dataset to extract the high-level features learned by the deep neural network (DNN) in the latter epochs. In this study, we propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data to encourage sequential acquisition of knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms state-of-the-art methods in various datasets, including SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet. Our code is available at https://github.com/shqii1j/seqmatch.

1 Introduction

Recent advancements in Deep Neural Networks (DNNs) have demonstrated their remarkable ability to extract knowledge from large-scale real-world data, as exemplified by the impressive performance of the large language model GPT-3, which was trained on a staggering 45 terabytes of text data [4]. However, the use of such massive datasets comes at a significant cost in terms of data storage, model training, and hyperparameter tuning.

The challenges associated with the use of large-scale datasets have motivated the development of various techniques aimed at reducing datasets size while preserving their essential characteristics. One such technique is dataset distillation [5, 6, 13, 22, 29, 35, 42, 48, 50, 51, 52, 53], which involves synthesizing a smaller dataset that effectively captures the knowledge contained within the original dataset. Models trained on these synthetic datasets have been shown to achieve comparable performance to those trained on the full dataset. In recent years, dataset distillation has garnered increasing attention from the deep learning community and has been leveraged in various practical applications, including continual learning [41, 52, 53], neural architecture search [21, 39, 40, 51, 53], and privacy-preserving tasks [12, 15, 31], among others.

Existing methods for dataset distillation, as proposed in [5, 11, 13, 33, 37, 38, 47, 51, 53], have improved the distillation performance through enhanced optimization methods. These approaches have achieved commendable improvements in consolidating knowledge from the original dataset and generating superior synthetic datasets. However, the knowledge condensed by these existing methods primarily originates from the easy instances, which exhibit a rapid reduction in training loss during the early stages of training. These easy instances constitute the majority of the dataset and typically encompass low-level, yet commonly encountered visual features (e.g. edges and textures [49]) acquired in the initial epochs of training. In contrast, the remaining, less frequent, but more challenging instances encapsulate high-level features (e.g. shapes and contours) that are extracted in the subsequent epochs and significantly impact the generalization capability of deep neural networks (DNNs). The findings depicted in Figure 1 reveal that an overemphasis on low-level features hinders the extraction and condensation of high-level features from hard instances, thereby resulting in a decline in performance.









































































In this paper, we investigate the factors that hinder the efficient condensation of high-level features in dataset distillation. Firstly, we reveal that DNNs are optimized through a process of learning from low-level visual features and gradually adapting to higher-level features. The condensation of high-level features determines the effectiveness of dataset distillation. Secondly, we argue that existing dataset distillation methods fail to extract high-level features because they treat the synthetic data as a unified entity and optimize each synthetic instance unvaryingly. Such static optimization makes the synthetic instances become coupled with each other easier in cases where more synthetic instances are optimized. As a result, increasing the size of synthetic dataset will over-condense the low-level features but fail to condense additional knowledge from the real dataset, let alone the higher-level features.

Building upon the insights derived from our analysis, we present a novel dataset distillation strategy, termed Sequential Subset Matching (Seq-

Match), which is designed to extract both low-level and high-level features from the real dataset, thereby improving dataset distillation. Our approach adopts a simple yet effective strategy for reorganizing the synthesized dataset S during the distillation and evaluation phases. Specifically, we divide the synthetic dataset into multiple subsets and encourage each subset to acquire knowledge in the order that DNNs learn from the real dataset. Our approach can be seamlessly integrated into existing dataset distillation methods. The experiments, as shown in Figure 1, demonstrate that SeqMatch effectively enables the latter subsets to capture high-level features. This, in turn, leads to a substantial improvement in performance compared to the baseline method MTT, which struggles to compress higher-level features from the real dataset. Extensive experiments demonstrate that SeqMatch outperforms state-of-the-art methods, particularly in high compression ratio 2 scenarios, across a range of datasets including CIFAR-10, CIFAR-100, TinyImageNet, and subsets of the ImageNet.

In a nutshell, our contribution can be summarized as follows.

we examine the inefficacy of current dataset distillation in condensing hard instances from the original dataset. We present insightful analyses regarding the plausible factors contributing to this inefficacy and reveal the inherent preference of dataset distillation in condensing knowledge.

We thereby propose a novel dataset distillation strategy called Sequential Subset Matching (SeqMatch) to targetedly encourage the condensing of higher-level features. SeqMatch seamlessly integrates with existing dataset distillation methods, offering easy implementa-

tion. Experiments on diverse datasets demonstrate the effectiveness of SeqMatch, achieving state-of-the-art performance.

2 Related work

Coreset selection is the traditional dataset reduction approach by selecting representative prototypes from the original dataset[2, 7, 17, 43, 45]. However, the non-editable nature of the coreset limits its performance potential. The idea of synthesizing the 'coreset' can be traced back to Wang et al. [47]. Compared to coreset selection, dataset distillation has demonstrated greatly superior performance. Based on the approach of optimizing the synthetic data, dataset distillation can be taxonomized into two types: data-matching methods and meta-learning methods [30].

Data-matching methods encourage the synthetic data to imitate the influence of the target data, involving the gradients, trajectories, and distributions. Zhao and Bilen [52] proposed distribution matching to update synthetic data. Zhao et al. [53] matched the gradients of the target and synthetic data in each iteration for optimization. This approach led to the development of several advanced gradientmatching methods[5, 20, 22, 51]. Trajectory-matching methods [5, 9, 13] further matched multi-step gradients to optimize the synthetic data, achieving state-of-the-art performance. Factorization-based methods [11, 29, 33] distilled the synthetic data into a low-dimensional manifold and used a decoder to recover the source instances from the factorized features.

Meta-learning methods treat the synthetic data as the parameters to be optimized by a meta (or outer) algorithm [3, 11, 32, 34, 37, 38, 54]. A base (or inner) algorithm solves the supervised learning problem and is nested inside the meta (or outer) algorithm with respect to the synthetic data. The synthetic data can be directly updated to minimize the empirical risk of the network. Kernel ridge regression (KRR) based methods [34, 37, 38] have achieved remarkable performance among meta-learning methods.

Both data-matching and meta-learning methods optimize each synthetic instance equally. The absence of variation in converged synthetic instances may lead to the extraction of similar knowledge and result in over-representation of low-level features.

3 Preliminaries

Background Throughout this paper, we denote the target dataset as T = { ( x i , y i ) } |T | i =1 . Each pair of data sample is drawn i.i.d. from a natural distribution D , and x i ∈ R d , y i ∈ Y = { 0 , 1 , · · · , C -1 } where d is the dimension of input data and C is the number of classes. We denote the synthetic dataset as S = { ( s i , y i ) } |S| i =1 where s i ∈ R d , y i ∈ Y . Each class of S contains ipc (images per class) data pairs. Thus, |S| = ipc × C and ipc is typically set to make |S| ≪ |T | .

We employ f θ to denote a deep neural network f with weights θ . An ideal training progress is to search for an optimal weight parameter ˆ θ that minimizes the expected risk over the natural distribution D , which is defined as L D ( f θ ) ≜ E ( x,y ) ∼D [ ℓ ( f θ ( x ) , y ) ] . However, as we can only access the training set T sampled from the natural distribution D , the practical training approach of the network f is to minimizing the empirical risk L T ( f θ ) minimization (ERM) on the training set T , which is defined as

where ℓ can be any training loss function; alg is the given training algorithm that optimizes the initialized weights parameters θ 0 over the training set T ; θ 0 is initialized by sampling from a distribution P θ 0 .

Dataset distillation aims to condense the knowledge of T into the synthetic dataset S so that training over the synthetic dataset S can achieve a comparable performance as training over the target dataset T . The objective of dataset distillation can be formulated as,

Gradient Matching Methods We take gradient matching methods as the backbone method to present our distillation strategy. Matching the gradients introduced by T and S helps to solve S

in Equation 2. By doing so, gradient matching methods achieve advanced performance in dataset distillation. Specifically, gradient matching methods introduce a distance metric D ( · , · ) to measure the distance between gradients. A widely-used distance metric [53] is defined as D ( X,Y ) = ∑ I i =1 ( 1 -⟨ X i ,Y i ⟩ ∥ X i ∥∥ Y i ∥ ) , where X,Y ∈ R I × J and X i , Y i ∈ R J are the i th columns of X and Y respectively. With the defined distance metric D ( · , · ) , gradient matching methods consider solving

where θ i is the intermediate weights which is continuously updated by training the network f θ 0 over the target dataset T . The methods employ M as the hyperparameter to control the length of teacher trajectories to be matched starting from the initialized weights θ 0 ∼ P θ 0 . L ( S .θ ) is the matching loss. The teacher trajectory { θ 0 , θ 1 , · · · , θ M } is equivalent to a series of gradients { g 1 , g 2 , · · · , g M } . To ensure the robustness of the synthetic dataset S to different weights initializations, θ 0 will be sampled from P θ 0 for many times. As a consequence, the distributions of the gradients for training can be represented as { P g 1 , P g 2 , · · · , P g M } .

Algorithm 1 Training with SeqMatch in Distillation Phase.

4 Method

Increasing the size of a synthetic dataset is a straightforward approach to incorporating additional high-level features. However, our findings reveal that simply optimizing more synthetic data leads to an excessive focus on knowledge learned from easy instances. In this section, we first introduce the concept of sequential knowledge acquisition in a standard training procedure (refer to subsection 4.1). Subsequently, we argue that the varying rate of convergence causes certain portions of the synthetic data to abandon the extraction of further knowledge in the later stages (as discussed in Figure 4.2). Finally, we present our proposed strategy, Sequential Subset Matching (referred to as SeqMatch), which is outlined in Algorithm 1 in subsection 4.3.

4.1 Features Are Represented Sequentially

Many studies have observed the sequential acquisition of knowledge in training DNNs. Zeiler et al. [49] revealed that DNNs are optimized to extract low-level visual features, such as edges and textures, in the lower layers, while higher-level features, such as object parts and shapes, were represented in the higher layers. Han et al. [16] leverages the observation that DNNs learn the knowledge from easy instances first, and gradually adapt to hard instances[1] to propose noisy learning methods. The sequential acquisition of knowledge is a critical aspect of DNN.

However, effectively condensing knowledge throughout the entire training process presents significant challenges for existing dataset distillation methods. While the synthetic dataset S is employed to learn from extensive teacher trajectories, extending the length of these trajectories during distillation can exacerbate the issue of domain shifting in gradient distributions, thereby resulting in performance degradation. This is primarily due to the fact that the knowledge extracted from the target dataset T varies across different epochs, leading to corresponding shifts in the domains of gradient distributions. Consequently, the synthetic dataset S may struggle to adequately capture and consolidate knowledge from prolonged teacher trajectories.

To enhance distillation performance, a common approach is to match a shorter teacher trajectory while disregarding knowledge extracted from the latter epochs of T . For instance, in the case of the CIFAR-10 dataset, optimal hyperparameters for M (measured in epochs) in the MTT [5] method were found to be 2 , 20 , 40 for ipc = 1 , 10 , 50 settings, respectively. The compromise made in matching a shorter teacher trajectory unexpectedly resulted in a performance gain, thereby confirming the presence of excessive condensation on easy instances.

Taking into account the sequential acquisition of knowledge during deep neural network (DNN) training is crucial for improving the generalization ability of synthetic data. Involving more synthetic data is the most straightforward approach to condense additional knowledge from longer teacher trajectory. However, our experimental findings, as illustrated in Figure 1, indicate that current gradient matching methods tend to prioritize the consolidation of knowledge derived from easy instances in the early epochs. Consequently, we conducted further investigations into the excessive emphasis on low-level features in existing dataset distillation methods.

4.2 Coupled Synthetic Dataset

The coupling issue within the synthetic dataset impedes its effectiveness in condensing additional high-level features. Existing dataset distillation methods optimize the synthetic dataset S as a unified entity, resulting in the backpropagated gradients used to update S being applied globally. The gradients on each instance only differ across different initializations and preassigned labels, implying that instances sharing a similar initialization within the same class will converge similarly. Consequently, a portion of the synthetic data only serves the purpose of alleviating the gradient matching error for the pre-existing synthetic data.

Consider a synthetic dataset S is newly initialized to be distilled from a target dataset T . The distributions of the gradients for distillation are { P g 1 , P g 2 , · · · , P g M } , and the sampled gradients for training is { g 1 , g 2 , · · · , g M } . Suppose that G is the integrated gradients calculated by S , by minimizing the loss function as stated in

0.000

0.005

0.010

0.015

0.020

0.025

E

si

+

[

‖

R

(

si

, f

θ

0

)

‖

1

]

-

E

si

[

‖

R

(

si

, f

θ

0

)

‖

1

]

-

2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

Acc.(

S

+

) - Acc.(

S

-

)

Distillation discrepancy between

S

+

and

S

-

MTT

SeqMatch

∈S

∈S -

Equation 3, the gradients used for updating s i when θ = θ m would be

we have ∂G ∂ ∇ θm ℓ ( f θm ( s i ) ,y i ) = 1 , because G is accumulated by the gradients of each synthetic data, i.e., G = ∇ θ m L S ( f θ m ) = ∑ ∥S∥ i =1 ∇ θ m ℓ ( f θ m ( s i ) , y i ) . Here we define the amplification function R ( s i , f θ m ) ⊂ R d . Then, the gradients on updating synthetic instance ∇ s i L ( S , θ m ) shares the same ∂ L ∂G and only varies in R ( s i , f θ m ) . The amplification function R ( s i , f θ m ) is only affected by the pre-assigned label and initialization of s i .

More importantly, the magnitude of R ( s i , f θ m ) determines the rate of convergence of each synthetic instance s i . Sorted by the l 1 -norm of amplification function ∥ R ( s i , f θ m ) ∥ 1 can be divided into two subsets S + and S -. S + contains the synthetic instances with greater values of R ( s i , f θ m ) than those in S -. That implies that instances in S + converge faster to minimize D ( ∇ θ m L S + ( f θ m ) , g m ) , and S + is optimized to imitate g m . On the other hand, the instances in S -converge slower and are optimized to minimize D ( ∇ θ m L S -( f θ m ) , ϵ ) , where ϵ represents the gradients matching error ϵ of S + , i.e., ϵ = g m -∇ θ m L S + ( f θ m ) . Therefore, S -is optimized to imitate ϵ and its effectiveness is achieved by compensating for the gradients matching error of S + . S -is coupled with S + and unable to capture the higher-level features in the latter epochs.

We conducted experiments to investigate whether S -solely compensates for the gradient matching error of S + and is unable to extract knowledge independently. To achieve this, we sorted S + and S -by the l 1 -norm of the amplification function ∥ R ( s i , f θ m ) ∥ 1 and trained separate networks with S + and S -. As depicted in Figure 2, we observed a significant discrepancy in accuracy, which increased with the difference in magnitude of R ( s i , f θ m ) . Further details and discussion are provided in subsection 5.3. These experiments verify the coupling issue wherein S -compensates for the matching error of S + , thereby reducing its effectiveness in condensing additional knowledge.

4.3 Sequential Subset Matching

We can use a standard deep learning task as an analogy for the dataset distillation problem, then the synthetic dataset S can be thought of as the weight parameters that need to be optimized. However, simply increasing the size of the synthetic dataset is comparable to multiplying the parameters of a model in an exact layer without architecting the newly added parameters, and the resulting performance improvement is marginal. We thereby propose SeqMatch to reorganize the synthetic dataset S to utilize the newly added synthetic data.

We incorporate additional variability into the optimization process of synthetic data to encourage the capture of higher-level feature extracted in the latter training progress. To do this, SeqMatch divides the synthetic dataset S into K subsets equally, i.e., S = S 1 ∪S 2 ∪···∪S K , |S k | = ⌊ |S| K ⌉ . SeqMatch optimizes each S k by solving

where S ( k -1) = S 1 ∪ S 2 ∪ · · · ∪ S k -1 , which represents the union set of the subsets in the former. S ( k -1) are fixed and only S k will be updated. The subset S k is encouraged to match the corresponding k th segment of the teacher trajectory to condense the knowledge in the latter epoch. Let n = ⌊ M K ⌉ denote the length of trajectory segment to be matched by each subset S k in the proposed framework. To strike a balance between providing adequate capacity for distillation and avoiding coupled synthetic data , the size of each subset S k is well-controlled by K .

In the distillation phase, each subset is arranged in ascending order to be optimized sequentially. We reveal that the first subset S 1 with 1 K size of the original synthetic dataset S is sufficient to condense adequate knowledge in the former epoch. For the subsequent subset S k , we encourage the k th subset S k condense the knowledge different from those condensed in the previous subsets. This is achieved by minimizing the matching loss L ( S k ∪ S ( k -1) , θ m ) while only S k will be updated.

During the evaluation phase, the subsets of the synthetic dataset are used sequentially to train the neural network f θ , with the weight parameters θ being iteratively updated by θ k = alg ( S k , θ k -1 ) . This training process emulates the sequential feature extraction of real dataset T during training. Further details regarding SeqMatch and the optimization of θ ∗ can be found in Algorithm.1.

5 Experiment

In this section, we provide implementation details for our proposed method, along with instructions for reproducibility. We compare the performance of SeqMatch against state-of-the-art dataset distillation methods on a variety of datasets. To ensure a fair and comprehensive comparison, we follow up the experimental setup as stated in [8, 30]. We provide more experiments to verify the effectiveness of SeqMatch including the results on ImageNet subsets and analysis experiments in Appendix A.1 due to page constraints.

5.1 Experimental Setup

Datasets: Weevaluate the performance of dataset distillation methods on several widely-used datasets across various resolutions. MNIST [28], which is a fundamental classification dataset, is included with a resolution of 28 × 28 . SVNH [36] is also considered, which is composed of RGB images of house numbers cwith a resolution of 32 × 32 . CIFAR10 and CIFAR100 [25], two datasets frequently used in dataset distillation, are evaluated in this study. These datasets consist of 50 , 000 training images and 10 , 000 test images from 10 and 100 different categories, respectively. Additionally, our proposed method is evaluated on the Tiny ImageNet [27] dataset with a resolution of 64 × 64 and on the ImageNet [24] subsets with a resolution of 128 × 128 .

Evaluation Metric: The evaluation metric involves distillation phase and evaluation phase. In the former, the synthetic dataset is optimized with a distillation budget that typically restricts the number of images per class ( ipc ). We evaluate the performance of our method and baseline methods under the settings ipc = { 10 , 50 } . We do not evaluate the setting with ipc = 1 since our approach requires ipc ≥ 2 . To facilitate a clear comparison, we mark the factorization-based baselines with an asterisk (*) since they often employ an additional decoder, following the suggestion in [30]. We employ 4-layer ConvNet [14] in Tiny ImageNet dataset whereas for the other datasets we use a 3-layer ConvNet [14].

In the evaluation phase, we utilize the optimized synthetic dataset to train neural networks using a standard training procedure. Specifically, we use each synthetic dataset to train five networks with random initializations for 1 , 000 iterations and report the mean accuracy and its standard deviation of the results.

Implementation Details. To ensure the reproducibility of SeqMatch, we provide detailed implementation specifications. Our method relies on a single hyperparameter, denoted by K , which determines the number of subsets. In order to balance the inclusion of sufficient knowledge in each segment with the capture of high-level features in the later stages, we set K = { 2 , 3 } for the scenarios where ipc = { 10 , 50 } , respectively. Notably, our evaluation results demonstrate that the choice of K remains consistent across the various datasets.

As a plug-in strategy, SeqMatch requires a backbone method for dataset synthesis. Each synthetic subset is optimized using a standard training procedure, specific to the chosen backbone method. The only hyperparameters that require adjustment in the backbone method are those that control the segments of the teacher trajectory to be learned by the synthetic dataset, whereas the remaining hyperparameters emain consistent without adjustment. Such adjustment is to ensure each synthetic subset effectively condenses the knowledge into stages. The precise hyperparameters of the backbone methods are presented in Appendix A.3. We conduct our experiments on the server with four Tesla V100 GPUs.

5.2 Results

Our proposed SeqMatch is plugged into the methods MTT [5] and IDC [23], which are denoted as SeqMatch-MTT and SeqMatch-IDC, respectively. As shown in Table 1, the classification accuracies of ConvNet [14] trained using each dataset distillation method are summarized. The results indicate that SeqMatch significantly outperforms the backbone method across various datasets, and even surpasses state-of-the-art baseline methods in different settings. Our method is demonstrated to outperform the state-of-the-art baseline methods in different settings among different datasets. Notably, SeqMatch achieves a greater performance improvement in scenarios with a high compression ratio (i.e., ipc = 50 ). For instance, we observe a 3 . 5% boost in the performance of MTT [5], achieving 51 . 2% accuracy on CIFAR-100. Similarly, we observe a 1 . 9% performance enhancement in IDC [23],

achieving 92 . 1% accuracy on SVNH, which approaches the 95 . 4% accuracy obtained using the real dataset. These results suggest that our method is effective in mitigating the adverse effects of coupling and effectively condenses high-level features in high compression ratio scenarios.

Cross-Architecture Generalization: We also conducted experiments to evaluate cross-architecture generalization, as illustrated in Table 2. The ability to generalize effectively across different architectures is crucial for the practical application of dataset distillation. We evaluated our proposed SeqMatch on the CIFAR-10 dataset with ipc = 50 . Following the evaluation metric established in [13, 46], three additional neural network architectures were utilized for evaluation: ResNet [19], VGG [44], and AlexNet [26]. Our SeqMatch approach demonstrated a significant improvement in performance during cross-architecture evaluation, highlighting its superior generalization ability.

5.3 Discussions

Sequential Knowledge Acquisition: We conducted experiments on CIFAR-10 with ipc = 50 , presented in Figure 1, to investigate the inability of existing baseline methods to capture the knowledge learned in the latter epoch, as discussed in subsection 4.1. Inspired by [16], we utilized the change in instance-wise loss on real dataset to measure the effectiveness of condensing high-level features. Specifically, we recorded the loss of each instance from the real dataset T at every epoch, where the network was trained with synthetic dataset for only 20

iterations in each epoch. To distinguish hard instances from easy ones, we employed k-means algorithm [18] to cluster all instances in the real dataset into two clusters based on the recorded instance-wise loss. The distribution of instances in terms of difficulty is as follows: 77% are considered easy instances, while 23% are classified as hard instances.

We evaluated MTT[5] and SeqMatch as mentioned above. Our results show that MTT[5] overcondenses the knowledge learned in the former epoch. In contrast, SeqMatch is able to successfully capture the knowledge learned in the latter epoch.

Coupled Synthetic Subsets: In order to validate our hypothesis that the synthetic subset S -is ineffective at condensing knowledge independently and results in over-condensation on the knowledge learned in the former epoch, we conducted experiments as shown in Figure 2. We sorted the subsets S + and S -of the same size by the l 1 -norm of the amplification function | R ( s i , f θ m ) | 1 as explained in Figure 4.2. We then recorded the accuracy discrepancies between the separate networks trained by S + and S -with respect to the mean l 1 -norm difference, i.e., E s i ∈S + [ | R ( s i , f θ 0 ) | 1] -E s i ∈S -[ | R ( s i , f θ 0 ) | 1 ] .

As shown in Figure 2, the accuracy discrepancies increased linearly with the l 1 -norm difference, which verifies our hypothesis that S -is coupled with S + and this coupling leads to the excessive condensation on low-level features. However, our proposed method, SeqMatch, is able to alleviate the coupling issue by encouraging S -to condense knowledge more efficiently.

Baseline

1

st

2

nd

3

rd

Synthetic Image Visualization: In order to demonstrate the distinction between MTT [5] and SeqMatch, we visualized synthetic images within the "car" class from CIFAR-10 [25] and visually compared them. As depicted in Figure 3, the synthetic images produced by MTT exhibit more concrete features and closely resemble actual "car" images. Conversely, the synthetic images generated by SeqMatch in the 2 nd and 3 rd subsets possess more abstract attributes and contain complex car shapes. We provide more visualizations of the synthetic images in Appendix A.2.

5.4 Limitations and Future Work

We acknowledge the limitations of our work from two perspectives. Firstly, our proposed sequential optimization of synthetic subsets increases the overall training time, potentially doubling or tripling it. To address this, future research could investigate optimization methods that allow for parallel optimization of each synthetic subset. Secondly, as the performance of subsequent synthetic subsets builds upon the performance of previous subsets, a strategy is required to adaptively distribute the distillation budget of each subset. Further research could explore strategies to address this limitation and effectively enhance the performance of dataset distillation, particularly in high compression ratio scenarios.

6 Conclusion

In this study, we provide empirical evidence of the failure in condensing high-level features in dataset distillation attributed to the sequential acquisition of knowledge in training DNNs. We reveal that the static optimization of synthetic data leads to a bias in over-condensing the low-level features, predominantly extracted from the majority during the initial stages of training. To address this issue in a targeted manner, we introduce an adaptive and plug-in distillation strategy called SeqMatch. Our proposed strategy involves the division of synthetic data into multiple subsets, which are sequentially optimized, thereby promoting the effective condensation of high-level features learned in the later epochs. Through comprehensive experimentation on diverse datasets, we validate the effectiveness of our analysis and proposed strategy, achieving state-of-the-art performance.

References

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017. 5

Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv:1703.06476 , 2017. 3

Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006.08572 , 2020. 3

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. 1

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022. 1, 2, 3, 5, 7, 8, 9, 14, 15

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3739-3748, 2023. 1

Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv preprint arXiv:1203.3472 , 2012. 3

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Dc-bench: Dataset condensation benchmark. arXiv preprint arXiv:2207.09639 , 2022. 2, 7

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet1k with constant memory. arXiv preprint arXiv:2211.10586 , 2022. 3

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009. 14

Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv:2206.02916 , 2022. 2, 3, 8

Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? arXiv preprint arXiv:2206.00240 , 2022. 1

Jiawei Du, Yidi Jiang, Vincent TF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. arXiv preprint arXiv:2211.11004 , 2022. 1, 2, 3, 8, 14

Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4367-4375, 2018. 7, 14

Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint arXiv:2008.04489 , 2020. 1

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems , 31, 2018. 5, 8

Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. In Proceedings of the twenty-first annual symposium on Computational geometry , pages 126-134, 2005. 3

John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics) , 28(1):100-108, 1979. 9

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 8

Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208.00311 , 2022. 3

Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1946-1956, 2019. 1

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning , pages 11102-11118. PMLR, 2022. 1, 3

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. arXiv preprint arXiv:2205.14959 , 2022. 7, 8

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision , pages 491-507. Springer, 2020. 7

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 and CIFAR-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html , 6(1):1, 2009. 2, 7, 9, 14

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012. 8

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015. 7, 14

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 7

Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208.10494 , 2022. 1, 3

Shiye Lei and Dacheng Tao. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301.05603 , 2023. 3, 7, 8

Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Soft-label anonymous gastric x-ray image distillation. In 2020 IEEE International Conference on Image Processing (ICIP) , pages 305-309. IEEE, 2020. 1

Ping Liu, Xin Yu, and Joey Tianyi Zhou. Meta knowledge condensation for federated learning. arXiv preprint arXiv:2209.14851 , 2022. 3

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. arXiv preprint arXiv:2210.16774 , 2022. 2, 3, 8, 14

Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint arXiv:2210.12067 , 2022. 3

Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06755 , 2023. 1

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011. 7

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. arXiv preprint arXiv:2011.00050 , 2020. 2, 3, 8

Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems , 34:5186-5198, 2021. 2, 3, 8

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning , pages 40954104. PMLR, 2018. 1

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) , 54(4):1-34, 2021. 1

Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. arXiv preprint arXiv:2103.15851 , 2021. 1

Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint arXiv:2301.04272 , 2023. 1

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017. 3

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 8

Ivor W Tsang, James T Kwok, Pak-Ming Cheung, and Nello Cristianini. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research , 6(4), 2005. 3

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12196-12205, 2022. 8

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018. 2, 3, 8

Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. arXiv preprint arXiv:2301.07014 , 2023. 1

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 , pages 818-833. Springer, 2014. 2, 4

Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11950-11959, 2023. 1

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , pages 12674-12685. PMLR, 2021. 1, 2, 3, 8

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6514-6523, 2023. 1, 3, 8

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR , 1(2):3, 2021. 1, 2, 3, 4, 8

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719 , 2022. 3

A More Experiements

A.1 ImageNet Subsets

To assess the efficacy of our approach, we conducted experiments on subsets of the ImageNet dataset [10]. These subsets were constructed by selecting ten pertinent categories from the ImageNet1k dataset [10], with a resolution of 128 × 128 . Consequently, the ImageNet subsets pose greater challenges compared to the CIFAR-10/100 [25] and Tiny ImageNet [27] datasets. We adhered to the configuration of the ImageNet subsets as suggested by previous studies [5, 13, 33], encompassing subsets such as ImageNette (diverse objects), ImageWoof (dog breeds), ImageFruits (various fruits), and ImageMeow (cats).

To synthesize the dataset, we employed a 5-layer ConvNet [14] with a parameter setting of ipc = 10 . The evaluation of the synthetic dataset involved performing five trials with randomly initialized networks. We compared the outcomes of our proposed method, referred to as SeqMatch, with the baseline approach MTT [5], as well as the plug-in strategies FTD [13] and HaBa [33] which build upon MTT.

The comprehensive results are presented in Table 3. Our proposed SeqMatch consistently outperformed the baseline MTT across all subsets. Notably, we achieved a performance improvement of 4 . 3% on the ImageFruit subset. Additionally, SeqMatch demonstrated superior performance compared to HaBa and achieved comparable results to FTD.

Table 3: The performance comparison trained with 5-layer ConvNet on the ImageNet subsets with a resolution of 128 × 128 . We cite the results as reported in MTT [5], FTD [13] and HaBa [33]. The latter two methods, FTD and HaBa, are plug-in strategies that build upon the foundation of MTT. Our proposed approach, SeqMatch, exhibits superior performance compared to both MTT and HaBa, demonstrating a significant improvement in results.

A.2 More Visualizations

Instance-wise loss change : We have presented the average loss change of easy and hard instances in Figure 1, revealing that MTT failed to effectively condense the knowledge learned from the hard instances. To avoid the bias introduced by averaging, we have meticulously recorded and visualized the precise loss change of each individual instance. This is accomplished by employing a heatmap representation, as demonstrated in Figure 6. Each instance is depicted as a horizontal line exhibiting varying colors, with deeper shades of blue indicating higher loss values. Following the same clustering approach as depicted in Figure 1, we proceed to visualize the hard instances at the top and the easy instances at the bottom.

The individual loss changes of MTT, depicted in Figure 6, remain static across epochs. The losses of easy instances decrease to a small value during the initial stages, while the losses of hard instances persist at a high value until the end of training. These results confirm that MTT excessively focuses on low-level features. In contrast, the visualization of SeqMatch clearly exhibits the effect of color gradient, indicating a decrease in loss for the majority of instances. Notably, the losses of hard instances experience a significant decrease when a subsequent subset is introduced. These results validate that SeqMatch effectively consolidates knowledge in a sequential manner.

MTT

0

5

10

15

20

25

30

35

40

45

Epoch

Instance loss from target dataset

0

5

10

15

20

25

30

35

40

45

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Instance-wise Loss

SeqMatch

IPC2 MTT

IPC3 MTT

1

st

2

nd

3

rd

Synthetic Dataset Visualization : We compare the synthetic images with ipc = { 2 , 3 } from the CIFAR10 dataset to highlight the differences between subsets of Seqmatch in Figure 5. We provide more visualizations of the synthetic datasets for ipc = 10 from the 128 × 128 resolution ImageNet dataset: ImageWoof subset in Figure 7 and ImageMeow subset in Figure 8. In addition, parts of the visualizations of synthetic images from the 32 × 32 resolution CIFAR-100 dataset are showed in Figure 6. We observe that the synthetic images generated by SeqMatch in the subsequent subset contains more abstract features than the previous subset.

A.3 Hyperparameter Details

The hyperparameters K of SeqMatch-MTT is set with { 2 , 3 } for the settings ipc = { 10 , 50 } , respectively. The optimal value of hyperparameter K is obtained via grid searches within the set { 2 , 3 , 4 , 5 , 6 } in a validation set within the CIFAR-10 dataset. We find that the subset with a small size will fail to condense the adequate knowledge from the corresponding segment of teacher trajectories,

resulting in performance degradation in the subsequent subsets. For the rest of the hyperparamters, we report them in Table 4.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel dataset distillation method called SeqMatch which focuses on extracting high-level features from later training trajectories. The authors highlight a limitation in state-of-the-art data distillation methods, which tend to condense low-level information from easy data while overlooking the high-level information contained in hard data. In response to this issue, the paper introduces a novel optimization technique that generates multiple small sets of synthetic data. Each of these sets distills distinct knowledge from various stages of the training trajectories. By addressing the inherent problem observed in previous dataset distillation methods, the authors conduct experiments to showcase the efficacy of SeqMatch.\n\nSTRENGTHS:\n1.\tThe idea of condensing different sub-datasets for different stages is novel and interesting.\n2.\tThe proposed method does not require extra computation cost.\n3.\tThe writing is good and easy to follow.\n\nWEAKNESSES:\n-\tThe paper lacks evaluation numbers for certain settings, such as SeqMatch-IDC, on datasets like CFAIR100 50IPC, Tiny-ImageNet, and ImageNet subset. (SeqMatch-IDC seems to be the most favorable setting) This omission makes it difficult to determine the performance of SeqMatch in comparison to other methods in these specific scenarios.\n\n-\tSeqMatch underperforms the baseline method FTD on Tiny-ImageNet and ImageNet subset, suggesting that SeqMatch may not scale well to larger datasets. This raises the question of why SeqMatch was chosen over FTD in these cases. \n\n-\tThe paper lacks an ablation study, which would provide valuable insights into the impact of the number of subsets, K, on the performance of SeqMatch. Including such an evaluation would enhance our understanding of how SeqMatch operates and how different parameter settings influence its performance. \n\n-\tThe caption of Table 1 is somewhat misleading. Although IDC is not categorized as a factorization-based method, it does employ data parameterization (factorization can be treated as a special data parameterization). Therefore, it would be more appropriate to compare IDC with RTP and HaBa rather than other methods distilling information into a single image.\n\n-\tMinor: There are citation inconsistencies in the appendix. For example, FTD is referred to as [12] in the appendix but as [11] in the main text. Additionally, the appendix lacks a reference section, making it difficult to trace the sources of the cited works accurately.\n\n- The reproductivity checklist is chosen as Yes, but no code is provided.\n\nQUESTIONS:\nHow is the repeated evaluation conducted to calculate the error bar?\n\nThe motivation of the paper is that SOTA methods fail to distill high-level information from hard data. From my perspective, a more natural and straightforward solution would be to divide the original training dataset into different subsets based on data difficulty and then distill distinct datasets based on these subsets. I am wondering if the authors explored this approach or considered it as an alternative solution in their research.\n\nLIMITATIONS:\nYes, the authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a change to existing dataset distillation methods by sequentially optimizing different subsets at a time. At each iteration, the existing subset is frozen and a new subset of data is *added* to it and optimized. This method allows different subsets of the synthetic data to capture different levels of features required by a network to learn during training time. This method boosts the state of the art for subsets with >1 IPC (since this method does not make sense with 1 IPC).\n\nSTRENGTHS:\nI like this paper a lot. The authors addressed an obvious problem with existing dataset algorithms (all samples capture the same level of features) in an elegant way that is adaptable to all existing and future backbone methods. The authors both empirically and theoretically show that jointly optimizing the entire set couples the gradients in a way that prevents the synthetic set from learning the necessary variety of features.\n\nThe visuals are all very nice, clearly illustrating the authors' points.\n\nWEAKNESSES:\nAlgorithm 1 is a bit confusing to read. According to line 4, it seems that a single network initialization is used to optimize the entire subset (but this can't be true since it would catastrophically overfit). It is also unclear what the $n$ parameter is. As the algorithm is currently presented, I cannot see how MTT can be slotted into it.\n\nMaybe it would be clearer if an additional inner loop was included along with a generic distillation loss and doing away with the sum over $m$?\n\nQUESTIONS:\nI would like for the authors to make Algorithm 1 more clear as described above.\n\nIt would also be nice to see visuals when there is just 1 sample per subset (i.e., K=IPC) even with just IPC 2 or 3 since this case would likely have the largest differences between the subsets.\n\nLastly, I recognize his surname is quite long, but Figure 1 should probably cite \"Cazenavette et al.\" rather than \"George et al.\" :)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a new method called sequential subset matching (SeqMatch) for dataset distillation. The proposed method is designed to continuously generate synthetic images at different training (distillation) iteration. This strategy is inspired by the general mechanism of optimization, which captures characteristics (low-level feature) of easy instances in an early stage, but takes characteristics (i.e., high-level feature) from increasingly difficult instances. SeqMatch was applied to various dataset distillation methods and showed marginal but better performance than the method in which SeqMatch was applied or other baseline methods in various four datasets.\n\nSTRENGTHS:\n1. The analysis about the general mechanism of optimization is so insightful that it deserves a lot of attention in other studies.\n\n2. The plots of several figures provide good support for the arguments in this paper. e.g., Figures 1 and 2 well illustrate the effect of the general mechanism of optimization on dataset distillation and the coupling issue, respectively.\n\nWEAKNESSES:\nFirst of all, I don't understand the motivation behind designing some SeqMatch.\n\n1. How was the claim for the optimization mechanism that captures low-level features in the early stages and high-level features in the later stages verified? Figure 1 seems to have been used to verify this claim, but can hard instances and easy instances represent high-level and low-level features, respectively?\n\n2. I don't understand how the analysis of the general mechanism of optimization was used to design the SeqMatch method. In particular, what is the motivation for applying the SeqMatch method when training f_theta in the evaluation phase?\n\n\nIn addition, in Table 1, most of the performance improvements acquired via SeqMatch are very marginal (~0.2). Performance improvement is not an absolute determining factor in judging the superiority of the proposed method, but it can be used to verify that the method works as claimed in this paper.\n\nQUESTIONS:\n1.  In \"Gradient Matching Methods\" in Section 3, are the gradients from g_1 to g_M the gradients from different layers of a network? Or gradients from another training iteration?\n\nLIMITATIONS:\nThis paper adequately addressed the limitation and promised to solve it in future work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates an issue with dataset distillation, where synthesized datasets tend to overly condense low-level features but fail to efficiently incorporate high-level ones. The authors argue that this is due to existing methods treating the synthetic dataset as a unified entity and equally optimizing each instance, leading to a coupling issue as the size of the synthetic dataset increases. To address this problem, they propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch). SeqMatch divides the synthetic dataset into multiple subsets and optimizes them in sequence, mimicking the learning process from low-level to high-level features. This approach allows each subset of the synthetic dataset to progressively capture more complex, high-level features, reducing the coupling issue and enhancing overall performance.\n\nSTRENGTHS:\nThis paper is well-written, well-motivated, and well-organized. The authors provide comprehensive experiments on various datasets such as CIFAR-10, CIFAR-100, TinyImageNet, and subsets of the ImageNet. They provide insightful analysis of the experimental results, discussing the impact of different factors. The authors provide a detailed algorithm description that translates their theoretical insights into practical application.\n\nWEAKNESSES:\n1. It's not clear how generalizable these results are to other tasks and datasets. Their experiments are based on specific DNN architectures and datasets, and the proposed method's effectiveness might vary under different conditions.\n2. Since the method divides the synthetic dataset into subsets and optimizes them sequentially, there might be a risk of overfitting, especially when the number of subsets is high or when subsets are small.\n3. The method proposed requires a series of sequential optimization processes, which could potentially increase the computational cost and time required for training.\n\nQUESTIONS:\n1. How do you account for the randomness in the initialization of the synthetic dataset, and how does it affect the rate of convergence of each synthetic instance?\n2. The authors mention that instances sharing a similar initialization within the same class will converge similarly. Could this lead to any form of bias in your findings?\n3. The authors identified that current gradient matching methods prioritize easy instances during early epochs. What is an effective mechanism for identifying or quantifying the 'difficult' instances to create a more balanced emphasis?\n\nLIMITATIONS:\nIn section 5.4, the authors discussed two limitations of their work. Their openness in acknowledging these limitations adds to the credibility of their research and provides useful guidance for follow-up studies in this field.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}