Title: You Only Condense Once: Two Rules for Pruning Condensed Datasets
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dataset Condensation

Dataset Pruning
Abstract: Dataset condensation is a crucial tool for enhancing training efficiency by reducing the size of the training dataset, particularly in on-device scenarios. However, these scenarios have two significant challenges: 1) the varying computational resources available on the devices require a dataset size different from the pre-defined condensed dataset, and 2) the limited computational resources often preclude the possibility of conducting additional condensation processes. We introduce You Only Condense Once (YOCO) to overcome these limitations. On top of one condensed dataset, YOCO produces smaller condensed datasets with two embarrassingly simple dataset pruning rules: Low LBPE Score and Balanced Construction. YOCO offers two key advantages: 1) it can flexibly resize the dataset to fit varying computational constraints, and 2) it eliminates the need for extra condensation processes, which can be computationally prohibitive. Experiments validate our findings on networks including ConvNet, ResNet and DenseNet, and datasets including CIFAR-10, CIFAR-100 and ImageNet. For example, our YOCO surpassed various dataset condensation and dataset pruning methods on CIFAR-10 with ten Images Per Class (IPC), achieving 6.98-8.89% and 6.31-23.92% accuracy gains, respectively. The code is available at: [https://github.com/he-y/you-only-condense-once](https://github.com/he-y/you-only-condense-once).

You Only Condense Once: Two Rules for Pruning Condensed Datasets

Abstract

Dataset condensation is a crucial tool for enhancing training efficiency by reducing the size of the training dataset, particularly in on-device scenarios. However, these scenarios have two significant challenges: 1) the varying computational resources available on the devices require a dataset size different from the pre-defined condensed dataset, and 2) the limited computational resources often preclude the possibility of conducting additional condensation processes. We introduce You Only Condense Once (YOCO) to overcome these limitations. On top of one condensed dataset, YOCO produces smaller condensed datasets with two embarrassingly simple dataset pruning rules: Low LBPE Score and Balanced Construction . YOCO offers two key advantages: 1) it can flexibly resize the dataset to fit varying computational constraints, and 2) it eliminates the need for extra condensation processes, which can be computationally prohibitive. Experiments validate our findings on networks including ConvNet, ResNet and DenseNet, and datasets including CIFAR10, CIFAR-100 and ImageNet. For example, our YOCO surpassed various dataset condensation and dataset pruning methods on CIFAR-10 with ten Images Per Class (IPC), achieving 6.98-8.89% and 6.31-23.92% accuracy gains, respectively. The code is available at: https://github.com/he-y/you-only-condense-once .

1 Introduction

Deep learning models often require vast amounts of data to achieve optimal performance. This data-hungry nature of deep learning algorithms, coupled with the growing size and complexity of datasets, has led to the need for more efficient dataset handling techniques. Dataset condensation is a promising approach that enables models to learn from a smaller and more representative subset of the entire dataset. Condensed datasets are especially utilized in on-device scenarios, where limited computational resources and storage constraints necessitate the use of a compact training set.

However, these on-device scenarios have two significant constraints. First, the diverse and fluctuating computational resources inherent in these scenarios necessitate a level of flexibility in the size of the dataset. But the requirement of flexibility is not accommodated by the fixed sizes of previous condensed datasets. Second, the limited computational capacity in these devices also makes extra condensation processes impractical, if not impossible. Therefore, the need for adaptability in the size of the condensed dataset becomes increasingly crucial. Furthermore, this adaptability needs to be realized without introducing another computationally intensive condensation process.

We introduce You Only Condense Once (YOCO) to enable the flexible resizing (pruning) of condensed datasets to fit varying on-device scenarios without extra condensation process (See Fig. 1). The first rule of our proposed method involves a metric to evaluate the importance of training samples in the context of dataset condensation.

From the gradient of the loss function, we develop the Logit-Based Prediction Error (LBPE) score to rank training samples. This metric quantifies the neural network's difficulty in recognizing each sample. Specifically, training samples with low LBPE scores are considered easy as they indicate that the model's prediction is close to the true label. These samples exhibit simpler patterns, easily captured by the model. Given the condensed datasets' small size, prioritizing easier samples with low LBPE scores is crucial to avoid overfitting.

A further concern is that relying solely on a metricbased ranking could result in an imbalanced distribution of classes within the dataset. Imbalanced

IPC_N

...

IPC_(N-1)

...

Epochs

Condense

IPC_1

Epochs

Condense

...

...

...

...

IPC_(N-1)

IPC_(N-1)

IPC_2

IPC_2

IPC_1

IPC_1

...

...

Previous: Require 200K   (N-1) Epochs

Condense: Require 200K Epochs

Our Goal

Scenarios

Size

Ours: Require 1K Epochs

Score_{1

,

2

,

...

,

N} for all IPC at once

Condensed

Dataset

Full

Dataset

Image

Per

Class

200K

Epochs

Condense

...

...

datasets can lead to biased predictions, as models tend to focus on the majority class, ignoring the underrepresented minority classes. This issue has not been given adequate attention in prior research about dataset pruning [3, 41, 34], but it is particularly important when dealing with condensed datasets. In order to delve deeper into the effects of class imbalance, we explore Rademacher Complexity [31], a widely recognized metric for model complexity that is intimately connected to generalization error and expected loss. Based on the analysis, we propose Balanced Construction to ensure that the condensed dataset is both informative and balanced.

The key contributions of our work are: 1) To the best of our knowledge, it's the first work to provide a solution to adaptively adjust the size of a condensed dataset to fit varying computational constraints. 2) After analyzing the gradient of the loss function, we propose the LBPE score to evaluate the sample importance and find out easy samples with low LBPE scores are suitable for condensed datasets. 3) Our analysis of the Rademacher Complexity highlights the challenges of class imbalance in condensed datasets, leading us to construct balanced datasets.

2 Related Works

2.1 Dataset Condensation/Distillation

Dataset condensation, or distillation, synthesizes a compact image set to maintain the original dataset's information. Wang et al. [45] pioneer an approach that leverages gradient-based hyperparameter optimization to model network parameters as a function of this synthetic data. Building on this, Zhao et al. [53] introduce gradient matching between real and synthetic image-trained models. Kim et al. [19] further extend this, splitting synthetic data into n factor segments, each decoding into n 2 training images. Zhao & Bilen [50] apply consistent differentiable augmentation to real and synthetic data, thus enhancing information distillation. Cazenavette et al. [1] propose to emulate the long-range training dynamics of real data by aligning learning trajectories. Liu et al. [24] advocate matching only representative real dataset images, selected based on latent space cluster centroid distances. Additional research avenues include integrating a contrastive signal [21], matching distribution or features [52, 44], matching multi-level gradients [16], minimizing accumulated trajectory error [7], aligning loss curvature [39], parameterizing datasets [6, 23, 51, 43], and optimizing dataset condensation [27, 32, 33, 42, 55, 25, 49, 4, 26]. Nevertheless, the fixed size of condensed datasets remains an unaddressed constraint in prior work.

2.2 Dataset Pruning

Unlike dataset condensation that alters image pixels, dataset pruning preserves the original data by selecting a representative subset. Entropy [3] keeps hard samples with maximum entropy (uncertainty [22, 38]), using a smaller proxy network. Forgetting [41] defines forgetting events as an accuracy drop at consecutive epochs, and hard samples with the most forgetting events are important. AUM[35] identifies data by computing the Area Under the Margin, the difference between the true label logits and the largest other logits. Memorization [11] prioritizes a sample if it significantly improves the probability of correctly predicting the true label. GraNd [34] and EL2N [34] classify samples as hard based on the presence of large gradient norm and large norm of error vectors, respectively. CCS [54] extends previous methods by pruning hard samples and using stratified sampling to

achieve good coverage of data distributions at a large pruning ratio. Moderate [47] selects moderate samples (neither hard nor easy) that are close to the score median in the feature space. Optimizationbased [48] chooses samples yielding a strictly constrained generalization gap. In addition, other dataset pruning (or coreset selection) methods [8, 28, 36, 17, 30, 18, 2, 29, 9, 10, 46, 15] are widely used for Active Learning [38, 37]. However, the previous methods consider full datasets and often neglect condensed datasets.

3 Method

3.1 Preliminaries

We denote a dataset by S = ( x i , y i ) N i =1 , where x i represents the i th input and y i represents the corresponding true label. Let L ( p ( w , x ) , y ) be a loss function that measures the discrepancy between the predicted output p ( w , x ) and the true label y . The loss function is parameterized by a weight vector w , which we optimize during the training process.

We consider a time-indexed sequence of weight vectors, w t , where t = 1 , . . . , T . This sequence represents the evolution of the weights during the training process. The gradient of the loss function with respect to the weights at time t is given by g t ( x, y ) = ∇ w t L ( p ( w t , x ) , y ) .

3.2 Identifying Important Training Samples

Our goal is to propose a measure that quantifies the importance of a training sample. We begin by analyzing the gradient of the loss function with respect to the weights w t :

We aim to determine the impact of training samples on the gradient of the loss function, as the gradient plays a critical role in the training process of gradient-based optimization methods. Note that our ranking method is inspired by EL2N [34], but we interpret it in a different way by explicitly considering the dataset size | S | .

Definition 1 (Logit-Based Prediction Error - LBPE) : The Logit-Based Prediction Error (LBPE) of a training sample ( x, y ) at time t is given by:

where w t is the weights at time t , and p ( w t , x ) represents the prediction logits.

Lemma 1 (Gradient and Importance of Training Samples) : The gradient of the loss function ∇ w t L ( p ( w t , x ) , y ) for a dataset S is influenced by the samples with prediction errors.

Proof of Lemma 1 : Consider two datasets S and S ¬ j , where S ¬ j is obtained by removing the sample ( x j , y j ) from S . Let the gradients of the loss function for these two datasets be ∇ S w t L and ∇ S ¬ j w t L , respectively. The difference between the gradients is given by (see Appendix A.1 for proof):

Let us denote the error term as: e j = p ( w t , x j ) -y j , the LBPE score for sample ( x j , y j ) is given by LBPE t ( x j , y j ) = E | e j | 2 , and the difference in gradients related to the sample ( x j , y j ) can be rewritten as:

If the sample ( x j , y j ) has a lower LBPE score, it implies that the error term e j is smaller. Let's consider the mean squared error (MSE) loss function, which is convex. The MSE loss function is

Algorithm 1 Compute LBPE score for samples over epochs

defined as L ( e j ) = 1 2 ( e j ) 2 . Consequently, the derivative of the loss function ∂ L ( e j ) ∂e j = e j would be smaller for samples with smaller LBPE scores, leading to a smaller change in the gradient ∆ ∇ w t L .

Rule 1: For a small dataset, a sample with a lower LBPE score will be more important. Let S be a dataset of size | S | , partitioned into subsets S easy (lower LBPE scores) and S hard (higher LBPE scores).

Case 1: Small Dataset - When the dataset size | S | is small, the model's capacity to learn complex representations is limited. Samples in S easy represent prevalent patterns in the data, and focusing on learning from them leads to a lower average expected loss. This enables the model to effectively capture the dominant patterns within the limited dataset size. Moreover, the gradients of the loss function for samples in S easy are smaller, leading to faster convergence and improved model performance within a limited number of training iterations.

Case 2: Large Dataset - When the dataset size | S | is large, the model has the capacity to learn complex representations, allowing it to generalize well to both easy and hard samples. As the model learns from samples in both S easy and S hard , its overall performance improves, and it achieves higher accuracy on hard samples. Training on samples in S hard helps the model learn more discriminative features, as they often lie close to the decision boundary.

Therefore, in the case of a small dataset, samples with lower LBPE scores are more important.

The use of the LBPE importance metric is outlined in Algorithm 1. LBPE scores over the epochs with the Top-K training accuracy are averaged. The output of this algorithm is the average LBPE score.

3.3 Balanced Construction

In this section, we prove that a more balanced class distribution yields a lower expected loss.

Definition 2.1 (Dataset Selection S A and S B ): S A is to select images from each class based on their LBPE scores such that the selection is balanced across classes, and S B is to select images purely based on their LBPE scores without considering the class balance. Formally, we have:

S A = ( x i , y i ) : x i ∈ X k , and LBPE t ( x i , y i ) ≤ τ k , S B = ( x i , y i ) : LBPE t ( x i , y i ) ≤ τ (5) where X k denotes the set of images from class k , and τ k is a threshold for class k , τ is a global threshold. Then S A is a more balanced dataset compared to S B .

Definition 2.2 (Generalization Error): The generalization error of a model is the difference between the expected loss on the training dataset and the expected loss on an unseen test dataset:

Definition 2.3 (Rademacher Complexity): The Rademacher complexity [31] of a hypothesis class H for a dataset S of size N is defined as:

Algorithm 2 Balanced Dataset Construction

where σ i are independent Rademacher random variables taking values in -1 , 1 with equal probability.

Lemma 2.1 (Generalization Error Bound): With a high probability, the generalization error is upper-bounded by the Rademacher complexity of the hypothesis class:

where O represents the order of the term.

Lemma 2.2 (Rademacher Complexity Comparison): The Rademacher complexity of dataset S A is less than that of dataset S B :

Theorem 2.1: The expected loss for the dataset S A is less than or equal to S B when both models achieve similar performance on their respective training sets.

Proof of Theorem 2.1: Using Lemma 2.1 and Lemma 2.2, we have:

Assuming that both models achieve similar performance on their respective training sets, the training losses are approximately equal:

Given this assumption, we can rewrite the generalization error inequality as:

Adding E [ L train ( w A )] to both sides, we get:

This result indicates that the balanced dataset S A is better than S B .

Theorem 2.2: Let S F and S C be the full and condensed datasets, respectively, and let both S F and S C have an imbalanced class distribution with the same degree of imbalance. Then, the influence of the imbalanced class distribution on the expected loss is larger for the condensed dataset S C than for the full dataset S F .

Proof of Theorem 2.2: We compare the expected loss for the full and condensed datasets, taking into account their class imbalances. Let L ( h ) denote the loss function for the hypothesis h . Let E [ L ( h ) | S ] denote the expected loss for the hypothesis h on the dataset S . Let n kF and n kC denote the number of samples in class k for datasets S F and S C , respectively. Let m F and m C denote the total number of samples in datasets S F and S C , respectively. Let r k = n kF m F = n kC m C be the class ratio for each class k in both datasets. The expected loss for S F and S C can be written as:

To show this, let's compare the expected loss per sample in each dataset:

This implies that the influence of the imbalanced class distribution is larger for S C than for S F .

Rule 2: Balanced class distribution should be utilized for the condensed dataset. The construction of a balanced class distribution based on LBPE scores is outlined in Algorithm 2. Its objective is to create an equal number of samples for each class to ensure a balanced dataset.

4 Experiments

4.1 Experiment Settings

IPC stands for 'Images Per Class'. IPC F → T means flexibly resize the condensed dataset from size F to size T . More detailed settings can be found in Appendix B.1.

Dataset Condensation Settings. The CIFAR-10 and CIFAR-100 datasets [20] are condensed via ConvNet-D3 [12], and ImageNet-10 [5] via ResNet10-AP [13], both following IDC [19]. IPC includes 10, 20, or 50, depending on the experiment. For both networks, the learning rate is 0 . 01 with 0 . 9 momentum and 0 . 0005 weight decay. The SGD optimizer and a multi-step learning rate scheduler are used. The training batch size is 64 , and the network is trained for 2000 × 100 epochs for CIFAR-10/CIFAR-100 and 500 × 100 epochs for ImageNet-10.

YOCO Settings. 1) LBPE score selection. To reduce computational costs, we derive the LBPE score from training dynamics of early E epochs. To reduce variance, we use the LBPE score from the topK training epochs with the highest accuracy. For CIFAR-10, we set E = 100 and K = 10 for all the IPC F and IPC T . For CIFAR-100 and ImageNet-10, we set E = 200 and K = 10 for all the IPC F and IPC T . 2) Balanced construction. We use S A in Eq. 5 to achieve a balanced construction. Following IDC [19], we leverage a multi-formation framework to increase the synthetic data quantity while preserving the storage budget. Specifically, an IDC-condensed image is composed of n 2 patches. Each patch is derived from one original image with the resolution scaled down by a factor of 1/ n 2 . Here, n is referred to as the 'factor" in the multi-formation process. For CIFAR-10 and CIFAR-100 datasets, n = 2 ; for ImageNet-10 dataset, n = 3 . We create balanced classes according to these patches. As a result, all the classes have the same number of samples. 3) Flexible resizing. For datasets with IPC F = 10 and IPC F = 20 , we select IPC T of 1 , 2 , and 5 . For IPC F = 50 , we select IPC T of 1 , 2 , 5 , and 10 . For a condensed dataset with IPC F , the performance of its flexible resizing is indicated by the average accuracy across different IPC T values.

Comparison Baselines. We have two sets of baselines for comparison: 1) dataset condensation methods including IDC[19], DREAM[24], MTT [1], DSA [50] and KIP [32] and 2) dataset pruning methods including SSP [40], Entropy [3], AUM [35], Forgetting [41], EL2N [34], and CCS [54]. For dataset condensation methods, we use a random subset as the baseline. For dataset pruning methods, their specific metrics are used to rank and prune datasets to the required size.

4.2 Primary Results

Tab. 1 provides a comprehensive comparison of different methods for flexibly resizing datasets from an initial IPC F to a target IPC T . In this table, we have not included ImageNet results on DREAM [24] since it only reports on Tiny ImageNet with a resolution of 64 × 64, in contrast to ImageNet's 224 × 224. The third column of the table shows the accuracy of the condensed dataset at the parameter IPC F . We then flexibly resize the dataset from IPC F to IPC T . The blue area represents the average accuracy across different IPC T values. For instance, consider the CIFAR-10 dataset with IPC F = 10 . Resizing it to IPC F = 1 , 2 , and 5 using our method yields accuracies of 42.28%, 46.67%, and 55.96%, respectively. The average accuracy of these three values is 48.30%. This value surpasses the 37.08% accuracy of SSP [40] by a considerable margin of 11.22%.

Ablation Study. Tab. 2 shows the ablation study of the LBPE score and the balanced construction across dataset condensation methods. In the first row, the baseline results are shown where neither the LBPE score nor the balanced construction is applied. 'Balanced only' (second row) indicates the selection method is random selection and the class distribution is balanced. 'LBPE only' (third row) means the

LBPE score is used for ranking samples, and the selection results are purely based on the LBPE score without considering the class balance. 'LBPE + Balanced' indicates that both elements are included for sample selection. The empirical findings conclusively affirm the effectiveness of the two rules, which constitute the principal contributions of our YOCO method.

Standard Deviation of Experiments. Different training dynamics and network initializations impact the final results. Therefore, the reported results are averaged over three different training dynamics, and each training dynamic is evaluated based on three different network initializations. See Appendix B.2 for the primary results table with standard deviation.

4.3 Analysis of Two Rules

4.3.1 Analysis of LBPE Score for Sample Ranking

Tab. 3 illustrates the robust performance of our YOCO method across diverse network structures, including ConvNet, ResNet, and DenseNet, demonstrating its strong generalization ability. Additionally, we present different sample ranking metrics from dataset pruning methods on these networks, demonstrating that our method outperforms both random selection and other data pruning methods.

In Tab. 4, we experiment with prioritizing easy samples over hard ones. We achieve this by reversing the importance metrics introduced by AUM [35], Forg. [41], and EL2N [34] that originally prioritize

ConvNet [12]

ResNet [13]

Random

SSP[40]

Entropy[3]

AUM[35]

Forg.[41]

EL2N[34]

Ours

28.23

27.83

30.30

13.30

16.68

16.95

42.28

24.14

24.64

30.53

15.04

16.75

19.98

34.53

DenseNet [14]

24.63

24.75

29.93

14.56

17.43

21.43

34.29

hard samples. Our results indicate that across various condensed datasets, including IDC [19], DREAM [24], MTT [1], and DSA [50], there is a distinct advantage in prioritizing easier samples over harder ones. These findings lend support to our Rule 1.

4.3.2 Analysis of Balanced Construction

Fig. 2 presents the class distributions with and without a balanced construction for different datasets and different IPC F → T . As explained in YOCO settings, our balanced construction is based on the multi-formation framework from IDC [19]. Therefore, the x-axis represents the count of images after multi-formation instead of the condensed images. It is evident that a ranking strategy relying solely on the LBPE score can result in a significant class imbalance, particularly severe in the ImageNet dataset. As depicted in Fig. 2(f), three classes have no image patches left. Our balanced construction method effectively mitigates this issue. Notably, in the case of ImageNet-10 10 → 1 , the balanced construction boosts the accuracy by an impressive 19.37%.

To better understand the impact of balanced class distribution on various dataset pruning methods, we conducted a comparative analysis, as presented in Tab. 5. Clearly, achieving a balanced class distribution significantly enhances the performance of all examined methods. Remarkably, our proposed method consistently outperforms others under both imbalanced and balanced class scenarios, further substantiating the efficacy of our approach.

4.4 Other Analysis

Sample Importance Rules Differ between Condensed Dataset and Full Dataset. In Fig. 3, we compare Sample Importance Rules for Condensed Datasets (IPC 10 , IPC 50 ) and the Full Dataset (IPC 5000 ), by adjusting the pruning ratio from 10% to 90%. Unmarked solid lines mean prioritizing easy samples, dashed lines suggest prioritizing hard samples, while marked solid lines depict the accuracy disparity between the preceding accuracies. Therefore, the grey region above zero indicates 'Prefer easy samples' (Rule easy ), while the blue region below zero represents 'Prefer hard samples'(Rule hard ). We have two observations. First, as the pruning ratio increases, there is a gradual transition from Rule hard to Rule easy . Second, the turning point of this transition depends on the dataset size. Specifically, the turning points for IPC 10 , IPC 50 , and IPC 5000 occur at pruning

0

2

4

6

Count

0

1

2

3

4

5

6

7

8

9

Classes

Balanced

42.28%

Imbalanced

37.63%

(a) CIFAR-10

10

→

1

0

2

4

6

8

10

12

14

16

Count

0

1

2

3

4

5

6

7

8

9

Classes

Balanced

46.67%

Imbalanced

42.29%

(b) CIFAR-10

10

→

2

0

4

8

12

16

20

24

28

Count

0

1

2

3

4

5

6

7

8

9

Classes

Balanced

55.96%

Imbalanced

53.86%

(c) CIFAR-10

10

→

5

0

2

4

6

8

10

Count

0

1

2

3

4

5

6

7

8

9

Classes

Balanced

38.77%

Imbalanced

30.70%

0

2

4

6

8

10 12 14 16 18

Count

0

10

20

30

40

50

60

70

80

90

99

Classes

Balanced

24.21%

Imbalanced

22.22%

0

3

6

9 12 15 18 21 24 27 30 33

Count

0

1

2

3

4

5

6

7

8

9

Classes

Balanced

53.91%

Imbalanced

34.90%

50

(d) CIFAR-10

→

1

(e) CIFAR-100

10

→

1

(f) ImageNet-10

10

→

1

10

20

30

40

50

60

70

80

90

Dataset pruning ratio (%)

10

0

10

20

30

40

50

60

70

80

90

Accuracy (%)

IPC10 Easy

IPC10 Hard

IPC10 Diff

IPC50 Easy

IPC50 Hard

IPC50 Diff

IPC5000 (Full) Easy

IPC5000 (Full) Hard

IPC5000 (Full) Diff

IPCDiff >0: Prefer easy samples

Dividing line for IPCDiff = IPCEasy -  IPCHard

IPCDiff <0: Prefer hard samples

ratios of 24%, 38%, and 72%, respectively. These experimental outcomes substantiate our Rule 1 that condensed datasets should adhere to Rule easy .

Performance Gap from Multi-formation. We would like to explain the huge performance gap between multi-formation-based methods (IDC [19] and DREAM[24]) and other methods (MTT [1], KIP [33], and DSA [50]) in Tab. 2 and Tab. 4. The potential reason is that a single image can be decoded to 2 2 = 4 low-resolution images via multi-formation. As a result, methods employing multi-formation generate four times as many images compared to those that do not use multi-formation. The illustration is shown in Fig. 4.

Why Use LBPE Score from the TopK Training Epochs with the Highest Accuracy? As shown in Eq. 2, different training epoch t leads to a different LBPE score. Fig. 5 illustrates the accuracy of the dataset selected via the LBPE score across specific training epochs. We select LBPE scores from the initial 100 epochs out of 1000 original epochs to reduce computational costs. We have two observations. First, the model's accuracy during the first few epochs is substantially low. LBPE scores derived from these early-stage epochs might not accurately represent the samples' true importance since the model is insufficiently trained. Second, there's significant variance in accuracy even after 40 epochs, leading to potential instability in the LBPE score selection. To address this, we average LBPE scores from epochs with topK accuracy, thereby reducing variability and ensuring a more reliable sample importance representation.

Speculating on Why LBPE Score Performs Better at Certain Epochs? In Fig. 7, we present the distribution of LBPE scores at various training epochs, with scores arranged in ascending order for each class to facilitate comparison across epochs. Our experiment finds the LBPE scores decrease as the epoch number increases. The superior accuracy of LBPE 90 is due to two reasons. First, the model at the 90 th epoch is more thoroughly trained than the model at the first epoch, leading to more accurate LBPE scores. Second, the LBPE 90 score offers a more uniform distribution and a wider range [0, 1], enhancing sample distinction. In contrast, the LBPE 1000 score is mostly concentrated within a narrow range [0, 0.1] for the majority of classes, limiting differentiation among samples. More possible reasons will be explored in future studies.

Visualization. Fig. 6 visualizes the easy and hard samples identified by our YOCO method. We notice that most easy samples have a distinct demarcation between the object and its background. This is particularly evident in the classes of 'vacuum cleaner' and 'cocktail shaker'. The easy samples in

0

20

40

60

80

100

Which Epoch for LBPE Score

28

30

32

34

36

38

40

42

Accuracy (%)

poke

bonnet

green

mamba

langur

Doberman

pinscher

gyromitra

gazelle

hound

cleaner

window

screen

cocktail

shaker

Easy

Original

garden

spider

Easy

Condensed

Hard

Original

Hard

Condensed

0

20

40

Sorted sample index

according to scores

0.0

0.5

1.0

Score

class 0: airplane

0

20

40

0.0

0.5

1.0

class 1: automobile

0

20

40

0.0

0.5

1.0

class 2: bird

0

20

40

0.0

0.5

1.0

class 3: cat

0

20

40

0.0

0.5

1.0

class 4: deer

0

20

40

Sorted sample index

according to scores

0.0

0.5

1.0

Score

class 5: dog

0

20

40

0.0

0.5

1.0

class 6: frog

0

20

40

0.0

0.5

1.0

class 7: horse

0

20

40

0.0

0.5

1.0

class 8: ship

0

20

40

0.0

0.5

1.0

class 9: truck

28.32%

Epoch 90

42.11%

Epoch 200

39.09%

Epoch 1000

32.25%

these two classes have clean backgrounds, while the hard samples have complex backgrounds. The visualization provides evidence of our method's ability to identify easy and hard samples.

5 Conclusion, Limitation and Future Work

We introduce You Only Condense Once (YOCO), a novel approach that resizes condensed datasets flexibly without an extra condensation process, enabling them to adjust to varying computational constraints. YOCO comprises two key rules. First, YOCO employs the Logit-Based Prediction Error (LBPE) score to rank the importance of training samples and emphasizes the benefit of prioritizing easy samples with low LBPE scores. Second, YOCO underscores the need to address the class imbalance in condensed datasets and utilizes Balanced Construction to solve the problem. Our experiments validated YOCO's effectiveness across different networks and datasets. These insights offer valuable directions for future dataset condensation and dataset pruning research.

We acknowledge several limitations and potential areas for further investigation. First, although our method uses early training epochs to reduce computational costs, determining the sample importance in the first few training epochs or even before training is interesting for future work. Second, we only utilize the LBPE score to establish the importance of samples within the dataset. However, relying on a single metric might not be the optimal approach. There are other importance metrics, such as SSP [40] and AUM [35], that could be beneficial to integrate into our methodology. Third, as our current work only covers clean datasets like CIFAR-10, the performance of our method on noisy datasets requires further investigation.

The border impact is shown in Appendix C.

References

G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2022.

K. Chitta, J. M. Álvarez, E. Haussmann, and C. Farabet. Training data subset search with ensemble active learning. IEEE Transactions on Intelligent Transportation Systems , 23(9):1474114752, 2021.

C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia. Selection via proxy: Efficient data selection for deep learning. In Proc. Int. Conf. Learn. Represent. , 2020.

J. Cui, R. Wang, S. Si, and C.-J. Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. arXiv preprint arXiv:2211.10586 , 2022.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , pages 248-255, 2009.

Z. Deng and O. Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In Proc. Adv. Neural Inform. Process. Syst. , 2022.

J. Du, Y. Jiang, V . T. Tan, J. T. Zhou, and H. Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2023.

M. Ducoffe and F. Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841 , 2018.

D. Feldman and M. Langberg. A unified framework for approximating and clustering data. In Proceedings of the Forty-Third Annual ACM Symposium on Theory of Computing , page 569-578, 2011.

D. Feldman, M. Schmidt, and C. Sohler. Turning big data into tiny data: Constant-size coresets for k-means, pca, and projective clustering. SIAM Journal on Computing , 49(3):601-657, 2020.

V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Proc. Adv. Neural Inform. Process. Syst. , pages 2881-2891, 2020.

S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , pages 4367-4375, 2018.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , pages 770-778, 2016.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , pages 4700-4708, 2017.

L. Huang and N. K. Vishnoi. Coresets for clustering in euclidean spaces: Importance sampling is nearly optimal. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , page 1416-1429, 2020.

Z. Jiang, J. Gu, M. Liu, and D. Z. Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208.00311 , 2022.

K. Killamsetty, S. Durga, G. Ramakrishnan, A. De, and R. Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In Proc. Int. Conf. Mach. Learn. , pages 5464-5474, 2021.

K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proc. AAAI Conf. Artif. Intell. , pages 8110-8118, 2021.

J.-H. Kim, J. Kim, S. J. Oh, S. Yun, H. Song, J. Jeong, J.-W. Ha, and H. O. Song. Dataset condensation via efficient synthetic-data parameterization. In Proc. Int. Conf. Mach. Learn. , 2022.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

S. Lee, S. Chun, S. Jung, S. Yun, and S. Yoon. Dataset condensation with contrastive signals. In Proc. Int. Conf. Mach. Learn. , pages 12352-12364, 2022.

D. D. Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum , pages 13-19, 1995.

S. Liu, K. Wang, X. Yang, J. Ye, and X. Wang. Dataset distillation via factorization. In Proc. Adv. Neural Inform. Process. Syst. , 2022.

Y. Liu, J. Gu, K. Wang, Z. Zhu, W. Jiang, and Y. You. DREAM: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302.14416 , 2023.

N. Loo, R. Hasani, A. Amini, and D. Rus. Efficient dataset distillation using random feature approximation. In Proc. Adv. Neural Inform. Process. Syst. , 2022.

N. Loo, R. Hasani, M. Lechner, and D. Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06755 , 2023.

J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics , pages 1540-1552, 2020.

K. Margatina, G. Vernikos, L. Barrault, and N. Aletras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 650-663, 2021.

K. Meding, L. M. S. Buschoff, R. Geirhos, and F. A. Wichmann. Trivial or impossible dichotomous data difficulty masks model differences (on imagenet and beyond). In Proc. Int. Conf. Learn. Represent. , 2022.

B. Mirzasoleiman, J. Bilmes, and J. Leskovec. Coresets for data-efficient training of machine learning models. In Proc. Int. Conf. Mach. Learn. , pages 6950-6960, 2020.

M. Mohri and A. Rostamizadeh. Rademacher complexity bounds for non-iid processes. In Proc. Adv. Neural Inform. Process. Syst. , 2008.

T. Nguyen, Z. Chen, and J. Lee. Dataset meta-learning from kernel ridge-regression. In Proc. Int. Conf. Learn. Represent. , 2021.

T. Nguyen, R. Novak, L. Xiao, and J. Lee. Dataset distillation with infinitely wide convolutional networks. In Proc. Adv. Neural Inform. Process. Syst. , pages 5186-5198, 2021.

M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Proc. Adv. Neural Inform. Process. Syst. , pages 20596-20607, 2021.

G. Pleiss, T. Zhang, E. Elenberg, and K. Q. Weinberger. Identifying mislabeled data using the area under the margin ranking. In Proc. Adv. Neural Inform. Process. Syst. , pages 17044-17056, 2020.

O. Pooladzandi, D. Davini, and B. Mirzasoleiman. Adaptive second order coresets for dataefficient machine learning. In Proc. Int. Conf. Mach. Learn. , pages 17848-17869, 2022.

P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang. A survey of deep active learning. ACM Comput. Surv. , 54(9), oct 2021.

B. Settles. Active Learning . Springer International Publishing, 2012.

S. Shin, H. Bae, D. Shin, W. Joo, and I.-C. Moon. Loss-curvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics , pages 8606-8628, 2023.

B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Proc. Adv. Neural Inform. Process. Syst. , pages 19523-19536, 2022.

M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. In Proc. Int. Conf. Learn. Represent. , 2019.

P. Vicol, J. P. Lorraine, F. Pedregosa, D. Duvenaud, and R. B. Grosse. On implicit bias in overparameterized bilevel optimization. In Proc. Int. Conf. Mach. Learn. , pages 22234-22259, 2022.

K. Wang, J. Gu, D. Zhou, Z. Zhu, W. Jiang, and Y. You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.04707 , 2023.

K. Wang, B. Zhao, X. Peng, Z. Zhu, S. Yang, S. Wang, G. Huang, H. Bilen, X. Wang, and Y. You. Cafe: Learning to condense dataset by aligning features. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , pages 12196-12205, 2022.

T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

M. Welling. Herding dynamical weights to learn. In Proc. Int. Conf. Mach. Learn. , pages 1121-1128, 2009.

X. Xia, J. Liu, J. Yu, X. Shen, B. Han, and T. Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In Proc. Int. Conf. Learn. Represent. , 2023.

S. Yang, Z. Xie, H. Peng, M. Xu, M. Sun, and P. Li. Dataset pruning: Reducing training data by examining generalization influence. In Proc. Int. Conf. Learn. Represent. , 2023.

L. Zhang, J. Zhang, B. Lei, S. Mukherjee, X. Pan, B. Zhao, C. Ding, Y. Li, and D. Xu. Accelerating dataset distillation via model augmentation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2023.

B. Zhao and H. Bilen. Dataset condensation with differentiable siamese augmentation. In Proc. Int. Conf. Mach. Learn. , pages 12674-12685, 2021.

B. Zhao and H. Bilen. Synthesizing informative training samples with GAN. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research , 2022.

B. Zhao and H. Bilen. Dataset condensation with distribution matching. In Proc. IEEE Winter Conf. Appl. Comput. Vis. , pages 6514-6523, 2023.

B. Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. In Proc. Int. Conf. Learn. Represent. , 2021.

H. Zheng, R. Liu, F. Lai, and A. Prakash. Coverage-centric coreset selection for high pruning rates. In Proc. Int. Conf. Learn. Represent. , 2023.

Y. Zhou, E. Nezhadarya, and J. Ba. Dataset distillation using neural feature regression. In Proc. Adv. Neural Inform. Process. Syst. , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a combination method for dataset pruning and dataset condensation. Both methods could reduce the size of training datasets. Experiments show that the proposed method achieves state-of-the-art performance on several datasets regardless of networks. The proposed rules are simple and effective, which might be commonly used in future works.\n\nSTRENGTHS:\n- This paper studies an interesting yet rarely studied problem, which is important for on-device scenarios. In addition, the authors have comprehensively analyze the challenges of this problem and the drawback of existing methods.\n\n- This paper is well-written and easy to follow. I generally enjoy reading the story, motivation, method and experiments of this paper.\n\n- Two novel, simple, and effective rules/metrics are introduced to select informative and balanced samples. Importantly, with this two rules, one can carry on various size requirements by one process.\n\n- Extensive experiments, including results and visualization, demonstrate the effectiveness and efficiency of the proposed method, and show the superiority over the state-of-the-art methods.\n\nWEAKNESSES:\nGenerally, I think the paper is strong. However, I have the following questions.\n\n- In Table 1, the ImageNet-10 on the DREAM method is not listed.\n\n- Some proofs, such as Proof of Theorem 1, can be shortened.\n\n- The core idea is first to condense the dataset before pruning the dataset. How about pruning the dataset first and then condensing it? Which way is better and why?\n\nQUESTIONS:\n- From Figure 1, previous methods require 200K *(N-1) epochs for extra condensation. Is there any way to reduce the extra condensation process without using dataset pruning? Dataset Pruning is just like a selection method.\n\n- Will the code be open-sourced?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper target at the dataset condensation for on-device scenarios.
It proposes two novel rules to prune the condensed dataset. The first rule is to rank the condensed images. The rule believes easy examples are better for pruning condensed dataset. The second rule is to choose balanced samples for different classes. The overall writing is clear. The experiments are sufficient.

STRENGTHS:
1. This is the first work to consider pruning the condensed dataset. The novelty is good.
2. The illustration and writing are clear. The motivation is clearly explained in the figures.
3. The paper provides solid proof for the LBPE metric for ranking the condensed images. 
4. The balanced construction is reasonable expecially when the dataset size is small.
5. The experiment looks good. On CIFAR-10, CIFAR-100, and ImageNet, the proposed YOCO method surpasses other methods by a large margin. For example, on CIFAR-10, the improvements are from 6.57% to 24.33%. On CIFAR-100, the improvements are from 4.77% to 22.13%.
6. The analysis is interesting. It's interesting to see that the rules are different between the condensed dataset and the full dataset.

WEAKNESSES:
1. The LBPE score and balanced construction. Which rule is more important? 
2. If we view the condensed dataset as a full dataset, will the rules change? Is there any threshold for when the rules will change?
3. The LBPE Score is from the Top-K Training Epochs with the Highest Accuracy. Why not use the loss to extract the LBPE score?

QUESTIONS:
Please see the Weaknesses.

LIMITATIONS:
The limitation part is covered in section 5.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper employs a dataset pruning approach on a condensed dataset. Specifically, the intent is to evaluate and rank the data within this condensed dataset based on their respective significance. This enables users to selectively choose any number of condensed datasets, offering both efficacy and flexibility in training the model.

STRENGTHS:
This paper claims that it's the first work to conduct dataset pruning on condensed datasets to fit varying computational constraints.
LBPE score is a reasonable metric to measure the importance of data in the condensed dataset.

WEAKNESSES:
1. The main weakness is the evaluation part. This paper compares their method: (1). dataset condensation + dataset pruning with the previous method (2). dataset condensation + random selection (line 202), which is not fair. For example, it's better to show the effectiveness of the proposed method by comparing (1) with IPCF = 10 to IPCT=1 or 2 or 5 with (3): directly generating the dataset condensation with the IPC=1 or 2 or 5. If it can get similar performance, this paper can claim its advantages of training time. 

2. It's unclear—the whole pipeline in Fig 1 or the rest of the paper. I can only understand the paper after reading the Appendix. It's more critical than putting the algorithm part in the main body. 

3. Proof of Theorem 1 is not solid. 

4. The imbalanced part is not thoroughly analyzed. The equal number of each class is a standard setting in the condensed dataset. This paper does not provide new insight from Section 3.3

QUESTIONS:
Please refer to Weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n--- Post-Rebuttal Edit ---\n\nAfter author rebuttals, I have updated my rating from 5 to 6 and confidence from 2 to 3.\n\n--- End Post-Rebuttal Edit ---\n\nThis paper considers a practical problem in dataset condensation/distillation, where edge devices have varying constraints that need to be flexibly met by a data distillation method. The authors propose two rules to prune the condensed datasets in order to save computational costs. The first rule uses logit-based prediction error (LBPE) to identify useful examples, where low LBPE samples are useful for smaller datasets and high LBPE samples are useful for larger datasets. The second rule focuses on the construction of balanced classes by ensuring an equal number of samples in each class via Rademacher complexity and generalization error.\n\nSTRENGTHS:\n- The writing is generally easy to follow and the experiments are thorough.\n- The proposed approaches are logical and appear useful. The first rule based on LBPE scores allows for flexible treatment based on dataset size and other characteristics; the second rule incorporates Rademacher complexity in a novel way in order to set up a theoretical foundation for the significance of balanced class construction.\n\nWEAKNESSES:\n- Perhaps this is because I am unfamiliar with data distillation literature, but the motivation and setting are somewhat unclear to me. There is not a single citation or brief explanation of what is meant by “on-device scenarios.” Even if this is obvious to well-informed readers, I believe that at minimum a citation or definition is needed.\n    - In what scenario would one need to store a training set on a device? Typically when deploying a machine learning model to a hardware-constrained device, the only thing being deployed is the model weights.\n- Related to the above point, I don’t understand the motivation for the experimental setting, where we are condensing an already-condensed dataset. If one of the central motivations mentioned in the Abstract is to “[eliminate] the need for extra condensation processes,” then I don’t see how this method achieves said goal – we are still ultimately performing two condensation steps!\n- The mixture of theoretical- and prose-style writing is awkward at times. For example, Theorem 1 on line 122 would be much more appropriately written as simple explanatory text. I don’t understand in what sense this is a theorem or how lines 123-124 constitute a “proof” for this claim. There is no need to present content in a mathematical style when it is not necessary.\n\nQUESTIONS:\n- What is an “on-device scenario?” What is the motivation for storing a training set (rather than, say, the trained model weights) on said device?\n- What is the motivation for condensing an already-condensed dataset?\n- How does this method avoid extra condensation steps if all experiments are performed after an initial condensation step with another method?\n    - Why not perform a single condensation step with the proposed method (from full -> desired size)? How does the method perform in this setting?\n\nLIMITATIONS:\n- See Questions above\n- There is no clear definition of what is meant by “small” and “large” datasets. A definition or rough empirical guideline would be useful.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}