You Only Condense Once: Two Rules for Pruning Condensed Datasets
Authors: Yang He
Lingao Xiao
Joey Tianyi Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Dataset Condensation
Dataset Pruning
TL;DR: Flexibly resize the condensed dataset without extra condensation process

Abstract: Dataset condensation is a crucial tool for enhancing training efficiency by reducing the size of the training dataset, particularly in on-device scenarios. However, these scenarios have two significant challenges: 1) the varying computational resources available on the devices require a dataset size different from the pre-defined condensed dataset, and 2) the limited computational resources often preclude the possibility of conducting additional condensation processes. We introduce You Only Condense Once (YOCO) to overcome these limitations. On top of one condensed dataset, YOCO produces smaller condensed datasets with two embarrassingly simple dataset pruning rules: Low LBPE Score and Balanced Construction. YOCO offers two key advantages: 1) it can flexibly resize the dataset to fit varying computational constraints, and 2) it eliminates the need for extra condensation processes, which can be computationally prohibitive. Experiments validate our findings on networks including ConvNet, ResNet and DenseNet, and datasets including CIFAR-10, CIFAR-100 and ImageNet. For example, our YOCO surpassed various dataset condensation and dataset pruning methods on CIFAR-10 with ten Images Per Class (IPC), achieving 6.98-8.89% and 6.31-23.92% accuracy gains, respectively. The code is available at: [https://github.com/he-y/you-only-condense-once](https://github.com/he-y/you-only-condense-once).

You Only Condense Once: Two Rules for Pruning Condensed Datasets
Yang He, Lingao Xiao, Joey Tianyi Zhou
CFAR, Agency for Science. Technology and Research, Singapore IHPC, Agency for Science, Technology and Research, Singapore {He__aue Yang, Joey_ Li__ZZooo@_fffaaa a-star.etreee aeaar.edd..e. edu.sg
Abstract
Dataset condensation is erucial tool for enhancing training efficiency by reducing the size of the training dataset, particularly in on-device scenarios. However, these scenarios have two significant challenges: 1) the varying computational resources available on the devices require dataset size different from the pre-defined con- densed dataset, and 2) the limited computational resources often preclude the pos- sibility of conducting additional condensation processes. We introduce You Only Condense Once (YOCO) to overcome these limitations. On top of one condensed dataset, YOCO produces smaller condensed datasets with embarrassingly sim- ple dataset pruning rules: Low LBPE Score and Balanced Construction. YOCO offers two key advantages: 1) it can flexibly resize the dataset to fit varying compu- tational constraints, and 2) it eliminates the need for extra condensation processes, which can be computationally prohibitive. Experiments validate our findings on networks including ConvNet, ResNet and DenseNet, and datasets including CIFAR- 10, CIFAR-100 and ImageNet. For example, our YOCO surpassed various dataset condensation and dataset pruning methods on CIFAR- with ten Images Per Class (IPC), achieving 6.98-8.89% and 6.31 6-31-23.0000 92% accuracy gains, respectively. The code is available at: hatps::/githie www/he-m/eee y/you onweeooodelseeeeeeeeeeeo once
Introduction
Deep learning models often require vast amounts of data to achieve optimal performance. This data-hungry nature of deep learning algorithms, coupled with the growing size and complexity of datasets, has led to the need for more efficient dataset handling techniques. Dataset condensation is a promising approach that enables models to Iearn from smaller and more representative subset of the entire dataset. Condensed datasets are especially utilized in on-device scenarios, where limited computational resources and storage constraints necessitate the use of compact training set.
However, these on-device scenarios have two significant constraints. First, the diverse and fluctuating computational resources inherent in these scenarios necessitate level of flexibility in the size of dataset. But the requirement of flexibility is not accommodated by the fixed sizes of previous condensed datasets. Second, the limited computational capacity in these devices also makes extra condensation processes impractical, if not impossible. Therefore, the need for adaptability in the size of the condensed dataset becomes increasingly crucial. Furthermore, this adaptability needs to be realized without introducing another computationally intensive condensation process.
We introduce You Only Condense Once (YOCO) to enable the flexible resizing (pruning) of condensed datasets fit varying on-device scenarios without extra condensation process (See Fig. T]. The first rule of our proposed method involves a metric to evaluate the importance of training samples in the context of dataset condensation.
Conresoonding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

From the gradient of the loss function, we develop the Logit-Based Prediction Error (LBPE) score to rank training samples. This metric quantifies the neural network's 's difficulty in recognizing each sam- ple. Specifically, training samples with low LBPE scores are considered easy as they indicate that the model's prediction is close to the true label. These samples exhibit simpler patterns, easily captured by the model. Given the condensed datasets' small size, prioritizing easier samples with low LBPE scores is crucial to avoid overfitting.
IPC
Condense: Roquire 200K Epochs
Condbnsee
PC_1
Condense
ZCCK
Proviouss Require 200Ks- Epochs
Ourut Require 1K Epochs
Condarue
PCL1
Store_.).2.... NO for IPC once
IPC_1
(PCLIN-)
further concern is that relying solely on a metric- based ranking could result in an imbalanced distri- bution of classes within the dataset. Imbalanced datasets can lead to biased predictions, models tend to focus on the majority class, ignoring the underrepresented minority classes. This issue has not been given adequate attention in prior research about dataset pruning 3 34], but it particularly important when dealing with condensed datasets. In order to delve deeper into the effects of class imbalance, we explore Rademacher Complexity [I], widely recognized metric for model complexity that is intimately connected to generalization error and expected loss. Based on the analysis, we propose Balanced Construction to ensure that the condensed dataset is both informative and balanced.
Figure I: Previous methods (left) require extra condensation processes, but ours (right) do not.
The key contributions of our work are: 1) To the best of our knowledge, it's the first work to provide a solution to adaptively adjust the size of condensed dataset to fit varying computational constraints. 2) After analyzing the gradient of the loss function, we propose the LBPE score to evaluate the sample importance and find out easy samples with low LBPE scores are suitable for condensed datasets. 3) Our analysis of the Rademacher Complexity highlights the challenges of class imbalance in condensed datasets, leading us to construct balanced datasets.
2 Related Works
2.1 Dataset Condensation/Distilaiie
Dataset condensation, or distillation, synthesizes a compact image set to maintain the original dataset's information. Wang et al. 455 pioneer an approach that leverages gradient-based hyper- parameter optimization to model network parameters as a function of this synthetic data. Building on this, Zhao et al. [53] introduce gradient matching between real and synthetic image-trained models. Kim et al. 19 further extend this, splitting synthetic data into n factor segments, each decoding into n² training images. Zhao & Bilen (50) apply consistent differentiable augmentation to real and synthetic data, thus enhancing information distillation. Cazenavette el al. 1 propose to emulate the long-range training dynamics of real data by aligning learning trajectories. Liu et al. 124] advocate matching only representative real dataset images, selected based on latent space cluster centroid distances. Additional research avenues include integrating a contrastive signal [2I], matching distribution or features (52 E matching multi-level gradients (16), minimizing
accumulated trajectory error 77), aligning loss curvature (39,, parameterizing datasets [6] 23 51 43].
and optimizing dataset condensation [27 32 33
42
55 25 49 26). Nevertheless, the fixed size
of condensed datasets remains an unaddressed constraint in prior work.
2.2 Dataset Pruning
Unlike dataset condensation that alters image pixels, dataset pruning preserves the original data by selecting a representative subset. Entropy [3] keeps hard samples with maximum entropy (un- certainty 22 38). ). using smaller proxy network. Forgetting [4] defines forgetting events as an accuracy drop at consecutive epochs, and hard samples with the most forgetting events are important.
AUM (35] identifies data by computing the Area Under the Margin, the difference between the true
label logits and the largest other logits. Memorization [II] prioritizes sample if significantly improves the probability of correctly predicting the true label. GraNd (34] and EL2N [4 classify
samples as hard based on the presence of large gradient norm and large norm of error vectors, respec-
tively. CCS (54) extends previous methods by pruning hard samples and using stratified sampling to

---

achieve good coverage of data distributions at large pruning ratio. Moderate 47 selects moderate samples (neither hard nor easy) that are close to the score median in the feature space. Optimization- based 48 chooses samples yielding a strictly constrained generalization gap. In addition, other
dataset pruning (or coreset selection) methods [8] 36
T8
10 46 [5] are widely
used for Active Learning [38] (3]. However, the previous methods consider full datasets and often neglect condensed datasets.
3 Method
3.1
Preliminaries
We denote a dataset by (Iig M)) where x; represents the jth input and Y/ represents the corresponding true label. Let C(((w, w), V) be a loss function that measures the discrepancy between the predicted output p(w,z) x)) and the true label y. The loss function parameterized by a weight vector w, which we optimize during the training process.
We consider time-indexed sequence of weight vectors, Wt, where & = l.... T. This sequence represents the evolution of the weights during the training process. The gradient of the loss function with respect to the weights time t is given by ge(x, y) V Ww. wwp,W, 2),,),))))
3.2 Identifying Important Training Samples
Our goal is to propose measure that quantifies the importance of training sample. We begin by analyzing the gradient of the loss function with respect to the weights w,:
V W. ((p(W,, aL(p(wt, Öp(w,,z)
Bp(w,,37)
Ow:
We aim to determine the impact of training samples on the gradient of the loss function, as the gradient plays critical role in the training process of gradient-based optimization methods. Note that our ranking method is inspired by EL.2N 34 but we interpret a different way by explicitly considering the dataset size [S].
Definition (Logit-Based Prediction Error LBPE): The Logit-Based Prediction Error (LBPE) of a training sample (x,, y) at time / is given by:
LBPE, E [p (w:,z) 2) 812
where w, is the weights time t, and p(w., 2)) represents the prediction logits.
Lemma 1 (Gradient and Importance of Training Samples): The gradient of the loss function V C(p(w,, x), y) for dataset S' is influenced by the samples with prediction errors.
Proof of Lemma 1: Consider two datasets S and S-jj. where S-j is obtained by removing the sample (xy, Uj) from S. Let the gradients of the loss function for these datasets be VS L and V L, respectively. The difference between the gradients is given by (see Appendix A.1 for proof):
DC(p(w,,x);1) Op( We, x)
OL (p,,, (. .y)) Op (w,,
AV w. L = [SI(S (n.y)E
+
Op(
Ow.
|S|
Op (w.,
Ow,
Let us denote the error term as: ej = p( w., m.) Vy. the LBPE score for sample (æ,, By By) is given by LBPE, = E lcylg. and the difference in gradients related to the sample (I,, Uj) can be rewritten as:
1 aC(e,) @p(w),ay) S de,
Ow,
If the sample (I.y. 1/) has a lower LBPE score, it implies that the error term e, is smaller. Let's consider the mean squared error (MSE) loss function, which is convex. The MSE loss function is

---

Algorithm 1 Compute LBPE score for samples over epochs
Require:
Training dataset S and its size |S|, weights wa, true labels y, model's predicted probabilities
((W,))) number of epochs E. Epochs with Top-K accuracy
1: Initialize matrix: LBPE= torch.zeros( (S))
D LLPE scores over samples and epochs P. Track the accuracy over epochs Looop through each epoch
2: Initialize accuracy: ACC= torch.zeros(E))
3: for each epoch in range E do
4: for each sample index in do
P Loop through each sample in the dataset
5: Compute error term for sample i epoch t: 6,, p( V/
6: Compute LBPE score for sample at epoch t with MSE loss: LBPE,, = EMSE |es,122
7: end for
8: Compute accuracy at epoch I: ACC,
9: end for
10: Top_K <- argsort (ACC)
> Find the epochs with the Top-K accuracy Average L.PE score over Top-K epochs
11: AVG LGG_ meannL.PP[[]Tl Top__,
12: return AVG LB__
defined as [(ez) = (e))². Consequently, the derivative of the loss function OCies) deg Ej would be smaller for samples with smaller LBPE scores, leading to a smaller change in the gradient AV C.
Rule 1: For a small dataset, a sample with a lower LBPE score will be more important. Let $ be a dataset of size [S], partitioned into subsets Seasy (lower LBPE scores) and Shard (higher LBPE scores).
Case 1: Small Dataset When the dataset size |S] is small, the model's capacity to learn complex representations is limited, Samples in Seasy represent prevalent patterns in the data, and focusing on learning from them leads lower average expected loss. This enables the model effectively cap- ture the dominant patterns within the limited dataset size. Moreover, the gradients of the loss function for samples in Seasy are smaller, leading to faster convergence and improved model performance within a limited number of training iterations.
Case 2: Large Dataset When the dataset size |S] is large, the model has the capacity to learn complex representations, allowing it to generalize well to both easy and hard samples. As the model learns from samples in both Seasy and Shard, its overall performance improves, and it achieves higher accuracy on hard samples. Training on samples in Spard helps the model learn more discriminative features, they often lie close the decision boundary.
Therefore, in the case small dataset, samples with lower LBPE scores are more important.
The use of the LBPE importance metric is outlined in Algorithm | LBPE scores over the epochs with the Top-K training accuracy are averaged. The output of this algorithm the average LBPE score.
3.3 Balanced Construction
In this section, we prove that a more balanced class distribution yields a lower expected loss.
Definition 2.1 (Dataset Selection SA and 3)) SA is to select images from each class based on their LBPE scores such that the selection is balanced across classes, and Sn is select images purely based on their LBPE scores without considering the class balance. Formally, we have:
Sn (x),y:) LBPE, M1) T (5) where X denotes the set of images from class k, and is a threshold for class k, T is global threshold. Then SA more balanced dataset compared to SB.
SA (Ii, y1) 2i Xs; and LBPE, y:) I Tk:
Definition 2.2 (Generalization Error):
The generalization error a model is the difference between the expected loss on the training dataset and the expected loss
unseen test dataset:
GenErr(w) = EE[Lest (w)) - E L..rriinww (w)]-
(6)
Definition 2.3 (Rademacher Complexity): The Rademacher complexity [1 of a hypothesis class H. for dataset S of size N defined as:
N
Rx (N())
sup MEH
a;h((xi)

---

Algorithm 2 Balanced Dataset Construction
Require: Condensed dataset S (2t, 96) with classes K, LBPE scores LBPE, class-specifer thresholds T Tkk=1 to ensure an equal number of samples for each class D Initialize the balanced subset as an empty set
1: Initialize SA = 9
2: for each class k K do
3: Isel + {i : 8/ = k and LBPE, U1) I Tk}
D Find indices of samples for class D Add the selected samples to the balanced subset
4: S AA U (z), :iE: E Ise
5: end for
6:
return SA
Return the balanced subset
where o; are independent Rademacher random variables taking values in -1, 1 with equal probability, Lemma 2.1 (Generalization Error Bound); With a high probability, the generalization error is upper-bounded by the Rademacher complexity of the hypothesis class:
GenErr(w) S NR (H) +
(8)
VN
where O represents the order of the term.
Lemma 2.2 (Rademacher Complexity Comparison): The Rademacher complexity of dataset SA is less than that dataset S3:
(H) S RNz (H).
(9)
Theorem 2.1:
The expected loss for the dataset A is less than or equal to Sg when both models achieve similar performance on their respective training sets.
Proof of Theorem 2.1: Using Lemma 2.1 and Lemma 2.2, have:
GenErr(w,)) A) S GenErr(WB).
(10)
Assuming that both models achieve similar performance on their respective training sets, the training losses approximately equal:
L (Wa)] x E[ (WLLLiiiWBB)) (wB)).
(11)
Given this assumption, we can rewrite the generalization error inequality as:
E Treea A)] - EE [T (w))] < EE [T w)]
E Tarai (w w)]]
(12)
Adding TE| [Tniin (w x)) to both sides, we get:
[E[Leaa W wx)] S E Luest (
(WB))
(13)
This result indicates that the balanced dataset SA better than Sg.
Theorem 2.2: Let SF and Se be the full and condensed datasets, respectively, and let both Sp and Se have imbalanced class distribution with the degree of imbalance. Then, the influence of the imbalanced class distribution on the expected loss larger for the condensed dataset Se than for the full dataset SF.
Proof of Theorem 2.2: We compare the expected loss for the full and condensed datasets, taking into account their class imbalances. Let L(h) denote the loss function for the hypothesis h. Let ([L(h) S] denote the expected loss for the hypothesis h on the dataset S. Let n& and T&c denote the number of samples in class &o for datasets Sp and Sc, respectively. Let my and mc denote the total number of samples in datasets Sp and Sc, respectively. Let ri n&F nac be the class ratio for each class both datasets. The expected loss for Sp and So can written as:
E [[)())))))
Σ (((((((()) [[[[())]cc] = E Em.B(((()))))))4) (14)
To show this, let's compare the expected loss per sample in each dataset:
E([((h))Sc)
[ [[[())]Sp]]
(15)
mc
m.
5

---

This implies that the influence of the imbalanced class distribution is larger for Se than for Sp.
Rule 2: Balanced class distribution should be utilized for the condensed dataset. The construction of a balanced class distribution based on LBPE scores is outlined Algorithm [ create equal number of samples for each class ensure balanced dataset.
Its objective is to
Experiments
4.1 Experiment Settings
IPC stands for "Images Per Class". IPCF--T means flexibly resize the condensed dataset from size F to size T. More detailed settings can be found in Appendix B.I.
Dataset Condensation Settings. The CIFAR-10 10 and CIFAR- 100 datasets 20 are condensed via
ConvNet-D3 [12, and ImageNet-10 [5] via ResNet1O-AP [3]. both following IDC [I9]. IPC includes 10, 20, or 50, depending on the experiment. For both networks, the learning rate is 0.01 with 0.9 momentum and 0.0005 weight decay. The SGD optimizer and a multi-step learning rate scheduler are used. The training batch size is 64, and the network is trained for 2000 x 100 epochs for 10//IA- 100 and 500 x 100 epochs for ImageNet-10.
YOCO Settings. 1) LBPE score selection. To reduce computational costs, we derive the LBPE score from training dynamics of early E epochs. To reduce variance, we use the LBPE score from the top-K training epochs with the highest accuracy. For CIFAR-10, 10, we set E = 100 and K = 10 for all the IPCr and IP For CIFAR 100 and ImageNet- 10, we set E 200 and K 10 for all the IPCF and I. 2) Balanced construction. We use S in Eq. to achieve a balanced construction. Following IDC [9. we leverage a multi-formation framework to increase the synthetic data quantity while preserving the storage budget. Specifically, an IDC-condensed image is composed of n² patches. Each patch is derived from one original image with the resolution scaled down by factor of 1/n2. Here, n is referred to as the "factor" in the multi-formation process. For CIFAR-10 10 and CIFAR- 100 datasets, n 2; for ImageNet-Il dataset, n = 3. We create balanced classes according to these patches. As result, all the classes have the same number of samples. 3) Flexible resizing. For datasets with IPCp = 10 and IPCp = 20, we select IPCT of 1, 2, and 5. For IPCf = 50, we select IPCr of 2. 5, and 10. For condensed dataset with IPCr, the performance of its flexible resizing indicated by the average accuracy across different IPCr values.
Comparison Baselines. We have two sets of baselines for comparison: 1) dataset condensation
methods including [DC[9] DREAM 24, MTT (T, DSA (50) and KIP [32] and 2) dataset pruning
methods including SSP 40 Entropy 3 AUM 135 Forgetting [4], EL.2N [34, and CCS [54] For dataset condensation methods, we use random subset as the baseline. For dataset pruning methods,
their specific metrics are used to rank and prune datasets to the required size.
4.2 Primary Results
Tab. M provides a comprehensive comparison of different methods for flexibly resizing datasets from an initial IPCr to target IPCr. In this table, we have not included ImageNet results on DREAM (24] since it only reports on Tiny ImageNet with a resolution of 64x64, in contrast to ImageNet's 224x224. The third column of the table shows the accuracy of the condensed dataset at the parameter IPCp. We then flexibly resize the dataset from IPCF to IPCT. The blue area represents average accuracy across different IPCT values. For instance, consider the CIFAR-10 dataset with IPCp = 10. Resizing it to IPCF = 1, 2, and 5 using our method yields accuracies of 42.28% 46.67%, and 55.96%, respectively. The average accuracy of these three values 48.30%. This value surpasses the 37. 37.08% accuracy of SSP (4) by considerable margin of 11.22%.
Ablation Study. Tab. 2 shows the ablation study of the LBPE score and the balanced construction the first
Table Ablation study on two rules. (CIFAR- 10: IPC10-1)
across dataset condensation methods. row, the baseline results are shown where neither the LBPE score nor the balanced construction is applied. "Balanced only" (second row) indicates the selection method is random selection and the class distribution is balanced. "LBPE only" (third row) means the
LBPE Balanced IDC (M]
DREAM 24]
MTT i1
KIP B
28.23
30.87
19.75
14.06
30.19
32.83
19.09
16.27
;
39.38
37.30
20.37
15.78
42.28
42.29
22.02
22.24

---

Table
IPC means "images per class". Flexibly resize dataset from IPCr to IPCr (IPCF-r)) The areas represent the average accuracy of listed 1 datasets for different values of T. The areas indicate the accuracy difference between the corresponding methods and ours.
blue
gray
Condensation
Pruning Method
Dataset
IPCF
Acc.
JPCT
[DC
DREAM
Entropy
AUM 35
Forg
Ours
28.23
30.87
27.83
30.30
13.30
16.68
16.95
33.54
42.28
2
37.10
38.88
34.95
38.88
18.44
22.13
23.26
39.20
46.67
10
67, .50
5
52.92
54.23
48.47
52.85
41.40
45.49
46.58
53.23
55.96
39.42
41.35
37.08
40.68
24.38
28.10
28.93
41.99
48.30
DilL.
-8.89
-6.98
-11.22
-7.63
-23.92
-20.20
-19.37
-6.31
CIFAR
29.45
27.61
28.99
17.95
7.21
12.23
7.95
31.28
38.77
34.27
36.11
34.51
24.46
8.67
12.17
9.47
38.71
44.54
45.85
48.28
46.38
34.12
12.85
15.55
16.03
48.19
53.04
50 74.50
10
57.71
59.11
56.81
47.61
22.92
27.01
31.33
56.80
61.10
Avg.
16.74
49.36
Diff.
-7.54
-6.58
-7.69
-18.33
-36.45
-32622
-33.17
-5.62
1
14.78
15.05
14,94
11.28
3.64
6.45
5.12
18.97
22.57
2
22.49
21.78
20.65
16.78
5.93
10.03
8.15
25.27
29.09
Avg.
24.06
24.12
22.02
19.34
8.96
12,64
11.89
26.75
30.06
-10.72
13.92
13.26
14.65
5.75
2.96
7.59
4.59
18.72
23.74
20.62
20.41
20.27
8.63
3.96
10.64
6.18
24.08
29.93
CIFAR-100
20
49.50
5
31.21
31.81
30.34
17.51
8.25
17.63
11.76
32.81
38.02
Diff.
-8.65
-8.74
-8.81
-19.93
-25.51
-18.61
-23.05
-5.36
13.41
13.36
15,90
1.86
2.79
9.03
4.21
19.05
23.47
2
20.38
19.97
21.26
2.86
3.04
12.66
5.01
24.32
29.59
29.92
29.63
37.52
50
52.60
10
37.79
37.85
36.97
13.31
8.56
29.11
11.72
38.05
42.79
Avg.
17.76
Diff.
-7.97
-8.08
-7,40
-27.33
-28.61
-15.59
-26.30
-5.01
44.93
45.69
40.98
17.84
32.07
41.00
44.27
53,91
2
$7.84
58.47
$2.04
29.13
44,89
54.47
56.53
59.69
Avg.
56.66
55.76
52.54
30:51
44.03
53.78
56.05
59.36
Dill.
2.70
3.60
6.82
28.85
15.33
5.58
3.31
ImageNet-10
1
42,00
43.13
36.13
14.51
24.98
24.09
34.64
53.07
54.82
20
76.60
5
$9.56
61.27
56.44
27.78
36.44
46.02
57.11
64.38
Avg.
51.83
53.07
30.90
34.42
58.80
Ddf.
6.97
5.73
12.31
38.34
27.90
24.38
14.14
LBPE score is used for ranking samples, and the selection results purely based on the LBPE score without considering the class balance. "LBPE Balanced" indicates that both elements are included for sample selection. The empirical findings conclusively affirm the effectiveness of the two rules, which constitute the principal contributions our YOCO method.
Standard Deviation of Experiments. Different training dynamics and network initializations impact the final results. Therefore. the reported results are averaged over three different training dynamics, and each training dynamic is evaluated based on three different network initializations. See Appendix B.2 for the primary results table with standard deviation.
4.3 Analysis of Two Rules
4.3.1 Analysis of LBPE Score for Sample Ranking
Tab. illustrates the robust performance of our YOCO method across diverse network structures, including ConvNet, ResNet, and DenseNet, demonstrating its strong generalization ability. Addition- ally, we present different sample ranking metrics from dataset pruning methods on these networks, demonstrating that method outperforms both random selection and other data pruning methods.
In Tab. we experiment with prioritizing easy samples over hard ones. We achieve this by reversing the importance metrics introduced by AUM [35], Forg. 41,, and EL2N [34] that originally prioritize

---

Table 3: Accuracies on different network structures and different sample ranking metrics. (IDC [9 condensed CIFAR-
Table Prioritizing easy samples is better for differ- ent dataset pruning and dataset condensation methods. "R?"" represents whether to reverse the metrics which prioritize hard samples. (CIFAR-10) 10: 1PC10->1)
10: 10)
ConvNet 12
ResNei 13
DenseNet
14
Method
R?
IDC
DREAM 24
MTT
DSA
Random
14.25
SSP
27.83
24.64
24.75
AUM
65
37.97
38.18
16.63
18.23
Eatropy
30.30
30.53
29.93
Forg
11
16.68
16.26
18.82
16.55
AUM
13.30
15.04
14.56
Forg.
41
36.69
36.15
16.65
17.03
Forg H1]
16.68
16.75
17.43
EL2N
16.95
18.13
16.98
13.14
Ours
42.28
34.53
34.29
Ours
42.28
42.29
22,02
22.40
Table 5: Balanced construction works on different dataset pruning methods. "B?" represents whether to use balanced construction. The subscript vvalue indicates the accuracy gain from balanced construction. (IDC 19 condensed CIFAR-IO: I0C1o-T)
1
B?
Random
4D)
Entrpy
AUM 35
Forg. 41
EL2N
Ours
IPCI
/
28.23
27.83
30.30
13.30
16.68
16.95
37.63
30.05+1.82
33.21 3s
33.67 37
15.64 34
19.09
18.43 48
42.28.44.85
IPC2
/ 39.44
37.10
40.57
34,95
42.17
38.88
23.84 40
18.44
28.06 6.08
22.13
26.54 3.
23,26
46.67 3.08
42.99
52.92
48.47
52.85
41.40
45.49
46.58
IPCS
/
52.64_...28
49.44.4
54.73.41.88
47.23
48.02
48.86
$5.96.2..10
hard samples. Our results indicate that across various condensed datasets, including IDC DREAM (24), MTT [1, and DSA [50], there is distinct advantage in prioritizing easier samples over harder ones. These findings lend support to our Rule
19
4.3.2 Analysis of Balanced Construction
Fig. presents the class distributions with and without a balanced construction for different datasets and different IPC As explained in YOCO settings, balanced construction is based the multi-formation framework from IDC 19) Therefore, the x-axis represents the count of images after multi-formation instead of the condensed images. is evident that ranking strategy relying solely on the LBPE score can result in a significant class imbalance, particularly severe in the ImageNet dataset. As depicted in Fig. 2
(f), three classes have no image patches left. Our balanced construction method effectively mitigates this issue. Notably, in the case of ImageNet-10 10 the balanced construction boosts the accuracy by an impressive 19.37%.
To better understand the impact of balanced class distribution on various dataset pruning methods, conducted comparative analysis, presented in Tab. Clearly, achieving balanced class distribution significantly enhances the performance all examined methods. Remarkably, our proposed method consistently outperforms others under both imbalanced and balanced class scenarios, further substantiating the efficacy of our approach.
4.4 Other Analysis
Sample Importance Rules Differ between Condensed Dataset and Full Dataset. In Fig. B compare Sample Importance Rules for Condensed Datasets (IPC IPCso) and the Full Dataset (IPCs000), by adjusting the pruning ratio from 10% to 90%. Unmarked solid lines mean prioritizing easy samples, dashed lines suggest prioritizing hard samples, while marked solid lines depict the accuracy disparity between the preceding accuracies. Therefore, the grey region above zero indi- cates "Prefer easy samples" (Rule while the blue region below zero represents "Prefer hard samples" (Rulees"(R We have two observations. First, as the pruning ratio increases, there is gradual transition from Rulehard Rule Second, the turning point of this transition depends on the dataset size. Specifically, the turning points for IPC IPCso, and IPCo000 occur at pruning
we

---

Relatced
wnbatoneed
Inbaanod
IPCID Easy IP5S Easy
Ircsece pull Easy
IPC2D Harz
inc2 Hard
pull Hard
IPCor > O: Prifer easy samples
Dividing line for IPCsn IPCaaay IPCaswo
IPCor Profer hard samples
90
Count
Count
Count
80
(a) CIFAR- 1010-1
(b) CIFAR-1000 1010-22
(c) CIFAR-1010--
70
38.77%
30.70%
24.35%
22.22%
53 53.9%%
34.90%
60-
Clasess
40-
20-
10
10
122 2015 20.2
10
Count
(d) CIFAR-1050-11
(c) CIPAR-10010-11 (f) ImageNet-101o..
10
-0 40
30 70 80
Dataset pruning ratio (96)
Figure 2: Balanced and imbalanced selection by ranking samples with LBPE score. D denotes resizing the dataset from IPCp IPCr. Accuracies for each setting also listed in the legend. (IDC [19 condensed datasets)
Figure 3: Different sample im- portance rules between condensed datasets and full datasets.
ratios of 24%, 38%, and 72%, respectively. These experimental outcomes substantiate our Rule 1 that condensed datasets should adhere to Rulegasy-
Performance Gap from Multi-formation. We would like to explain the huge performance gap be- tween multi-formation-basee methods (IDC [19 and DREAM (44) and other methods (MTT [1, KIP (33 and DSA (50) in Tab. P and Tab. The potential
Uniform
formation
reason that single image can be decoded to 2 4 low-resolution images via multi nulli-formation. As a
Condensed data
Synthetic data
sult, methods employing multi-formation generate four times many images compared those that do not use multi-formation. The illustration is shown in Fig.
Figure 4: Illustration of the multi-formation
with factor of 2. (Taken from IDC [19]
Why Use LBPE Score from the Top-K K Training Epochs with the Highest Accuracy? As shown in Eq. 2 different training epoch / leads to different LBPE score. Fig- iillutrisssss the accuracy of the dataset selected via the LBPE score across specific training epochs. We select L.PE scores from the initial 100 epochs out of 1000 original epochs to reduce computational costs. We have two observations. First, the model's 's accuracy during the first few epochs is substantially low. LBPE scores derived from these carly-stage epochs might not accurately represent the samples' true importance since the model is insufficiently trained. Second, there's significant variance in accuracy even after 40 epochs, leading to potential instability in the LBPE score selection. To address this, average LBPE scores from epochs with top-K K accuracy, thereby reducing variability and ensuring a more reliable sample importance representation.
Speculating on Why LBPE Score Performs Better at Certain Epochs? Fig. we present the distribution of LBPE scores at various training epochs, with scores arranged in ascending order for each class to facilitate comparison across epochs. Our experiment finds the LBPE scores decrease as the epoch number increases. The superior accuracy of LBPE30 is due to two reasons. First, the model at the 90,h epoch is more thoroughly trained than the model at the first epoch, leading to more accurate LBPE scores. Second, the LBPEgo score offers more uniform distribution and wider range [O, 1], enhancing sample distinction. In contrast, the LBPE1000 score is mostly concentrated within narrow range [0, 0.1] for the majority of classes, limiting differentiation among samples. More possible reasons will be explored in future studies.
Visualization. Fig. 6 visualizes the easy and hard samples identified by our YOCO method. We notice that most easy samples have distinct demarcation between the object and its background. This is particularly evident in the classes of "vacuum cleaner" and "cocktail shaker". The easy samples in

---

42-
parti
green manite
bangin
Deseren
pinechar
miccina
VAsscum
windon
Glarden spitter
Exsy Original
Accuracy (8)
36
Condlerion
Original
10
Ward Condersed
40
80 100
Which Epoch for LaPE Score
Figure 5: Accuracy of the dataset selected with LBPE score at specific epochs.
Figure 6: Visualization hard and easy samples of ImageNet dataset selected by method. Both the original and condensed images are shown for comparison.
class 0: airplane
class 1: automobile
class 2: bird
class 3: cat
class 4: deer
1.0
1.0
1.0
Score
0.5
0.5
0.0
0.0
0.0
0.0
20
40
20
40
20
40
20
40
20
40
Sorted sample index according scores
class dog
class frog
class horse
class 8: ship
class 9: truck
1.0
1.0
1.0
Score
0.5
0.5
0.5
0.0
20
according
sample
Epoch 1 28. 32%
Epoch 90 42.11%
Epoch 200 39.09%
Epoch 1000
32 32.25%
Figure 7: LBPE scores at different epochs (LBPE for ten classes of the CIFAR-10 dataset.
these two classes have clean backgrounds, while the hard samples have complex backgrounds. The visualization provides evidence of our method's ability to identify easy and hard samples.
5 Conclusion, Limitation and Future Work
We introduce You Only Condense Once (YOCO), novel approach that resizes condensed datasets flexibly without an extra condensation process, enabling them to adjust to varying computational constraints. YOCO comprises two key rules. First, YOCO employs the Logit- Based Prediction Error (LBPE) score to rank the importance of training samples and emphasizes the benefit of prioritizing easy samples with low LBPE scores. Second, YOCO underscores the need to address the class imbalance in condensed datasets and utilizes Balanced Construction to solve the problem. Our experiments validated YOCO's effectiveness across different networks and datasets. These insights offer valuable directions for future dataset condensation and dataset pruning research.
We acknowledge several limitations and potential areas for further investigation. First, although method uses early training epochs to reduce computational costs, determining the sample importance in the first few training epochs even before training is interesting for future work. Second, we only utilize the LBPE score to establish the importance of samples within the dataset. However, relying on single metric might not be the optimal approach. There other importance metrics, such as SSP [40] and AUM (35,, that could beneficial to integrate into our methodology. Third, as our current work only covers clean datasets like CIFAR-10, the performance of method noisy datasets requires further investigation.
The border impact is shown in Appendix C.
10

---

6 Acknowledgement
This work is partially supported by Joey Tianyi Zhou's A*STAR SERC Central Research Fund (Use- inspired Basic Research), the Singapore Government's 's Research, Innovation and enterprise 2020 Plan (Advanced Manufacturing and Engineering domain) under Grant A AI8AI Ib0045, and A*STAR CFAR Internship Award for Research Excellence (CIARE). The computational work for this article was partially performed on resources of the National Supercomputing Centre (NSCC), Singapore (https://ww/ http://wwwwnnccc.cc ng)...g).
References
[1] G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[2] K. Chitta, J. M. Álvarez, E. Haussmann, and C. Farabet. Training data subset search with en- semble active learning. IEEE Transactions on Intelligent Transportation Systems, 23(9):1474) 14752, 2021.
[3] C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia. Selection via proxy: Efficient data selection for deep learning. In Proc. Int. Conf. Learn. Represent., 2020.
[4] J. Cui, R. Wang, S. Si, and C.-J. Hsich. Scaling up dataset distillation to imagenet- 1k with constant memory. arXiv preprint arXiv:22/1. 2022.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 248-255, 2009.
[6] Z. Deng and O. Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In Proc. Adv. Neural Inform. Process. Syst., 2022.
[7] J. Du Y. Jiang, V. T. Tan, J. T. Zhou, and H. Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proc. IEEE Conf. Comput. Vis. Puttern Recog., 2023.
[8] M. Ducoffe and F. Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv: 1802.0940, 2018
[9] D. Feldman and M. Langberg. A unified framework for approximating and clustering data. In Proceedings of the Forty-Third Annual ACM Symposium on Theory of Computing, page 569-578, 2011.
[10] D. Feldman, M. Schmidt, and C. Sohler. Turning big data into tiny data: Constant-size coresets for k-means, pea, and projective clustering. SIAM Journal on Computing, 49(3):601-657, 2020.
[11] V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Proc. Adv. Neural Inform. Process. Syst., pages 2881-2891, 2020.
[12] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4367-4375, 2018.
[13] K. He, Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pages 770-778, 2016.
[14] G. Huang, 7. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4700-4708, 2017.
[15] L. Huang and N. K. Vishnoi. Coresets for clustering in euclidean spaces: Importance sampling is nearly optimal. Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, page 1416-1429, 2020.
[16] 7. Jiang, J. Gu, M. Liu, and D. Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208.00311 2022.

---

[17] K. Killamsetty, S. Durga, G. Ramakrishnan, A. De, and R. Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In Proc. lnt. Conf. Mach. Learn., pages
2021.
[18] K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proc. AAAI Conf. Artif. Intell., pages 81 1110-8118. 8118, 2021.
[19] J.-H. Kim, Kim, S. J. Oh, S. Yun, H. Song, J. Jeong, J.-W. Ha, and H. O. Song. Dataset condensation via efficient synthetic-date parameterization. In Proc. Int. Conf. Mach. Learn., 2022.
[20] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[21] S. Lee, S. Chun, S. Jung, S. Yun, and S. Yoon. Dataset condensation with contrastive signals. In Proc. Int. Conf. Mach. Learn., pages 12352-12364, 2022.
[22] D. D. Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Aem Sigir Forum, pages 13-19, 1995.
[23] S. Liu, K. Wang, X. Yang, Ye, and X. Wang. Dataset distillation via factorization. In Proc. Adv. Neural Inform. Process. Syst., 2022.
[24] Y. Liu, J. Gu, K. Wang. Z. Zhu, W. Jiang, and You. DREAM: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302.144111 2023.
[25] N. Loo, R. Hasani, A. Amini, and D. Rus. Efficient dataset distillation using random feature approximation. Proc. Adv. Neural Inform. Process. Syst., 2022.
[26] N. Loo, R. Hasani, M. Lechner, and D. Rus. Dataset distillation with convexified implicit gradients. arXiv preprint 230iv:2302.06755 2023.
[27] Lorraine, Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pages 1540-1552, 2020.
[28] K. Margatina, G. Vemikos, L. Barrault, and N. Aletras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 650-663, 2021.
[29] K. Meding, L. M. S. Buschoff, R. Geirhos, and F, A. Wichmann. Trivial or impossible dichotomous data difficulty masks model differences (on imagenet and beyond). In Proc. Int. Conf. Learn. Represent., 2022.
[30] B. Mirzasoleiman, J. Bilmes, and J. Leskovec. Coresets for data-efficient training of machine learning models. Proc. Int. Conf. Mach. Learn., pages 1960, 2020.
[31] M. Mohri and A. Rostamizadeh. Rademacher complexity bounds for non-iid processes. In Proc. Adv. Neural Inform. Process. Syst., 2008
[32] Nguyen, Z. Chen, and J. Lee. Dataset meta-learning from kernel ridge-regression. In Proc. Int. Conf. Learn. Represent., 2021.
[33] T. Nguyen, R. Novak, L. Xiao, and J. Lee. Dataset distillation with infinitely wide convolutional networks. Proc. Ady. Neural Inform. Process. Syst., pages 5186-5198, 2021.
[34] M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Proc. Adv. Neural Inform. Process. Syst., pages 20596-20607, 2021.
[35] G. Pleiss, T. Zhang, E. Elenberg, and K. Q, Weinberger. Identifying mislabeled data using the area under the margin ranking. Proc. Adv. Neural Inform. Process. Syst., pages 17044-17056. 2020.
12

---

[36] O. Pooladzandi, D. Davini, and B. Mirzasoleiman. Adaptive second order coresets for data- efficient machine learning. In Proc. Int. Conf. Mach. Learn., pages 17848-17866 2022.
[37] Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. Gupta, Chen, and X. Wang. survey of deep active learning, ACM Comput, Surv. 54(9), oct 2021.
[38] B. Settles. Active Learning. Springer International Publishing, 2012.
[39] S. Shin, H. Bae, D. Shin, W. Joo, and I.-C. Moon. Loss-curvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics, pages 1606-8628, 2023.
[40] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Proc. Adv. Neural Inform. Process. Syst., pages 15523-19536 2022.
[41] M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. Gordon. An empirical study of example forgetting during deep neural network learning. In Proc. Int. Conf. Learn. Represent., 2019.
[42] P. Vicol, J. P. Lorraine, F. Pedregosa, D. Duvenaud, and R. B. Grosse. On implicit bias in overparameterized bilevel optimization. In Proc. Int. Conf. Mach. Learn., pages 22234-22259, 2022.
[43] K. Wang, J. Gu, D. Zhou, Z. Zhu, W. Jiang. and You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303 04707, 2023.
[44] K. Wang, B. Zhao, X. Peng, Z. Zhu, S. Yang. S. Wang, G. Huang, H. Bilen, X. Wang, and
Y. You. Cafe: Learning condense dataset by aligning features. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pages 22196-12205, 2022.
[45] T. Wang. J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint 18Xiv:1811. 10959, 2018.
[46] M. Welling. Herding dynamical weights to learn. In Proc. Int. Conf. Mach. Learn., pages 1121-1128, 2009.
[47] X. Xia, J. Liu, J. Yu, X. Shen, B. Han, and T. Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In Proc. Int. Conf. Learn. Represent., 2023.
[48] S. Yang, Z. Xie, H. Peng, M. Xu, M. Sun, and P. Li. Dataset pruning: Reducing training data by examining generalization influence. In Proc. Int. Conf. Learn. Represent., 2023.
[49] L. Zhang, J. Zhang, B. Lei, S. Mukherjee, X. Pan, B. Zhao, C. Ding, Y. Li, and D. Xu. Accelerating dataset distillation via model augmentation. In Proc. IEEE Conf. Comput. Vis. Puttern Recog., 2023.
[50] B. Zhao and H. Bilen. Dataset condensation with differentiable siamese augmentation. In Proc. Int. Conf. Mach. Learn., pages 12674-12685, 2021.
[51] B. Zhao and Bilen. Synthesizing informative training samples with GAN. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022.
[52] B. Zhao and H. Bilen. Dataset condensation with distribution matching. In Proc. IEEE Winter Conf. Appl. Comput. Vis., pages 6514-6523, 2023.
[53] B. Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. In Proc. Int. Conf. Learn. Represent., 2021.
[54] H. Zheng, R. Liu, F. Lai, and A. Prakash. Coverage-centric coreset selection for high pruning rates. In Proc. Int. Conf. Learn. Represent, 2023.
[55] Y. Zhou, Nezhadarya, and Ba. Dataset distillation using neural feature regression, In Proc. Adv. Neural Inform. Process. Syst., 2022.
13


Reviews:
[{"review_id": "CMoF5RWb2v", "reviewer": "Reviewer_h1cK", "summary": "--- Post-Rebuttal Edit ---\n\nAfter author rebuttals, I have updated my rating from 5 to 6 and confidence from 2 to 3.\n\n--- End Post-Rebuttal Edit ---\n\nThis paper considers a practical problem in dataset condensation/distillation, where edge devices have varying constraints that need to be flexibly met by a data distillation method. The authors propose two rules to prune the condensed datasets in order to save computational costs. The first rule uses logit-based prediction error (LBPE) to identify useful examples, where low LBPE samples are useful for smaller datasets and high LBPE samples are useful for larger datasets. The second rule focuses on the construction of balanced classes by ensuring an equal number of samples in each class via Rademacher complexity and generalization error. ", "questions": "- What is an “on-device scenario?” What is the motivation for storing a training set (rather than, say, the trained model weights) on said device?\n- What is the motivation for condensing an already-condensed dataset?\n- How does this method avoid extra condensation steps if all experiments are performed after an initial condensation step with another method?\n    - Why not perform a single condensation step with the proposed method (from full -> desired size)? How does the method perform in this setting?\n", "limitations": "- See Questions above\n- There is no clear definition of what is meant by “small” and “large” datasets. A definition or rough empirical guideline would be useful.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing is generally easy to follow and the experiments are thorough.\n- The proposed approaches are logical and appear useful. The first rule based on LBPE scores allows for flexible treatment based on dataset size and other characteristics; the second rule incorporates Rademacher complexity in a novel way in order to set up a theoretical foundation for the significance of balanced class construction.\n", "weaknesses": "- Perhaps this is because I am unfamiliar with data distillation literature, but the motivation and setting are somewhat unclear to me. There is not a single citation or brief explanation of what is meant by “on-device scenarios.” Even if this is obvious to well-informed readers, I believe that at minimum a citation or definition is needed.\n    - In what scenario would one need to store a training set on a device? Typically when deploying a machine learning model to a hardware-constrained device, the only thing being deployed is the model weights.\n- Related to the above point, I don’t understand the motivation for the experimental setting, where we are condensing an already-condensed dataset. If one of the central motivations mentioned in the Abstract is to “[eliminate] the need for extra condensation processes,” then I don’t see how this method achieves said goal – we are still ultimately performing two condensation steps!\n- The mixture of theoretical- and prose-style writing is awkward at times. For example, Theorem 1 on line 122 would be much more appropriately written as simple explanatory text. I don’t understand in what sense this is a theorem or how lines 123-124 constitute a “proof” for this claim. There is no need to present content in a mathematical style when it is not necessary. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and for raising concerns about the clarity of the motivation and the terminology used in our paper.  \n\n>  1. What is an “on-device scenario?” \n\nWe acknowledge that the term \"on-device scenarios\" may not be clear to all readers. What we mean by this phrase is the application of machine learning in environments where both the model training and inference occur on a local device, such as a smartphone or embedded system, without relying on cloud-based resources. We added a few citations [1-7] here and will make sure to include them in the revised manuscript. \nThe key advantages of on-devices learning can be explained as follows:\n\n1. **Continual Adaptation to New Data**: By leveraging on-device learning, edge devices can continually adapt the AI models to new data. This provides a dynamic and flexible system that can evolve with changing data patterns and user behaviors.\n2. **Privacy Protection**: Since the training and adaptation of the model occur directly on the device, there is no need to transfer data to the cloud, thereby reducing the risk of unauthorized access or breaches. \n\n> 2. What is the motivation for storing a training set (rather than, say, the trained model weights) on said device?\n\nThe scenario you pointed out, where only the model weights are deployed to a hardware-constrained device, is indeed a common one. However, in some on-device scenarios such as:\n\n1. Incremental learning: adapting the model to new data as it comes in.  \n2. Federated learning: decentralized training\n\nIn such cases, having a distilled or compressed version of the training set on the device can be valuable. Dataset condensation allows us to retain essential information while minimizing storage requirements. We will include a more detailed explanation of the specific scenarios where on-device storage of a training set might be needed.\n\n\n> 3. What is the motivation for condensing an already-condensed dataset?\n\n1. **Limitations in Computational Power**: Existing methods may only reduce datasets to specific IPCs, and further condensation is needed for devices with lower computational abilities.\n2. **Avoiding Extra Processes and Storage**: Traditional further condensation can cause performance losses or require additional complex processes and storage.\nTo overcome these challenges, dataset pruning is used to find a representative subset without changing image pixels or needing extra storage, effectively eliminating extra condensation steps.\n\n\n> 4. How does this method avoid extra condensation steps if all experiments are performed after an initial condensation step with another method? Why not perform a single condensation step with the proposed method (from full -> desired size)? How does the method perform in this setting?\n\nThe reason is that (from full -> desired size) is not **flexible**. If we condense the dataset to IPC5, we would not have a dataset of IPC10 if more computing resources are available. With our YOCO method, both IPC5 and IPC10 can be easily found as a subset of IPC50.\n\n\n> 5. There is no clear definition of what is meant by “small” and “large” datasets. A definition or rough empirical guideline would be useful.\n\nPlease take a look at Fig. 4 and section 4.4.\n\n### Reference:\n\n[1] Cai, Han, et al. \"TinyTL: Reduce memory, not parameters for efficient on-device learning.\" _Advances in Neural Information Processing Systems_ 33 (2020): 11285-11297.\n\n[2] Lin, Ji, et al. \"On-device training under 256kb memory.\" _Advances in Neural Information Processing Systems_ 35 (2022): 22941-22954.\n\n[3] Yang, Li, Adnan Siraj Rakin, and Deliang Fan. \"Rep-net: Efficient on-device learning via feature reprogramming.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2022.\n\n[4] Yang, Yuedong, Guihong Li, and Radu Marculescu. \"Efficient On-device Training via Gradient Filtering.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2023.\n\n[5] Qiu, Xinchi, et al. \"ZeroFL: Efficient on-device training for federated learning with local sparsity.\" _International Conference on Learning Representations_ (2022).\n\n[6] Lee, Jinsu, and Hoi-Jun Yoo. \"An overview of energy-efficient hardware accelerators for on-device deep-neural-network training.\" _IEEE Open Journal of the Solid-State Circuits Society_ 1 (2021): 115-128.\n\n[7] Dhar, Sauptik, et al. \"A survey of on-device machine learning: An algorithms and learning theory perspective.\" _ACM Transactions on Internet of Things_ 2.3 (2021): 1-49.\n\n\n"}, {"review_id": "S97BWThvpf", "reviewer": "Reviewer_9GkH", "summary": "This paper employs a dataset pruning approach on a condensed dataset. Specifically, the intent is to evaluate and rank the data within this condensed dataset based on their respective significance. This enables users to selectively choose any number of condensed datasets, offering both efficacy and flexibility in training the model.", "questions": "Please refer to Weakness. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper claims that it's the first work to conduct dataset pruning on condensed datasets to fit varying computational constraints.\nLBPE score is a reasonable metric to measure the importance of data in the condensed dataset.\n", "weaknesses": "1. The main weakness is the evaluation part. This paper compares their method: (1). dataset condensation + dataset pruning with the previous method (2). dataset condensation + random selection (line 202), which is not fair. For example, it's better to show the effectiveness of the proposed method by comparing (1) with IPCF = 10 to IPCT=1 or 2 or 5 with (3): directly generating the dataset condensation with the IPC=1 or 2 or 5. If it can get similar performance, this paper can claim its advantages of training time. \n\n2. It's unclear—the whole pipeline in Fig 1 or the rest of the paper. I can only understand the paper after reading the Appendix. It's more critical than putting the algorithm part in the main body. \n\n3. Proof of Theorem 1 is not solid. \n\n4. The imbalanced part is not thoroughly analyzed. The equal number of each class is a standard setting in the condensed dataset. This paper does not provide new insight from Section 3.3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments regarding the evaluation section of our paper. We will explain your questions point by point.\n\n> 1. The main weakness is the evaluation part. This paper compares their method: (1). dataset condensation + dataset pruning with the previous method (2). dataset condensation + random selection (line 202), which is not fair. For example, it's better to show the effectiveness of the proposed method by comparing (1) with IPCF = 10 to IPCT=1 or 2 or 5 with (3): directly generating the dataset condensation with the IPC=1 or 2 or 5. If it can get similar performance, this paper can claim its advantages of training time.\n\nThanks for your question. We will explain our comparison is fair in two parts.\n\n### Comparison with dataset pruning method\nWe would like to clarify that this paper, as the title suggests, utilizes **dataset pruning** techniques to fit condensed datasets on various computational budgets. So we compare the results with **dataset pruning** methods, including SSP [1],  Entropy [2], AUM [3], Forg. [4], and EL2N [5] in **Table 1** and **Table 5**. It looks like the reviewer ignored these comparisons. We want to emphasize that these comparisons are fair.\n\n### Comparison with dataset condensation method\nRegarding the comparison with dataset condensation methods, we believe it is also fair. In the table below, we list the required time for the condensation process on GPU 3090. We have these observations below:\n\nA. Condensation on IPC-10 requires 12 hrs 36 mins. \n\nB. Our method requires 12 hrs 37 mins. It consists of the 12 hrs 36 mins required by process A, and **only 1 minute required by our pruning method**.\n\nFollowing the suggestion from the reviewer that we should not use random selection as a baseline, we set the accuracies of other IPCs to zeros. We also include average accuracy over IPC from 1 to 10 as the metric. It is clear that our method improve the average accuracy by **+50.03%** (from 6.75% to 56.78%) with almost **the same training time** required compared to \"condense ipc10 only\".\n\n\n|                        | ipc1  | ipc2  | ipc3 | ipc4  | ipc5  | ipc6  | ipc7 | ipc8  | ipc9  | ipc10 | average | time            |   |\n|------------------------|-------|-------|------|-------|-------|-------|------|-------|-------|-------|---------|-----------------|---|\n| condense ipc1 only     | 50.8  | 0     | 0    | 0     | 0     | 0     | 0    | 0     | 0     | 0     | 5.08    | 11 hrs 46 mins  |   |\n| condense ipc2 only     | 0     | 54.8  | 0    | 0     | 0     | 0     | 0    | 0     | 0     | 0     | 5.48    | 11 hrs 48 mins  |   |\n| condense ipc3 only     | 0     | 0     | 59.8 | 0     | 0     | 0     | 0    | 0     | 0     | 0     | 5.98    | 11 hrs 50 mins  |   |\n| condense ipc4 only     | 0     | 0     | 0    | 61.8  | 0     | 0     | 0    | 0     | 0     | 0     | 6.18    | 11 hrs 51 mins  |   |\n| condense ipc5 only     | 0     | 0     | 0    | 0     | 62.5  | 0     | 0    | 0     | 0     | 0     | 6.25    | 11 hrs 53 mins  |   |\n| condense ipc6 only     | 0     | 0     | 0    | 0     | 0     | 64.6  | 0    | 0     | 0     | 0     | 6.46    | 12 hrs 02 mins  |   |\n| condense ipc7 only     | 0     | 0     | 0    | 0     | 0     | 0     | 65.5 | 0     | 0     | 0     | 6.55    | 12 hrs 19 mins  |   |\n| condense ipc8 only     | 0     | 0     | 0    | 0     | 0     | 0     | 0    | 66.3  | 0     | 0     | 6.63    | 12 hrs 23 mins  |   |\n| condense ipc9 only     | 0     | 0     | 0    | 0     | 0     | 0     | 0    | 0     | 66.8  | 0     | 6.68    | 12 hrs 33 mins  |   |\n| condense ipc10 only    | 0     | 0     | 0    | 0     | 0     | 0     | 0    | 0     | 0     | 67.5  | 6.75    | 12 hrs 36 mins  |   |\n| **ours**                   | 42.79 | 46.28 | 51.8 | 54.27 | 55.29 | 59.61 | 61.1 | 63.68 | 65.44 | 67.5  | **56.78**   | 12 hrs 37 mins  |   |\n\n\n\n### Reference\n\n[1] Beyond neural scaling laws: beating power law scaling via data pruning, NeurIPS 2022\n\n[2] Selection via Proxy: Efficient Data Selection for Deep Learning, ICLR 2020\n\n[3] Identifying Mislabeled Data using the Area Under the Margin Ranking, NeurIPS 2020\n\n[4] An Empirical Study of Example Forgetting during Deep Neural Network Learning, ICLR 2019\n\n[5] Deep Learning on a Data Diet: Finding Important Examples Early in Training, NeurIPS 2021\n\n\n> 2. It's unclear—the whole pipeline in Fig 1 or the rest of the paper. I can only understand the paper after reading the Appendix. It's more critical than putting the algorithm part in the main body.\n\nThanks for your question. But we are sorry that we don't quite understand this question.\n**Concerning Figure 1**: While we recognize that the reviewer found this figure unclear, we are unsure which part is unclear. As commented by Reviewer hLEg, \"The motivation is clearly explained in the figures.\"\n\n\n> 3. Proof of Theorem 1 is not solid.\n\nThanks for your question. But it's difficult for us to provide a response based on the general judgment of \"not solid\". Could you please provide more information?\n\n \n> 4. The imbalanced part is not thoroughly analyzed. The equal number of each class is a standard setting in the condensed dataset. This paper does not provide new insight from Section 3.3\n\nThanks for the clear question. Regarding the \" imbalanced part\", please take a look at **Section 4.3.2.**, **Table 3**, and **Figure 5**. We use these to analyze the imbalanced part. Could the reviewer please provide more information on which aspect of the analysis is missing for the imbalanced part?\n\nRegarding your comment \"The equal number of each class is a standard setting in the condensed dataset\", we agree with this. However, the equal number of each class is **NOT** a standard setting in **dataset pruning** methods. We find that the equal number of each class is quite useful for **pruning** condensed dataset. This is our new insight, and we sincerely hope the reviewer will recognize this."}, {"review_id": "5DYx8hEdiC", "reviewer": "Reviewer_hLEg", "summary": "This paper target at the dataset condensation for on-device scenarios.\nIt proposes two novel rules to prune the condensed dataset. The first rule is to rank the condensed images. The rule believes easy examples are better for pruning condensed dataset. The second rule is to choose balanced samples for different classes. The overall writing is clear. The experiments are sufficient.\n", "questions": "Please see the Weaknesses.", "limitations": "The limitation part is covered in section 5.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This is the first work to consider pruning the condensed dataset. The novelty is good.\n2. The illustration and writing are clear. The motivation is clearly explained in the figures.\n3. The paper provides solid proof for the LBPE metric for ranking the condensed images. \n4. The balanced construction is reasonable expecially when the dataset size is small.\n5. The experiment looks good. On CIFAR-10, CIFAR-100, and ImageNet, the proposed YOCO method surpasses other methods by a large margin. For example, on CIFAR-10, the improvements are from 6.57% to 24.33%. On CIFAR-100, the improvements are from 4.77% to 22.13%.\n6. The analysis is interesting. It's interesting to see that the rules are different between the condensed dataset and the full dataset. ", "weaknesses": "1. The LBPE score and balanced construction. Which rule is more important? \n2. If we view the condensed dataset as a full dataset, will the rules change? Is there any threshold for when the rules will change?\n3. The LBPE Score is from the Top-K Training Epochs with the Highest Accuracy. Why not use the loss to extract the LBPE score?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your high score, and we'd like to answer your question below.\n\n> 1. The LBPE score and balanced construction. Which rule is more important?\n\nThanks for the question. Table 2 in the paper provides an ablation study on the importance of LBPE score and Balanced Construction. These results indicate a higher importance of LBPE score over balanced construction.\n\n> 2.  If we view the condensed dataset as a full dataset, will the rules change? Is there any threshold for when the rules will change?\n\nThanks for the question.  In our setting, we indeed view the \"condensed dataset\" IPCF as the \"full dataset\". As reported in line 261-262, the threshold is 24%, 38%, and 72% of total data pruned for condensed CIFAR10 IPCF=10, IPCF=50, and full CIFAR10 dataset, respectively. The threshold tends to occur at a large pruning ratio if the dataset is large.\n \n> 3.  The LBPE Score is from the Top-K Training Epochs with the Highest Accuracy. Why not use the loss to extract the LBPE score?\n\nThanks for the question. We agree that the loss could be used to extract the LBPE score. In the following two tables, we compare the performance for using \"accuracy\" and \"loss\" as metrics.\n\nTable 1. CIFAR10 IPCF=10\n\n| (IPC10)  | ipc1  | ipc2  | ipc5  | average |\n| -------- | ----- | ----- | ----- | ------- |\n| accuracy | 42.79 | 46.28 | 55.29 | 48.12   |\n| loss     | 41.42 | 47.12 | 55.96 | 48.16   |\n\n\nTable 2. CIFAR10 IPCF=50\n\n| (IPC50)  | ipc1  | ipc2  | ipc5  | ipc10 | average |\n| -------- | ----- | ----- | ----- | ----- | ------- |\n| accuracy | 39.88 | 45.68 | 53.28 | 60.31 | 49.79   |\n| loss     | 37.01 | 45.47 | 53.86 | 60.96 | 48.16   | \n\nThe results show that these two metrics achieve similar performance on small IPCF, but  \"accuracy\" metric is better on large IPCF."}, {"review_id": "dtcj3Ub8q3", "reviewer": "Reviewer_hpsG", "summary": "This paper proposes a combination method for dataset pruning and dataset condensation. Both methods could reduce the size of training datasets. Experiments show that the proposed method achieves state-of-the-art performance on several datasets regardless of networks. The proposed rules are simple and effective, which might be commonly used in future works. ", "questions": "- From Figure 1, previous methods require 200K *(N-1) epochs for extra condensation. Is there any way to reduce the extra condensation process without using dataset pruning? Dataset Pruning is just like a selection method.\n\n- Will the code be open-sourced?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper studies an interesting yet rarely studied problem, which is important for on-device scenarios. In addition, the authors have comprehensively analyze the challenges of this problem and the drawback of existing methods.\n\n- This paper is well-written and easy to follow. I generally enjoy reading the story, motivation, method and experiments of this paper.\n\n- Two novel, simple, and effective rules/metrics are introduced to select informative and balanced samples. Importantly, with this two rules, one can carry on various size requirements by one process.\n\n- Extensive experiments, including results and visualization, demonstrate the effectiveness and efficiency of the proposed method, and show the superiority over the state-of-the-art methods.\n", "weaknesses": "Generally, I think the paper is strong. However, I have the following questions.\n\n- In Table 1, the ImageNet-10 on the DREAM method is not listed.\n\n- Some proofs, such as Proof of Theorem 1, can be shortened.\n\n- The core idea is first to condense the dataset before pruning the dataset. How about pruning the dataset first and then condensing it? Which way is better and why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your comment. We will answer the questions below.\n\n>1. In Table 1, the ImageNet-10 on the DREAM method is not listed.\n\nDREAM uses tiny-ImageNet (resolution: 64x64) instead of a subset of ImageNet (resolution: 224 x 224). This is because DREAM performs clustering during condensation, and the increase in the image resolution incurs large memory costs. \n\n > 2. Some proofs, such as Proof of Theorem 1, can be shortened. \n\nThanks for the suggestion. We will modify this part.\n \n> 3  The core idea is first to condense the dataset before pruning the dataset. How about pruning the dataset first and then condensing it? Which way is better and why?\n\nThanks for the insightful question. We believe conducting dataset pruning before dataset condensation is better.\nDataset pruning has roughly two times compression, and dataset condensation has 100 times compression. If we do dataset pruning before dataset condensation, the final compression ratio is still 100 times. If we do dataset condensation before dataset pruning, the final compression ratio can reach 200 times. \n\n> 4  From Figure 1, previous methods require 200K\\*(N-1) epochs for extra condensation. Is there any way to reduce the extra condensation process without using dataset pruning? Dataset Pruning is just like a selection method.\n\nYes, it is possible. Thanks for your suggestion.\n\n>  5. Will the code be open-sourced?\n\nYes. The code will be released after acceptance."}]

Author rebuttal:

Meta-review: The paper proposes a new approach to dataset condensation that adapts to varying computational constraints. The authors introduce two rules for pruning condensed datasets: the Importance Preservation Criterion (IPC) and the Loss-Based Sample Selection (LBSS) rule. The IPC rule prioritizes easy samples, while the LBSS rule selects samples based on their loss function gradient and balances samples for different classes. The authors demonstrate that their approach outperforms other dataset condensation and pruning methods on various networks and datasets, especially in imbalanced class scenarios. 

One of the main strengths of this paper is its clear and concise presentation of the proposed approach and its evaluation. The authors provide a thorough analysis of the effectiveness of their approach and compare it to other state-of-the-art methods. Additionally, the paper offers valuable insights into the challenges of dataset condensation and the importance of balancing class distributions. Reviewers initially raised questions on the clarity of motivation, comparison with more dataset pruning/ condensation methods, and the accuracy drop that challenges the practicality of this pruning method. The authors performed great due diligence and provided an extensive set of new experiments, that managed to convince the concerned reviewers. After rebuttal, all reviewers unanimously rate this paper positively, and AC therefore recommends acceptance.
Decision: Accept (poster)