SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yang He, Lingao Xiao, Joey Tianyi Zhou
CFAR, Agency for Science, Technology and Research, Singapore
IHPC, Agency for Science, Technology and Research, Singapore
{He_Yang, Joey_Zhou}@cfar.a-star.edu.sg
Dataset condensation is a crucial tool for enhancing training efficiency by reducing the size of the training dataset, particularly in on-device scenarios. However, these scenarios have two significant challenges: 1) the varying computational resources available on the devices require a dataset size different from the pre-defined condensed dataset, and 2) the limited computational resources often preclude the possibility of conducting additional condensation processes. We introduce You Only Condense Once (YOCO) to overcome these limitations. On top of one condensed dataset, YOCO produces smaller condensed datasets with two embarrassingly simple dataset pruning rules: Low LBPE Score and Balanced Construction. YOCO offers two key advantages: 1) it can flexibly resize the dataset to fit varying computational constraints, and 2) it eliminates the need for extra condensation processes, which can be computationally prohibitive. Experiments validate our findings on networks including ConvNet, ResNet and DenseNet, and datasets including CIFAR-10, CIFAR-100 and ImageNet. For example, our YOCO surpassed various dataset condensation and dataset pruning methods on CIFAR-10 with ten Images Per Class (IPC), achieving  $6.98 - 8.89\%$  and  $6.31 - 23.92\%$  accuracy gains, respectively. The code is available at: https://github.com/he-y/you-only-condense-once.
Deep learning models often require vast amounts of data to achieve optimal performance. This data-hungry nature of deep learning algorithms, coupled with the growing size and complexity of datasets, has led to the need for more efficient dataset handling techniques. Dataset condensation is a promising approach that enables models to learn from a smaller and more representative subset of the entire dataset. Condensed datasets are especially utilized in on-device scenarios, where limited computational resources and storage constraints necessitate the use of a compact training set.
However, these on-device scenarios have two significant constraints. First, the diverse and fluctuating computational resources inherent in these scenarios necessitate a level of flexibility in the size of the dataset. But the requirement of flexibility is not accommodated by the fixed sizes of previous condensed datasets. Second, the limited computational capacity in these devices also makes extra condensation processes impractical, if not impossible. Therefore, the need for adaptability in the size of the condensed dataset becomes increasingly crucial. Furthermore, this adaptability needs to be realized without introducing another computationally intensive condensation process.
We introduce You Only Condense Once (YOCO) to enable the flexible resizing (pruning) of condensed datasets to fit varying on-device scenarios without extra condensation process (See Fig. 1). The first rule of our proposed method involves a metric to evaluate the importance of training samples in the context of dataset condensation.
From the gradient of the loss function, we develop the Logit-Based Prediction Error (LBPE) score to rank training samples. This metric quantifies the neural network's difficulty in recognizing each sample. Specifically, training samples with low LBPE scores are considered easy as they indicate that the model's prediction is close to the true label. These samples exhibit simpler patterns, easily captured by the model. Given the condensed datasets' small size, prioritizing easier samples with low LBPE scores is crucial to avoid overfitting.
A further concern is that relying solely on a metric-based ranking could result in an imbalanced distribution of classes within the dataset. Imbalanced
Figure 1: Previous methods (left) require extra condensation processes, but ours (right) do not.
datasets can lead to biased predictions, as models tend to focus on the majority class, ignoring the underrepresented minority classes. This issue has not been given adequate attention in prior research about dataset pruning [3, 41, 34], but it is particularly important when dealing with condensed datasets. In order to delve deeper into the effects of class imbalance, we explore Rademacher Complexity [31], a widely recognized metric for model complexity that is intimately connected to generalization error and expected loss. Based on the analysis, we propose Balanced Construction to ensure that the condensed dataset is both informative and balanced.
The key contributions of our work are: 1) To the best of our knowledge, it's the first work to provide a solution to adaptively adjust the size of a condensed dataset to fit varying computational constraints. 2) After analyzing the gradient of the loss function, we propose the LBPE score to evaluate the sample importance and find out easy samples with low LBPE scores are suitable for condensed datasets. 3) Our analysis of the Rademacher Complexity highlights the challenges of class imbalance in condensed datasets, leading us to construct balanced datasets.
2.1 Dataset Condensation/Distillation
Dataset condensation, or distillation, synthesizes a compact image set to maintain the original dataset's information. Wang et al. [45] pioneer an approach that leverages gradient-based hyperparameter optimization to model network parameters as a function of this synthetic data. Building on this, Zhao et al. [53] introduce gradient matching between real and synthetic image-trained models. Kim et al. [19] further extend this, splitting synthetic data into  $n$  factor segments, each decoding into  $n^2$  training images. Zhao & Bilen [50] apply consistent differentiable augmentation to real and synthetic data, thus enhancing information distillation. Cazenavette et al. [1] propose to emulate the long-range training dynamics of real data by aligning learning trajectories. Liu et al. [24] advocate matching only representative real dataset images, selected based on latent space cluster centroid distances. Additional research avenues include integrating a contrastive signal [21], matching distribution or features [52, 44], matching multi-level gradients [16], minimizing accumulated trajectory error [7], aligning loss curvature [39], parameterizing datasets [6, 23, 51, 43], and optimizing dataset condensation [27, 32, 33, 42, 55, 25, 49, 4, 26]. Nevertheless, the fixed size of condensed datasets remains an unaddressed constraint in prior work.
2.2 Dataset Pruning
Unlike dataset condensation that alters image pixels, dataset pruning preserves the original data by selecting a representative subset. Entropy [3] keeps hard samples with maximum entropy (uncertainty [22, 38]), using a smaller proxy network. Forgetting [41] defines forgetting events as an accuracy drop at consecutive epochs, and hard samples with the most forgetting events are important. AUM [35] identifies data by computing the Area Under the Margin, the difference between the true label logits and the largest other logits. Memorization [11] prioritizes a sample if it significantly improves the probability of correctly predicting the true label. GraNd [34] and EL2N [34] classify samples as hard based on the presence of large gradient norm and large norm of error vectors, respectively. CCS [54] extends previous methods by pruning hard samples and using stratified sampling to
achieve good coverage of data distributions at a large pruning ratio. Moderate [47] selects moderate samples (neither hard nor easy) that are close to the score median in the feature space. Optimization-based [48] chooses samples yielding a strictly constrained generalization gap. In addition, other dataset pruning (or coreset selection) methods [8, 28, 36, 17, 30, 18, 2, 29, 9, 10, 46, 15] are widely used for Active Learning [38, 37]. However, the previous methods consider full datasets and often neglect condensed datasets.
We denote a dataset by  $S = (x_{i},y_{i})_{i = 1}^{N}$ , where  $x_{i}$  represents the  $i^{th}$  input and  $y_{i}$  represents the corresponding true label. Let  $\mathcal{L}(p(\mathbf{w},x),y)$  be a loss function that measures the discrepancy between the predicted output  $p(\mathbf{w},x)$  and the true label  $y$ . The loss function is parameterized by a weight vector  $\mathbf{w}$ , which we optimize during the training process.
We consider a time-indexed sequence of weight vectors,  $\mathbf{w}_t$ , where  $t = 1,\dots ,T$ . This sequence represents the evolution of the weights during the training process. The gradient of the loss function with respect to the weights at time  $t$  is given by  $g_{t}(x,y) = \nabla_{\mathbf{w}_{t}}\mathcal{L}(p(\mathbf{w}_{t},x),y)$ .
3.2 Identifying Important Training Samples
Our goal is to propose a measure that quantifies the importance of a training sample. We begin by analyzing the gradient of the loss function with respect to the weights  $\mathbf{w}_t$ :
$$
\nabla_ {\mathbf {w} _ {t}} \mathcal {L} (p (\mathbf {w} _ {t}, x), y) = \frac {\partial \mathcal {L} (p (\mathbf {w} _ {t} , x) , y)}{\partial p (\mathbf {w} _ {t} , x)} \cdot \frac {\partial p (\mathbf {w} _ {t} , x)}{\partial \mathbf {w} _ {t}}. \tag {1}
$$
We aim to determine the impact of training samples on the gradient of the loss function, as the gradient plays a critical role in the training process of gradient-based optimization methods. Note that our ranking method is inspired by EL2N [34], but we interpret it in a different way by explicitly considering the dataset size  $|S|$ .
Definition 1 (Logit-Based Prediction Error - LBPE): The Logit-Based Prediction Error (LBPE) of a training sample  $(x,y)$  at time  $t$  is given by:
$$
\mathrm {L B P E} _ {t} (x, y) = \mathbb {E} | p (\mathbf {w} _ {t}, x) - y | _ {2}, \tag {2}
$$
where  $\mathbf{w}_t$  is the weights at time  $t$ , and  $p(\mathbf{w}_t, x)$  represents the prediction logits.
Lemma 1 (Gradient and Importance of Training Samples): The gradient of the loss function  $\nabla_{\mathbf{w}_t}\mathcal{L}(p(\mathbf{w}_t,x),y)$  for a dataset  $S$  is influenced by the samples with prediction errors.
Proof of Lemma 1: Consider two datasets  $S$  and  $S_{\neg j}$ , where  $S_{\neg j}$  is obtained by removing the sample  $(x_j, y_j)$  from  $S$ . Let the gradients of the loss function for these two datasets be  $\nabla_{\mathbf{w}_t}^S \mathcal{L}$  and  $\nabla_{\mathbf{w}_t}^{S \neg j} \mathcal{L}$ , respectively. The difference between the gradients is given by (see Appendix A.1 for proof):
$$
\Delta \nabla_ {\mathbf {w} _ {t}} \mathcal {L} = \frac {- 1}{| S | (| S | - 1)} \sum_ {(x, y) \in S _ {\neg j}} \frac {\partial \mathcal {L} (p \left(\mathbf {w} _ {t} , x\right) , y)}{\partial p \left(\mathbf {w} _ {t} , x\right)} \cdot \frac {\partial p \left(\mathbf {w} _ {t} , x\right)}{\partial \mathbf {w} _ {t}} + \frac {1}{| S |} \frac {\partial \mathcal {L} (p \left(\mathbf {w} _ {t} , x _ {j}\right) , y _ {j})}{\partial p \left(\mathbf {w} _ {t} , x _ {j}\right)} \cdot \frac {\partial p \left(\mathbf {w} _ {t} , x _ {j}\right)}{\partial \mathbf {w} _ {t}} \tag {3}
$$
Let us denote the error term as:  $e_j = p(\mathbf{w}_t, x_j) - y_j$ , the LBPE score for sample  $(x_j, y_j)$  is given by  $\mathrm{LBPE}_t(x_j, y_j) = \mathbb{E}|e_j|_2$ , and the difference in gradients related to the sample  $(x_j, y_j)$  can be rewritten as:
$$
\frac {1}{| S |} \frac {\partial \mathcal {L} \left(e _ {j}\right)}{\partial e _ {j}} \cdot \frac {\partial p \left(\mathbf {w} _ {t} , x _ {j}\right)}{\partial \mathbf {w} _ {t}}. \tag {4}
$$
If the sample  $(x_{j},y_{j})$  has a lower LBPE score, it implies that the error term  $e_j$  is smaller. Let's consider the mean squared error (MSE) loss function, which is convex. The MSE loss function is
Algorithm 1 Compute LBPE score for samples over epochs
Require: Training dataset  $S$  and its size  $|S|$ , weights  $\mathbf{w}_t$ , true labels  $y$ , model's predicted probabilities  $p(\mathbf{w}_t, x)$ , number of epochs  $E$ , Epochs with Top-K accuracy  
1: Initialize matrix: LBPE = torch.zeros((E, |S|)) ▷ LBPE scores over samples and epochs  
2: Initialize accuracy: ACC = torch.zeros(E) ▷ Track the accuracy over epochs  
3: for each epoch  $t$  in range  $E$  do ▷ Loop through each epoch  
4: for each sample index  $i$  in  $S$  do ▷ Loop through each sample in the dataset  
5: Compute error term for sample  $i$  at epoch  $t$ :  $e_{i,t} = p(\mathbf{w}_t, x_i) - y_i$   
6: Compute LBPE score for sample  $i$  at epoch  $t$  with MSE loss:  $\mathrm{LBPE}_{i,t} = \mathbb{E}_{MSE}|e_{i,t}|_2$   
7: end for  
8: Compute accuracy at epoch  $t$ :  $\mathrm{ACC}_t$   
9: end for  
10: Top_K ← argsort(ACC)[-k:] ▷ Find the epochs with the Top-K accuracy  
11: AVG_LBPE ← mean(LBPE[Top_k,:]) ▷ Average LBPE score over Top-K epochs  
12: return AVG_LBPE
defined as  $\mathcal{L}(e_j) = \frac{1}{2} (e_j)^2$ . Consequently, the derivative of the loss function  $\frac{\partial\mathcal{L}(e_j)}{\partial e_j} = e_j$  would be smaller for samples with smaller LBPE scores, leading to a smaller change in the gradient  $\Delta \nabla_{\mathbf{w}_t}\mathcal{L}$ .
Rule 1: For a small dataset, a sample with a lower LBPE score will be more important. Let  $S$  be a dataset of size  $|S|$ , partitioned into subsets  $S_{easy}$  (lower LBPE scores) and  $S_{hard}$  (higher LBPE scores).
Case 1: Small Dataset - When the dataset size  $|S|$  is small, the model's capacity to learn complex representations is limited. Samples in  $S_{easy}$  represent prevalent patterns in the data, and focusing on learning from them leads to a lower average expected loss. This enables the model to effectively capture the dominant patterns within the limited dataset size. Moreover, the gradients of the loss function for samples in  $S_{easy}$  are smaller, leading to faster convergence and improved model performance within a limited number of training iterations.
Case 2: Large Dataset - When the dataset size  $|S|$  is large, the model has the capacity to learn complex representations, allowing it to generalize well to both easy and hard samples. As the model learns from samples in both  $S_{easy}$  and  $S_{hard}$ , its overall performance improves, and it achieves higher accuracy on hard samples. Training on samples in  $S_{hard}$  helps the model learn more discriminative features, as they often lie close to the decision boundary.
Therefore, in the case of a small dataset, samples with lower LBPE scores are more important.
The use of the LBPE importance metric is outlined in Algorithm 1. LBPE scores over the epochs with the Top-K training accuracy are averaged. The output of this algorithm is the average LBPE score.
3.3 Balanced Construction
In this section, we prove that a more balanced class distribution yields a lower expected loss.
Definition 2.1 (Dataset Selection  $S_A$  and  $S_B$ ):  $S_A$  is to select images from each class based on their LBPE scores such that the selection is balanced across classes, and  $S_B$  is to select images purely based on their LBPE scores without considering the class balance. Formally, we have:
$$
S _ {A} = \left(x _ {i}, y _ {i}\right): x _ {i} \in \mathcal {X} _ {k}, \text {a n d} \mathrm {L B P E} _ {t} \left(x _ {i}, y _ {i}\right) \leq \tau_ {k}, \quad S _ {B} = \left(x _ {i}, y _ {i}\right): \mathrm {L B P E} _ {t} \left(x _ {i}, y _ {i}\right) \leq \tau \tag {5}
$$
where  $\mathcal{X}_k$  denotes the set of images from class  $k$ , and  $\tau_k$  is a threshold for class  $k$ ,  $\tau$  is a global threshold. Then  $S_A$  is a more balanced dataset compared to  $S_B$ .
Definition 2.2 (Generalization Error): The generalization error of a model is the difference between the expected loss on the training dataset and the expected loss on an unseen test dataset:
$$
\operatorname {G e n E r r} (\mathbf {w}) = \mathbb {E} \left[ L _ {\text {t e s t}} (\mathbf {w}) \right] - \mathbb {E} \left[ L _ {\text {t r a i n}} (\mathbf {w}) \right]. \tag {6}
$$
Definition 2.3 (Rademacher Complexity): The Rademacher complexity [31] of a hypothesis class  $\mathcal{H}$  for a dataset  $S$  of size  $N$  is defined as:
$$
\mathcal {R} _ {N} (\mathcal {H}) = \underset {\sigma} {\mathbb {E}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{N} \sum_ {i = 1} ^ {N} \sigma_ {i} h \left(\mathbf {x} _ {i}\right) \right], \tag {7}
$$
Algorithm 2 Balanced Dataset Construction
Require: Condensed dataset  $S = (x_{i},y_{i})_{i = 1}^{m}$  with classes K, LBPE scores LBPE, class-specific thresholds  $\pmb{\tau} = \tau_{k}\pmb{\tau}_{k = 1}^{K}$  to ensure an equal number of samples for each class
1: Initialize  $S_A = \emptyset$  ▷ Initialize the balanced subset as an empty set
2: for each class  $k \in \mathbf{K}$  do
3:  $I_{\text{sel}} \gets \{i : y_i = k \text{ and } \mathbf{LBP} \mathbf{E}_t(x_i, y_i) \leq \tau_k\}$  ▷ Find indices of samples for class  $k$
4:  $S_A\gets S_A\cup (x_i,y_i):i\in I_{sel}$  Add the selected samples to the balanced subset
5: end for
6: return  $S_A$ $\triangleright$  Return the balanced subset
where  $\sigma_{i}$  are independent Rademacher random variables taking values in  $-1, 1$  with equal probability.
Lemma 2.1 (Generalization Error Bound): With a high probability, the generalization error is upper-bounded by the Rademacher complexity of the hypothesis class:
$$
\operatorname {G e n E r r} (\mathbf {w}) \leq 2 \mathcal {R} _ {N} (\mathcal {H}) + \mathcal {O} \left(\frac {1}{\sqrt {N}}\right), \tag {8}
$$
where  $\mathcal{O}$  represents the order of the term.
Lemma 2.2 (Rademacher Complexity Comparison): The Rademacher complexity of dataset  $S_A$  is less than that of dataset  $S_B$ :
$$
\mathcal {R} _ {N _ {A}} (\mathcal {H}) \leq \mathcal {R} _ {N _ {B}} (\mathcal {H}). \tag {9}
$$
Theorem 2.1: The expected loss for the dataset  $S_A$  is less than or equal to  $S_B$  when both models achieve similar performance on their respective training sets.
Proof of Theorem 2.1: Using Lemma 2.1 and Lemma 2.2, we have:
$$
\operatorname {G e n E r r} \left(\mathbf {w} _ {A}\right) \leq \operatorname {G e n E r r} \left(\mathbf {w} _ {B}\right). \tag {10}
$$
Assuming that both models achieve similar performance on their respective training sets, the training losses are approximately equal:
$$
\mathbb {E} \left[ L _ {\text {t r a i n}} \left(\mathbf {w} _ {A}\right) \right] \approx \mathbb {E} \left[ L _ {\text {t r a i n}} \left(\mathbf {w} _ {B}\right) \right]. \tag {11}
$$
Given this assumption, we can rewrite the generalization error inequality as:
$$
\mathbb {E} \left[ L _ {\text {t e s t}} (\mathbf {w} _ {A}) \right] - \mathbb {E} \left[ L _ {\text {t r a i n}} (\mathbf {w} _ {A}) \right] \leq \mathbb {E} \left[ L _ {\text {t e s t}} (\mathbf {w} _ {B}) \right] - \mathbb {E} \left[ L _ {\text {t r a i n}} (\mathbf {w} _ {B}) \right]. \tag {12}
$$
Adding  $\mathbb{E}[L_{\mathrm{train}}(\mathbf{w}_A)]$  to both sides, we get:
$$
\mathbb {E} \left[ L _ {\text {t e s t}} \left(\mathbf {w} _ {A}\right) \right] \leq \mathbb {E} \left[ L _ {\text {t e s t}} \left(\mathbf {w} _ {B}\right) \right]. \tag {13}
$$
This result indicates that the balanced dataset  $S_A$  is better than  $S_B$ .
Theorem 2.2: Let  $S_F$  and  $S_C$  be the full and condensed datasets, respectively, and let both  $S_F$  and  $S_C$  have an imbalanced class distribution with the same degree of imbalance. Then, the influence of the imbalanced class distribution on the expected loss is larger for the condensed dataset  $S_C$  than for the full dataset  $S_F$ .
Proof of Theorem 2.2: We compare the expected loss for the full and condensed datasets, taking into account their class imbalances. Let  $L(h)$  denote the loss function for the hypothesis  $h$ . Let  $\mathbb{E}[L(h)|S]$  denote the expected loss for the hypothesis  $h$  on the dataset  $S$ . Let  $n_{kF}$  and  $n_{kC}$  denote the number of samples in class  $k$  for datasets  $S_F$  and  $S_C$ , respectively. Let  $m_F$  and  $m_C$  denote the total number of samples in datasets  $S_F$  and  $S_C$ , respectively. Let  $r_k = \frac{n_{kF}}{m_F} = \frac{n_{kC}}{m_C}$  be the class ratio for each class  $k$  in both datasets. The expected loss for  $S_F$  and  $S_C$  can be written as:
$$
\mathbb {E} [ L (h) | S _ {F} ] = \sum_ {k = 1} ^ {K} r _ {k} \mathbb {E} [ l (h (x), y) | \mathcal {X} _ {k} ], \quad \mathbb {E} [ L (h) | S _ {C} ] = \sum_ {k = 1} ^ {K} r _ {k} \mathbb {E} [ l (h (x), y) | \mathcal {X} _ {k} ], \tag {14}
$$
To show this, let's compare the expected loss per sample in each dataset:
$$
\frac {\mathbb {E} [ L (h) \mid S _ {C} ]}{m _ {C}} > \frac {\mathbb {E} [ L (h) \mid S _ {F} ]}{m _ {F}}. \tag {15}
$$
This implies that the influence of the imbalanced class distribution is larger for  $S_{C}$  than for  $S_{F}$ .
Rule 2: Balanced class distribution should be utilized for the condensed dataset. The construction of a balanced class distribution based on LBPE scores is outlined in Algorithm 2. Its objective is to create an equal number of samples for each class to ensure a balanced dataset.
IPC stands for "Images Per Class".  $\mathrm{IPC}_{\mathbf{F}\rightarrow \mathbf{T}}$  means flexibly resize the condensed dataset from size  $\mathbf{F}$  to size T. More detailed settings can be found in Appendix B.1.
Dataset Condensation Settings. The CIFAR-10 and CIFAR-100 datasets [20] are condensed via ConvNet-D3 [12], and ImageNet-10 [5] via ResNet10-AP [13], both following IDC [19]. IPC includes 10, 20, or 50, depending on the experiment. For both networks, the learning rate is 0.01 with 0.9 momentum and 0.0005 weight decay. The SGD optimizer and a multi-step learning rate scheduler are used. The training batch size is 64, and the network is trained for  $2000 \times 100$  epochs for CIFAR-10/CIFAR-100 and  $500 \times 100$  epochs for ImageNet-10.
YOCO Settings. 1) LBPE score selection. To reduce computational costs, we derive the LBPE score from training dynamics of early  $E$  epochs. To reduce variance, we use the LBPE score from the top-  $K$  training epochs with the highest accuracy. For CIFAR-10, we set  $E = 100$  and  $K = 10$  for all the  $\mathrm{IPC}_{\mathbf{F}}$  and  $\mathrm{IPC}_{\mathbf{T}}$ . For CIFAR-100 and ImageNet-10, we set  $E = 200$  and  $K = 10$  for all the  $\mathrm{IPC}_{\mathbf{F}}$  and  $\mathrm{IPC}_{\mathbf{T}}$ . 2) Balanced construction. We use  $S_{A}$  in Eq. 5 to achieve a balanced construction. Following IDC [19], we leverage a multi-formation framework to increase the synthetic data quantity while preserving the storage budget. Specifically, an IDC-condensed image is composed of  $n^2$  patches. Each patch is derived from one original image with the resolution scaled down by a factor of  $1/n^2$ . Here,  $n$  is referred to as the "factor" in the multi-formation process. For CIFAR-10 and CIFAR-100 datasets,  $n = 2$ ; for ImageNet-10 dataset,  $n = 3$ . We create balanced classes according to these patches. As a result, all the classes have the same number of samples. 3) Flexible resizing. For datasets with  $\mathrm{IPC}_{\mathbf{F}} = 10$  and  $\mathrm{IPC}_{\mathbf{F}} = 20$ , we select  $\mathrm{IPC}_{\mathbf{T}}$  of 1, 2, and 5. For  $\mathrm{IPC}_{\mathbf{F}} = 50$ , we select  $\mathrm{IPC}_{\mathbf{T}}$  of 1, 2, 5, and 10. For a condensed dataset with  $\mathrm{IPC}_{\mathbf{F}}$ , the performance of its flexible resizing is indicated by the average accuracy across different  $\mathrm{IPC}_{\mathbf{T}}$  values.
Comparison Baselines. We have two sets of baselines for comparison: 1) dataset condensation methods including IDC[19], DREAM[24], MTT [1], DSA [50] and KIP [32] and 2) dataset pruning methods including SSP [40], Entropy [3], AUM [35], Forgetting [41], EL2N [34], and CCS [54]. For dataset condensation methods, we use a random subset as the baseline. For dataset pruning methods, their specific metrics are used to rank and prune datasets to the required size.
4.2 Primary Results