SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tab. 1 provides a comprehensive comparison of different methods for flexibly resizing datasets from an initial  $\mathrm{IPC}_{\mathbf{F}}$  to a target  $\mathrm{IPC}_{\mathbf{T}}$ . In this table, we have not included ImageNet results on DREAM [24] since it only reports on Tiny ImageNet with a resolution of  $64\times 64$ , in contrast to ImageNet's  $224\times 224$ . The third column of the table shows the accuracy of the condensed dataset at the parameter  $\mathrm{IPC}_{\mathbf{F}}$ . We then flexibly resize the dataset from  $\mathrm{IPC}_{\mathbf{F}}$  to  $\mathrm{IPC}_{\mathbf{T}}$ . The blue area represents the average accuracy across different  $\mathrm{IPC}_{\mathbf{T}}$  values. For instance, consider the CIFAR-10 dataset with  $\mathrm{IPC}_{\mathbf{F}} = 10$ . Resizing it to  $\mathrm{IPC}_{\mathbf{F}} = 1, 2$ , and 5 using our method yields accuracies of  $42.28\%$ ,  $46.67\%$ , and  $55.96\%$ , respectively. The average accuracy of these three values is  $48.30\%$ . This value surpasses the  $37.08\%$  accuracy of SSP [40] by a considerable margin of  $11.22\%$ .
Ablation Study. Tab. 2 shows the ablation study of the LBPE score and the balanced construction across dataset condensation methods. In the first row, the baseline results are shown where neither the LBPE score nor the balanced construction is applied. "Balanced only" (second row) indicates the selection method is random selection and the class distribution is balanced. "LBPE only" (third row) means the
Table 2: Ablation study on two rules. (CIFAR-10:  $\mathrm{IPC}_{10\rightarrow 1}$ )
LBPE	Balanced	IDC [19]	DREAM [24]	MTT [1]	KIP [32]
-	-	28.23	30.87	19.75	14.06
-	✓	30.19	32.83	19.09	16.27
✓	-	39.38	37.30	20.37	15.78
✓	✓	42.28	42.29	22.02	22.24
Table 1: IPC means “images per class”. Flexibly resize dataset from  $\mathrm{IPC}_{\mathbf{F}}$  to  $\mathrm{IPC}_{\mathbf{T}}$  ( $\mathrm{IPC}_{\mathbf{F} \rightarrow \mathbf{T}}$ ). The blue areas represent the average accuracy of listed  $\mathrm{IPC}_{\mathbf{T}}$  datasets for different values of  $\mathbf{T}$ . The gray areas indicate the accuracy difference between the corresponding methods and ours.
Dataset	\( {\mathrm{{IPC}}}_{\mathrm{F}} \)	Acc.	\( {\mathrm{{IPC}}}_{\mathrm{T}} \)	Condensation	Pruning Method
IDC[19]	DREAM[24]	SSP[40]	Entropy[3]	AUM[35]	Forg.[41]	EL2N[34]	CCS[54]	Ours
CIFAR-10	10	67.50	1	28.23	30.87	27.83	30.30	13.30	16.68	16.95	33.54	42.28
2	37.10	38.88	34.95	38.88	18.44	22.13	23.26	39.20	46.67
5	52.92	54.23	48.47	52.85	41.40	45.49	46.58	53.23	55.96
Avg.	39.42	41.33	37.08	40.68	24.38	28.10	28.93	41.99	48.30
Diff.	-8.89	-6.98	-11.22	-7.63	-23.92	-20.20	-19.37	-6.31	-
50	74.50	1	29.45	27.61	28.99	17.95	7.21	12.23	7.95	31.28	38.77
2	34.27	36.11	34.51	24.46	8.67	12.17	9.47	38.71	44.54
5	45.85	48.28	46.38	34.12	12.85	15.55	16.03	48.19	53.04
10	57.71	59.11	56.81	47.61	22.92	27.01	31.33	56.80	61.10
Avg.	41.82	42.78	41.67	31.04	12.91	16.74	16.20	43.75	49.36
Diff.	-7.54	-6.58	-7.69	-18.33	-36.45	-32.62	-33.17	-5.62	-
CIFAR-100	10	45.40	1	14.78	15.05	14.94	11.28	3.64	6.45	5.12	18.97	22.57
2	22.49	21.78	20.65	16.78	5.93	10.03	8.15	25.27	29.09
5	34.90	35.54	30.48	29.96	17.32	21.45	22.40	36.01	38.51
Avg.	24.06	24.12	22.02	19.34	8.96	12.64	11.89	26.75	30.06
Diff.	-6.00	-5.93	-8.03	-10.72	-21.09	-17.41	-18.17	-3.31	-
20	49.50	1	13.92	13.26	14.65	5.75	2.96	7.59	4.59	18.72	23.74
2	20.62	20.41	20.27	8.63	3.96	10.64	6.18	24.08	29.93
5	31.21	31.81	30.34	17.51	8.25	17.63	11.76	32.81	38.02
Avg.	21.92	21.83	21.75	10.63	5.06	11.95	7.51	25.20	30.56
Diff.	-8.65	-8.74	-8.81	-19.93	-25.51	-18.61	-23.05	-5.36	-
50	52.60	1	13.41	13.36	15.90	1.86	2.79	9.03	4.21	19.05	23.47
2	20.38	19.97	21.26	2.86	3.04	12.66	5.01	24.32	29.59
5	29.92	29.88	29.63	6.04	4.56	20.23	7.24	31.93	37.52
10	37.79	37.85	36.97	13.31	8.56	29.11	11.72	38.05	42.79
Avg.	25.38	25.27	25.94	6.02	4.74	17.76	7.05	28.34	33.34
Diff.	-7.97	-8.08	-7.40	-27.33	-28.61	-15.59	-26.30	-5.01	-
ImageNet-10	10	72.80	1	44.93	-	45.69	40.98	17.84	32.07	41.00	44.27	53.91
2	57.84	-	58.47	52.04	29.13	44.89	54.47	56.53	59.69
5	67.20	-	63.11	64.60	44.56	55.13	65.87	67.36	64.47
Avg.	56.66	-	55.76	52.54	30.51	44.03	53.78	56.05	59.36
Diff.	2.70	-	3.60	6.82	28.85	15.33	5.58	3.31	-
20	76.60	1	42.00	-	43.13	36.13	14.51	24.98	24.09	34.64	53.07
2	53.93	-	54.82	46.91	19.09	31.27	33.16	42.22	58.96
5	59.56	-	61.27	56.44	27.78	36.44	46.02	57.11	64.38
Avg.	51.83	-	53.07	46.49	20.46	30.90	34.42	44.66	58.80
Diff.	6.97	-	5.73	12.31	38.34	27.90	24.38	14.14	-
LBPE score is used for ranking samples, and the selection results are purely based on the LBPE score without considering the class balance. "LBPE + Balanced" indicates that both elements are included for sample selection. The empirical findings conclusively affirm the effectiveness of the two rules, which constitute the principal contributions of our YOCO method.
Standard Deviation of Experiments. Different training dynamics and network initializations impact the final results. Therefore, the reported results are averaged over three different training dynamics, and each training dynamic is evaluated based on three different network initializations. See Appendix B.2 for the primary results table with standard deviation.
4.3 Analysis of Two Rules
4.3.1 Analysis of LBPE Score for Sample Ranking
Tab. 3 illustrates the robust performance of our YOCO method across diverse network structures, including ConvNet, ResNet, and DenseNet, demonstrating its strong generalization ability. Additionally, we present different sample ranking metrics from dataset pruning methods on these networks, demonstrating that our method outperforms both random selection and other data pruning methods.
In Tab. 4, we experiment with prioritizing easy samples over hard ones. We achieve this by reversing the importance metrics introduced by AUM [35], Forg. [41], and EL2N [34] that originally prioritize
Table 3: Accuracies on different network structures and different sample ranking metrics. (IDC [19] condensed CIFAR-10:  $\mathrm{IPC}_{10\rightarrow 1}$ )
ConvNet [12]	ResNet [13]	DenseNet [14]
Random	28.23	24.14	24.63
SSP[40]	27.83	24.64	24.75
Entropy[3]	30.30	30.53	29.93
AUM[35]	13.30	15.04	14.56
Forg.[41]	16.68	16.75	17.43
EL2N[34]	16.95	19.98	21.43
Ours	42.28	34.53	34.29
Table 4: Prioritizing easy samples is better for different dataset pruning and dataset condensation methods. "R?" represents whether to reverse the metrics which prioritize hard samples. (CIFAR-10:  $\mathrm{IPC}_{10\rightarrow 1}$ )
Method	R?	IDC[19]	DREAM[24]	MTT[1]	DSA[50]
AUM [35]	-	13.30	14.43	15.33	14.25
AUM [35]	✓	37.97	38.18	16.63	18.23
Forg. [41]	-	16.68	16.26	18.82	16.55
Forg. [41]	✓	36.69	36.15	16.65	17.03
EL2N [34]	-	16.95	18.13	16.98	13.14
EL2N [34]	✓	33.11	34.36	19.01	21.29
Ours	-	42.28	42.29	22.02	22.40
Table 5: Balanced construction works on different dataset pruning methods. “ $\mathcal{B}$ ?” represents whether to use balanced construction. The subscript  $^+\mathrm{value}$  indicates the accuracy gain from balanced construction. (IDC [19] condensed CIFAR-10:  $\mathrm{IPC}_{10\rightarrow T}$ )
\(IPC_T\)	\(\mathcal{B}?\)	Random	SSP [40]	Entropy [3]	AUM [35]	Forg. [41]	EL2N [34]	Ours
IPC1	-	28.23	27.83	30.30	13.30	16.68	16.95	37.63
\(\checkmark\)	\(30.05+1.82\)	\(33.21+5.38\)	\(33.67+3.37\)	\(15.64+2.34\)	\(19.09+2.41\)	\(18.43+1.48\)	\(42.28+4.65\)
IPC2	-	37.10	34.95	38.88	18.44	22.13	23.26	42.99
\(\checkmark\)	\(39.44+2.34\)	\(40.57+5.62\)	\(42.17+3.29\)	\(23.84+5.40\)	\(28.06+5.93\)	\(26.54+3.28\)	\(46.67+3.68\)
IPC5	-	52.92	48.47	52.85	41.40	45.49	46.58	53.86
\(\checkmark\)	\(52.64-0.28\)	\(49.44+0.97\)	\(54.73+1.88\)	\(47.23+5.83\)	\(48.02+2.53\)	\(48.86+2.28\)	\(55.96+2.10\)
hard samples. Our results indicate that across various condensed datasets, including IDC [19], DREAM [24], MTT [1], and DSA [50], there is a distinct advantage in prioritizing easier samples over harder ones. These findings lend support to our Rule 1.
4.3.2 Analysis of Balanced Construction
Fig. 2 presents the class distributions with and without a balanced construction for different datasets and different  $\mathrm{IPC}_{\mathbf{F}\rightarrow \mathbf{T}}$ . As explained in YOCO settings, our balanced construction is based on the multi-formation framework from IDC [19]. Therefore, the x-axis represents the count of images after multi-formation instead of the condensed images. It is evident that a ranking strategy relying solely on the LBPE score can result in a significant class imbalance, particularly severe in the ImageNet dataset. As depicted in Fig. 2(f), three classes have no image patches left. Our balanced construction method effectively mitigates this issue. Notably, in the case of ImageNet- $10_{10\to 1}$ , the balanced construction boosts the accuracy by an impressive  $19.37\%$ .
To better understand the impact of balanced class distribution on various dataset pruning methods, we conducted a comparative analysis, as presented in Tab. 5. Clearly, achieving a balanced class distribution significantly enhances the performance of all examined methods. Remarkably, our proposed method consistently outperforms others under both imbalanced and balanced class scenarios, further substantiating the efficacy of our approach.
4.4 Other Analysis
Sample Importance Rules Differ between Condensed Dataset and Full Dataset. In Fig. 3, we compare Sample Importance Rules for Condensed Datasets  $\left(\mathrm{IPC}_{10}, \mathrm{IPC}_{50}\right)$  and the Full Dataset  $\left(\mathrm{IPC}_{5000}\right)$ , by adjusting the pruning ratio from  $10\%$  to  $90\%$ . Unmarked solid lines mean prioritizing easy samples, dashed lines suggest prioritizing hard samples, while marked solid lines depict the accuracy disparity between the preceding accuracies. Therefore, the grey region above zero indicates "Prefer easy samples" (Rule\_easy), while the blue region below zero represents "Prefer hard samples" (Rule\_hard). We have two observations. First, as the pruning ratio increases, there is a gradual transition from Rule\_hard to Rule\_easy. Second, the turning point of this transition depends on the dataset size. Specifically, the turning points for  $\mathrm{IPC}_{10}$ ,  $\mathrm{IPC}_{50}$ , and  $\mathrm{IPC}_{5000}$  occur at pruning
(d) CIFAR-  $10_{50\to 1}$
(e) CIFAR-100 $_{10 \to 1}$
(f) ImageNet-  $10_{10\to 1}$
Figure 2: Balanced and imbalanced selection by ranking samples with LBPE score.  $\mathrm{Dataset}_{\mathbf{F} \rightarrow \mathbf{T}}$  denotes resizing the dataset from  $\mathrm{IPC}_{\mathbf{F}}$  to  $\mathrm{IPC}_{\mathbf{T}}$ . Accuracies for each setting are also listed in the legend. (IDC [19] condensed datasets)
Figure 3: Different sample importance rules between condensed datasets and full datasets.
ratios of  $24\%$ ,  $38\%$ , and  $72\%$ , respectively. These experimental outcomes substantiate our Rule 1 that condensed datasets should adhere to Rule<sub>easy</sub>.
Performance Gap from Multi-formation. We would like to explain the huge performance gap between multi-formation-based methods (IDC [19] and DREAM [24]) and other methods (MTT [1], KIP [33], and DSA [50]) in Tab. 2 and Tab. 4. The potential reason is that a single image can be decoded to  $2^{2} = 4$  low-resolution images via multi-formation. As a result, methods employing multi-formation generate four times as many images compared to those that do not use multi-formation. The illustration is shown in Fig. 4.
Figure 4: Illustration of the multi-formation with a factor of 2. (Taken from IDC [19])
Why Use LBPE Score from the Top-  $K$  Training Epochs with the Highest Accuracy? As shown in Eq. 2, different training epoch  $t$  leads to a different LBPE score. Fig. 5 illustrates the accuracy of the dataset selected via the LBPE score across specific training epochs. We select LBPE scores from the initial 100 epochs out of 1000 original epochs to reduce computational costs. We have two observations. First, the model's accuracy during the first few epochs is substantially low. LBPE scores derived from these early-stage epochs might not accurately represent the samples' true importance since the model is insufficiently trained. Second, there's significant variance in accuracy even after 40 epochs, leading to potential instability in the LBPE score selection. To address this, we average LBPE scores from epochs with top-  $K$  accuracy, thereby reducing variability and ensuring a more reliable sample importance representation.
Speculating on Why LBPE Score Performs Better at Certain Epochs? In Fig. 7, we present the distribution of LBPE scores at various training epochs, with scores arranged in ascending order for each class to facilitate comparison across epochs. Our experiment finds the LBPE scores decrease as the epoch number increases. The superior accuracy of  $\mathrm{LBPE}_{90}$  is due to two reasons. First, the model at the  $90_{th}$  epoch is more thoroughly trained than the model at the first epoch, leading to more accurate LBPE scores. Second, the  $\mathrm{LBPE}_{90}$  score offers a more uniform distribution and a wider range [0, 1], enhancing sample distinction. In contrast, the  $\mathrm{LBPE}_{1000}$  score is mostly concentrated within a narrow range [0, 0.1] for the majority of classes, limiting differentiation among samples. More possible reasons will be explored in future studies.
Visualization. Fig. 6 visualizes the easy and hard samples identified by our YOCO method. We notice that most easy samples have a distinct demarcation between the object and its background. This is particularly evident in the classes of "vacuum cleaner" and "cocktail shaker". The easy samples in
Figure 5: Accuracy of the dataset selected with LBPE score at specific epochs.
Figure 6: Visualization of hard and easy samples of ImageNet dataset selected by our method. Both the original and condensed images are shown for comparison.
Figure 7: LBPE scores at different epochs (LBPE<sub>epoch</sub>) for ten classes of the CIFAR-10 dataset.
these two classes have clean backgrounds, while the hard samples have complex backgrounds. The visualization provides evidence of our method's ability to identify easy and hard samples.
5 Conclusion, Limitation and Future Work
We introduce You Only Condense Once (YOCO), a novel approach that resizes condensed datasets flexibly without an extra condensation process, enabling them to adjust to varying computational constraints. YOCO comprises two key rules. First, YOCO employs the Logit-Based Prediction Error (LBPE) score to rank the importance of training samples and emphasizes the benefit of prioritizing easy samples with low LBPE scores. Second, YOCO underscores the need to address the class imbalance in condensed datasets and utilizes Balanced Construction to solve the problem. Our experiments validated YOCO's effectiveness across different networks and datasets. These insights offer valuable directions for future dataset condensation and dataset pruning research.
We acknowledge several limitations and potential areas for further investigation. First, although our method uses early training epochs to reduce computational costs, determining the sample importance in the first few training epochs or even before training is interesting for future work. Second, we only utilize the LBPE score to establish the importance of samples within the dataset. However, relying on a single metric might not be the optimal approach. There are other importance metrics, such as SSP [40] and AUM [35], that could be beneficial to integrate into our methodology. Third, as our current work only covers clean datasets like CIFAR-10, the performance of our method on noisy datasets requires further investigation.
The border impact is shown in Appendix C.
This work is partially supported by Joey Tianyi Zhou's A*STAR SERC Central Research Fund (Use-inspired Basic Research), the Singapore Government's Research, Innovation and enterprise 2020 Plan (Advanced Manufacturing and Engineering domain) under Grant A18A1b0045, and A*STAR CFAR Internship Award for Research Excellence (CIARE). The computational work for this article was partially performed on resources of the National Supercomputing Centre (NSCC), Singapore (https://www.nscc.sg).
[1] G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[2] K. Chitta, J. M. Álvarez, E. Haussmann, and C. Farabet. Training data subset search with ensemble active learning. IEEE Transactions on Intelligent Transportation Systems, 23(9):14741-14752, 2021.
[3] C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia. Selection via proxy: Efficient data selection for deep learning. In Proc. Int. Conf. Learn. Represent., 2020.
[4] J. Cui, R. Wang, S. Si, and C.-J. Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. arXiv preprint arXiv:2211.10586, 2022.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 248–255, 2009.
[6] Z. Deng and O. Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In Proc. Adv. Neural Inform. Process. Syst., 2022.
[7] J. Du, Y. Jiang, V. T. Tan, J. T. Zhou, and H. Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[8] M. Ducoffe and F. Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841, 2018.
[9] D. Feldman and M. Langberg. A unified framework for approximating and clustering data. In Proceedings of the Forty-Third Annual ACM Symposium on Theory of Computing, page 569-578, 2011.
[10] D. Feldman, M. Schmidt, and C. Sohler. Turning big data into tiny data: Constant-size coresets for k-means, pca, and projective clustering. SIAM Journal on Computing, 49(3):601-657, 2020.
[11] V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Proc. Adv. Neural Inform. Process. Syst., pages 2881-2891, 2020.
[12] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4367-4375, 2018.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 770-778, 2016.
[14] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4700-4708, 2017.
[15] L. Huang and N. K. Vishnoi. Coresets for clustering in euclidean spaces: Importance sampling is nearly optimal. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, page 1416-1429, 2020.
[16] Z. Jiang, J. Gu, M. Liu, and D. Z. Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208.00311, 2022.