ICML 2025

Lightweight Dataset Pruning without Full Training
via Example Difficulty and Prediction Uncertainty
Yeseul Cho * 1 Baekrok Shin * 1 Changmin Kang 1 Chulhee Yun 1
Abstract
Recent advances in deep learning rely heavily
on massive datasets, leading to substantial stor-
age and training costs. Dataset pruning aims to
alleviate this demand by discarding redundant ex-
amples. However, many existing methods require
training a model with a full dataset over a large
number of epochs before being able to prune the
dataset, which ironically makes the pruning pro-
cess more expensive than just training the model
on the entire dataset. To overcome this limitation,
we introduce the Difficulty and Uncertainty-
Aware Lightweight (DUAL) score, which aims
to identify important samples from the early train-
ing stage by considering both example difficulty
and prediction uncertainty. To address a catas-
trophic accuracy drop at an extreme pruning ra-
tio, we further propose a pruning ratio-adaptive
sampling using Beta distribution. Experiments
on various datasets and learning scenarios such
as image classification with label noise and im-
age corruption, and model architecture general-
ization demonstrate the superiority of our method
over previous state-of-the-art (SOTA) approaches.
Specifically, on ImageNet-1k, our method reduces
the time cost for pruning to 66% compared to
previous methods while achieving a SOTA 60%
test accuracy at a 90% pruning ratio. On CIFAR
datasets, the time cost is reduced to just 15% while
maintaining SOTA performance. Implementation
is available at github/dual-pruning.
1. Introduction
Advancements in deep learning have been significantly
driven by large-scale datasets. However, recent studies have
*Equal contribution
1Kim Jaechul Graduate School of AI,
KAIST, Seoul, South Korea. Correspondence to: Chulhee Yun
<chulhee.yun@kaist.ac.kr>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by...