PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity
Authors: Mustafa Burak Gurbuz
Xingyu Zheng
Constantine Dovrolis
Venue: ICML 2025 poster (2025)
Keywords: data pruning
coresets
data selection
deep learning
Primary area: deep_learning->algorithms
TL;DR: We pose the Incremental Data Selection problem, where examples arrive as a continuous stream, and we propose PEAKS, a principled and efficient method tailored for this problem.

Abstract: As deep learning continues to be driven by ever-larger datasets, understanding which examples are most important for generalization has become a critical question. While progress in data selection continues, emerging applications require studying this problem in dynamic contexts. To bridge this gap, we pose the Incremental Data Selection (IDS) problem, where examples arrive as a continuous stream, and need to be selected without access to the full data source. In this setting, the learner must incrementally build a training dataset of predefined size while simultaneously learning the underlying task. We find that in IDS, the impact of a new sample on the model state depends fundamentally on both its geometric relationship in the feature space and its prediction error. Leveraging this insight, we propose PEAKS (Prediction Error Anchored by Kernel Similarity), an efficient data selection method tailored for IDS. Our comprehensive evaluations demonstrate that PEAKS consistently outperforms existing selection strategies. Furthermore, PEAKS yields increasingly better performance returns than random selection as training data size grows on real-world datasets. The  code is available at https://github.com/BurakGurbuz97/PEAKS.

PEAKS: Selecting Key Training Examples Incrementally via Prediction Error Anchored by Kernel Similarity
Mustafa Burak Gurbuz'
Xingyu Zheng?
Constantine Dovrolis 13
Abstract
Data Selection (time
Model Update (time =
Low High Sim
As deep learning continues to be driven by ever- larger datasets, understanding which examples are most important for generalization has become a critical question. While progress in data se- lection continues, emerging applications require studying this problem in dynamic contexts. To bridge this gap, we pose the Incremental Data Se- lection (IDS) problem, where examples arrive continuous stream, and need to be selected with- out access to the full data source. In this setting, the learner must incrementally build a training dataset of predefined size while simultaneously learning the underlying task. We find that in IDS, the impact of new sample on the model state both its geometric rela- tionship in the feature space and its prediction er- ror. Leveraging this insight, we propose PEAKS (Prediction Error Anchored by Kernel Similar- ity), an efficient data selection method tailored for IDS. Our comprehensive evaluations demonstrate that PEAKS consistently outperforms existing se- lection strategies. Furthermore, PEAKS yields increasingly better performance returns than dom selection as training data size grows on real- world datasets. The code is available at ht tps:: / //github. http:aa....oooouurauuuuur// PPEASSS
A
High Eir
High Sim
A
A
A
A
A
High Err
A
Low Sim
A
Figure I. Illustration of PEAKS:
The solid line denotes deci- sion boundary. Empty markers represent current training samples, and the star indicates the mean embedding of the circular class. During the selection step, among the three potential candidates for inclusion in the training dataset, the green circle is selected due to its high prediction error (Err) and strong similarity (Sim) to the mean embedding. The model subsequently updated using the newly selected sample along with prior examples. This iterative process is repeated until a total of new examples are collected.
depends fundamentally
tional requirements, and environmental impact from energy consumption (Yang al., 2023). Thus, there a growing need to improve the data efficiency of DNNs ensure their development remains accessible and sustainable.
This pressing need for data efficiency has sparked renewed interest in coreset selection, a well-studied problem in tra- ditional machine learning that is now finding applications in deep learning. A common approach static coreset selection, which identifies a representative subset of the training data in one shot using trained model (Guo et al., 2022). However, this approach assumes example's value remains fixed regardless of the training trajectory and model state an assumption that holds for traditional algorithms (e.g., SVMs) but not for DNNs. Adaptive methods ad- dress this limitation by dynamically reselecting coresets throughout training (Yang al., 2023; Killamsetty et al., 2021b). Yet both approaches require complete datasets be available upfront and need process the entire dataset for selection, making the selection process itself impractical as datasets grow to millions or billions of examples.
1. Introduction
Deep neural networks (DNNs) have achieved remarkable success in recent years, fueled in large part by the availabil- ity of massive datasets (Zhai et al., 2022; Touvron et al., 2023; Liu et al., 2024). However, scaling up training data incurs substantial costs: data collection, increased computa-
SScooo of Computer Science, Georgia Institute of Technology, USA "School of Biological Sciences, Cold Spring Harbor Labo- ratory, USA 'The Cyprus Institute, Cyprus. Correspondence to: Mustafa Burak Gurbuz <mgurbuz66 <mguubbz66@attcc...d. edu.
Online batch selection methods offer a more practical ap- proach by evaluating examples in each training batch and selectively using only the most informative examples for
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PEAKS: Prediction Error Anchored by Kernel Similarity
updates (Nguyen et al., 2024; Wang et al., 2024). However, when data cannot be collected and shuffled beforehand, this local focus becomes problematic. Real-world data is in- herently heterogeneous, with varying quality levels across sources and potential for sudden distribution shifts (Goyal et al., 2024; Delange et al., 2021). In such cases, batches may consist entirely of low-quality samples or reflect tem- porary distribution changes, making selections and training updates based solely on the local batch context unreliable.
for IDS. Finally, we validate our approach through compre- hensive experiments on four datasets, ranging from clean to noisy and imbalanced, to reflect real-world complexity. Our evaluation primarily focuses on fine-tuning pre-trained models with a limited data budget for complex image classi- fication tasks, where the value of principled data selection becomes particularly evident. While our method shows its greatest impact in fine-tuning scenarios, it also provides consistent benefits when training from scratch.
As deep learning applications continue to grow, they often operate in a dynamic and restrictive context that require studying data selection in new scenarios. Consider on- device learning and personalization, where devices continu- ously receive user data that cannot be shuffled beforehand, highly redundant, but needs to be filtered under resource constraints to capture high-value signals for model improve- ment (Moon et al., 2024). Similarly, computer vision ap- plications relying on web-scraped data must contend with streams of images of varying quality, from high-resolution to noisy or irrelevant ones (Li et al., 2017). In such settings, while identifying promising samples is erucial, the greater challenge lies in retaining them effectively-valuabe data should not be discarded after an update, as models typically require multiple exposures to learn effectively, with one- time updates potentially causing disruptive changes (e.g., large biased updates) or being overwritten in subsequent training (Toneva et al., 2019). Instead, we argue that these samples should be retained and repeatedly integrated into training, where their collective interactions over multiple passes help develop more effective representations.
2. Problem Formulation and Related Work
We study data selection for classification; however, the prob- lem formulations and mathematical analysis presented in the next section are applicable to regression as well. To study incremental data selection in a principled way, this work proposes version of IDS that captures the essential aspects of sequentially selecting data with fixed budget. The setting considers a data source Dxource of unknown size. Access to Doource is limited to incremental sampling, where labeled examples (x. y) ~ Dsource are observed one at a time. The goal is to select examples to train [a:; 0), such that it learns the underlying task and performs well on a clean testing set. Following standard deep learning training practices, IDS has three phases (see Figure 2):
1) Initialization: The process begins by randomly sampling m examples from Duounee (m & k) to form the initial train- ing set To. The initial model (with or without pretraining) trains on Tu until convergence, resulting in f(: 8g). In this step. [=;; 0g) learns basic task-relevant features before data selection begins. Next, To grows by selecting new samples.
To address these challenges, we propose Incremental Data Selection (IDS), a novel and pragmatic problem setting for data-efficient learning. In IDS, data arrives as continuous stream, requiring the learner to make incremental selection decisions while training. The streaming nature of IDS ties the utility of each example to the model's evolving state, as selection choices must build upon and refine the current rep- resentation space. Unlike adaptive or online methods where selections are often temporary, IDS retains chosen examples in the training set. These samples are then mixed with newly arriving data during subsequent updates, providing stable representation learning (see Figure 2). Furthermore, this permanent selection mechanism incrementally constructs a coreset of previously seen examples up predefined size, enabling systematic study of how training data size influences overall performance.
2) Data Selection: selection and model update. Consider selection step t:
This phase alternates between example
Example Selection: A potential example (xpp Up) is sam- pled from Dsauce Tr. The decision to keep this sample is made using an acquisition function A(Ip, yp. fe, AA). where A is the parameters of this function, which is up- dated after each selection (e.g.. a selection threshold). This process is repeated until 5 new examples are selected.
Model and Dataset Update: After collecting 5 examples, y a training batch of size is formed by
Snew = {x,,,,)) combining these new examples with b 8 older examples sampled uniformly at random' from T,. The model is then updated using this batch, and the training set is updated as T(+1 T, S.cw- This process is repeated until [Te+1l = k, forming the final set Tena for the next phase.
In the next section, we formalize the IDS problem and re- view related work. Section 3 studies IDS by examining how new samples affect the network's 's predictions and its inter- nal representations, providing insights into the relationship between sample utility and model state. Building on these insights. Section 4 presents PEAKS, an efficient algorithm
3) Final Training: In final phase, the model is fine-tuned on the final training set Tana- This ensures the model learns all selected examples, including the most recent additions.
Alternative sampling mechanism is discussed in Section D.4
2

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Initialization
Data Selection (step = t)
Final Training
Random Samples
Train fo
Data Source
(xp:)l))
Select?
Yes add to Te
New Samples?
Yes
Update
Train fend
A Ax yp. f:: 8x)
fi+1 ft
Tenal = k
[To 77L
IT S k
No
Figure 2. Incremental data selection consists of three phases. First, the model is trained on m randomly selected examples. During the main data selection phase, examples iteratively selected from the data source using the acquisition function A. The model updated each time ó new examples are collected. Once the data budget ls is reached, the model is fine-tuned on all the selected examples.
The objective of IDS is to maximize test accuracy through an acquisition function. The framework involves two hyper- parameters: & (data budget) and ō (selection increment). We set 8 to ensure half of the total training updates (%) occur during data selection. For example, with k 1000 and 200, set 5 = 10. This approach performs consis- tently well and simplifies experimentation by making k the only free variable. See Section B.7 for practical considera- tions for efficiently scaling IDS in large-scale experiments.
high- and low-value samples (Nguyen et al., 2024; Wang et al., 2024). However, when data cannot be collected and shuffled beforehand, batches may not be representative of the broader dataset distribution. In such cases, individual batches might consist primarily of low-quality redundant samples, reflect short-temn distribution shifts, rendering local batch-level selection suboptimal.
Furthermore, these methods typically make temporary se- lection decisions, either discarding samples after a single update (Wang et al., 2024) in single-pass learning or allow- ing previously selected samples to be rejected in subsequent epochs (Jiang et al., 2019). This presents two limitations. First, in single-pass scenarios. discarding samples after one update may be suboptimal, as many models and tasks re- quire multiple exposures to learn training examples. Second, in multi-epoch settings, despite aiming to reduce per-epoch computation, these methods typically process most of the dataset across epochs through continuous rotation and recon- sideration of selected samples. In contrast, IDS constructs training set of predefined size while enabling repeated usage of selected examples. This allows for systematic study of the relationships between dataset size, sample repetition, and model performance-a topic of growing interest given empirically observed neural scaling laws (Sorscher et al., 2022; Muennighoff et al., 2024; Zhai al., 2022).
2.1. Related Work
Static and Adaptive Coreset Selection: Coreset selection aims to identify a subset of the training data that allows a model to achieve performance comparable to training on the full dataset. This is typically approached in one of two ways. The static method selects subset once after training a model on the full dataset, then trains new model from scratch on the reduced dataset (Paul et al., 2021; Sorscher et al., 2022; Meding et al., 2022; Coleman et al., 2020; Mindermann et al., 2022; Xia et al., 2024b; Zheng et al., 2023; Yang et al., 2024). A key limitation of static methods is their assumption that the importance of samples independent of the model's state, as selection is performed in isolation from the training.
In contrast, adaptive methods periodically reselect subsets from the full dataset during training to remain their effec- tiveness (Mirzasoleiman et al., 2020; Yang et al., 2023; Kil- lamsetty et al., 2021b;a). However, both approaches require processing the entire dataset multiple times either during the training phase through repeated reselection-resulfiee in significant computational costs. Furthermore, these ap- proaches become inapplicable when the full dataset is not available upfront.
Connections to Broader Learning Settings: Our work also connects to several related research directions. The streaming nature of data selection shares similarities with streaming active learning (Settles, 2009), while the presence of noisy or imbalanced examples in Doource relates closely to robust and long-tailed learning (Song et al., 2022: Zhang et al., 2023). Although our current formulation assumes a stationary distribution for Doource and focuses exclusively on data efficiency, the IDS framework naturally aligns with con- tinual learning settings where the data distribution evolves over time. While PEAKS is not designed for continual learn- ing, its incremental update mechanism bears resemblance to replay-based strategies commonly used that domain (Delange et al., 2021). We elaborate on these connections and important distinctions in Appendix C.
These methods evaluate examples at the batch level, selecting only a subset of each batch for model updates (Jiang al., 2019; Katharopoulos & Fleuret, 2018). This approach has proven particularly effective for LLM training, where it offers practical way to reduce computational costs when batches contain a diverse mix of
Online Batch Selection:
3

---

PEAKS: Prediction Error Anchored by Kernel Similarity
3. Analyzing Impact of New Data on Model
encouraging logit decreases for incorrect classes. This moti- vates our scoring function for example Tpp
In IDS, after the initialization phase, the model should have improved largely from random guessing and acquired decent zero-shot performance. In line with two recent work (Dong et al., 2024; Xia et al., 2024a), tion at this point is close to the optimal parameter space, and thus the model's behavior between consecutive updates can be approximated via first-order Taylor expansion, operating in the kernel regime".
(ppp x.) = Afolcal EAAM
(3)
assume the network func-
where Co :- yeu Through eq. (2), we can see that the high scoring examples should satisfy two criteria: they should have significant prediction errors (indicating potential for improvement,, and gradient patterns that align with valida- tion examples. The prediction error term identifies examples where the model needs improvement, while kernel coupling term naturally emphasizes examples that are geometrically related in the model's parameter space. Our derivation pro- vides principled way to analyze how training on new example influences the model's predictions.
To effectively select examples in the IDS setting, we need to understand how incorporating a potential example (Ip: Mp) into training impacts the model's predictions validation samples, and therefore able to assess its value for future learning. Here, y E RC is the one-hot-encoded label. The model is defined as [(x; 0z) at training time I, and the pre- dicted class probabilities are given by (z;; 0c)). where a represents the softmax operator, and o (a;; 0,)) i denotes the probability for class i.
3.1. Simplification for Tractable Algorithm
Computing Af. requires calculating the Jacobians of all logits with respect to the network parameters. With modern DNNs containing millions of parameters and hundreds of output classes, this quickly becomes intractable. Therefore, we consider the following simplification.
Let Afe f(x., 0c+1)= f(z,, 01) E RC denote the change in model output for validation sample after one SGD update using example Zp. Using first-order Taylor expansion and gradient descent with learning rate vy:
We assume updates are limited to the classification layer L(x) = W ((x), where W Rdxc and d(x) is the penultimate layer representation (T((rr) re- lates to the network function: fa; W) = L(o(>; 1))- This approximation is justified by empirical findings that later layers undergo more significant changes during train- ing while early layers remain relatively stable (Yosinski et al., 2014), whether using a pretrained model or after suffi- cient initial training. We focus on how training on (Ip, yp) affects validation samples of the same class (i.e., where Co yp). Under last-layer training, K( xpp) factorizes as ((r,)" TT(pp) when i = j. and is zero elsewhere. The kernel simplifies to:
Last Layer Training:
Aʃe = - Vol 0,) V (xp. 01) (o(s(xp 02)) - Up)
(1)
The first term, Vo (of 0,) )a Vo (Ip, 0t), is the Neural Tan-
gent Kernel K( xp) E RCC which characterizes how training on input xp induces changes in the model's predic- tions for input v, through parameter updates (Jacot et al., 2018). The second term, (o 0t)) - Mp). is the gradient of the cross-entropy loss with respect to model outputs. To understand how updating on example Vp affects specific logits, we examine Afo with the kernel entries:
A = Σ 2,
0))(() (2)
K( xp)lt, = (otx)) ifi=
(4)
10
i
Here, [f represents the change in the i-th logit of the validation example, which depends on both the kernel inter-
With this, our objective (Zzp. x.,) becomes:
action K(r,, xx)) pp) j] and the logit gradient for each class j.
The latter measures how much the model's predicted proba- bility distribution deviates from the class distribution, which we refer to as the prediction error. Putting together, the change in logit depends on logit gradients for all classes weighted by their kernel coupling K( xp)) [ j). thus influ- enced by both the geometric relationship between examples in parameter space and the model's 's prediction error.
off 0.))b+
(5)
(,(x,)) Σ of((z, $p
(6)
which approximates the net positive effect of training on Tp for validating example Tu. This is simply:
(zpp x.) E ((zz) ((z,))
(7)
When selecting beneficial training examples, a natural objec- tive is to maximize logit increases for correct classes while
where E(x,)) captures the model's prediction error:
These assumptions require a larger initial To when training from scratch, compared to fine-tuning a pre-trained model.
El(pp) = (1-o(f(zp 0.))(u1)) + Σ [((((pp 0.)1 (8)
4010
4

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Therefore, taking the expectation over all validation exam- ples of class Up-
4. PEAKS for Incremental Data Selection
While traditional coreset selection require access to the full dataset, the scoring function from our theoretical analysis offers a readily usable way evaluate examples incremen- tally. Based on this insight, we propose PEAKS (Prediction Error Anchored by Kernel Similarity), which leverages the- oretical insights from Section 3 that establish a sample's utility depends on both geometric relationships and errors. We introduce two variants: PEAKS-V, V, which requires a vali- dation set and involves periodic computation of class means (eq. (9)), and PEAKS, which eliminates this requirement (eq. (11)). Our experiments primarily focus on PEAKS, which provides an efficient solution for IDS (see Figure 1),
E P. [A(pp: 2.01= = E(xp) p(op); ExrEDiT o(x,)
D
This shows that up improves predictions based on its error p())) and similarity to the class mean embedding. Interest- ingly, these terms compete: maximizing E(x,) aligns with the EL.2N score in (Paul et al., 2021), while the feature prod- uct term captures how close Tp's 's features are the class prototype (mean embedding) (Sorscher al., 2022). Our method suggests using the product of these scores rather than considering them independently.
4.1. Data Selection with PEAKS and PEAKS-V
We argue that combining prediction error with embedding similarity enables more effective selection of informative examples. This product emphasizes examples that are both challenging (i.e., the model currently predicts them poorly) and representative (i.e., their embeddings align with the class prototype). Such examples are likely to induce ben- eficial updates that generalize across many other typical samples in the same class.
Consider potential example (xp: Up) presented selection step l. As discussed in Section 3, E D (Tpp. esti- mates how much a sample improves predictions for class Vp, However, this score is inherently class-dependent, and comparing it directly across classes can be misleading. For instance, change of A in the activation of an output neuron might significantly improve predictions for one class while having minimal impact for another class. Since the IDS sce- nario does not allow comparison across a pool of examples with different classes, we propose a simple solution: weight- ing E [A xx)] by (.) where Cyp (t) represents the number of samples already selected from class yp at time t. Thus, for a sample, we define the score s(x Up) as:
In contrast, relying solely on prediction error could priori- tize high-error examples that may be mislabeled, atypical, partbuuionn-attiulllaly common in real-world data. On the other hand, selecting purely based em- bedding similarity tends to favor overly typical or already well-leareed examples, filtering out noise but also discard- ing hard cases that could drive model improvement. By integrating both criteria, our scoring function selectively promotes hard-but-clean examples.
or
8 s(Ip; Up) = E (1pp Fiv)
(12)
Cy, (1)
Ep,"" (3) is calculated via eq. (11) (9) for PEAKS and PEAKS-V, respectiivel.. This adjustment avoids extremely imbalanced selection and provides consistent ben- efits on balanced and imbalanced datasets (see Section D.5).
3.2. Eliminating the Need for a Validation Set
Our scoring function Ep [A(pp x.)]] requires validation examples to compute class mean embeddings. However, the weight vector Wi,,,1 already serves as a learned pro- totype for class Yp, approximating the true class prototype after initial training on randomly selected data. Therefore, the logit fzz 8,) 0,) = W F. ((x) measures fcature align- ment with this prototype.
4.2. Score-based Dynamic Selection Criterion
Selection decisions require a dynamic threshold that adapts to evolving score distributions during model training'. To achieve this, we maintain a cache C of scores from recently seen examples and make decisions based on percentile rank- ings. For sample (I'p. Yp) with score s(xp, Up), we com- pute its percentile rank within C. Given a selection ratio p%, the acquisition function A(rp, Up- fo.; C) is:
Using this geometric interpretation for the weight vectors, we can approximate our score using x 's features and logits:
E Egg (AIp) Ix]] se E(zp) (0(xp), WENDI)
(10)
Percentile(s(#pp.. Up), C) 2 100- - p
(13)
= =(xx)) (zp. vppppp
(11)
Otherwise, the sample is discarded. Every T updates, the recent score cache C is cleared. For PEAKS-V, we also recompute class mean embeddings. This dynamic selection
We empirically validate the quality of these approximations in Appendix D. showing strong agreement with the exact scoring function.
"The evolution of selection scores is analyzed in Section D.2.
5

---

PEAKS: Prediction Error Anchored by Kernel Similarity
criterion is adapted for all score-based methods in our ex- periments, where each method defines its own acquisition function p( Up, fe; C) with two hyperparameters: the se- lection rate p% and the refresh period r. The complete IDS procedure is formalized in Algorithm 1.
Training From Scratch ResNet-18
-0 0,.80
60-
Accuracy 1%)
+0,40
5. Experimental Results
50-
45
Random PEAKS
Uncertainty R12N
Our experiments spans four datasets: CIFARIO0, Food101 (Bossard et al., 2014), Food101-N (Lee et al., 2018), and WebVision (Li et al., 2017), chosen to assess IDS across varying data quality. CIFARIO0 and Food101 serve as bal- anced, relatively clean datasets, while Food101-N (a distinct dataset from Food-101 collected from different sources) in- troduces class imbalance and label noise. WehVision repre- sents large-scale real-world scenario. Dataset details and selection criteria are discussed Section B.5.
Moderata
CIFRIIO 13k
CFARIDO 30k
Foodio1 20k
Food 10 40k
foos101 -i 40k
Food101 i sok
Figure 3. ResNet-18 without pretraining. Accuracy across three seeds. |m|: 7.5k (CIFAR10O), 10k (Food101), 20k (Food101-N). WebVision experiments are omitted due to the computational cost of training from scratch.
We compare PEAKS against eight baselines. Three vari- ants utilize penultimate layer embeddings-Easy (Welling, 2009), Hard (Sorscher et al., 2022), and Moderate (Xia et al., (023)-selecting examples based on their cosine similarity (highest, lowest, and intermediate, respectively) to class mean embeddings. These are computed either via validation set or approximated using output layer weights as described in Section 3.2. The remaining baselines include EL2N (er- ror magnitude), GraNd (gradient norm) (Paul et al., 2021), uncertainty-based selection (prediction confidence) (Guo et al., 2022), wrong and low confidence and uniform ran- dom selection. The rationale behind baseline selection and details are presented in B.6.
PEAKS are modest, more so on the lower data size end and on the well-curated CIFARI00. This observation reveals once again that random selection serves as a surprisingly strong baseline for data selection on simpler datasets (Guo et al., 2022). Our result could also potentially be explained by the analytic theory of data pruning developed by Sorscher et al. (2022), which postulates that although in theory ag- gressive data pruning with high quality pruning metric can lead to bigger improvement compared to random pruning, in practice achieving such reliable metrics becomes chal- lenging, especially with limited data. Furthermore, with a highly unreliable scoring function, beating random selection with small data budget might even be impossible.
We set the selection rate p% to 20 across all experiments, while 7 is configured to ensure 10 cache refresh operations occur during the data selection phase, unless otherwise spec- ified. While PEAKS algorithm's 's derivation assumes training only the classification layer to derive an efficient selection method, our experiments train the entire architecture. Addi- tional experimental details are provided in their respective subsections and comprehensively documented in Section B.
This creates a fundamental tension: while aggressive prun- ing is desirable for selecting even more efficient data subset, stringent data budgets undermine pruning effectiveness, as training with less data would render the score unreliable. This paradox poses a limitation on all baseline methods that depend on model performance. We address this challenge by utilizing pretrained model for our subsequent experi- ments with more aggressive pruning schemes, specifically ViT-B/16 pretrained on ImageNet- 1k using DINO (Caron et al., 2021) see Section B.2 for details. The pretrained foundation provides a reliable feature space, enabling higher quality data selection decisions even with aggressive prun- ing rates.
5.1. Training from Scratch Results
We first perform IDS by training a ResNet-18 from scratch. In this setting, larger initial training set size m is necessary to ensure the model achieves sufficient performance for meaningful selection decisions, as both PEAKS and other baseline methods rely on model outputs being sufficiently informative to assess sample utility. Experiment details are provided in Section B.
This approach not only resolves the technical paradox but also aligns with contemporary practice, where pretrained models are increasingly ubiquitous and training from scratch is becoming less common. Thus, studying data selection in the context of fine-tuning with limited data offers both practical relevance and an ideal setting for understanding the full potential of data selection strategies.
Figure 3 presents results across three datasets with varying data quality and complexity. PEAKS consistently outper- forms random selection and other baselines, with one ex- ception on CIFAR100 at the 30k data budget. However, the performance gains from data selection methods including
6

---

PEAKS: Prediction Error Anchored by Kernel Similarity
5.2. Results With Pretraining
Table 1. Test accuracy using validation set. Results averaged across
With Validation Set: Table presents results where a portion of the test set is held out as validation data. This val- idation set enables us to establish performance upper bounds for data selection. Specifically, for Food101, Food and WebVision, the validation examples are human-curated representing clear, artifact-free images with correct labels. We evaluate two upper bound baselines: Static, which trains on the entire validation set using epoch-based training, and Incremental, which performs IDS selecting only from the validation examples. These upper bounds are not computed for CIFARIO0 100, as its test set is of similar quality to its train- ing set. For WebVision, upper bounds are only available for the smallest data budget due to limited test set size. We utilize these validation sets to compute class prototypes for PEAKS-V and the three embedding- based methods.
5 seeds. x 1, x2, and x4 refer to dataset sizes of 2.5k, 5k, and
10k examples for CIFARI00. Food101, and Food101-N, and 25K,
50k, and 100k examples for WebVision, respectively.
With Validation Set
CIFARIOO
Food101
Food101-N
WebVision
Upper-Srd (x
N/A
(-.6)) 15.6))
(+007)
Upper Inc (x1)
N/A
(11.)1
(11.0 (-.2(-1.))
Random (x1)
57.0
(226()))1)
(611.) (-.)
Easy Emd (x.).
(+7.7)
(+1.7)
(9.6)
(+36
Mederate Fimb. (x1)
56.7 (4:8.7)
50.1 (42.4)
(-.0
38.8 (+-0.)
Hard Emb. (x1)
55.7 (1:1.2))
40.6 (42.1)
222((18))
21.7 (1111)
PEAKS-V (x1)
61.0 (1:9.9)
53.3 (+2.4)
45.4
45,9 (+004)
Upper-Std (x2)
N/A
66.5 (+100)
(6.(0)
N/A
Upper-Ine Inc (x2)
N/A
(18(220) (42.1)
N/A
Random (x2)
71.1 (1:2.8)
(1222)
(-26)
(4.0()10.4
Easy Emd. (x2)
55.0 (+3.9)
(+1.))
46.7 (=1.)
35.1 (+0.2)
Mederate Fimb, (x2)
68.9 (+-2.4)
47.5 (-.5 (+2.0
Hard Emb. (x2)
71.8 (1:2.5)
49.1 (12.7)
(-15))
18.7 (1.0.)
PEAKS-V (x2)
75.5 (+-2.1)
(4220)
(-2.6
(40.3)
Upper-Std (x4)
N/A
74.3 (+0.6)
74.3
N/A
Upper- Inc (x4)
N/A
(+0.7)
(-.7
N/A
The results show that PEAKS-V significantly outperforms all embedding baselines that are based on distance, demon- strating that such distance to class prototype alone is in- sufficient for data selection in the transfer learning setting. The integration of prediction error weighting in PEAKS pro- vides substantial improvements. However, the performance gap between PEAKS and the upper bounds. particularly noticeable in Food- 101N and WebVision, indicates room for improvement on complex real-world datasets. Moreover, we observe a considerable gap between static and incremen- tal upper bounds in WebVision, which may indicate that the model's convergence on earlier selected samples hinders learning from newer examples. These findings suggest that IDS for complex datasets might benefit from more sophisti- cated regularization techniques or methods improve the plasticity of gradient descent (Dohare et al., 2021).
Rundom (x4)
81.1 (1:0.5)
69.7 (11.8)
55.00-).))
(1.0.)
Easy Emd. (xc)
(::1.)
(29(=16) (6140)
(t.1.)
Moderate Emb. (x4)
77.7 (+0.6)
70.5 (+1.4)
(-.4
53.3 (+0.4)
Hard Emb. (x4)
82.1 (#:07)
61.5 (+2.0)
(118)
16.0 (+0.6)
PEAKS-V V (xc4)
(++0.))
73.1 (+1.5)
(-23)
(+-06)
Furthermore, consistent with recent findings (Xia et al., 2023), selecting examples with intermediate distances to class means (Moderate Emb.) provides reliable baseline across all real-world datasets. Overall, while PEAKS achieves its best results with validation data, it maintains su- perior performance even in scenarios where validation data is unavailable, demonstrating its robustness and practical utility.
5.3. Impact of Selection Ratio
In previous experiments, we used fixed selection rate of p = 20%. This hyperparameter significantly influences the pace of the selection process for fixed data budget k. Higher values (p > 50%) make selection less discriminative by allowing lower-scored examples to be selected, lead- ing to greater overlap with random selection. Conversely, lower values of p focus strictly on the highest-scored sam- ples, making selection more focused but potentially limiting diversity. If selection method truly captures dataset char- acteristics and their relation to previously selected samples and the model state, we would expect it to perform bet- ter with lower p and not benefit from the inclusion of low scored examples. We examine how varying this parameter affects performance across different methods on Food101 and Food datasets, as shown Figure 4.
Without Validation Set: Next, we evaluate our efficient PEAKS variant against baselines that do not require valida- tion data. We also consider the best west-performing embedding method (Moderate) from previous experiments, but use read- out weights rather than validation set to measure distances.
Comparing Tables 1 and 2, we observe that PEAKS shows some performance degradation compared to PEAKS-V. This gap is most noticeable when samples per class are very lim- ited (e.g., x 11), where output layer weights cannot accurately approximate class means. Despite this, PEAKS outperforms all baselines in most cases.
Notably, in the CIFAR100 experiments, the "wrong and low confidence" selection strategy outperforms PEAKS. This result can be attributed to the clean and well-curated nature of CIFARI00, where misclassifications often provide a reli- able signal of sample difficulty. In such clean settings, low model confidence tends to correlate well with informative- ness. However, this advantage diminishes on more complex datasets that contain label noise or outliers.
The results show that none of the methods benefit from stricter selection (p 10%). In IDS, methods rely on the model state for selection, which is shaped by previously se- lected samples, but making optimal decisions incrementally without access to the entire data source makes it challenging to capture true data distributions.
7

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Food101
Table Test accuracies averaged seeds without validation
75.0
set. Data budgets x1, x2 and x4 are same with Table 1.
Without Validation Set
CIFAR100
Food101
Food101-N N
WebVision
Accuracy (%)
72.5
Random
58.0
(811))
(#330)
366
Moderate Finh. (x1)
57.6 (++..)
508 (+2.1)
19.7 (+3.1)
(++04)
70.0
(x1)
60.1 (#6.7)
462 (#2.))
(22.4)
32.2 (+2.0)
GraNd (xc)
(+-5.9)
(42.3) (+23))
35.7 (40.5)
67.5
Uncertainty (XC)
(+:6.)
49.1 (#18)
3633 (433.)
35.2 (+0.3)
PEAKS
Uncertainty EL2N
Wrong Low Conf. (x1)
(1:7.0)
(4:31)
(40.4) (4-3.7
65.0
Moderate
PEAKS (x1)
(++6.)
50.9 (+
41.4 (+3.2)
(+0.0)
Random (x2)
59.6 (+-3.3)
46.5 (+3.0)
44.1 (+0.3)
Food101-N
Moderate Emb. (x2)
67.2
(119)))
50.5 (#3.2)
(40.0)
70
EL2N (x2)
74.2 (4:3.)
(+-.4) (6.6)))
37.5 (+0.2)
GraNd (x2)
76.4 (#2.0)
(+203)
42.3 (+200)
(#0.2)
Accuracy (%)
Uncertainty (x2)
75.9 (d:3.2)
61.1 (4:2.)
45.7 (43.3)
40.6
Wrong Low Conf. (x2)
77.0 (+:2.))
60.3 (+2.9)
448 (+3.7)
40.8 (+0.1)
60
PEAKS (x2)
(1:3.2)
(21..)))
(1:277)
(100.2)
Random (x4)
79.4 (1:1.3)
(1.1.)
55.8 (1:.1))
50.1 (10.1)
Moderate Embh (x4)
(+:1.8)
70.5 (+1.2)
62.0
54.8 (+0.2)
50
EL2N
66.5 (22211)
GraNd (x44
(1:0.))
69.9 (c:1.7)
50.0
48.8 (10.2)
Uncertainty (x4)
82.6 (++0.)
(+118))))
58.7 (+3.0)
45.6
40
Wrong Low Conf. (x4)
83,9 (1:10)
(1.58))))) (1223)
45.3 (10.3)
10%
20%
30%
40% 50% 60% 70% 80% 90%
PRAKS (x:4)
82.7 (++0.)
72.9
62.2 (+227)
(+0.3)
Selection Rate p%
Moreover, since the scoring functions are approximations of the true utility, larger selection fraction helps compen- sate for this uncertainty by accepting more samples that could be valuable. PEAKS maintains superior performance across most selection rates, achieving its peak performance at 30%. As approaches 100%, all methods converge to similar accuracy levels, resembling random sampling. Notable, EL2N improves with higher selection rates. This can be explained by EL2N's focus on high-error examples when selection strict, which may often correspond to noisy or atypical samples. As the selection rate increases, it naturally incorporates more diverse examples, leading to better performance.
Figure 4. Performance comparison using a fixed budget of 10k samples from Food101 and Food101-N datasets with varying se- lection rates (p%). Results averaged across three seeds.
Food101
Accuraicy g
72.5
70.0
67.5
65.0
5-50
T-100
T-300
Food101-N
65
Accuracc 193
55
50
5.4. Impact of Refresh Period (r)
40
(default) Hyperparameter Values
T-300
Another hyperparameter in IDS is the refresh period т, which determines how frequently the score cache is reset during selection. This cache is used to compute percentile thresholds for dynamic selection criteria. While we do not expect selection quality to be highly sensitive to т, as it only affects range of recent scores considered when computing percentiles, we nonetheless evaluate its impact empirically. We vary from 50 to 300 (default is 100 for all datasets except VisVin on two datasets following the same configuration described in Section 5.3. As shown in Figure 5, we observe results are consistent across range of values.
Figure 5. Impact of refresh period performance. Results averaged over three seeds, using a fixed selection budget of 10k examples from the Food101 and Food101-N datasets.
thermore, randomness in the sampling process means not all samples are guaranteed to encountered. To alleviate these variations, our analysis focuses on selecting 30% of samples from CIFAR-100 (15k) and Food- 101 (~23k)). This larger data budget, compared to earlier experiments (Section 5.2). increases the likelihood of most samples being encountered at least once. Under these conditions, approximately 85% of all samples are seen by each method, with a high overlap (~~90%) in the samples encountered across methods (see Section B.3 for details). Figure 6 illustrates the Jaccard similarity between the final training sets Tend selected.
5.5. Overlap in Selection
We next compare which samples are selected by differ- ent strategies. Since examples are sampled as (xp, yp) ~ D Dsouce Te, and T, varies across methods, the order which samples appear differs, even with fixed random seeds. Fur-
8

---

PEAKS: Prediction Error Anchored by Kernel Similarity
PEAKS shows partial overlap (0.36-0.40) with EL2N, in- dicating that kemel similarity plays significant role in guiding selection, too, rather than prediction error alone. Our analysis of Food101 samples selected exclusively by PEAKS (see Section D.3 in Appendix) suggests PEAKS may favor examples near decision boundaries between simi- lar classes, while EL2N tends to select poorly framed or mis- labeled samples. Interestingly. EL2N and PEAKS achieve the same accuracy (rounded CIFAR 100. Similarly, EL2N and Moderate exhibit com- parable performance on Food101, despite having very low overlap in selection (0. 18). These findings underscore an important aspect of data selection: multiple distinct subsets can serve as qualitatively equivalent training sets. Once a sample selected and influences the model state, it inher- ently changes the utility of the remaining samples, leading to different but likely equally effective selection paths.
Food107-N
PEAKS
Random
70
Accuracy (%)
60
50
40
single decimal point)
4k
8k
38k
32k
64k
128k
Webvision
65
PEAKS Random
60
Accuracy (%6
45
40
35
CIFAR100 30% Detaset
Food101 30% of Dataset
PEMS
AGMS
12.5k
25k
50k
100k
200k
400k
Number of Training Samples
Uncent Socertt Erald 11.M
0.40
1.50
6.52
0.20
0.45
R22
0.30
1:00
0.5
Figure 7. Accuracy vs. training dataset size for Food- 101 N and WebVision, averaged across seeds (shaded areas show std.)
0.30
0.32
3.02
0.19
0.45
Urcent Miceoon Grati
0:35
1:50
0.15
0.43
0.36
0.45
5.45
0.21
1.00
0.43
PEARS
0.29
Grald
Moderate (05.30 283
Uncer (25.88]
PEAKS 182.0%1
C12M
Grald (72,35)
Vederate (76,9%)
Uncer: B%)
text where examples arrive as continuous stream and must be selected without access to the full data source. IDS bridges the gap between the traditional coreset selection procedure and the practical needs of DNNs. Our analysis revealed fundamental relationships between newly arriving data samples and their potential for model improvement. These insights led to the development of PEAKS, an effi- cient data selection method tailored for IDS. Most notably, on WebVision a large-scale weal-world dataset, PEAKS achieves comparable performance while requiring only one- fourth of the data needed by random selection. Our results suggest that principled data selection strategies can substan- tially reduce data requirements for modern DNNs while maintaining model performance.
(85.4%)
(85
(96.00
(78.4%)
Figure 6. Jaccard similarity (intersection over union) between se- lected samples across different methods. Accuracies rounded to single decimal point shown in parentheses (single seed).
5.6. Scaling Against Random Selection
Finally, we evaluate PEAKS against random selection across broader range of data budgets, focusing on Food101-1 -N and WebVision due to their larger sizes. Figure 7 shows that the advantage of PEAKS over random selection be- comes more significant as the dataset size increases. This is particularly evident in WebVision, where random selection requires approximately 4x more data to achieve similar per- formance levels. For instance, PEAKS with 50k and 100k samples achieves comparable accuracy to random selection using 200k and 400k samples, respectively. Similarly, on 101NN random selection consistently needs double the data to achieve comparable performance. These results demonstrate that effective data selection strategies can sub- stantially reduce data requirements for DNNs, offering practical path toward more efficient and sustainable training.
A Acknowledgements
Xingyu Zheng is supported by a William R. Miller Fellow- ship at the School of Biological Sciences at Cold Spring Harbor Laboratory. The authors are grateful to the ICML 2025 reviewers for their constructive comments.
Impact Statement
6. Conclusion
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
We introduced the Incremental Data Selection (IDS) prob- lem, addressing data selection for DNNs in dynamic con-

---

PEAKS: Prediction Error Anchored by Kernel Similarity
References
Guo, C., Zhao, B., and Bai, Y. Deepcore: A comprehensive library for coreset selection in deep learning. In Strauss, C., Cuzzocrea, A., Kotsis, G., Tjoa, A. M., and Khalil, I. (eds.), Database and Expert Systems Applications, PP. 181-195 Springer International Publishing, 2022.
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
Cacciarelli, D. and Kulahci, M. Active learning for data streams: survey. Machine Learning, 113(1):185-23) 2024.
Huang, S.JJ., Zhao, J.-W., and Liu, Z.-Y. Cost-effective training of deep cnns with active model adaptation. In Proceedings of the 24th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, KDD *18. pp. 1580-1588, New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 9781450355520.
Caron, M., Touvron, H.. Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 165009660,00 2021.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances neural information processing systems, 31, 2018.
Chen, Y., Yuille, A., and Zhou, Z. Which layer is learning faster? a systematic exploration of layer-wise conver- gence rate for deep neural networks. In The Eleventh International Conference on Learning Representations, 2023.
Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., et al. Accelerating deep learning by focusing the biggest losers. arXiv preprint arXiv:1910.00762, 2019.
Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Selec- tion via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020. URL https: http://pppeeeeeeeeeeeeee Forum? id=HJg2b0VYDr.
Katharopoulos, A. and Fleuret, Not all samples are cre- ated equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525- 2534. PMLR, 2018.
Killamsetty, K., Durga, S., Ramakrishnan, G., De, A., and Iyer, R. Grad-match: Gradient matching based data sub- set selection for efficient deep model training. In Interna- tional Conference on Machine Learning, pp. PMLR, 2021a.
Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in clas- sification tasks. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, pp. 1-1, 2021. doi: 10.1 109/T09MM..2000 2021.307466.
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset selection for efficient and robust learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8110-8118, 2021b.
Dohare, S., Sutton, R. S., and Mahmood, Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.00 06325, 2021.
Dong, Y., Phan, H., Pan, X., and Lei, Q. Sketchy moment matching: Toward fast and provable data se- lection for finetuning. In The eiihth- Annual Conference on Neural Information Processing Systems, 2024. URL /tctts::/openreeee httpreewwwwwwweeeeeeeeeeeeeeeeee net/forun id=yAAgWBMGiT.
Lee, K.-H., He, X., Zhang, L., and Yang. L. Cleannet: Transfer learning for scalable image classifier training with label noise. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Li, W., Wang, L., Li, W., Agustsson, E.,, and Van Gool, L. Webvision database: Visual learning and understanding from web data, arXiv preprint arXiv: 1708 020020202222..... 2017.
Goerttler, T. and Obermayer, K. Unveiling the dynamics of transfer learning representations. In ICLR 2024 Workshop on Representational Alignment, 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXhv:2412.19437 2024.
Goyal, S., Maini, P., Lipton, Z. C., Raghunathan, A., and Kolter, J. Z. Scaling laws for data filtering-date cura- tion cannot be compute agnostic. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2771112771, 2024.
Meding, K., Buschoff, L. M. S., Geirhos, R., and Wichmann, F. A. Trivial or impossible - dichotomous data difficulty masks model differences (on imagenet and beyond). In
10

---

PEAKS: Prediction Error Anchored by Kernel Similarity
International Conference on Learning Representations, 2022. URL https::////... http//pppeeeeeeeeeeeeeeeeeeeee.e forum? id-C_vsGwEI
Song, H., Kim, M., Park, D., Shin, Y., and Lee, J.-G. Learn- ing from noisy labels with deep neural networks: A sur- vey. IEEE transactions neural networks and learning systems, 34(11):8135-81)) 8153, 2022.
Mindermann, S., Brauner, M., Razzak, M. T., Sharma, M.,, Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In In- ternational Conference on Machine Learning, pp. 15630- 15649. PMLR, 2022.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems, volume 35, pp. 16796- 16808, 2022.
Mirzasoleiman, B., Bilmes, J., and Leskovec, J. Coresets for data-efficient training of machine learning models. In Proceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6950-6960. PMLR, 2020.
Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., and Gordon, G. An empirical study of example forgetting during deep neural network learning, In International Conference on Learning Representations, 2019.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B.. Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint UrXiv:2302.13977 2023.
Moon, J., Lee, H. S., Chu, J., Park, D., Hong, S., Sco, H., Jeong. D., Kong. S., and Ham, M. A New Frontier of AI: On-Device AI Training and Personalization. In 2024 IEEF/ACM 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE- SEIP), pp. 323-333, Los Alamitos, CA, USA, April 2024. IEEE Computer Society. doi: 10. 10.11145 00.1145//663447736339666
Wang, J. T., Wu, T., Song, D., Mittal, P., and Jia, R. GREATS: Online selection of high-quality data for LLM training in every iteration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https::/p/// /pwnrevieweeeeeeeeeeeeeeee net/ forum? id=232VcN8tSx
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Wang. L.. Zhang. X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: Theory, method and application, 2023.
Neyshabur, B., Sedghi, H., and Zhang, C. What is being transferred in transfer learning? In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 512-523. Curran Associates, Inc., neurips..
Welling, M. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Confer- ence Machine Learning, ICML '09, pp. 1121-1128, New York, NY, USA, 2009, Association for Comput- ing Machinery. ISBN 9781605585161. doi: 10. /tttts://doi.. http://oooooo/o/o/
2020. URL /t/ppppee_flllespppppp////////11 ce cc/paper_fiiiiccccccc 000TTAA773555531aaaffaaaa2222222 pdf.
1 1145/1553374115535117 URL https: /
10. 1..11451155374.155555
Nguyen, D., Yang, W., Anand, R., Yang, Y., and Mirza- soleiman, B. Mini-batch coresets for memory-efficient training of large language models. arXiv preprint arXiv:2407. 195.0 2024.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: selecting influential data for targeted instruc- tion tuning. Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024a.
Paul, M., Ganguli, S., and Dziugaite, G.K. Deep learning on data diet: Finding important examples early in training. Advances in neural information processing systems, 34: 20596-20607, 2021.
Xia, X., Liu, J.. Yu, J., Shen, X., Han, B., and Liu, T. Mod- erate coreset: A universal method of data selection for real-worke data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2023. URL https: hwwpr/veee.eeeeeeeoo/eeeeeoorewm net forum? i id= 70=7EEECbba...
Ren, P., Xiao, Y., Chang, X.. Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., and Wang, X. A survey of deep active learning. ACM Comput. Surv, 54(9), oct 2021. ISSN 0360-0300. doi: 10.1 10.1145/3472291. URL https:// doi. org/10.....0
Xia, X., Liu, J., Zhang, S., Wu, Q., Wei, H., and Liu, T. Refined coreset selection: Towards minimal coreset size under model performance constraints. In Forty-first Inter- national Conference on Machine Learning, 2024b.
Settles, B. Active learning literature survey. 2009.

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Yang, S., Cao, Z., Guo, S., Zhang, R., Luo, P., Zhang. S., and Nie, L. Mind the boundary: Coreset selection via reconstructing the decision boundary. In Forty-first International Conference on Machine Learning, 2024.
Yang, Y., Kang, H., and Mirzasoleiman, B. Towards sustain- able learning: Coresets for data-efficient deep learning- In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 39314-39330. PMLR, 23-29 Jul 2023.
Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How trans- ferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
Zhang, Y., Kang, B., Hooi, B., Yan, S., and Feng, J. Deep long-tailed learning: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(9):10795- 10816, 2023.
Zheng, H., Liu, R., Lai, F., and Prakash, A. Coverage-centriic coreset selection for high pruning rates. In The Eleventh International Conference on Learning Representations, 2023. URL https:://///.....t. www.eeereeeew.eeeeferreeeeeeeeee id=QwKvL6wC8Y1.
Zhou, Z.-H. A brief introduction to weakly supervised learning. National science review, 5(1):44-53, 2018.
12

---

PEAKS: Prediction Error Anchored by Kernel Similarity
A. IDS Algorithm
Algorithm Incremental Data Selection
1: Input: Initial set To. data budget k selection rate p, refresh period Acquisition function A Vp fo: C)
2: Train initial model fu on To
3: Initialize C <- 0, i <- O, Snew <- 9
4: while [Tel < k do
5: neselected t 0
6: while Niselocted < and [Tyl Dsellcaeed < le do
Sample (zp. Up) from Douure
8:
if A Up Jr; C) accepts then
9:
Snew t- Snow U (Tpp 8p)}
10:
nselected t Mselected +1
11:
end if
12:
end while
13:
batch f U Random(Ti b 6)
Update model fi+1 using batch
15:
L ++1
16:
T. T--1 U Snew
17:
Snew +0 0
18:
if't mod т -= 0 then
19:
C 5-0 0
20:
end if
21:
end while
Fine-tune final model on Tend
B. Experiment Details
B.1. Details of Training from Scratch Experiments
We conducted our initial experiments using a ResNet-18 architecture trained from scratch on CIFAR100, Food101, and Food101-N datasets. For CIFAR100, following standard practice for low-resolution images, we modified the network architecture by replacing the initial 7x7 convolution layer (stride 2) and 3x3 max pooling layer (stride 2) with single 3x3 convolution layer (stride 1).
Training Configuration:
Optimizer: AdamW W with learning rate 0.001 and weight decay 0.01
Total training steps: 30000
Batch size: 128
Data normalization: Per-channel mean and standard deviation using ImageNet- 1k statistics
Data augmentation: Random cropping and random horizontal flipping (p 0.5)
Learning rate schedule: Reduced by factor of 10 at the start of final training phase
While training for 30000 steps allows models to overfit the training dataset, we did not observe any adverse effects on validation performance. We picked the optimizer, learning rate, and weight decay based on initial phase training performance (same for all baselines, with randomly selected m examples).
Dataset Confiuurations:
CIFAR-100: 15k and 30k samples
13

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Food-101: 20k and 40k samples
:::d 40k and 80k samples
We selected these dataset sizes to represent two distinct scenarios: minimal viable budget that enables successful training from scratch (first number), and a doubled budget to study larger budget. In the literature, dataset size is commonly reported as a fraction of the total dataset. However, our datasets vary drastically in terms of total samples. For example, 30% of CIFARIO0 means 15000 samples, while in Food101-N the same ratio means 90000 samples. Therefore, we focus on absolute sample counts rather than fractions. Due computational constraints, across wider range of dataset sizes. Training from scratch is particularly resource-ineessive compared to fine-tuning pre-trained models, which influenced our decision to focus on these specific dataset sizes. For similar computational reasons, we excluded WebVision from our
could conduct exhaustive experiments
experiments.
IDS-Specific Parameters:
Selection increment (6):
4 for CIFAR-100 and Food-101
8 for Food- 101N (adjusted for larger dataset size)
Cache Refresh Period (r): 400 steps
Selection Rate (p): 20%
Initial Dataset Size (m)) CIFAR-100 7.5k, Food-101 10k, Food-10IN; 20k. In other words, half of the low budget scenario.
Initial Training Steps: 10000
The cache refresh period proved robust across wide range of values. However, if is set too high cache refreshing is not performed frequently enough, selection rates consistently drop it becomes harder collect samples. This occurs because selection scores (PEAKS, Moderate, EL.2N, and Uncertainty) typically decrease as the model learns, and without regular cache refreshes, high scores from early training prevent newer examples from being selected.
B.2. Details of Pretraining Experiments
Our primary focus was data selection for fine-tuning pretrained models, reflecting their ubiquity in practice training from scratch becomes increasingly uncommon. For these experiments, we used ViT-B/16 architecture pretrained using DINO self-supervised learning on ImageNet-lk.
Training Configuration:
Optimizer:
WebVision: AdamW (Ir=0.0001, weight decay=0.01)
All other datasets: SGD with momentum (Ir=0.001, weight decay=0.0001)
Training steps:
Weiiiison: 6000
All other datasets: 2000
Batch size: 128
Data normalization: ImageNet-1k 1k per-channel mean and standard deviation. We upscaled CIFAR100 images to 224 x 224 x 3.
Data augmentation: None
Learning rate schedule: None
14

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Notably, omitted data augmentation and learning rate scheduling, which less critical when fine-tuning pretrained models compared to training from seratch. This simplification also enables clearer analysis of the relationship between dataset size and model performance. Similar to the meining-from-sereee case, we picked the optimizer, learning rate, and weight decay based on initial phase training performance.
Dataset Configurations:
CIF,R-100, Food-101, Food- 2.5k, 5k, 10k samples
Vision:::: 25k, 50k, 100k samples
These sizes correspond to approximately 25, 50, and 100 samples per class in the training dataset.
IDS-Specific Parameters:
Selection increment (6):
CIFAR-100, Food-101 101, Food-10IN: 2, 4, and for respective data budgets WebVision: 14, and 30.
These values ensure approximately half the training steps occur during the data selection phase.
Cache Refresh period (T):
WebVision 300
Other datasets: 100
Configured for roughly 10 refreshes during selection
Selection rate (p): 20%
Initial dataset size (mn):
WesVision: 10000
Other datasets: 1000
Approximately 10 samples number of classes, though not necessarily balaned
Initial training steps:
WebVini: 1000
Other datasets: 100
These hyperparameters were intentionally set to round values based on dataset characteristics without extensive optimization. This approach avoids potential bias, as different selection methods may achieve optimal performance under different configurations. Additionally, it provides more realistic evaluation, optimal parameter tuning may be impractical in real-world scenarios where data sources can be unbounded.
The above configurations were consistent across all experiments with or without validation set. For experiments involving validation sets, we made the following additional considerations:
Validation Set Configuration:
Rood-101 and Food-1011 Reserved clean, human-annotated samples from test matching training budget (2.5k, 5k, 10k)
WibVis::n: Fixed 25k validation set due to limited test set size (50k total)
CIFAR-100: Reserved 2.5k, 5k, 10k from training set due to test set size limitations (10k) and similar distribution to training data.
For validation set experiments, we used 5 random seeds (versus for other experiments) to obtain more reliable generalization estimates given the reduced test set size. The validation sets were used to compute class mean embeddings every т steps for embedding- based methods (Table 1).
15

---

PEAKS: Prediction Error Anchored by Kernel Similarity
B.3. Details of Overlap in Selection
In Section 5.5, we analyze the overlap between samples selected by different methods using Jaccard Similarity (intersection over union). It is important to note that Jaccard Similarity operates sets and is therefore insensitive ordering. For instance, if two methods both select the same sample but assign it different importance rankings, the Jaccard Similarity treats these equally as selections.
Our experimental setup follows the configurations detailed in Section B.2 for pretrained models, with one key modification: we increased the selection increment ò to 16 to ensure sufficient examples could be selected within the allocated training steps. The initial training dataset of size m 1000 was excluded from Jaceard Similarity calculations as these samples were randomly collected rather than chosen by selection methods.
To validate that methods encountered similar samples during the selection phase, we also computed the Jaccard Similarity of seen samples (see Figure 8). This analysis confirmed that approximately 90% of samples seen by one method were also encountered by others, ensuring a fair comparison of selection strategies despite the sequential nature of IDS.
CIFAR100 30% of Dataset Seen Indices
Food101 30% of Dataset- Seen Indices
PEAKS
PEAMS
BL2N
8:28
Uncert Mederate: Gretc
Grate
0.87
1:00
0.90
0.89
Moderale
0.86
0:87
0.58
0.89
1.00
Uncert.
0.87
0:38
0:89
PLAKS
L1.2N
Grafd
Moderaate
Uncert
PLAKS
L1.2N
Greld
Moderate
Uncertt
Figure 8. Jaceard similarity between sets of samples encountered (scen) by different selection methods during training on CIFAR- (left) and Food- 101 (right) when selecting 30% of the dataset. High values ( 0.87-0.90) across all method pairs indicate that methods encounter largely overlapping sets of samples during selection, despite making different selection decisions.
B.4. Details of Scaling Against Random Selection Experiments
For the scaling experiments presented in Section 5.6, we followed the experimental configuration described in Section B.2, adjusting only the selection increment ö based on dataset size to ensure balanced data collection throughout the training process.
For Food-101N we started with = 1 for the 2k budget and doubled it for each subsequent budget doubling. For Web Vision:
12.5k budget: 8 = 5
25k-100k budgets: Same configuration Section B.2
200k budget: 60
400k budget: 8 100
These adjustments ensure that data selection keeps pace with model training, avoiding both too rapid and too slow data collection.
B.5. Choice of Datasets
Our experiments require datasets that enable thorough assessment of IDS across varying data quality conditions while main- taining reliable evaluation metrics. We selected four datasets CiciuasttsCCFAROOOO CIFARI00, Food101, Food101-N, and WebVision based on several criteria that align with our experimental objectives (see Table 3 for dataset statistics).
16

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Num Classes
Train Data
Dataset
Train Size
Test Size
Train Label Balance
Noise
CIFARI00
100
50k
10k
Yes Yes Mild Imbalance
00%
10od101
101
75k
25k
8%
Food101-N
101
310k
25k
20%
Web Vision
1000
2.4M
50k
Strong Imbalance
200
Table 3. Dataset statistics and characteristics used in our experiments.
The primary consideration in our dataset selection was the need to evaluate IDS across a spectrum data quality scenarios. CIFARI00 serves as our clean, well-curated baseline dataset, being both balanced and widely accepted data selection literature. Food101 represents middle ground. containing approximately 8% label noise primarily manifesting color intensity variations and occasional labeling errors. Food101-N introduces substantial quality variation with an estimated 20% noise level, while VisVision provides a large-scale real-world scenario with naturally occurring noise patterns.
Several crucial factors influenced our dataset selection: First, reliable evaluation necessitates clean and sufficiently large test sets, particularly as we partition these for validation PEAKS- V experiments and upper bound analysis. Both Food101-N and Web Vision satisfy this requirement by providing human-verified test sets of relatively large size.
Second, explicitly avoided datasets elosely related to ImageNet (such Tiny ImageNet) to prevent potential bias, as ImageNet serves as pretraining source.
Third, the datasets needed to contain sufficient training examples to make the selection process meaningful. We excluded several otherwise suitable datasets that contained not enough examples (e.g., less than 50k).
Finally, we prioritized well-established datasets to ensure reproducibility and contextual relevance. While Food101 and WisVin are less commonly used in coreset selection literature, they are widely recognized in broader computer vision and machine learning research, particularly in studies involving learning from noisy data and real-world distributions.
B.6. Choice of Baselines
We compare PEAKS against eight baseline methods that can be naturally adapted the IDS setting. Our selection prioritizes methods that can make independent decisions for each example without requiring access to a pool of candidates. The baselines can be categorized into four groups based on their selection criteria:
Embedding-basee Methods. We implement three variants that utilize penultimate layer embeddings:
Selects examples with the highest cosine similarity to their class embedding (also known as "Herding" in literature). The class embedding computed either via validation set means or output layer weights, depending on the context.
Easy Embedding:
Hard Embedding: Selects examples with the lowest cosine similarity to their class embedding. targeting the least prototypical examples.
Selects examples with intermediate similarity scores. For selection rate of p%. it chooses examples whose distances fall between the (50 th and (50 + )thh percentiles in cache C.
Moderate Embedding:
Error-based Method. include EL2N, which selects examples based on the L2 norm of the error vector (difference between predicted probabilities and one-hot encoded labels).
Gradient- Method. GraNd selects examples with larger gradient norm magnitudes. This method is computationally most expensive as requires computing per-example gradients through backpropagation.
Uncertainty-bacee -based Method. We implement the "least confidence" metric that selects examples based on cooofmaaxloogittt) This uniquely unsupervised, requiring no label information.
17

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Wrong and Low Confidence Method. Wrongly predicted samples but with lower confidence. We assign a score of 0 correctly predicted examples and score of 1 max(softmax) incorrect ones.
Random Selection. This is uniform random sampling from D Dourrce-
Several popular data selection methods were not included in our evaluation due to incompatibility with the IDS setting:
Methods requiring pool of examples to approximate dateset-wide characteristies
Approaches based on bilevel optimization or multiple model training
Methods requiring auxiliary reference models
These exclusions stem from either computational constraints or fundamental incompatibility with the sequential nature of IDS. Rather than potentially misrepresenting these methods by modifying them beyond their intended use, we focus on baselines that naturally adapt to the incremental setting while maintaining reasonable computational requirements.
B.7. IDS Practical Considerations
When implementing IDS scale, several practical considerations become important. First, processing individual examples through forward passes is computationally inefficient, as it fails to leverage the parallelization capabilities of modern deep learning frameworks and hardware. To address this, we process potential examples in batches during the selection phase. While we compute model outputs for the entire batch in parallel, the selection process remains sequential: we evaluate examples in order from the start of the batch using our acquisition function, select the first examples that meet our criteria, and discard the remaining examples. This approach maintains the incremental nature of our selection process and does not utilize any information across samples in the batch while allowing us to benefit from batched forward passes for computational efficiency.
Second, instead of updating the training set T; immediately after each selection, maintain buffer of recently selected examples and integrate them into T; during the cache C refresh operation every updates. This deferred update strategy aligns better with typical data loader implementations, which often prefetch data for future updates. Our empirical evaluation shows no significant performance degradation compared to immediate updates, while the implementation becomes more efficient and robust.
Finally, we sample potential examples from D sourcee \ TJ rather than directly from D source. This ensures that already selected examples are not reconsidered, eliminating the need for explicit duplicate detection. We do not require any method to detect exact duplicates as in practice this can be handled perfectly with simple data structures.
These tricks are consistently applied across all baseline methods in our experiments, ensuring fair comparisons. These practical considerations enable efficient scaling of IDS while preserving its core theoretical properties and selection behavior.
C. Extended Related Work
Active Learning: In Active Learning (AL), model queries an oracle label the most informative examples from large unlabeled pool (Ren et al., 2021; Settles, 2009; Huang al., 2018). Streaming AL scenarios (Cacciarelli & Kulahci, 2024), which evaluate examples incrementally as they arrive, are most related to our work. However, key differences exist. AL aims to minimize labeling costs, while IDS assumes labels are readily available, simplifying sample evaluation.
However, IDS introduces unique challenges. In AL, decisions about labeling a sample can often be deferred until more information is gathered. In contrast, IDS requires immediate decisions. Also, while AL typically emphasizes minimizing labeling costs and often involves resetting and retraining the model on the final dataset, IDS simultaneously trains the model efficiently during the data selection process. Thus, while AL and IDS have complementary objectives, they operate under distinct goals and constraints.
Robust, Maig,,alld,, and Continual Learning: In IDS, the data source Dsouree may be noisy and class-imbalanced. Robust learning with noisy labels has been extensively studied (Song et al., 2022; Zhou, 2018). Similarly, Long- Tailed Learning addresses scenarios where a subset of classes has disproportionately many samples, while most classes are underrepresented (Zhang et al., 2023). Although these challenges are relevant to IDS, they are not its primary focus.
18

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Continual Learning (CL) methods aim to train DNNs on evolving data distributions while retaining previously acquired knowledge (Wang al., 2023; Delange al., 2021). A common strategy in CL is replay past examples by mixing them with new data during training- approach that bears resemblance to the model update mechanism in IDS, where previously selected samples are combined with new selections. This similarity, along with IDS's inherently incremental setup, suggests a potential connection between the two paradigms.
However, our focus in this work solely on data efficiency under the assumption that Douuce remains stationary. is important to note that PEAKS is explicitly designed to select samples that maximize changes in model logits- an intuitive objective for standard learning scenarios. In the context of CL, however, such samples may not be optimal for mitigating catastrophic forgetting promoting knowledge retention and transfer- key objectives in CL. Addressing these challenges would likely require alternative selection criteria. Extending IDS and PEAKS to the CL setting is promising direction for future work, but one that would necessitate significant conceptual and algorithmic adjustments.
D. Additional Results
D.I. Approximate vs. Exact PEAKS Scores.
In Section 3, we introduced an exact score for evaluating example utility (Eq. (3)), derived directly from first-order approximations of model updates. However, computing Eq. (3) requires the Jacobian of every validation sample with respect all model parameters matrix of size (number of parameters) (number of classes) mlaknn computationally prohibitive for modern networks.
To address this, we made two simplifications. First, we assumed that most learning occurs in the final classification layer, allowing us to approximate the effect sample using only the final layer weights (Eq. (9)). This is supported by prior work showing that (i) early layers are more transferable across tasks (Yosinski et al., 2014; Neyshabur et al., 2020), (ii) pretraining leads shallow layers converge more quickly (Chen al., 2023), and (iii) early layers change less during fine-tuning (Goerttler& & Obermayer, 2024).
Second, avoid the need for validation set, we approximated class mean embeddings using the output weights of the final layer, yielding Eq. (11). This formulation underlies the main variant of PEAKS used in our experiments.
To empirically assess the quality of these approximations, we compare all three scoring variants Eq. (3), Eq. (9), and Eq. (11)-in a controlled setup using CIFARIO and CIFAR100. Due to computational constraints, we use ResNet-18 instead of VIT.
We begin by pretraining ResNet- 18 CIFAR100, then fine-tune it on 3000 CIFARIO examples to simulate the initialization phase used in IDS. We hold out 500 additional CIFAR 10 samples for validation and evaluate the selection scores of 1000 unseen examples using all three scoring variants.
Figure presents our comparison. On the left, we report Spearman rank correlations between scoring methods. The center and right plots show pairwise scatter plots of example rankings under exact vs. approximate methods. We observe strong agreement ranking between the exact and approximate scores, indicating that our simplifications preserve the key ordering needed for effective selection.
Spearman Rank Correlation (across seeds)
Scatser Piot Rankings
Scatter Piot Rankings
Equation 3
1300
1000
Perfect Agreement
1.00
0.98
0.92
300
Ranking by Eouation
Equation 11 Equattoon 7
0.98
1.00
0.93
400
Ranking
400
200
200
0.92
0.93
1.00
Agreement
0
220
403
sčo
1000
250
<00
800
8do
1010
Equation
Equation 7
Equation 11
Ranking by Equation 3
Ranking by Equetian 3
Figure 9. Left: Spearman rank correlation of 1000 examples scored by exact and approximate methods. Middle: Score ranking agreement between Eq. (3) and Eq. (9). Right: Agreement between Eq. (3) and Eq. (11). Scatter plots are based on a single seed.
19

---

PEAKS: Prediction Error Anchored by Kernel Similarity
D.2. Why Do We Need Cache C Refresh?
To empirically demonstrate the necessity of cache refreshing discussed in Section 4.2, we analyze how selection scores evolve during training. We focus selecting 30% of samples from CIFAR-100 (15k) and Food-101 ( (23k), consistent with our overlap analysis setting in Section 5.5. Since different methods operate vastly different scales (e.g.. Moderate selection's s cosine distance is bounded while GraNd can be arbitrarily large), we normalize each score by its maximum value of rolling average for comparison.
Figure 10 illustrates the evolution these normalized scores across training. Several key patterns emerge. All methods show general decreasing trend in their scores as the model learns, though different rates. PEAKS exhibits the sharpest decline, reflecting its strong dependence prediction error which naturally decreases during training. Moderate selection maintains relatively stable scores compared to other methods, though still showing gradual decrease. This declining pattern highlights why cache refreshing is crucial: without periodic resets, early high scores would dominate the cache's s high percentile band, making it increasingly difficult to select new samples. While our chosen works well in practice, these observations suggest that optimal refresh rates could potentially be method-specific, given their different temporal dynamics.
CIFAR100 30% of Dataset
Food101 30% of Dataset
Normalized Rolling Avg (Window- 5500)
Nornalized Rolling Avg [Vindow=500)
1.0
3.0
0.8
PEAKS
PEAKS
GraNi
GraNd
Moderate
Uncert.
Uncert.
100000
120009
Selection Decisions (Index)
Selection Decisions (Index)
Figure 10. Evolution of normalized selection scores across training for different methods. Each line shows rolling average (window size=500) of selection scores, scaled relative to their maximum rolling mean to bound scores between and 1. This normalization enables comparison of temporal trends across methods despite their different scales.
D.3. What Does PEAKS Capture Additionally Compared to EL2N?
To better understand how PEAKS' kernel similarity term influences selection decisions compared to pure error-based methods like EL2N, we conducted a detailed analysis of their selections on Food-101. We chose this dataset for its visual interpretability, opposed the low-resolution CIFAR-100 or the complex 1000-class WeiVision dataset.
We analyzed selections during the middle 100 steps of the data selection phase when selecting 30% of Food-101 (~23k samples), consistent with our setting in Section 5.5. During this period, both methods encountered approximately 4000 examples. Due to the sequential nature of IDS, we focused on the 167 examples seen by both methods. Among these, 23 were exclusively selected by PEAKS and rejected by EL2N, while 24 showed the opposite pattern. Figure 11 visualizes the top 10 highest-scoring examples from each group, annotated with ground-truth labels and top predicted classes.
PEAKS' exclusive selections reveal an interesting pattern: they often represent examples that lie at the boundary between visually similar classes. For instance, the model shows uncertainty between spaghetti carbonara and bolognese (image- 1), club sandwich and grilled cheese sandwich (image-2), or poutine and french fries (image-3). While this observation is somewhat anecdotal, it suggests that PEAKS identifies examples that could help establish more robust decision boundaries between similar classes. In contrast, EL2N's exclusive selections tend to favor examples that are poorly framed, potentially mislabeled, or outliers. This difference highlights a limitation of using prediction eттоr alone as a selection criterion for real-world datasets, where high error might indicate problematic samples rather than informative examples.
20

---

PEAKS: Prediction Error Anchored by Kernel Similarity
PEAKS Only (Top by Score)
GT: spaghetti carbonara speghetti, carbonarate 53) spaghetti bolcgneseiO pad
GT: club sandwich grilled cheese wwnchrichl0. 65) mal_sanddcci0220
ct poutine french fnesto 63)
Gi:: nachos
Gii: tecos
chicken, quesadillst0.00 (.0)) nachosio hambungeri0.100
nachozio 401
pourine 23)
caesar se chicken csaacilaa00014 14)
GF chicken curry shripp gritsil curyyo.50 remeni0.10
GT. ch doen, eiings poking duckio.5 53) wingsip fried calem 0...
GT: hamburger s0ew600 46) herm falatee(ti. U11l
GT. omdlette omelattorD 40) chicken quecadilato.Ba 36) breakfest, buriloo 121
GT. macarons macarensiO cream10.22 donuts/0.16)
EL2N Only (Top Score) cub sandwich
GT shrimp_ard grit shrimp and grisiic. 01) rootloo o0)
chocolate cake cras10.93 donutsid 02) baklevat0.00) 01)
cog(0.93) hamEurgerj0.04 04) french, frie(o.003
ruliso bruschettalD 03)
anddutaa00
breakfast bur ritato og)
Falefel(D. D2!
grilld_chense cheese sandwichio hitdeeihhioo)))) US))
GT: steak orime, ribil 7) peking 10) steak(0.08)
GT: odamame sushild 44) dumoingsio 10y baktvvar
GE breakfast burrito greek, sellelo d6) tune Jartarell) 1Y; falafelic (6)
GT: shrimp and grits reottoli 54) ravelk0.07 07] shrimp and prttsoo..s
ome talateli 31)
chicken, qww.adullllll.ooo 19)
hummusio
Figure 1.. Shown above are two separate 2x55 grids of images from the Food101 dataset encountered in the middle the data selection phase. The first figure highlights the top 10 images (hy score) that were exclusively selected by PEAKS but not by EL12,, while the second figure shows the top 10 images (by score) exclusively selected by EL2N but not by PEAKS. Each image is annotated with its ground-truth label and the top three predicted classes (with their respective softmax probabilities). This comparison illustrates the differences in selection criteria between the two methods.
D.4. Alternative Sampling Mechanism
During the data selection phase of IDS, each training batch combines ò newly selected examples with 6 - ô randomly sampled examples from the existing training set, where is the batch size. However, uniform random sampling of previous examples leads to an uneven usage distribution: some carly-selected samples are repeatedly chosen while others remain underutilized. Figure 12 illustrates this phenomenon, showing significant variation in sample usage counts. While our final fine-tuning phase ensures the model eventually learns from all selected examples, the uneven sample distribution during the selection phase may bias the model's learning. To address this, we implement simple count- count-based sampling strategy. For each example i, we track its usage count ct) and assign sampling probabilities inversely proportional to these counts:
pt(t) = 1/((())
(14)
1/c
This approach reduces the probability of selecting frequently used samples while promoting the use of underutilized ones. As shown in Table 4, this simple modification improves accuracy in 8 out of dataset and budget combinations tested with
21

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Table 4. Alternative sampling using counts. Test accuracies averaged 3 seeds without validation set. Data budgets x1, x2, and x4 refer to dataset sizes of 2.5k, 5k, and 10k examples
Without Validation Set
CIFAR100
Food101
Food101-N
PEAKS Alt. Sampling (x1)
60.0
51.5
42.0
PEAKS (x1)
59.0
50.9
41.4
PEAKS Alt. Sampling (x2)
73.8
63.0
52.8
PEAKS (x2)
72.3
62.6
52.6
PEAKS Alt. Sampling (x4)
83.5
73.1
61.9
PEAKS (x4)
82.7
72.9
62.2
PEAKS. These results suggest that the batch formation strategy plays important role in IDS Future work could explore more sophisticated batch formation functions B(T;, fe, 0B), where 0 represents adaptive sampling parameters that evolve with the model state and training dynamics.
[WebVision- 100k] Histogram of Training Sample Usage (Exluding Fine-tuning]
[CIFAR100 - 10k] Histogram of Training Sample Usage (Exluding Fine-tuning)
0.25
0.06
Probabiltty Density
Pobaaiiity Density
0.15
0.03
0.02
0.05
5
10
15
20
25
30
10
20
30
40
50
60
70
Number of Times Sample Used
Number Times Semple Used
Figure 12. Sample repetition during training (excluding fine-tuning phase) for WebVision and CIFAR100 under random sampling.
D.5. Ablation Study Class Normalization Term for PEAKS
In PEAKS, normalize the expected improvement score p [A(p, x.)] by (e) where Cyp (t) represents the number of samples already selected from class Vp at time t. This normalization accounts for varying score ranges across different classes. Table 5 shows ablation results across three datasets (Web Vision omitted due to computational constraints). The results demonstrate that this term consistently improves performance of PEAK on both balanced and imbalanced datasets.
22

---

PEAKS: Prediction Error Anchored by Kernel Similarity
Table 5. Impact class normalization PEAKS. accuracies averaged across seeds without validation set. Data budgets x1, x2, and x4 refer datasct sizes of 2.5k, 5k, and 10k examples
Without Validation Set
CIFAR100
Fo0d101
Fol-
PEAKS w/o Cyp (x1)
57.8
49.8
39.2
PEAKS (x1)
59.0
50.9
41.4
PEAKS w/o Cyp (x2)
71.8
61.7
50.5
PEAKS (x2)
72.3
62.6
52.6
PEAKS (x4)
82.7
72.3
59.6
PEAKS (x4)
82.7
72.9
62.2
23


Reviews:
[{"review_id": "zAc6Uyc23j", "reviewer": "Reviewer_WR8r", "summary": "This paper focuses on data selection of DNNs that data arrives as a continuous stream and must be selected without access to the full data source. Based on this, the incremental data selection (IDS) problem is formulated as a three-stage process which including initialization with random samples, (streaming) data selection with model update and final training. To resolve IDS problem, this paper analyzes the impact of new data on model and proposes a score function based on the prediction error and penultimate representation similarity for measuring data importance. As a result, a new algorithm PEAKS is proposed. Experiments on CIFAR-100, FOOD101, Webvision demonstrate the effectiveness of PEAKS compared to random selection, embedding similarity based method, EL2N and GraNd.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their time and constructive comments.\n\n**Lack of analysis about the empirical time complexity**\n\nWe thank the reviewer for highlighting this concern. We will clarify in the main text that all baselines are subject to similar time complexity constraints. All methods require only a forward pass to make selection decisions—with GraNd being the only exception, as it requires an additional backward pass. Furthermore, we fixed the selection rate of IDS constant for all baselines (e.g., 20\\% in main experiments). Thus, the sample acquisition speed is consistent across all baselines.\n\n**There lacks analysis of how close the approximated scoring function (i.e., Eq.7 or Eq.11) compares to the original one (Eq.3).**\n\nWe agree with the reviewer's concern and have conducted an additional analysis to address this point.\n\nOur intuition for using last layer weight updates as an indicator for the whole network is supported by several studies in feature learning/transfer learning: early layers are more transferable [1, 2]; with pre-training, shallower layers approach optimality faster [3]; and early layers undergo less structural change than later ones during training [4]. Therefore, late layers in comparison are subjected to a lot of changes, thus can be representative of the whole network changes.\n\nTo quantify the amount of change analytically is out of the scope of this paper. Additionally, note that explicitly computing Eq.3 requires calculating the Jacobian for every validation sample, which is of size (number of parameters × number of classes). Due to computational constraints, *we performed the following analysis on a toy MNIST scenario* rather than on large datasets and the ViT architecture.\n\nWe first trained a two-hidden-layer MLP with 400 neurons each on 5k MNIST examples for 10 epochs. This step replicates a pretrained model that was trained on a few examples at the initialization phase to achieve some performance on the underlying task. Next, we held out 500 examples for validation and considered another 500 samples for selection using 3 methods:\n1. Computing Eq.3 exactly.\n2. Eq.9 (last layer training assumption).\n3. Eq.11 (last layer training assumption and avoiding validation set).\n\nWe used a 20% selection following our setting in the paper.\n\nAcross 10 runs with different seeds, we found high overlap between selected subsets: $\\mathbf{98 (\\pm 1.4)}$\\% between Eq.3 and Eq.9, and $\\mathbf{92.9 (\\pm 0.9)}$\\% between Eq.3 and Eq.11. This demonstrates that although we make some simplifying assumptions, our approximate selection mechanisms closely match decisions with exact computation. We are committed to expanding this analysis and scaling it to more realistic networks and datasets such as CIFAR10 and ResNet-18 in the revised paper as computational resources permit.\n\n[1] How transferable are features in deep neural networks?, NeurIPS 2014.\n\n[2] What is being transferred in transfer learning?, NeurIPS 2020.\n\n[3] Which Layer is Learning Faster? A Systematic Exploration of Layer-wise Convergence Rate for Deep Neural Networks, ICLR 2023.\n\n[4] Unveiling the Dynamics of Transfer Learning Representations, ICLR 2024 Workshop Re-Align.\n\n**IDS still needs to finetune the model on the selected data subset, which does not follow a purely streaming setting.**\n\nWe would like to clarify the goal of this final training phase and believe that in our writing, calling it \"fine-tuning\" might have led to misrepresentation. This final training phase is optional and we added it to ensure that the model also has a chance to iterate a few times over samples selected toward the end of data selection so that final accuracy is representative of the corresponding dataset size. However, this only provides an incremental improvement in performance. Below we report the performance of PEAKS before this final training and after. As seen from these results, PEAKS can also work on purely streaming performance with a minimal accuracy drop due to recently selected examples not yet being learned.\n\n| **Dataset** | **Before Final Training** | **After Final Training** |\n|----------------------|:---------------------------:|:--------------------------:|\n| CIFAR100 (×1)        | 57.56  | 58.99 |\n| CIFAR100 (×2)        | 70.03| 72.73|\n| CIFAR100 (×4)        | 80.71| 82.74|\n| Food101 (×1)         | 49.88| 50.85 |\n| Food101 (×2)         | 60.55| 62.64 |\n| Food101 (×4)         | 69.39| 72.88|\n| Food101N (×1)        | 40.71| 41.42|\n| Food101N (×2)        | 50.92 | 52.63|\n| Food101N (×4)        | 59.16| 62.16|\n| WebVision (×1)       | 37.75| 43.92|\n| WebVision (×2)       | 47.33 | 53.11|\n| WebVision (×4)       | 54.53| 59.02|\n\n*Average over three seeds.*", "claims_and_evidence": "The claims about IDS and PEAKS are clear and supported by convincing evidence.", "methods_and_evaluation_criteria": "The proposed method (i.e., PEAKS) makes sense but the evaluation can be further improved to compare their empirical time complexity.", "theoretical_claims": "I have checked the theoretical analysis in section 3. The derivation is correct but the claim is not well supported. Approximations are introduced, including replacing the Jacobian w.r.t. the network parameters with parameters only in the last layer and replacing the mean embeddings of validation data with the weight vector. However, there lacks analysis of how close the approximated scoring function (i.e., Eq. 7 or Eq. 11) compares to the original one (Eq. 3).", "experimental_designs_or_analyses": "Lack of analysis of time complexity", "relation_to_broader_scientific_literature": "The idea of incremental data selection is also related to online/continual learning", "essential_references_not_discussed": "To the best of my knowledge, essential references are included", "other_strengths_and_weaknesses": "Strength:\n1. Incremental data selection is a novel setting for data efficient learning\n2. The paper is well structured and easy to follow\n\nWeakness:\n1. IDS still needs to finetune the model on the selected data subset, which does not follow a purely streaming setting.\n2. Lack of experimental evidence for the rationality of approximation in deriving scoring function\n3. Lack of analysis about the empirical time complexity w.r.t. IDS and PEAKS", "other_comments_or_suggestions": "None", "questions_for_authors": "My concerns are listed in the weakness section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCnNFGsmRg", "reviewer": "Reviewer_Cx9C", "summary": "This paper poses Incremental Data Selection (IDS) problem where examples arrive continuously during training. Then it proposes a prediction error-based method PEAKS to address the problem, showing that a sample’s impact is influenced by both its position in feature space and its prediction error. Experimental results demonstrate the effectiveness of proposed method compared to naive baselines (such as random selection and Moderate).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough feedback.\n\n**Weakness-1**\n\nWe thank the reviewer for pointing out that the draft lacks discussion of why PEAKS is effective. In the revised manuscript, we will explicitly clarify this. We believe PEAKS' main advantage is its ability to discriminate hard examples from outliers and noise. By combining two scores, it effectively picks typical but hard examples. In contrast, error-based methods like EL2N and GraNd purely focus on high-error samples that might be mislabeled. Moderate selection is conservative, filtering out noise but also ignoring valuable examples. Uncertainty is an unsupervised metric that does not exploit label information, making it immune to label noise but also limiting the information it can leverage.\n\n**Weakness-2**\n\n*We ran some additional experiments in the offline setting.* We first trained a ResNet-18 on the full Food101-N dataset (our largest dataset for training from scratch experiments, focusing on one dataset due to time limitations). Second, we used PEAKS and other baselines to rank the full dataset based on scores. Finally, we trained a model from scratch using 50%, 60%, 70%, and 80% of the top samples.  As seen from the results below (across 3 seeds), PEAKS performs best across most of the data budgets.\n||50%|60%|70%|80%|\n|---|---|---|---|---|\n|Random|70.1 (±0.1)|71.1 (±0.1)|72.4 (±0.1)|73.6 (±0.1)|\n|Moderate|**71.0** (±2.0)|72.2 (±1.6)|73.6 (±1.1)|74.9 (±0.9)|\n|EL2N|64.9 (±4.9)|69.2 (±2.3)|71.9 (±0.9)|73.8 (±0.2)|\n|Uncertainty|66.5 (±3.5)|69.7 (±1.1)|71.9 (±0.2)|73.5 (±0.3)|\n|PEAKS|70.5 (±1.5)|**72.3** (±1.1)|**74.1** (±1.3)|**76.0** (±0.1)|\n\nWhile these preliminary results are promising, we want to emphasize that PEAKS was designed for the incremental setting, where an example's value is tied to the current model state. Offline data selection has different constraints and opportunities, such as the ability to globally optimize selection across the entire dataset.\n\n**Weakness-3**\n\nWe ran a small grid search to select the optimizer, lr, and weight decay that optimized performance during the initial training phase (Appendix B). This approach was motivated by two key considerations:\n\n1. In realistic streaming scenarios, hyperparameters must be selected based on early performance signals rather than retrospectively after seeing final results.\n2. The initial phase is method-independent. This ensures a fair comparison where no method is advantaged by the optimizer choice.\n\nWe found AdamW to perform better in our ResNet-18 and WebVision experiments, likely due to differences from the typical epoch-based full-dataset training. For ViT experiments, we used SGD with momentum (lr 0.001), which aligns with lr options explored by the original ViT authors. Notably, they do not consider a high lr such as 0.1 for fine-tuning (see Appendix Table 4 of [1]). Thus, PEAKS demonstrates strong performance with both AdamW and SGD.\n\n[1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021\n\n**Weakness-4**\n\nWe would like to highlight that the WebVision dataset contains 2.4 million images across 1000 classes, making it larger than ImageNet-1K. Furthermore, PEAKS shows the most promising results on this large real-world dataset. As explained in Appendix B.5, we deliberately avoided datasets closely related to ImageNet to prevent a potential bias, since ImageNet serves as our pretraining source.\n\n**Weakness-5**\n\nWe thank the reviewer for this insightful suggestion. Studying IDS under distribution shift would indeed be interesting. However, we would like to clarify that we propose IDS as a problem setting, where data arrives incrementally but from a relatively stable distribution. While we noted that the scenario extends to continual learning (CL), this was to acknowledge the relationship rather than claim PEAKS would directly transfer to that setting without modification. PEAKS was designed to select samples that lead to maximum change in logits. However, such samples may not necessarily be good for avoiding catastrophic forgetting or providing knowledge transfer, which are crucial for CL. CL would likely need different considerations. Extending our work to CL would be a significant research direction requiring substantial modifications. We've noted this as important future work.\n\n\n**Weakness-6**\n\nThank you for suggesting this baseline. *We implemented it exactly as described*: by assigning a score of 0 to correctly predicted examples and a score of 1 - max(softmax) to incorrect ones. Results below replicate our setting from Section 5.2.\n\n||C100|F101|F101-N|WebVision|\n|---|---|---|---|---|\n|Data x1|62.7 (±7.0)|48.7 (±3.1)|35.0 (±3.7)|35.2 (±0.4)|\n|Data x2|77.0 (±2.8)|60.3 (±2.9)|44.8 (±3.7)|40.8 (±0.1)|\n|Data x4|83.9 (±1.0)|70.7 (±2.3)|53.5 (±3.8)|45.3 (±0.3)|\n\nCompared with Table-2 in our paper, this baseline performs strongly on CIFAR100, on par with GraNd. However, on the three larger datasets, it significantly lags behind PEAKS.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Fair", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n1. Incremental Data Selection problem is interesting and seems to be important in real world applications.\n2. The paper is well organized and written.\n\nWeakness\n1. It is unclear why and how the proposed method (PEAKS) is better than other baselines in IDS setting. The experimental results have verified the effectiveness of PEAKS, while further explanation and demonstration is needed.\n2. How the proposed method performs on standard offline data selection setting, can it still outperforms other baselines?\n3. The experimental setup is not aligned with existing works. For classification like CIFAR-100, a standard setup is SGD\noptimizer with a momentum of 0.9, weight decay of 5e-4, and an initial learning rate of 0.1. While this work adopts adamw optimizer, which achieves worse performance than SGD on classification datasets. It remains unclear if the proposed method performs other baselines when using SGD optimizer.\n4. Lack of experiments on large-scale real-world datasets like ImageNet-1K. \n5. This paper focuses on online setting and the authors argue that the scenario naturally extends to continual learning\nwhen the input distribution evolves over time. Therefore, it is helpful to conduct experiments on online continual learning benchmarks to demonstrate the performance, e.g., can PEAKS enhances the ER [1] and DER++ [2] by replacing random selection with PEAKS.\n6. The reviewer suggests a simple baseline: selecting training samples which are wrongly predicted but with lower confidences (maximum posterior probability).\n\n[1] Learning to learn without forgetting by maximizing transfer and minimizing interference.\n\n[2] Dark Experience for General Continual Learning: a Strong, Simple Baseline.", "other_comments_or_suggestions": "The reviewer suggests that the authors add a summary of the paper's contributions at the end of the introduction. This summary would help emphasize the key points and guide readers through the rest of the manuscript.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOeJ3EYxcR", "reviewer": "Reviewer_YBNe", "summary": "This work introduces Incremental Data Selection (IDS) and proposes PEAKS, a method that selects training samples based on prediction error and kernel similarity. PEAKS efficiently builds training datasets while improving model performance. Experiments show it outperforms existing methods, significantly reducing data needs while maintaining accuracy, especially on large-scale datasets like WebVision.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's supportive and positive evaluation of our work.\n\n**The Neural Tangent Kernel seems to be applicable only to infinitely wide networks. Are there alternative methods for other networks?**\n\nThe reviewer raises an important clarification point regarding the Neural Tangent Kernel (NTK) being theoretically guaranteed only for infinitely wide networks. We acknowledge that our networks do not operate in the strict NTK regime. However, based on intuition that in the model fine-tuning phase, weight changes happen less dramatically, our approach uses a first-order Taylor approximation as a practical tool to model network behavior between consecutive updates, and is not claiming the network exists in the kernel regime. [1] also used first-order approximations to derive methods and shed insight for practical networks like ours. Our empirical results also support this position.\n\n[1] Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning, NeurIPS 2024", "claims_and_evidence": "This paper supports its claims clearly with a combination of mathematical derivations and detailed experimental setups.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are meaningful for incremental data selection. PEAKS is well-suited for streaming data, and benchmark datasets ensure comprehensive evaluation.", "theoretical_claims": "The paper does not involve highly complex mathematical derivations. The presented formulas are correctly applied and support the method’s claims. No issues were found with their usage.", "experimental_designs_or_analyses": "The experimental design appears detailed and supports the reliability of the conclusions.", "relation_to_broader_scientific_literature": "PEAK combines similarity and uncertainty, offering an innovative approach not widely explored in existing methods, advancing related research.", "essential_references_not_discussed": "There are no essential references missing in the paper.", "other_strengths_and_weaknesses": "Strengths:\n1: The idea is explained clearly and is also innovative. \n2: The experiments are thorough, making the conclusions highly convincing.", "other_comments_or_suggestions": "None.", "questions_for_authors": "The Neural Tangent Kernel seems to be applicable only to infinitely wide networks. Are there alternative methods for other networks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kX4MDUiNPX", "reviewer": "Reviewer_dKNe", "summary": "The paper introduces an algorithm for Incremental Data Selection (IDS) that selects training examples from a continuous data stream by combining prediction error and kernel similarity. The problem is important for the machine learning community. IDS addresses the challenge of efficient data utilization in deep learning, particularly in scenarios where full datasets are unavailable upfront. PEAKS dynamically balances samples with high prediction errors (indicating model uncertainty) and those aligned with class prototypes (via kernel similarity). Experiments on image datasets (CIFAR100, Food101, WebVision) demonstrate PEAKS' superiority over baselines like EL2N and GraNd, especially in low-data regimes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to the reviewer for their detailed assessment and helpful suggestions.\n\n**Weakness-1**\n\nWe acknowledge that our theoretical contribution builds incrementally on existing frameworks rather than proposing an entirely new theoretical foundation. Our primary contributions are twofold. First, we formulate the Incremental Data Selection (IDS) problem — a practical, previously unexplored setting that addresses real-world constraints in data selection. Second, we devise the PEAKS algorithm tailored for the IDS.\n\nWe also thank the reviewer for highlighting the connection to AL. We currently have a brief discussion of AL in Appendix C and will move this to the main paper, expanding it to include the suggested reference.\n\n**Weakness-2**\n\nWe selected image classification as a testbed for IDS as it is a prevalent and well-established task in the data selection literature (e.g., [1, 2]). While PEAKS' design is not specific to the image domain, we acknowledge that the absence of empirical evaluation on other tasks is a limitation. We commit to adding a discussion of this limitation in the revised paper.\n\n[1] Beyond neural scaling laws: beating power law scaling via data pruning, NeurIPS 2022.\n\n[2] Deep Learning on a Data Diet: Finding Important Examples Early in Training, NeurIPS 2021.\n\n\n**Weakness-3**\n\nWe thank the reviewer for raising this point and will clarify these details in the paper. Due to the approximations in Section 3, kernel similarity in PEAKS reduces to the product of the error and output logit (Eq.11), incurring no extra cost. Regarding cache and thresholding (which are not specific to PEAKS but inherent to the IDS), the required overhead is minimal. \n1. The cache stores only $\\tau \\times \\delta$ floating-point values. For example, in the Section 5.2 experiments, the smallest cache stores 200 values (0.78 KB), while the largest (WebVision, 100k budget) stores 9000 values (35 KB).\n2. Dynamic thresholding requires maintaining a sorted cache for percentile computation. These overheads are negligible compared to the training cost.  \n\n**Weakness-4 (performance of PEAKS)**\n\nWe would like to respectfully clarify that the validation-free PEAKS variant consistently demonstrates strong performance across experiments. As demonstrated in Figure 3, Table 2, and Figure 4, the validation-free PEAKS consistently outperforms all baselines across datasets and budgets, performing slightly worse only on the simpler CIFAR100 dataset. Notably, on the challenging real-world WebVision dataset, PEAKS significantly outperforms the closest baselines by margins of 4.9%, 5.2%, and 4.2% across the three evaluated data budgets. This demonstrates that PEAKS is the most robust among baselines for real-world scenarios.\n\n**Weakness-4 (accuracy variance)**\n\nThank you for this important point about reporting statistical variance. Due to space limitations in this response, we cannot present tables here. However, we can confirm that the accuracy variance is within an acceptable range compared to the performance differences between baselines. The complete variance data for all experiments will be included in the revised manuscript.\n\n**Weakness-5**\n\nWe address selection rate sensitivity in Section 5.3, where Figure 4 shows PEAKS consistently outperforms other methods across selection rates ranging from 10% to 90%.\n\nRegarding refresh period, we wouldn't expect methods to be highly sensitive to this parameter as it only determines how many recently seen samples are considered for modeling the score distribution for percentile selection. *We also conducted an additional ablation study* varying $\\tau$ from 50 to 300 (default was 100) on two datasets following the setting in Section 5.3. As the table below shows, the results remain consistent across different $\\tau$ values.\n\n|Method|F101 (τ=50)|F101 (τ=200)|F101 (τ=300)|F101-N (τ=50)|F101-N (τ=200)|F01-N (τ=300)|\n|---|---|---|---|---|---|---|\n|PEAKS|**73.0 (±1.4)**|**72.8 (±1.3)**|**72.3 (±1.1)**|**62.0 (±2.6)**|**62.4 (±2.5)**|**61.9 (±2.4)**|\n|Moderate|70.4 (±1.3)|70.6 (±1.2)|71.1 (±1.2)|61.8 (±2.7)|62.2 (±3.4)|61.8 (±2.9)|\n|EL2N |66.3 (±1.4)|66.8 (±1.5)|67.5 (±1.8)|44.4 (±2.3)| 45.4 (±2.5)|45.4 (±2.7)|\n|Uncertainty|71.3 (±1.4)|71.8 (±1.9)|72.1 (±1.8)|54.6 (±2.7)|55.3 (±2.2)|55.7 (±3.0)|\n\n**Weakness-6**\n\nIn Section 3, we assume that after the initialization phase (training on a few random samples), our pre-trained model acquires \"decent\" performance. We will clarify in the revised paper that this is far from satisfactory performance. To illustrate this clearly, we provide *additional results* below, comparing the accuracy of the pre-trained model after the initialization phase, and after IDS using PEAKS (budget x4). As shown, the model's performance after initialization does not approach the performance we report after IDS\n\n|Dataset|After Init|After IDS|\n|---|---|---|\n|C100|41.7 ±4.0|82.7 ±1.1|\n|F101|37.8 ±1.1|72.9 ±1.4|\n|F101-N|24.9 ±2.0|62.2 ±3.3|\n|WebVision|26.9 ±3.9|59.0 ±0.4|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper introduces an algorithm for Incremental Data Selection (IDS) that selects training examples from a continuous data stream by combining prediction error and kernel similarity. The problem is important for the machine learning community. IDS addresses the challenge of efficient data utilization in deep learning, particularly in scenarios where full datasets are unavailable upfront. PEAKS dynamically balances samples with high prediction errors (indicating model uncertainty) and those aligned with class prototypes (via kernel similarity). Experiments on image datasets (CIFAR100, Food101, WebVision) demonstrate PEAKS' superiority over baselines like EL2N and GraNd, especially in low-data regimes.", "essential_references_not_discussed": "[1] Cost-Effective training of deep CNNs with active model adaptation. KDD 2018\n\nSee weaknesses.", "other_strengths_and_weaknesses": "# Strengths \n1.\tThe problem of IDS is important for the machine learning community.\n2.\tThe proposed method employs a measure that considers two core factors.\n3.\tThe proposed method is evaluated on several vision datasets against other baselines.\n# Weaknesses\n1.\tThe core idea of PEAKS combines prediction error and kernel similarity. While the integration is practical, the theoretical contribution is incremental rather than groundbreaking, lacking a novel framework. Besides, the proposed method shares concepts in the active learning field [1], while proper discussions are not included in this paper.\n2.\tThe experiments are restricted to image classification tasks. There is no validation on non-vision domains (e.g., NLP, time-series) or regression tasks, raising concerns about generalizability. \n3.\tWhile PEAKS is framed as efficient, computational overhead from kernel similarity calculations, cache maintenance, and dynamic thresholds is not quantified. This omission leaves scalability for large-scale models/datasets unclear.\n4.\tPEAKS-V relies on a validation set for class prototypes, which is often impractical in streaming scenarios. The validation-free PEAKS variant underperforms other baselines, suggesting robustness issues in real-world deployments. Besides, the authors only report the average accuracy while ignoring the variance in the Tables, making it hard to understand the robustness.\n5.\tThe paper does not address how sensitive PEAKS is to the choices on hyperparameters like selection rate and refresh period, limiting reproducibility and adaptability to diverse settings.\n6.\tWhen using pre-trained models as the initialization, another concern also emerges, i.e., the is overlapping between the pre-training dataset and the downstream tasks. If the pre-trained model has already known them, why bother conducting further learning processes?\n\n[1] Cost-Effective training of deep CNNs with active model adaptation. KDD 2018", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel IDS problem and proposes an effective method, PEAKS. It shows strong performance in experiments but lacks generalizability and detailed analysis on computational overhead and hyperparameter sensitivity. Overall, it’s a promising contribution with room for improvement.
Decision: Accept (poster)