Inconsistency Biases in Dynamic Data Pruning
Authors: Qing Zhou
Tao Yang
Bingxuan Zhao
Hongyuan Zhang
Junyu Gao
Qi Wang
Venue: ICLR 2026 Poster (2026)
Keywords: dynamic data pruning
efficient training
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Dynamic data pruning accelerates training by focusing on informative samples. However, comparing importance scores across different model states introduces inconsistency (score context drift), and variable selection rates bias gradient dynamics over time (temporal gradient bias). We introduce RePB (Resolving Pruning Biases), a framework addressing these issues. RePB performs pruning decisions within local windows (short sequences of batches) during training, using loss scores computed with a near-constant model state within each window to ensure valid comparisons. These decisions determine the data subset used in the subsequent training phase. To counteract temporal gradient bias arising from non-uniform sample inclusion, cumulative temporal rescaling reweights sample losses during training based on their historical selection frequency. We provide theoretical grounding for RePB's consistency in score comparison and gradient alignment. Experiments show RePB achieves near-full-dataset accuracy using reduced data (most above 30%) across 16 datasets, 17 models and 13 tasks, offering a robust and scalable approach to efficient deep learning. Code is available at https://github.com/mrazhou/RePB.

Published as a conference paper at ICLR 2026
INCONSISTENCY BIASES IN DYNAMIC DATA PRUNING
Qing Zhou' Tao Yang' Bingxuan Zhao 1 Hongyuan Zhang Junyu Gao Qi Wang'*
NNotthwestern Polytechnical University "hhe University of Hong Kong
{chautsing, yta09464, bxuanzhao202, hyzhang98, g3y3035, crabwq) @gmail com
ABSTRACT
Dynamic data pruning accelerates training by focusing on informative samples. However, comparing importance scores across different model states introduces inconsistency (score context drift), and variable selection rates bias gradient dy- namics over time (temporal gradient bias). We introduce RePB (Resolving Prun- ing Biases), framework addressing these issues. RePB performs pruning de- cisions within local windows (short sequences batches) during training, us- ing loss scores computed with near-constant model state within each window to ensure valid comparisons. These decisions determine the data subset used the subsequent training phase. To counteract temporal gradient bias arising from non-uniform sample inclusion, cumulative temporal rescaling reweights sample losses during training based their historical selection frequency. We provide theoretical grounding for RePB's s consistency in score comparison and gradient alignment. Experiments show RePB achieves near-full-datase accuracy using re- duced data (most above 30%) across 16 datasets, 17 models and 13 tasks, offering robust and scalable approach efficient deep learning. Code available attps://gitai www/mmaawwo//eeeeeeeooo
INTRODUCTION
The remarkable success of deep learning models across diverse applications (He et al., 2016; Doso- vitskiy et al., 2021a) often comes at the cost of immense computational resources and vast datasets (Kaplan et al., 2020; Russakovsky al., 2015; Feng al., 2019). As models continue to grow in complexity and datasets expand, improving training efficiency without compromising model quality becomes paramount. Data selection, which aims to train models on smaller, carefully chosen data subsets, represent a promising direction for achieving such efficiency gains (Coleman et al., 2020; Raju al., 2021; Sener & Savarese, 2018; He al., 2024).
Among data selection techniques, dy- namic methods that adapt the train- ing subset during the learning pro- cess hold particular appeal (Qin al., 2024: Zhou et al., 2025; Raju et al., 2021). By responding to the chang- ing importance or redundancy of data points as the model evolves, dynamic pruning can potentially offer greater efficiency than static selection fixed before training (Coleman al., 2020; Paul et al., 2021: Toneva et al., 2019). However, the practical effectiveness and reliability of dynamic pruning are often undermined by two fundamen- tal consistency issues. First, score context drift: importance metrics (e.g., sample loss Qin et al. (2024); Zhou al. (2025); Raju et al. (2021), gradient norm Katharopoulos & Fleuret (2018)) used for pruning decisions are typically evaluated using the current model state. Because the model pa- rameters (the context) drift significantly during training, comparing scores computed at different
Dataset
Uniifrrm Resumpling
insccton
Resamphin Candita
DVDE
Lacal Window Pruning
Irning Condition
D"
'Dg
O
F+1
g+1
-Uk Q
Backwaard
Cumulative Temporul Resculing
Pruned Dataset
w (/M.
Next Epach
Figure 1: Overview of RePB framework.
"Co-corresponding author

---

Published as a conference paper at ICLR 2026
steps lacks statistical validity and can lead to suboptimal pruning choices (Blalock et al., 2020). Second, temporal gradient bias: iteratively selecting non-uniform subsets epoch after epoch alters the effective sampling distribution over time compared to standard uniform sampling from the full dataset. This introduces a bias in the expected cumulative gradient trajectory, potentially hindering convergence or leading the model to different optima (Qin et al., 2024; Hu et al., 2020). These consistency violations represent fundamental obstacles to achieving reliable and theoretically sound acceleration with dynamic data pruning.
To overcome these critical limitations, particularly the inconsistency biases inherent in scoring context and temporal gradient dynamics, we propose RePB (Resolving Pruning Biases). a frame- work characterized by pruning within a batch and rescaling over epochs (As shown in the Figure 1). Specifically, to ensure reliable pruning decisions despite score context drift, it employs local window pruming (LWP): importance scores (sample losses) used to determine the data for the next epoch are computed and compared only within short local windows of the current epoch (e.g., one or few batches). Leveraging the minimal model drift within these short windows allows for valid comparisons of scores generated at slightly different steps, enabling reliable identification of less informative samples relative to their peers within the same window. Complementing this selection process, RePB maintains data diversity and prevents sample pool collapse through uniform prob- ability resampling. At the end of each epoch, it explicitly reintroduces samples from the original dataset that were not part of the current training set with a fixed probability, guaranteeing long-term exploration. Finally, to counteract the temporal bias in gradient dynamics arising from selecting potentially non-uniform subsets over epochs, RePB utilizes cumulative temporal rescaling (CTR). During training in the subsequent epoch, CTR reweights the loss of each sample based on its histor- ical selection frequency, effectively applying an empirical inverse probability weighting that aims to align the expected gradient trajectory with that of full-dataset training over time. By systematically addressing both score comparison validity and long-term gradient bias, this synergistic approach allows RePB to deliver the efficiency benefits of dynamic pruning without sacrificing the stability and reliability of standard training.
2 RELATED WORK
Data Selection and Pruning. Selecting informative data subsets is a core theme in machine learn- ing, pursued for efficiency (Konyushkova al., 2017; Kousar et al., 2025) and generalization (Jiang et al., 2018). Static subset selection methods choose data once before training. Early static meth- ods were often based on criteria like core-sets (Huggins et al., 2016), uncertainty (Coleman et al., 2020), or gradient information (Katharopoulos & Fleuret, 2018; Paul et al., 2021). More recent work in static pruning has developed sophisticated criteria that move beyond just efficiency, focusing on enhancing model generalization and data quality. For instance, methods have been proposed to im- prove generalization by severing spurious correlations (Mulchandani Kim, 2025) or by carefully selecting data in specialized domains like molecular modeling (Chen et al., 2024). Other advanced static criteria include perplexity- based scoring with small reference models (Ankner et al., 2025), information maximization (Tan et al., 2025), and influence-based metrics like moving-one-sampe out (Tan et al., 2023). While these static approaches are powerful, they select data once and cannot adapt to the model's evolving understanding of the dataset during training. In contrast, dynamic methods, like InfoBatch, update the subset during training (Toneva et al., 2019; Qin et al., 2024). Early work often focused on identifying forgettable samples (Toneva et al., 2019), while more re- cent approaches use loss dynamics (Qin et al., 2024; Zhou et al., 2025). RePB falls into the dynamic category, but it distinguishes itself by addressing the fundamental optimization consistency biases inherent in the dynamic process itself, rather than focusing solely on static data quality.
Addressing Score Inconsistency and Staleness. The challenge of comparing importance scores computed across differing model states is recognized in the literature (Jiang et al., 2019; Borsos et al., 2020). Prior attempts to mitigate this include using score moving averages (Jiang et al., 2019) or employing infrequent score updates (Borsos et al., 2020). However, these strategies often provide indirect mitigation rather than a direct structural guarantee of comparable scoring contexts. RePB's s local window pruning offers more direct and foundational solution by explicitly restricting score comparisons to windows of model state stability, thereby ensuring local decision consistency and tackling the root cause of this specific inconsistency.
2

---

Published as a conference paper at ICLR 2026
Addressing Gradient Bias and Variance. Dynamic subset selection inevitably leads to uniform sampling over time, potentially biasing gradient estimates (Qin et al., 2024). Importance Sampling (IS) is classical technique used in stochastic optimization to reduce variance cor- rect bias by weighting gradient steps inversely to sampling probabilities (Needell et al., 2016; Katharopoulos & Fleuret, 2018; Qin al., 2024; Salaün et al., 2023; 2024). IS often uses in- stantaneous importance or sampling probabilities to adjust the current gradient step, primarily for variance reduction. CTR, conversely, uses cumulative selection frequencies (counts of inclusion in past epochs' datasets) to correct the long-term temporal bias introduced by the selection process itself across epochs. aims align the entire training trajectory, not just individual steps (detailed in Appendix A).
In conclusion, RePB offers a refined approach to dynamic data pruning by addressing critical con- sistency challenges. While methods like InfoBatch (Qin et al., 2024) tackle temporal gradient bias through instantaneous rescaling, RePB introduces a distinct cumulative temporal rescaling mecha- nism. This approach ensures long-term gradient alignment with enhanced applicabilityy as it operates effectively without relying on explicit knowledge or modeling of precise selection probability dis- tributions, which may be intractable or unavailable. Crucially, RePB also identifies and resolves the challenge of score context drift- aa limitation particularly evident in global comparison strategies like infoBatch's-thooee parisons within stable model contexts. RePB thus provides a more robust, theoretically grounded, and broadly applicable path to efficient training than prior dynamic pruning methods.
its local window scoring strategy, which ensures valid importance com-
REPB: RESOLVING INCONSISTENCY BIASES IN DYNAMIC PRUNING
Dynamic data pruning seeks efficiency gains by training subsets of data identified as most infor- mative. However, two fundamental consistency challenges hinder its reliability: (1) Score context drift:
Importance scores (c.g., sample losses) computed different training stages (with different model parameters, i.c., different contexts) are not directly comparable, making pruning decisions unreliable. (2) Temporal gradient bias: Selecting subsets non-uniformly over time alters the ex- pected cumulative gradient compared to full-dataset training. RePB (Resolving Pruning Biases, as shown Figure 1) tackles these issues systematically using local window pruning, uniform probability resampling, and Cumulative Temporal Rescaling (CTR).
3.1 LOCAL WINDOW PRUNING FOR SELECTION CONSISTENCY
During the training process on dataset DE (in epoch F), RePB identifies samples within DE for potential inclusion in DE+1- Pruning decisions are localized within windows. A window k may consist of single batch B, or span W consecutive batches (e.g. from step t, to f14w-1): the latter is particularly useful with small training batch sizes to ensure sufficient sample pool for effective pruning. We denote the set of samples processed in window ic Wa S DE- For each sample (I,, y4) E We processed at step t E [tz, tw+ 1l. its importance score is computed using the model state that time: 8; f(x., yı; 0.). After processing all samples in the window, these scores ( Jew are collected. The core assumption is that the parameter drift ||0, 0zl for any t within the window [t,, 11] is small. This allows for meaningful comparison of the scores 8; collected across the window.
Within each processing window We. importance scores = ((1,, yt, @window) are computed for all constituent samples (a;; 3/a) E We- The window-specific mean score then determined as μa W E(e, ((r,,,))ww, 8j- Let p [0, 1] be a pre-defined hyperparameter representing the probability with which sample (zt, y/) E Wj having s; < 14k is pruned. For each sample (æ+,, y:) E Wa. an independent random variable U, ~ ((00, 1) is drawn. The sample is retained and included in the candidate set D'511 for the subsequent epoch E + 1 the following condition holds:
(si 2 mk) V (s: < A
(1)
Otherwise, the sample is temporarily pruned. This selection rule is applied to all samples across all windows processed during epoch E, and the union of retained samples forms D D E+1
3

---

Published as a conference paper at ICLR 2026
3.2 UNIFORM PROBABILITY RESAMPLING FOR DIVERSITY
After processing Dg and collecting retained samples into EE at the end of epoch E, resampling occurs to ensure diversity and counteract sample pool shrinkage. We identify the set of samples from the full dataset D that were not part of the training set for the current epoch E, i.c., D Dg. Every
sample (T Mj) 'D \ 'De is added to form the final dataset DE+1 with probability Presample = 1-p-
Final D = D'e+1 U {(1g,1) M5) E D Dr random((,) 1) < Presample
(2)
This step guarantees that samples outside the current training focus have a chance to re-enter, pre- venting convergence to a empty subset.
3.3 CUMULATIVE TEMPORAL RESCALING FOR GRADIENT ALIGNMENT
Dynamic selection can cause samples to be used with varying frequencies over epochs, deviating from the uniform usage in standard training and potentially biasing the cumulative gradient. CTR counteracts this during the training phase of epoch + by amplifying the gradient contribution of under-selected samples and dampening that of over-selected ones.
Let ( be the total number of times sample (I%, Ui) has been included in the training datasets from epoch I to E:
N.(E) Σ [[( E Del,
where [|-] is the indicator function. Assuming the initial dataset D, contains all samples from the full dataset D, every sample i has N:(E)= 2 for E 2
The core idea of CTR is to reweight each sample's contribution inversely proportional to its observed historical selection frequency relative to the baseline (heing selected in every epoch). We define the CTR weight for sample i to be used during epoch E' + 1 as: w (B) = EE), This weight w CTR (E)) represents the ratio of the number of epochs elapsed to the number of times sample i was actually used. If (E(
E, the sample was under-selected, and its weight will be greater than 1. If (E) >> E (possible if resampling allows multiple inclusions), it was over-selected, and its weight is less than 1. If (E) = E, its weight is I, matching standard training.
During training in epoch E' + 1, when processing mini-batch B, C DE+1, the gradient update is computed using these weights directly:
g1
w. CTR
(E).
(4)
Be
iEB.
The standard mini-batch averaging 1/|Bel is retained, but each sample's gradient V& is scaled by its individual historical weight (F) This approach directly uses the historical counts N2(F) to adjust gradient magnitudes, offering practical mechanism to mitigate temporal selection bias without requiring knowledge of the underlying selection probabilities.
4
THEORETICAL FOUNDATIONS
4.1 LOCAL WINDOW PRUNING ENSURES SCORE CONSISTENCY
By restricting pruning decisions to comparisons of scores collected within local win- dows, RePB leverages the bounded and typically small parameter drift within such windows. This makes the comparison of scores computed at slightly different model states significantly more reli- able than comparisons across larger time intervals (c.g., epochs), thus mitigating the impact of score context drift.
Proposition:
Consider the k-th window We spanning steps to f1- For sample (I,, Mi) Wa processed at step t E [tk.tx+w-11] its score is s; = E(x,, yi; 0.). A comparison might involve 8; and 8j = 0zz Uy; (o)) where t, t' [t,, -1l, or comparing s; to the window mean A WW Ejew, (z))));00.).
Proof:
4

---

Published as a conference paper at ICLR 2026
The validity of such comparisons relies on the parameter drift within the window being small. As- sume the loss function ((r, y: 0) is L-Lipschitz continuous with respect to parameters 8. If gradient norms bounded by G and learning rate is 77, the parameter change between any two steps t' within the window is bounded: ||0, 0.l || S |t - |n s WiJG. The difference in loss for the same sample i evaluated two different model states 02, 02 within the window is bounded:
[e(z; Ms 8,) ((x,, yus 0v)) = 0 0. & LWnG.
(5)
If the window size W and learning rate 1/ are sufficiently small, this difference is small. This implies that the score = f(x; y/s 0,) is reasonably stable measure of the sample's importance relative to the model states encountered within that window. Comparing 83 8y (computed with 0p) or the mean f4 (an average over slightly different ('s) is meaningful because the score context drift within the window limited. The relative ordering of sample importance is likely preserved. The efficacy of LWP relies on minimal intra- wirdow parameter drift. This is perfectly achieved when window single batch (W- 1). as scores are computed with fixed model state before any parameter update, resulting in zero drift. Since extremely small batch sizes are uncommon in many settings, W = often serves as an ideal and practical default for RePB. For windows spanning multiple batches (W > 1), the anticipated drift (proportional to WnGG) remains significantly smaller than cumulative drift over an entire epuch. Thus, unlike epoch- wide score aggregation which suffers from substantial score context drift, LWP ensures much higher degree of score comparability, providing more statistically sound basis for selection.
4.2 CUMULATIVE TEMPORAL RESCALING REDUCES GRADIENT BIAS
Proposition: By reweighting gradient contributions during training using weights w. (E) (E) derived from empirically observed historical selection counts, CTR ensures that the ex- pectation of the stochastic gradient updates aligns with the direction of the gradient computed over the full dataset DD correcting the bias induced by non-uniform dynamic selection frequencies.
Proof: The target unbiased gradient over the full dataset is g" (0) [D] ΣiED (O) The CTR method computes the stochastic gradient at step t (within epoch + 1) using the weight niCTR (F) (F) based on history up epoch E:
g:
Σ E
CTR CTR (E)VE,)))) W
(6)
B
where B, is sampled uniformly from the current epoch's 's dataset DE+1. We analyze the expectation of this gradient E[g, 8.). Taking expectations over the batch sampling and the dataset selection process, and assuming [DE+1l x= SE+1. we have:
Elg:
Gi] ~ SE+1 iED
ΣPLE+ Elu CTR (E)) (e.(
The weight w (F)) (F) is the reciprocal of the empirical selection frequency ME)) = N.(F)/E. By the Law of Large Numbers, f(( - P, as E - co, where ir lime E De= Pi,e is the long-term average selection probability of sample i. This makes " (FT a practical, computable estimator for 1/p..
Here, we use the approximation Blw CTR (E) B(/S((E) & 1/E[](() 1/p. As noted by Jensen's inequality, that (E)) Z (E) This approximation becomes increasingly accurate E + oo, because the Strong Law of Large Numbers ensures that the random variable f(( concentrates tightly around its mean p., reducing the error of the approximation." "
Substituting this approximation into Eq. 7:
E g: 0.)
1 SEt1
Σ PL (O.( RED P
"Practically, the inequality implies that our estimator may slightly overweight samples, especially those with low or highly variable selection counts. This can be viewed as beneficial conservative correction, as it gives a slightly larger voice to under-selected samples, helping to prevent catastrophic forgetting and aiding robust convergence.
5

---

Published as a conference paper at ICLR 2026
As the selection process stabilizes, the current selection probability P,,+1 will fluctuate around the long-term average pt. Assuming Pi, R ps. the ratio pt, Pr = 1. Then:
1
[D] Sg+1
Elge 0r]
V(((0))
g" (0.).
5 5
EED
This demonstrates that the expected gradient under CTR, Elg:]. is approximately proportional to the true unbiased gradient g' (8,). The proportionality constant is C' = Di/Sg+1 If the selection strategy maintains roughly constant expected subset size fraction = SE+1/ [D), then C' se 1/a, a stable factor.
This factor C' acts a global scaling of the gradient. The outcome of CTR is the correction of the relative contributions of different samples in the expected gradient, aligning the optimization direction with that of full-dataset training. The resampling mechanism helps ensure (E) grows over time for all i, keeping the weights well-behaved and ensuring the concentration of ft(
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets and Tasks, Our experiments span a multitude of learning paradigms and data modalities. For image classification, we use CIFARIO, CIFARI00 (Krizhevsky et al.), and ImageNet- 1K (Rus- sakovsky et al., 2015). Vision-language understanding is evaluated on zero-shot captioning with ToCa (Zhou al., 2024), cross-domain captioning with SSIM (Feng et al., 2019), image caption- ing on COCO (Chen et al., 2015), and video captioning on MSR-VTT (Xu et al., 2016). Scene text recognition experiments utilize the MJ+ST dataset (Jaderberg et al., 2014; Gupta et al., 2016). We further test RePB on multi-view stereo with WHU-MVS (Liu & Ji. 2020), cross-view geo- localization with CVACT (Liu & Li, 2019), and image generation on MNIST (Deng, 2012) and CIFAR10. Finally, we explore supervised and semi-supervised learning for image (EuroSAT AT (Hel- ber et al.. 2019), CIFARI00), text (AG News (Zhang et al., 2015), Yelp Review (Yelp, Inc.)), and audio (ESC-50 (Piczak, 2015)).
Models. We demonstrate RePB's architecture independence by applying to Convolutional Neural Networks (CNNs: ResNet18, 8, ResNet50 (He et al., 2016), EfficientNet (Tan & Le, 2019)), Trans- formers (ViT (Dosovitskiy et al., 2021b), Swin Transformer (Liu al., 2021), ViECap (Fei et al., 2023), ABINet (Fang al., 2021), BERT (Devlin et al., 2019), HuBERT (Hsu et al., 2021), GeoDTR (Zhang et al., 2023)), State Space Models (Mamba-based Vim (Zhu et al., 2024)), and other task- specific architectures like Ada-MVS (Liu et al., 2023), VAE Kingma & Welling (2014); Ho & Sali- mans (2022). DDPM Ho et al. (2020), FixMatch (Sohn et al., 2020), FlexMatch (Zhang et al., 2021), and Dash (Xu al., 2021).
Pruning Rate. We use the pruning rate (or Pruned %), defined as the percentage of data points skipped during training, our primary indicator of efficiency. RePB's computational overhead for score calculation (reusing existing sample losses) and rescaling is minimal and largely amortized. Therefore, the pruning rate serves as hardware-indeepeddnn, and easily comparable metric that directly reflects the reduction in computational load (e.g., forward/backwarrd passes) and, consequently, potential training time speedup. Higher pruning rates, while maintaining or improving task performance, indicate greater efficiency (detailed justification in B)
Efficiency Metric:
5.2 COMPARATIVE ANALYSIS WITH OFATE-OF-THE-AEE METHODS
Dynamic Data Pruning Benchmarks. Table presents results on CIFARIO and CIFAR100 using ResNet18. RePB consistently outperforms other static or dynamic pruning methods (e.g.. GraNd- 4, EL2N-20, DP, Random*, UCB), across all pruning rates. Notably, RePB achieves performance comparable to or even slightly exceeding full dataset training at 30% and 50% pruning rates. Com- pared to InfoBatch (when run for the same number of epochs, denoted by 1), RePB demonstrates superior accuracy, especially higher pruning rates (e.g., 50% on CIFAR 100: 78.1% for RePB vs. 77.7% for InfoBatch"). ) Even when InfoBatch uses adjusted (typically more) epochs (1), RePB often matches or surpasses its performance with less data (e.g., CIFAR10 70%: RePB 94.9%

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of RePB with state-of-the-art dynamic data pruning methods on CIFARIO and CIFAR100 using ResNet18 across various pruning rates. t; Adjusted epochs as reported in the original paper. i; Same number of epochs as other compared methods. \: Indicates an unattainable pruning rate. Random indicates dynamic random pruning Qin et al. (2024).
CIFARIO
CIFARI00
Method
30%
50%
70%
30%
50%
70%
ResNet18
95.6
78.2
Random
94.6 _1.0
93.3 12.3
90.2 J5.4
73,8 14.4
72.1 JG.1
69.7
GraNd-4 (Paul et al., 2021)
95.3 .0.3
94.6 41.0
91.2
14.4
74.6
13.6
71.4 J6.8
68.8 .9.4
E1,2N-20 (Toneva al., 2019)
95.3 _0.3
95.1
10.5
91.9
13.7
77.2
11.0
72.1
16.1
DP (Yang et al., 2023)
94,9 .0.7
93.8
11.8
90.8
14.8
77.2
11.0
73.1
13.1
Random
94.8 .0.8
94.5 11.1
93.0
12.8
77.3
10.9
75.3
e-greedy Raju et al. (2021)
95.2 J0.4
94.9 10.7
94.1
11.5
76.4 11.8
74,8 13.4
UCB Raju al. (2021)
95.3 _0.3
94.7 10.9
93.9
11.7
77.3 10.9
75.3 12.9
InfoBatch Qin al. (2024)
95.6 -0.0
95.1 10.5
94.7 10.9
78.2 10.0
78. 10.1
76.5 L1.7
InfoBatch* Qin et al. (2024)
95.6 +0.0
95.0 10.6
94.4 11.2
78.3 70.1
77.7 40.5
RePB
95.6 -0.0
95.4 40.2
94.9 10.7
78.4 10.2
78.1 10.1
77.2 U1.0
InfoBatch' 94.7%). This highlights RePB's ability to effectively prune data without significant per- formance degradation, showcasing the benefits of its consistency-preserviig mechanisms.
Large-Scale Vision-Language Tasks vs. InfoBatch. We further benchmark RePB against In- foBatch on three challenging large-scale vision-language datasets: ToCa (zero-shot captioning). MJ+ST (scene text recognition), and SSIM (cross-domain captioning). detailed in Table 2. Across these demanding tasks, RePB consistently demonstrates superior efficiency or performance. For instance, ToCa (Table 2a), RePB achieves higher pruning rate (35,8% vs. 34.1% % for Info- Batch) while simultaneously outperforming both InfoBatch and full-dataset training on key metrics like NoCaps CIDEr (70.5 69.2 for InfoBatch, 70.5 for full). Similarly, for scene text recogni- tion on the 15M-sample MJ+ST dataset (Table 2b), RePB prunes substantial 44.4% of data-a greater reduction than InfoBatch (38.1%) (38.1%)-hiile while maintaining performance equivalent to full train- ing across all benchmarks (c.g., 96.1% on IIT5k).. On the SSIM cross coos-domain captioning task (Table 2c), RePB again achieves a higher pruning rate (34.8% vs. 23.0%) and yields improved or comparable CIDEr scores relative to both full training and InfoBatch. These results underscore RePB's enhanced effectiveness in handling large-scale, complex datasets where InfoBatch exhibits more modest pruning or comparatively lower performance. RePB's 's capacity for substantial data reduction while maintaining high fidelity highlights the robustness of its consistency-fouus local pruning and temporal rescaling strategies.
Table 2: Comparative performance RePB and InfoBatch on diverse large-scale vision-language and scene text recognition tasks. (a) Zero-shot captioning on 3M samples. (b) Scene text recognition on 15M samples. (c) Cross-domain captioning on 3M samples.
(a) ViECap on ToCa Dataset
(b) ABINet on MJ+ST
(c) ViECap on SSIM
NoCaps B@4
COCO B@4 C
IITTSk IC15 SVTP C80
COCO
Flickr30k
Method Pruned %
Pruned %
Pruned %
Accuracy
B@4
C
B@4 C
Full
26.6 70.5 27.1 95.2
96.1
85.4
88.7
89.2
9.6 45.1
6.5 22.3
InfoBatch
34.1
26.4 69.2 26.9 93.7
38.1
95.9
84.2
88.0
88.4
23.0
9.2 44.4
6.4 22.4
RePB
35.8
27.0 70.5 28.0 95.1
44.4
96.1
85.3
89.1
89.2
34.8
9.5
46.3
6.7 22.6
5.3 BROAD GENERALIZATION AND APPLICABILITY OF REPB
A key strength of principled data pruning framework is its ability to generalize across diverse tasks, model architectures, and learning settings. We showcase RePB 's wide-ranging applicability through an extensive set of experiments.
7

---

Published as a conference paper at ICLR 2026
Table Cross-architctture generalization on ImageNet-IK and CIFARI00 with CNNs, Transform- ers, and Mamba. RePB maintains accuracy while significantly reducing data across architectures.
ImageNet-IK
CIFAR100
Method
CNN
Transformer
Mamba
R18
R50
EfficientNet
VIT
Swin
Vim
R18
R50
Full
69.5
78.6
76.1
73.3
80.0
75.7
78.2
80.6
+RePB
69.5 32.1
32.1 78.5
30.2
76.1 / 28.0
73.3 23.3
80.0/ 38.3
75.6
31.3
78.3/ 39.6
80.8
49.5
Cross-Architecture Generalization. Table 3 demonstrates RePB's performance on ImageNet- K and CIFAR100 across various architectural families: CNNs (ResNet18, ResNet50, EfficientNett, Transformers (VIT, Swin, ViECap), and Mamba-based models (Vim). RePB consistently achieves performance nearly identical full dataset training while pruning significant portion of data (e.g., 23-38% on ImageNet- IK, 38-50% on CIFARI00), This consistent, near-lossless performance across diverse architectures underscores RePB's model-agnostic nature, stemming from its fundamental approach to addressing consistency issues rather than relying on architecture-spectee heuristics.
Diverse Downstream Tasks. Beyond standard classification, RePB exhibits strong performance across a variety of other vision and vision-language tasks (Tables 4, 5, 6). In multi-view stereo on WHU-MVS (Table 4), RePB with 37.3% pruning slightly improves MAE (0. 1147m vs. 0.1585m) and coverage. Similarly, for cross-view geo-localization of data while maintaining or slightly improving recall metrics. For image captioning on COCO and video captioning
CVACT (Table 5), RePB prunes 39.2%
MSR-VTT (Table 6, bottom panel), RePB achieves improved comparable results to full training with substantial pruning (e.g.. 24.3% pruned on COCO captioning improving CIDEr; 48.1 88.1% % pruned MSR-VTT video captioning improving BLEU @4 and CIDEr). These results highlight RePB's 's versatility in adapting to tasks with complex objectives and data structures.
Table 4: Performance on the multi-view stereo task using the Ada-MVS model on the WHU- MVS dataset (28K image-depth maps).
Table 5: Performance on the cross-view geo- localization task using the GeoDTR model on the CVACT dataset (35K images).
Method Pruned % MAE (m) +1 <0.6m (%)
Method Pruned % R@!
R R@5
R@ 10
R@ 1%
Ada-MVS
0.1185
97.38
GeoDTR RePB
86.21 95,44 96.72
98.77 96.90
RePB
37.3
0.1147
97.54
39.2
86.21
95.68
98.74
Table 6: Performance of RePB on advanced vision-language tasks including image captioning and video captioning. RePB demonstrates strong performance with substantial data pruning.
(a) Image Captioning on COCO
(b) Video Captioning on MSR-VTT
NoCaps Val (CIDEr)
COCO
MSR- VTT
Method Pruned %
Pruned %
Near Out Overall
B@4 M
B@4
M
ViECap RePB
58.4 63.1 65.3
65.2
27.1 24.6 91.5 18.0
23.4
20.8
27.9
24.3
58.4
63.7
65.9
65.8
27.6
24.6 92.8 18.0
48.1
25.9
21.6
29.2
5.0
Image Generation. Table shows RePB's application to image generation. For VAE on MNIST, DDPM on CIFAR1O. and DDPM with Classifier Guidance on CIFARIO, RePB prunes 27-40% of the data while maintaining nearly identical FID scores compared to full dataset train- ing. This demonstrates RePB's utility in generative modeling where data distribu- tion fidelity is crucial.
Table 7: Performance in image generation tasks across different model architectures and datasets: VAE on MNIST, DDPM and DDPM with CFG on CIFARIO.
VAE
DDPM DDPM-CFC
Method
Pruned % FID J Pruned % FID Pruned % FID
Full RePB 39.8
35.34
16.38
14.89
35,33
27.3
16.22
27.3
14,80
Supervised and remi-Supervised Learning across Modalities. Finally, Table 8 showcases RePB's effectiveness in standard supervised and sup-ssueee learning scenarios across image, text, and
8

---

Published as a conference paper at ICLR 2026
Table 8: Performance (a) supervised and (b) semi-super learning across diverse modalities: image, text, and audio. RePB consistently maintains accuracy across paradigms and modalities.
(a) Supervised learning
(b) Semi-supervised learning
ImagerunSAT
Textag News
Audiogsc.so HuBERT
ImageciFARio FixMatch
Textyep Review
Audiopsc-so
Method
ViT
BERT
FlexMatch
Dash
Pruned % Ace| Pruned % Acc Pruned % Acc
Pruned % Acc Pruned % Acc Pruned % Acc
Full
96.8
89.6
67.5
61.9
53.7
64.5
RePB
35.6
96.9
35.0
89.5
34.3
67.6
39.8
61.9
36.3
54.4
27.3
64.5
audio. In supervised learning (Table 8a), RePB applied to ViT (image), BERT (text), and HuBERT (audio) consistently prunes 35% of the data while maintaining or slightly improving accuracy. In semi-supervised learning (Table 8b), RePB integrated with FixMatch (image), FlexMatch (text), and Dash (audio) also demonstrates robust performance, achieving significant pruning (27-40%) with negligible or even positive impacts on accuracy. These results suggest that RePB's principles are broadly beneficial, irrespective of the learning paradigm or data modality.
In summary, the extensive experimental validation demonstrates that RePB not only outperforms existing dynamic data pruning methods on established benchmarks but also exhibits remarkable generalization capabilities across wide spectrum of tasks, datasets, model architectures, and learn ing settings. Its ability to significantly reduce data requirements while preserving or even enhancing performance underscores its potential as a valuable tool for efficient and reliable machine learning.
Compatibility with Sampling Strategies. Beyond standard architecture and task generalization, we evaluate RePB's s flex- ibility meta-framework by integrating it with recent im- portance sampling methods (Salaün et al., 2023; 2024). We conducted experiments CIFAR- 100 using ResNet18 un- der three configurations: (1) The original global optimal sam- pling strategy proposed in these works; (2) A hybrid approach where their resampling mechanism is applied locally within RePB's windows and reweighted by CTR; and (3) Standard RePB using gradient norm instead of loss. As shown in Table 9, integrating this external sampling strategy into the RePB framework yields consistent performance improvements across all pruning rates (e.g., +0.5% accuracy 30% pruning). This indicates that RePB's core mechanisms-
Table 9: Compatibility with alter- native sampling and gradient norm on CIFAR-100 (ResNet18).
Method
30%
50% 70%
Salaun et al.
77.7
77.3 76.7
RePB Salaun et al.
78.2
77.7 77.2
RePB (Grad Norm) RePB (Loss)
78.6
78.0 77.0
78.4
78.1 77.2
score context drift via local windows and correcting temporal bias via CTR-provine structural benefits that complement even sophisticated sampling algorithms. Fur- thermore, RePB using gradient norm achieves performance comparable to the loss-based default. This demonstrates that RePB metric-agnostic: it serves as a robust, general-purpon framework capable of enhancing various scoring criteria and selection strategies.
5.4 ABLATION STUDY
Effeet of each operation. An ablation study on CIFAR100 (Table 10, R18/R50 at 30%/50% pruning rate) reveals the dis- tinct contribution of each RePB component. Pruning without resampling (Res) results in sample pool depletion and per- formance collapse (19.0% adding LWP to address
Table 10: Ablation of proposed op- eration on CIFAR100.
Operation LWP
Accuracy R18
Building on resampling, context drift boosts accuracy to 78. 1%....2%% while adding CTR to mitigate temporal gradi- bias achieves 78.2% The full RePB framework attains the highest performance, matching or exceeding full dataset training and substantially outperforming random prun- ing (Random*). This demonstrates that the principled strate- gies of LWP for valid score comparisons and CTR for tempo- ral dynamic consistency are synergistically vital for achieving SOTA efficacy.
Res
CTR
R50
19.0
8.7
77.5
79.9
78.1
80.2
78.2
80.3
78.4
80.8
Random
77.3
79.7
Full Dataset
78.2
80.6

---

Published as a conference paper at ICLR 2026
Impact of Window Size. Figure 2 compares RePB (varying window sizes W, multiples B = 128) against InfoBatch (Annealing) and InfoBatch* (No Annealing, for higher prun- ing limits) on CIFAR 100 100 (ResNet50). RePB consistently out- performs both InfoBatch variants across most pruning percent- ages, regardless of its window size. For instance, at 40% prun- ing, all RePB configurations achieve +778%+ accuracy, sur- passing InfoBatch ( (778%% and InfoBatch* (78.0%), with this advantage widening at higher pruning rates (50-70%) Within RePB, smaller windows (W 1B to 4B) generally yield optimal performance, particularly up to 55% pruning. The
79
70
Accuracy (N6)
76
Ra7B (W1-12: 123)
R6-8 180- 2563
(W--22
75
R:58 328: 0156)
Htars 180-541: 03521
infobocch (A (knrealing)
I+Fobacdr" (No acneal rg)
30
40
50
60
70
80
W = 1B setting, which theoretically eliminates score con- text drift by using scores from a single pre-update batch, often core
Pruned Data Percentage (%)
defines the upper performance envelope, validating hypothesis. As W increases (e.g.. W 2 16B), a marginal performance decrease is observed, suggesting that even slight increases in potential intra-window drift can have a subtle impact, further underscoring the benefit of RePB's local context approach. Crucially, even RePB with larger windows largely maintains supe- riority over the global comparison methods of InfoBatch. This indicates RePB's 's relative robustness the window size hyperparameter, and consistently affirms the advantage of its principled local pruning strategy over global score aggregation.
Figure 2: Impact window size.
6 CONCLUSION
In this work, we addressed two critical consistency challenges in dynamic data pruning: score con- text drift, which invalidates inter-batch importance comparisons, and temporal gradient bias, which skews training dynamics. We proposed RePB, framework that systematically resolves these issues by integrating local window pruning for valid score comparisons within stable model contexts, and cumulative temporal rescaling to align the expected gradient trajectory with full-dataset training via inverse historical sampling frequency weighting. Extensive empirical validation across diverse datasets, tasks (classification, vision-language generation), and architectures (CNNs, Transformers, SSMs) demonstrates RePB's consistent superiority over state-of-the-an methods like InfoBatch, of- ten matching exceeding full-dataset performance with significant data reduction. By offering thcoretically grounded solution to previously underappreciated consistency violations, RePB estab- lishes more principled, robust, and effective paradigm for dynamic data pruning, enabling more reliable and efficient training of various models across diverse applications.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant 62471394, U21B2041, 62306241 and 62576284.
REFERENCES
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Man- sheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. In The Thirteenth International Conference on Learning Representations, 2025.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 2:12-1146, 2020.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 14879-14890. Curran Associates, Inc., 2020.
Dingshuo Chen, Zhixun Li, Yuyan Ni, Guibin Zhang, Ding Wang. Qiang Liu, Shu Wu. Jeffrey Yu, and Liang Wang. Beyond efficiency: Molecular data pruning for enhanced generalization. Advances in Neural Information Processing Systems, 37: 17:18033-18061. 2024.
10

---

Published as a conference paper at ICLR 2026
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint
2015.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. International Conference on Learning Representations, 2020.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6)11411142, 2012. doi: 10. 199/MS...2. 2211477.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021a.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2021b.
Talfan Evans, Shreya Pathak, Hamza Merzic, Jonathan Schwarz, Ryutaro Tanno, and Olivier Hénaff. Bad students make great teachers: Active learning accelerates large-scale visual under- standing. In Proceedings of the 18th European Conference on Computer Vision, volume 15075, pp. 264-280. Springer, 2024.
Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7094-7103, 2021.
Junjie Fei, Teng Wang, Jinrui Zhang, Zhenyu He, Chengjie Wang, and Feng Zheng. Transferable decoding with visual entities for zero-shot image captioning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3136-3146, 2023.
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4125-4134, 2019.
Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2315-2324, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
the
Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition Workshop, pp. 7713-7722, 2024.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266, 2019.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598. 2022.
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840- 68100-6881,10 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. /EEE/ACM transactions audio, speech, and language processing, 19:3451-3460, 2021.
Yawei Hu. Chi Zhang, Song Liu, and Mohan S Kankanhalli. Biased stochastic gradient descent for conditional stochastic optimization. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4199-42015 2020.
Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances in neural information processing systems, 29, 2016.
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. ArXiv, abs/1406.2227, 2014.
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary Lipton, al. Accelerating deep learning by focusing the biggest losers. arXiv preprint urXiv: 1910.0070 2019.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning. pp. 2304-234232 2313. PMLR, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2538-2547, PMLR, 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations, 2014.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. Ad- vances in neural information processing systems, 30, 2017.
Humaira Kousar, Hasnain Irshad Bhatti, and Jackyun Moon. Pruning-based data selection and net- work fusion for efficient deep learning. arXiv preprint arXiv:2501.01116 2025.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 100 (canadian institute for advanced research).
Jin Liu and Shunping Ji. novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from open aerial dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6050-6059, 2020.
Jin Liu, Jian Gao, Shunping Ji. Chang Zeng. Shaoyi Zhang, and Jian Gong. Deep learning based multi-view stereo matching and 3d scene reconstruction from oblique acrial images. ISPRS Journal of Photogrammetry and Remote Sensing, 204:42 -60, 2023.
Liu Liu and Hongdong Li. Lending orientation to neural networks for cross-view geo-localization. In CVPR, pp. 3624-5633, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Varun Mulchandani and Jung-Eun Eun Kim. Severing spurious correlations with data pruning. arXiv preprint arXiv:2503.1
(tl.:255...828. 18258, 2025.
12

---

Published as a conference paper at ICLR 2026
Deanna Needell, Nati Srebro, and Rachel Ward. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. Mathematical Programming. 155(1-2):549-577 2016.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet:
Finding important examples early in training. Advances neural information processing systems, 34:20596-20600 2021.
Karol Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM infernational conference on Multimedia, pp. 1015-1018, 2015.
Ziheng Qin, Kai Wang. Zangwei Zheng, Jianyang Gu, Xiangyu Peng, xu Zhao Pan, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, and Yang You. Infobatch: Lossless training speed up by unbiased dynamic data pruning. In The Twelfth International Conference on Learning Represen- fations, 2024.
Ravi S Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning. arXiv preprint arXiv:2111./2621 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, volume 115, pp. 211-252, 2015.
Corentin Salaün, Xingchang Huang, Iliyan Georgiev, Niloy Mitra, and Gurprit Singh, Online im- portance sampling for stochastic gradient optimization. arXiv preprint arXiv:2311.1446. 2023.
Corentin Salaün, Xingchang Huang, Iliyan Georgiev, Niloy J Mitra, and Gurprit Singh. Multiple importance sampling for stochastic gradient estimation. arXiv preprint arXiv:2407.15525 2024.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 608, 2020.
Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi. Data pruning via moving-one-sanee oan-saaple- Advances in neural information processing systems, 36: 18251-18262, 2023.
Haoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, and XIAOJUAN QL Data pruning by information maximization. In The Thirteenth International Conference Learning Representations, 2025.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114, 2019.
Mariya Toneva, Alessandro Sordoni, Aaron Comport, Adam Trischler, Yoshua Bengio, and Ge- offrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1288-52960 2016.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi- supervised learning with dynamic thresholding. In International conference machine learning, pp. 11525-11536. 2021.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reduc- ing training data by examining generalization influence. In The Eleventh International Conference Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2026
Yelp, Inc. Yelp Open Dataset. Web Page. URL
Ac-
cessed: 2025-05-12.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling. Advances in neural information processing systems, 54:18408-18419 2021.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances neural information processing systems, 28, 2015.
Xiaohan Zhang, Xingyu Li, Waqas Sultani, Zhou, and Safwan Wshah. Cross-view geo- localization via learning disentangled geometric layout correspondence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3480-3488, 2023.
Qing Zhou, Junlin Huang, Qiang Li. Junyu Gao, and Qi Wang. Text-only synthesis for image captioning. arXiv preprint arXiv:2405. 18258, 2024.
Qing Zhou, Junyu Gao, and Qi Wang. Scale efficient training for large datasets. CoRR, abs/2503. 1a5/2503.11385,,, 2025.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vi- sion mamba: Efficient visual representation learning with bidirectional state space model. In Proceedings of the 41st International Conference Machine Learning, 2024.
14

---

Published as a conference paper at ICLR 2026
A DETAILED COMPARISON WITH INFOBATCH RESCALING
While both RePB and InfoBatch (Qin et al., 2024) employ form of loss/gradient rescaling in con- junction with dynamic data pruning, their rescaling mechanisms, theoretical motivations, and practi- cal implications differ significantly. This appendix elucidates these distinctions, focusing on RePB's s Cumulative Temporal Rescaling (CTR) versus InfoBatch's 's instantameous expectation rescaling.
A.1 INFOBATCH: INSTANTANEOUS UNBIASED GRADIENT ESTIMATION
InfoBatch aims to achieve an unbiased estimate of the full batch gradient at each training step As described in their work (see Figure I and Section 2.3 of Qin et al. (2024)), given a dataset (or current epoch's data pool), InfoBatch maintains scores (e.g.. loss values) for samples. It then soft prunes by stochastically discarding portion of low-score samples. Specifically, if a sample has a score (=) below a threshold Hi (e.g., mean loss) and its pruning probability is r E (0, 1), its gradient is scaled by 1/(1 r) if it is kept. Samples with scores H₂(2) 2 He are not modified (implicitly, their pruning probability is 0, so 1/(1-0) = 1).
The core objective of InfoBatch's 's rescaling is to ensure that the expected gradient computed on the pruned subset S₁ at step approximates the gradient that would have been computed on the original (pre-pruning) data pool D, for that step:
P((z) V()))((2,00.) vo EEND, (V((2, 0.)
(10)
where (z) the probability of pruning sample step (if H₂(2) < He, P((z) r; otherwise (2( = 0), and (2) 1/(1 - (z)) is the rescaling factor. The expectation is taken over the stochastic pruning decisions step t, conditioned on model 0, and the scores (=)) This strategy focuses on making each individual gradient update step an unbiased estimator relative to the data available for pruning at that instant. The pruning probabilities and rescaling factors are typically determined based on the current batch epoch's 's score distribution and are not explicitly dependent on the long-tere selection history of individual samples across multiple epochs. InfoBatch also employs annealing schedule, training the full dataset in the last few epochs to reduce variance and potential remaining bias
A.2 REPB: CUMULATIVE TEMPORAL RESCALING FOR LONG-TERE DYNAMIC STABILITY
RePB's Cumulative Temporal Rescaling (CTR) mechanism serves a different primary purpose: to ensure the long-term expected cumulative gradient dynamics of training with pruned data align with those of training on the full dataset uniformly over the entire training trajectory. It addresses the temporal bias introduced by the fact that dynamic pruning causes different samples to be seen with varying frequencies across epochs.
In RePB, each sample (I;; y4) in the selected batch Sy global iteration k (where k typically corresponds to an epoch or a longer period over which rescaling weights are updated) has its loss f(r,, Mi, 0t) scaled by wi.
1/Sak- where f(, q/kk is the empirical cumulative selection frequency of sample up to iteration & (i.c., selected times in k rescaling periods). The rescaled loss is LRePB(S, 0c) SST Eies; Wi,k ((2,, y1, 0c).
The key differences and advantages of RePB's CTR are:
1. Addressing Long-Term Temporal Bias: CTR explicitly accounts for the entire selection history of each sample. If sample has been historically under-selected relative to uni- form draw from the dataset, its contribution up- up-weighted when it finally appears in a batch. This aims to correct for biases not just within single pruning step epoch, but over the entire course of training. InfoBatch's 's rescaling, being instantancous, does not in- herently correct for such long-term under- or overreepresnntation of specific samples across epochs.
2. Stabilizing Cumulative Gradient Magnitude:
By using Wi.k R3 1/(avg- probability of seeing sample i), CTR aims make the expected total gradi- ent signal contributed by each sample over many epochs proportional to what it would have contributed if seen uniformly. This helps to maintain consistent training dynamics
15

---

Published as a conference paper at ICLR 2026
and prevents the learning trajectory from being unduly skewed by the pruning process itself. The goal is EIE, A0, PJ = E,E, A6f"")] ensuring the overall learning path remains faithful to full-dataset training in terms of expected update magnitudes.
3. Independence from Instantaneous Pruning Probabilities: RePB's CTR relies on ob- served historical frequencies rather than needing to precisely define or estimate the proba- bility (=) with which a sample is pruned the current step t for the purpose of calculating an inverse probability weight. This can be an advantage because in complex dynamic prun- ing schemes (especially with local windowing like RePB's LWP), the exact instantaneous selection probability of a given sample can be difficult to model precisely, as it depends the composition of its current local window and the model state. CTR's empirical, historical approach sidesteps this modeling challenge.
4. Synergy with Local Window Pruning (LWP): RePB's LWP component addresses score context drift by ensuring score comparisons are valid within local window. CTR then complements this by addressing the longer-term consequences of these local (but still po- tentially non-uniform over the whole dataset) selections. InfoBatch, typically performing epoch-wide score comparison for pruning, faces the score context drift issue that LWP is designed solve. While its rescaling aims for unbiasedness relative to that epoch's 'ss (potentially flawed) pruning decision, it doesn't 't correct the upstream issue of score incom- parability within the epoch, nor the downstream issue of long-term sample frequency bias in the same direct way as CTR.
A.3 ILLUSTRATIVE EXAMPLE
Consider a scenario where, due to the dynamics of the model and data, a specific subset of infor- mative samples (e.g.. hard negatives) is consistently assigned low loss scores for several epochs by InfoBatch's 's global scoring, leading to them being frequently pruned (even if stochastically). While InfoBatch's rescaling would make the gradient unbiased for the steps where these samples are se- lected, it doesn't inherently compensate for their overall reduced exposure across these epochs. In contrast, RePB's s CTR would track that these samples have low o values. When they are eventually selected (perhaps due to resampling changes in their loss within local window), their Wi,k would be high, significantly boosting their contribution to compensate for past under-selection. This helps ensure their cumulative impact the model is not diminished over the long term. Furthermore, LWP makes it more likely that their true importance (relative to their local peers) is accurately as- sessed when they do appear in window, avoiding premature dismissal based comparison with globally easier samples.
A.4 SUMMARY OF ADVANTAGES OF REPB's APPROACH
In summary, RePB's ' dual strategy offers distinct advantages:
LWP ensures valid score comparisons, mitigating score context drift, problem not di- rectly addressed by InfoBatch's epoch-wide score aggregation for pruning.
CTR provides long-term dynamic stability, correcting for temporal biases in sample exposure across epochs, which more holistic approach than InfoBatch's ' focus on stantaneous unbiasedness at each pruning step.
CTR is empirically driven by historical data, making it robust and not reliant on explicit modeling of complex, instantaneous selection probabilities.
These aspects contribute to RePB's strong performance and generalization, particularly its ability to maintain high accuracy at significant pruning rates across diverse and large-scale tasks, as demon- strated in our experiments. While InfoBatch's 's annealing (training full data at the end) can recover some performance lost due to earlier biases, RePB aims to maintain a more consistent and less biased training trajectory throughout the pruning phase itself.
16

---

Published as a conference paper at ICLR 2026
B EFFICIENCY METRIC
B.1 COMPUTATIONAL EFFICIENCY
Reproducing exact wall-clock time reductions for dynamic data pruning methods can be notoriously challenging due to variations in hardware, software implementations, and system-level optimiza- tions. We therefore primarily adopt the percentage of data pruned (Pruned %) as our main efñ- ciency metric, reflecting the reduction in samples processed. This choice is strongly supported by the minimal computational overhead introduced by RePB.
As shown in Table 11, which details the overhead for processing 1 million samples RTX 3090 GPU, RePB's pruning logic is exceptionally lightweight. Its overhead measured at mere 0.082 seconds, which is even lower than that of InfoBatch (0.236s). More critically, this overhead is negligible when compared to the backbone processing time for standard architectures. For ResNet18 (734.1s for 1M samples), RePB's overhead constitutes only 0.01 1% of the backbone time, and for ResNet50 22),.4s), this ratio drops further to an almost imperceptible 0.004%. This substantially lower than InfoBatch's 's ratios (0.032% for ResNet18 and 0.01 for ResNet50).
NVIDIA
This extremely low overhead ensures that RePB is compute-positivy (Evans et al., 2024); the computational cost of executing RePB's prun- ing and rescaling logic vastly outweighed by the significant savings achieved from pro- cessing a reduced dataset (typically 220-30% pruned in our experiments, often much higher). For more complex models or those with larger, slower backbones than ResNet18/50, or tasks involving multiple intricate components be- yond a single backbone (c.g., in large vision- language models), the relative pruning over- head of RePB would be even smaller, further amplifying its efficiency benefits. Consequently, the Pruned % serves as a robust, hardware-agnostic, and readily comparable indicator of the substantial computational savings and training acceleration potential offered by RePB.
Table 11: Computational overhead of RePB (IM samples, NVIDIA RTX 3090 GPU). Overhead is the time for the pruning logic itself. Back- bone Time is provided for ResNet18 (R18) and ResNet50 (R50) for processing 1M samples. P/B the ratio of Overhead to Backbone Time.
R18 P/B
R50 P/B
Method
Overhead
734.1s
2122.4s
InfoBatch
0.236s
0.032%
0.011%
RePB
0.082s
0.01 1%
0.004%
C LIMITATIONS AND FUTURE WORK
While RePB robustly addresses key consistency challenges in dynamic pruning, future work could explore several refinements. The optimal local window size (W) for LWP, though RePB shows good tolerance, might benefit from adaptive selection based on dataset or training phase characteris- ties. Similarly, while CTR effectively provides long-term gradient aligmment, its adaptation speed in highly non-stationary scenarios could be investigated, potentially by incorporating short-term bias indicators. Finally, extending LWP to incorporate diverse or multiple importance metrics beyond sample loss presents an interesting avenue for enhancing its mecision-making process. These direc- tions aim to further broaden RePB's applicability and refine its efficiency while building upon its demonstrated strengths in principled, consistent data pruning.
D THE USE OF LARGE LANGUAGE MODELS (LLMS)
The authors affirm that this manuscript was written primarily by human authors. However, sec- tions of this paper underwent language refinement and grammatical corrections with the assistance of LLM, specifically Gemini-2.5 Pro. This tool used exclusively for improving clarity and readability and did not contribute to the scientific content, methodology, or core conclusions of the research. All scholarly responsibility for the content remains with the human authors.
17


Reviews:
[{"review_id": "enKUZayezK", "reviewer": "Reviewer_Gn4Z", "summary": "This paper identifies two fundamental consistency issues in dynamic data pruning: score context drift (incomparability of importance scores computed under different model states) and temporal gradient bias (distortion of gradient dynamics due to non-uniform sample selection over epochs). To address these, the authors propose RePB (Resolving Pruning Biases), a framework that combines (1) Local Window Pruning (LWP) to ensure valid score comparisons within short training windows where model parameters are nearly constant, (2) Uniform Probability Resampling to maintain data diversity, and (3) Cumulative Temporal Rescaling (CTR) to reweight sample losses based on historical selection frequency, thereby aligning long-term gradient expectations with full-dataset training. The method is theoretically motivated and evaluated across 16 datasets, 17 models, and 13 tasks, showing strong performance—often matching or exceeding full-dataset accuracy while pruning 30% or more of the data.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly articulates two underappreciated but critical pitfalls in dynamic data pruning and provides a principled, theoretically grounded solution.\n\n2. RePB’s design is elegant: LWP directly tackles score inconsistency by restricting comparisons to stable model contexts, while CTR offers a practical, history-based inverse weighting scheme that avoids the need to model complex instantaneous selection probabilities.\n\n3. The empirical evaluation is impressively broad in terms of tasks (classification, captioning, generation, semi-supervised learning), modalities (vision, text, audio), and architectures (CNNs, Transformers, Mamba, VAEs, diffusion models), demonstrating strong generalization.\n\n4. Ablation studies and comparisons with SOTA methods like InfoBatch convincingly validate the necessity and effectiveness of each component. \n\n5. Computational overhead is minimal, making RePB highly practical for real-world deployment.", "weaknesses": "1. Despite the diversity of tasks, the scale of some experiments remains limited. For instance, while ImageNet-1K is included, there is no evaluation on larger-scale vision benchmarks (e.g., ImageNet-21K) or billion-parameter language models, which are increasingly standard in efficiency research. The largest dataset used (MJ+ST with 15M samples) is promising, but more large-scale LLM or multimodal experiments would strengthen claims about scalability.\n\n2. The paper assumes sample loss as the sole importance metric. While common, this may not capture all aspects of sample utility (e.g., diversity, influence, or representativeness). Extending LWP to other scoring functions (as hinted in the limitations) would be valuable.\n\n3. The theoretical analysis relies on assumptions like Lipschitz continuity and bounded gradients, which are standard but may not always hold in practice (e.g., with unstable training or aggressive learning rates). A brief discussion of robustness under violation of these assumptions would be helpful. \n\n4. Missing some data pruning methods: \n\nSevering Spurious Correlations with Data Pruning\n\nPerplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models\n\nData Pruning by Information Maximization\n\nBeyond Efficiency: Molecular Data Pruning for Enhanced Generalization\n\nPruning-based Data Selection and Network Fusion for Efficient Deep Learning\n\nData Pruning via Moving-one-Sample-out", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OlJQepvZfC", "reviewer": "Reviewer_THQf", "summary": "This paper identifies two important problems in dynamic data pruning: inconsistency biases in the scoring context, and temporal gradient dynamics. It proposes to use local window pruning and cumulative temporal rescaling to address these two problems. The authors evaluate its effect on various datasets and demonstrate its effectiveness. Overall, this is an insightful and meaningful work.", "questions": "1. In the original InfoBatch, there was a trick mentioned: one can further downsample the lower-loss samples (E.g., keeping only 25% samples for 25% low-loss samples). Is this trick also compatible with RePB? It could further enhance the saving ratio. This is a complementary question, out of my curiosity, but not required.\n2. CTR weight could encounter a more extreme value than InfoBatch's rescaling factor (which is fixed to 1/(1-r)). Is there any observation on this potential problem? \nFor example, an update with a large factor on a sample not sampled for many epochs could lead to a large update step, leading to higher loss, and the subsequent updates cannot immediately reduce the CTR weight, which will cause this sample to be updated with much higher importance than sampling it earlier.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work identifies two important problems in current dynamic data pruning methods and solves them. The insight is quite accurate, and the meaningful improvement further enhances the robustness and generalization of dynamic data pruning.\n2. The experimental evaluation is comprehensive. This work extends the application to a lot more scenarios, including other vision and vision-language tasks, and semi-supervised learning.\n3. The presentation is good.", "weaknesses": "1. According to the ablation experiment demonstrated in Tab. 9, RePB relies on resampling more than other dynamic pruning methods. It is worth a little bit more discussion: whether it is because of the inner window score stability, or cumulative temporal rescaling (rescaling factor too high in some cases)?\n2. Currently, the pruning factor $\\rho$ is used, but this factor has no ablation, and there is no discussion of its value and tuning characteristics.\n3. For proof of 4.2, a critical problem is that in this scenario $E[1/X] \\neq 1/E[x]$, the substitution may not hold.\n\nMinor:\n1. The citation of \"Scale efficient training for large datasets\" would be better to use the accepted version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DQDo57vgNM", "reviewer": "Reviewer_kLUp", "summary": "This paper introduces RePB (Resolving Pruning Biases), a framework that tackles two fundamental consistency problems in dynamic data pruning: score context drift (where importance scores computed at different training stages aren't comparable) and temporal gradient bias (where non-uniform sample selection alters training dynamics). RePB addresses these issues through Local Window Pruning, which restricts score comparisons to short windows (a few batches), and Cumulative Temporal Rescaling, which reweights samples based on their historical selection frequency to align gradient trajectories with full-dataset training. The authors provide theoretical guarantees for both components and demonstrate RePB's effectiveness across 16 datasets, 17 model architectures, and 13 diverse tasks spanning classification", "questions": "- What are the formal statements of the propositions ?\n- How does the framework deal with flexible pruning rate $r$ ?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper’s main strength is its comprehensive and diverse experimental validation, covering many datasets, architectures, and tasks. RePB seems to consistently deliver good performance and hence is a promissing approach.", "weaknesses": "Overall, I found that the paper needs further refinement and clarification before it is ready for publication.\n\nThe main weaknesses of the paper lie in its theoretical clarity and methodological presentation. The theoretical analysis lacks rigour: propositions are not clearly stated, assumptions are introduced informally within the proofs, and the proofs rely on approximations rather than precise derivations. This makes it difficult to assess what is actually proven and under what conditions the results hold. \n\nIn addition, the methodology section presents the pruning criterion using a fixed mean-based threshold $\\mu_k$, which limits flexibility in controlling the pruning or compression rate $r$. While most experiments appear to use this fixed rule, some dynamic data pruning benchmarks suggest a variable rate, creating inconsistency between the method’s description and its implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2W3jKdtcpr", "reviewer": "Reviewer_XfsH", "summary": "Deep learning models rely their success mainly on the variety of data used in vast quantities. \nThis dataset expansion has direct impact on the training efficiency. This work focuses on data selection which aims to train models on smaller but carefully chosen data subsets. There are dynamic methods that perform such data selection during the learning process, called online data pruning. Giving wrong importance to data samples can introduce biased gradients. This submission identifies two fundamental consistency issues called: score context drift and temporal gradient bias. \n\nScore context drift focuses on the parameters drift during training whereas temporal gradient bias takes into account the bias introduced by non-uniform sampling distribution over time (due to different data selection).", "questions": "- How sensitive is RePB to the local window size?\n- How the method ensures that computing scores over a small window does not accumulate error over time?\n- How the method compares to Salaun et al. 2025a,b?\n- Does it make sense to combine RePB with Salaun et al.?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n- Identify fundamental issues of inconsistency biases in dynamic data pruning\n- Propose a framework (RePB) to mitigate the inconsistency bias due to parameter drift during training and non-uniform sampling distribution over time\n- RePB maintains data diversity and prevents sample pool collapse\n- Theoretical foundations are provided that justifies comparison of scores collection within local windows\n- Comparisons are reported over different methods.", "weaknesses": "Weaknesses:\n- Comparisons with the most recent SOTA (Salaun et al.2025a) is missing\n- How sensitive is RePB to the local window size?\n- How the method ensures that computing scores over a small window does not accumulate error over time?\n- Missing citations: \n\nRecent work by Salaun et al. 2025a, b, develop better online importance mechanisms driven by Multiple importance sampling (MIS) and Optimal MIS. The paper seems to completely ignore these references.\n\n@misc{salaun2025a,\n      title={Online Importance Sampling for Stochastic Gradient Optimization}, \n      author={Corentin Salaün and Xingchang Huang and Iliyan Georgiev and Niloy J. Mitra and Gurprit Singh},\n      year={2025},\n      eprint={2311.14468},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2311.14468}, \n}\n\n@misc{salaun2025b,\n      title={Multiple Importance Sampling for Stochastic Gradient Estimation}, \n      author={Corentin Salaün and Xingchang Huang and Iliyan Georgiev and Niloy J. Mitra and Gurprit Singh},\n      year={2025},\n      eprint={2407.15525},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2407.15525}, \n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)