Batch Pruning by Activation Stability
Authors: Md Mustakin Alam
Shaker Islam
Aminul Islam
Venue: ICLR 2026 Poster (2026)
Keywords: Batch Pruning
Activation Stability
Neural Network
Activation
Deep Learning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: A dynamic data pruning method for deep learning training that adaptively removes low-utility batches based on activation stability, significantly reducing data usage while maintaining the accuracy.

Abstract: Training deep neural networks remains costly in terms of data, time, and energy, limiting their deployment in large-scale and resource-constrained settings. To address this, we propose Batch Pruning by Activation Stability (*B-PAS*), a dynamic plug-in strategy that accelerates training by removing batches that contribute less to learning. *B-PAS* monitors the stability of activation representations across epochs and prunes batches whose activation variance exhibits minimal change, indicating diminishing learning utility. Applied to ResNet-18, ResNet-50, and the Convolutional vision Transformer (CvT) on CIFAR-10, CIFAR-100, SVHN, and ImageNet-1K, *B-PAS* reduces training batch usage by up to 57\% with no loss in accuracy, and by 47\% while slightly improving accuracy. Moreover, it achieves up to 61\% savings in GPU node-hours, outperforming prior state-of-the-art pruning methods with up to 29\% higher data savings and 21\% greater GPU node-hour savings. We further demonstrate the generalization of *B-PAS* by extending it to GPT-2 fine-tuning, showing that activation stability can serve as an effective pruning signal beyond vision models. These results highlight activation stability as a powerful internal signal for efficient training, offering a practical and sustainable path toward data and energy-efficient deep learning.

Published as a conference paper at ICLR 2026
BATCH PRUNING BY ACTIVATION STABILITY
Md Mustakin Alam Shaker Islam Aminul Islam'
l University of Louisiana at Lafayette BRAC University
(md- und-mustakin.alam aminul) @all)@lla @louisiana.eeee edu, shaker.islam@eg www.:gg....a...bbh..aamamm
ABSTRACT
Training deep neural networks remains costly in terms of data, time, and energy, limiting their deployment in large-scale and resource-constrained settings. To address this, we propose Batch Pruning by Activation Stability (B-PAS), dynamic plug-in strategy that accelerates training by removing batches that contribute less learning. B-PAS monitors the stability of activation representations across epochs and prunes batches whose activation variance exhibits minimal change, indicating diminishing learning utility. Applied to ResNet- 18, ResNet-50, and the Convolutional vision Transformer (CvT) on CIFAR-10, CIFAR-100, 100, SVHN, and ImageNet-1K K, B-PAS reduces training batch usage by up to 57% with no loss accuracy, and by 47% while slightly improving accuracy. Moreover, it achieves up to 61% savings in GPU node-hours, outperforming prior state-of-the-art pruning methods with up to 29% higher data savings and 21 % greater GPU node-hour savings. We further demonstrate the generalization of B-PAS by extending it to GPT-2 fine-tum munittunnn, showing that activation stability can serve an effective pruning signal beyond vision models. These results highlight activation stability as powerful internal signal for efficient training, offering a practical and sustainable path toward data and energy-efficien deep learning. Our code is publicly available at http:///iwubbooooo/mssaaaaaaaaaa Bathhahmmaaaaaaaaeaeeaa BwricchhPPnnn----- Pruning-by-Activatie -cttAiiiationnSSaaale
INTRODUCTION
Deep learning has emerged as a powerful paradigm for solving complex tasks across a variety of domains. These models, while highly effective, are inherently resource and time-intensive, frequently consuming significant GPU hours and memory bandwidth during both training and inference phases. They utilize large amounts of computation even on redundant or less informative data, leading to inefficiencies in resource-constrannel environments. Among deep learning models, Convolutional Neural Networks (CNNs) have achieved remarkable success a wide range of computer vision tasks, including image classification, object detection, and segmentation (Krizhevsky et al. 2012, Simonyan & Zisserman 2014 He et al. 2016 2017 Law & Deng 2018). However, these performance gains often come at the cost of increased computational demands. State-of-tthe-ar CNN architectures typically require large-scale datasets such as ImageNet (Krizhevsky et al.) (2012) for effective training and involve millions of parameters, making them expensive to train and deploy. This poses a significant barrier for practitioners with limited computational resources. Reducing training costs without sacrificing performance remains a longstanding challenge in deep learning.
straightforward approach to address this challenge is to reduce the amount of training data. Techniques such as dataset distillation (Nguyen et al.. 2021 Zhao & Bilen
2023] Wang et al. 2022)
and coreset selection (Har-Peled & Mazumdar] [004] Park et al., [0022 Xia et al. 2022) aim to synthesize or select compact, informative subset of the original dataset. While effective in reducing
data volume, these methods often introduce nontrivial computational overhead and may result in
degraded model performance. Weighted sampling methods (Zhao & Zhang 2015 Csiba & Richtárik 2018; Johnson & Guestrin 2018) improve convergence by increasing the sampling frequency of informative samples, but their performance is highly sensitive to the choice of model and dataset.
Another line of research focuses on reducing the number training iterations through data pruning. Static pruning methods estimate sample utility scores and remove low-utility samples before training
"Corresooniigg author

---

Published as a conference paper at ICLR 2026
begins (Toneva et al., 2018: Paul et al. 2021). ) but often incur high preprocessing costs and lack adaptability during training. Dynamic pruning approaches mitigate these issues by adjusting the pruning process on the fly. For example, InfoBatch (Qin et al., 2024) dynamically prunes low-utiliity samples using a soft-pruning strategy combined with expectation rescaling to maintain unbiased gradients. Similarly, [He et al. (2024) leverages prediction uncertainty and training dynamics to prune up to 25% of data from large-scale datasets such as ImageNet without sacrificing accuracy. However, these approaches often rely on complex heuristics.
Early stopping is a widely used strategy for reducing data usage by terminating training once performance plateaus (Duvenaud al. [2066] Mahsereci et al., 2017 Bonet al. 202I). However, most existing approaches rely on gradient- gradient-based signals that often fail to generalize across modern optimizers or require carefully tuned hyperparameters and specialized frameworks. More recently, Ahmaa et al. (2024) proposed an early stopping criterion based on the stability of convolutional activations, highlighting the potential of internal network dynamics as reliable indicators of training progress. This line of work builds on the broader concept of activation stability, which is closely linked to the phenomenon of Neural Collapse (Papyan et al., 2020), wherein class-specific representations become increasingly aligned and activation patterns stabilize as training converges. Together, these findings suggest that activation stability offers a promising direction for analyzing data utility through internal model dynamics.
While prior data pruning approaches such as InfoBatch (Qin et al. 2024 rely on per-sample loss statistics, gradient rescaling, or fixed heuristics, often incurring additional computation requiring explicit loss tracking, our work introduces a different perspective: Can the stability of internal activations serve as a signal for assessing the informativeness of batches during training, thereby enabling effective and dynamic pruning of redundant batches? We answer this affirmatively by introducing lightweight, plug-and-play method that dynamically prunes low-utility batches based on the stability of activation variances across network layers. Unlike strategies that depend on difficulty scores, auxiliary models, or manually crafted rules, our approach is non-intrusive, as it operates on activation statistics already available from the forward pass, thereby introducing negligible overhead. The pruning is performed on-the-fly during training. without pretraining phases, validation labels, or static schedules. Our framework continuously monitors the mean standard deviation of flattened activations across layers for each batch across consecutive epochs and prunes batches if the change in this statistic is negligible across any neural network architecture. For the demonstration, applied to convolutional neural networks, this activation-dii ddiiven pruning framework reduces redundant data usage across epochs while preserving training effectiveness and generality, offering a practical and efficient solution that establishes a foundation for extending stability- praning to broader deep learning architectures. Our key contributions are as follows:
Activation Stability-Guided Dynamic Batch Pruning, We introduce a lightweight, model-internal pruning strategy, 'Batch Pruning by Activation Stability' (B-PAS) that dynamically identifies and removes low-utility batches during training by leveraging the stability of activation representations. Specifically, we track the mean standard deviation of activations across hidden layers and consecutive epochs to assess whether a batch continues to contribute meaningful learning gradients. Batches exhibiting negligible change activation variance are deemed redundant and pruned on-the-fly, without relying on auxiliary networks or handcrafted difficulty metrics. The pruning behavior is controlled by a tunable threshold hyperparameter 6, enabling adaptability across datasets and model architectures.
Comprehensive Evaluation on Benchmark Models. We evaluate B-PAS on ResNet-18, 18, ResNet-50, and CvT (Wu et al. 2021) across C1 CIF (Krizhevsky et al. b). SVHN (Netzer et al. 2), and ImageNet- 1K (Krizhessky et al. 2012), with extensive threshold sweeps (45 settings on CIFAR-10 10 and eight on ImageNet- K). Results show that B-PAS prunes up to 57% of training batches without accuracy loss, and up to 47% while slightly improving accuracy, while reducing computational cost by as much as % in GPU node-hours. These findings highlight both the robustness and tunability of activation stability as pruning signal, delivering substantial training efficiency gains across scales and architectures. We further extend B-PAS to GPT-2 fine-tuning on the Alpaca dataset, demonstrating its applicability to LLMs and text-based tasks.
Data Savings Index (DSI), We introduce DSI, a new metric that quantifies the cumulative fraction of training data saved during learning. DSI provides direct measure of data efficiency, with higher values indicating greater reductions in training cost and computational resources.
2

---

Published as a conference paper at ICLR 2026
2 BATCH PRUNING BY ACTIVATION STABILITY (B-PAS)
In this section, we introduce the concept of a plug-in to CNN architectures: Batch Pruning by Activation Stability (B-PAS), detailing its formulation and adaptation for image applications. Figure illustrates the working mechanism of a conventional CNN (labeled 'A') alongside the proposed plug-in (labeled 'B').
A
kernel
Convolution Openaiinss
Inpus
Pooling
Pooling
Output
Convolation
Convolution I
0.7 .X..
ReLUU
ReL.U
..9
Softmax Activation
Training
Flatten
feethackk
Fully Connected Layers
B
Flattened Output Tensors
Prune Batch: Bag
x
AX
Qpondly
(0, Inn
B1
B2
...
B
B
B
Keep Batch: B,,
No
6(1)
Butch
Coav
Conv
Meen of Std.
Hatch
Batch
Coav
Cony
Mean of Std. per
Batch
Coav
Coay
Meen of Std.
Batch
B
041
(01
ota))
B
(0
B
0
(0
B2
012
022
1 1 0t2)
B2
01
012
10 11a
B
012
2
Bs
013
013
(
013))
By
013 mn
01,3
(01,3) d1.))
B 113
01,3
(0) 03.)
B
B -1
B
B B
X-- (0) (01,0t-11-- -o alana))
B
X; (a (0 iy [nnn
B Basa
X (0) (01,00011 iy alanon
Fpoch i
Epachi +
2 74--1
M4 744-1
R: :Baccess
spoch
#Butches
#Batthhes in current epoch, previous spoeii
#Butches current spoch. i+ is less than or equal to previous spoch
1. than or equal to
equal
spoch
Figure 1: Overview of the proposed Batch Pruning by Activation Stability (B-PAS) plug-in integrated
into prevalent Convolutional Neural Network (CNN) training pipeline. (A) The conventional CNN architecture processes input images in batches through convolutional, ReLU activation, and pooling layers, followed by fully connected classifier.
(B) The B-PAS module monitors the standard deviation of ReLU-activated convolutional outputs for each batch across epochs. For each epoch, standard deviation is recorded per convolutional layer (columns) and aggregated across layers to compute the mean standard deviation for each batch (rows). These per-batch means compared across consecutive epochs to assess activation stability. the change (AXX) for a batch B,, falls below dynamic threshold d(i), the batch is deemed to have low learning utility and is pruned from subsequent training. By updating the dataset at the end of each epoch using this feedback, the process dynamically focuses training batches that continue to provide meaningful gradient information, thereby improving efficiency without compromising accuracy.
then

---

Published as a conference paper at ICLR 2026
2.1 ACTIVATION STABILITY
Building on the observation from Neural Collapse (Papyan et al. 2020) that activation patterns stabilize as training converges, we adopt the idea of (Ahmad et al. 2024). which links data variation across CNN layers to near-optimal learning capacity. Extending this principle to the batch level, we observe that as training progresses, certain batches show diminishing changes in activation variance across consecutive epochs, signaling little additional learning. To capture this effect, we use the widely adopted Rectified Linear Unit (ReLU) (Agarap 2018). where the variance of ReL.U-activated outputs provides meaningful measure of feature stability. When this variance remains nearly unchanged across epochs, the corresponding batch is deemed converged, as its contribution to weight updates has effectively stabilized.
We compute variance after ReLU since Since it not only introduces non-linearity but also suppresses inactive neurons, ensuring that variance reflects sparse, meaningful features rather than noisy pre-activation values. Thus, monitoring the standard deviation of post-ReLU activations provides a reliable estimate of batch learning utility: once changes become negligible, the batch is pruned. This enables the model to focus computation on informative batches while discarding redundant ones, improving efficiency without harming performance.
In Figure D [). the 'Convolution Operations' module illustrates the ReLU-activated convolutional feature maps input images, organized batch-wise across training epochs. At each convolutional layer, training images are processed in batches denoted by B,...., Bag where BR, refers to the final batch in epoch i. For instance, if epoch i contains 400 batches, then Ba, = B300. indicating that the th0th batch is the last batch of that epoch. Each batch consists of multiple images, which are represented as tensors containing numerical activation values. To quantify activation variability, we compute the standard deviation of these values by first flattening the output tensors of each ReLU-
activated convolutional layer. The standard deviation is then computed using o = V 1 Em (tx (t=--)² p)?. where l denotes each individual value from the flattened tensor (for k = 1,...,N), mean of these values, N is the total number of values in the flattened tensor, and represents the resulting standard deviation. This computation is performed independently for each batch and each convolutional layer.
is the
In Figure B), each epoch table (i.e., Epochs i 1, i, and i++1) 1) presents the standard deviation of the Rel.U--ctiivted outputs from the convolutional layers. In each table, rows correspond to data batches (e.g., BI..... Br,). ). and columns represent individual convolutional layers, except the final column. The last column contains the mean standard deviation for each batch, computed across all convolutional layers. For example, the row corresponding to batch B,, includes standard deviations OIn.s.... L.n,, where di.,. denotes the standard deviation of the output from the 1-th convolutional layer for batch Br,, and / is the total number of convolutional layers. The mean standard deviation for the final batch B,, in epoch is denoted as X:( ol,,)), indicating the mean standard deviation across all convolutional layers.
These mean standard deviations are used to track the variance behavior of each batch over time. Beginning with epoch one, the mean standard deviation is computed for every batch. From epoch two onward, the current epoch's mean is compared against the previous epoch's 's mean for each batch. If the change is negligible, the batch is considered to have converged and may be pruned from subsequent training epochs. This pruning process continues iteratively for the remaining epochs.
2.2 BATCH PRUNING
The decision prune batch is based on the stability of its mean activation standard deviation across consecutive epochs. As illustrated in Figure BB), we consider three epochs: 1-1, i, and i+1. If the absolute difference between these means across consecutive epochs falls below a dynamic threshold 8(1), the batch is deemed to have converged and is excluded from subsequent training.
To illustrate, consider the standard deviations of batch BR, across the | convolutional lay- ers as O1. .t.n. The mean standard deviation for this batch at epoch i is denoted by X1 I.n.)), and at epoch i-1 by (o1. Ila,)) The change in mean standard deviation/variance for the batch BRg is computed as:
AX
Xe(a - ,la,) X,- (01 .t.... amm) s 5(1)
4

---

Published as a conference paper at ICLR 2026
If AX for the batch B,; falls below the threshold (1),, the batch is pruned and excluded from training in epoch 1+1. This criterion is applied to all batches at the end of each epoch. The pruning decisions update the dataset by retaining only the informative batches, resulting in reduced set of training data for the next epoch and a progressively more efficient training process. As training progresses, the number of retained batches decreases monotonically. Formally, for any epoch i, the number of batches n, satisfies S. SR Because the pruning is dynamic, the composition of the final batch may change across epochs. For instance, for epoch let the last batch be BR, and the second last B,,-1; similarly, in epoch i 1, they are Br and - -1 1. Without pruning, these batches are identical, whereas pruning one batch in epoch shifts the indexing so that in epoch Bn, =B
2.3 DYNAMIC THRESHOLD FOR PRUNING
To determine the negligible difference between activation variances across epochs, introduce a tolerance hy- perparameter to formalize pruning. Instead of requiring exact invariance, 5 captures variance saturation with a small non-zero margin.
10
6
6==x10-
6 10-5
66
=-=5x00
6=5x
6 10
6.-5x 10-6
6.-10-1
6.-10
6=
=5=5
=6=100
=6=5110 10
No pruning
- -Harsh prusing
1250
Number of Batches
1000
We empirically analyze the effect of different ó values on pruning dynam- ics. Very small leads to overly con- servative pruning, discarding batches only when variance changes are nearly imperceptible, thus retaining most batches. Conversely, large 5 induces aggressive pruning, prematurely re- moving informative batches and risk- ing early training collapse.
750
500
250
50
100
150
200
Epoch
Figure 2: Pruning dynamics under different 6 settings for ResNet-50 on ImageNet-IKK (200 epochs). Lower thresholds (e.g., 8 [10-6 5 10-%) lead to conservative pruning, retaining most batches until late epochs, while higher thresh- olds (e.g.. ō [5x 10 1010005.10111110001 50) cause aggressive pruning and premature training termination. The dynamic schedule (6, 5x10-6 de 5x10- (x10=))rrovidess provides a balanced trajectory, stcadily reducing data.
To balance this trade-off, we adopt a dynamic schedule: 6(1) = 6, eas a = ( where 5, and 5e the initial and final threshold values, is the current epoch, and the total number of epochs. This schedule is conservative in early train- ing, when features are broadly learned, and more aggressive in later stages, when learning stabilizes. At epoch i, batches with AX < 5(1) are pruned.
Figure 2 illustrates these dynamics using eight sets of values on ImageNet- IK with ResNet-50. The horizontal line shows no pruning, while the sharp drop at epoch 2 depicts extreme aggressiveness (all batches removed). Lower ranges (e.g., whereas higher ranges (e.g., 5 E [5x 5x 1x100-1) prune aggressively and often terminate training early. A balanced configuration of 8, = x 10 and & = x 10-5 achieves steady pruning without sacrificing stability. Nonetheless, 8 remains a tunable hyperparameter, adaptable to dataset complexity, architecture, and resource budgets.
[10 5 x ")) yield delayed, conservative pruning.
EXPERIMENT
We evaluate B-PAS as a plug-in module on ResNet-18. 18, ResNet-50, and the Convolutional vision Transformer (CvT) to assess robustness and generality. In the networks, we add Batch Normalization layer, which normalizes per-batch variance and has strong impact on pruning dynamics, Experiments span CIFAR-10, CIFAR-100, SVHN (32 x 32), and ImageNet- (~1.3M images, 224 x 224). Unless specified, we use E [10- 6, x 10 5] for CIFAR-10/10 and SVHN, and E [5 10 6 5 x 10 for ImageNet-IK. 1K, corresponding demonstrate the generalization of B-PAS PAS, it is extended to GPT-2 fine-tuning with the Alpaca dataset. For evaluation, report validation accuracy and GPU node-hours, but since hardware and system
empirically validated balanced pruning regimes. To further
5

---

Published as a conference paper at ICLR 2026
factors confound GPU time, also introduce the Data Savings Index (DSI), which quantifies skipped data across epochs and batches as a continuous measure of efficiency. To ensure well-defined variance tracking. batches fixed once initialization (rather than regenerated each epoch) while intra-batch shuffling is applied to preserve stochasticity and avoid overfitting In contrast, full-dataset baseline training follows the standard batch shuffligg protocol. Full experimental details provided in Appendix[A.1
GPU node-hours. It is measure of computa- tional cost, defined as GPU node-hours = 9 x h where g the number of GPUs used during training and h is the total training time (in hours). The percentage of node-hours saved reported relative to the full dataset baseline.
= 10-4
6 = 10
10-1
6 10-
Apcuracy (%)
95.0
92.5
Data Savings Index (DSI), To directly quan- tify the reduction in training data usage, we in- troduce the Data Savings Index: DSI
90.0
100
carne where n, is the number of batches retained in epoch i, no is the total number of batches before training begins, eg the epoch at which training stops, and eo is the scheduled number of epochs in the absence of pruning or early stopping (c, S cp). The DSI value lies in (0, 1). with higher values indicating greater savings in data usage. For example, if a model is scheduled to train for 5 epochs with 200 batches per epoch, but training stops at epoch 3 after processing 200, 190, and 180 batches, the DSI 200+190+ 180 0.43, meaning that 43% 5x 200
75
DSI (%)
50-
25
0-
1
5
13 17 21
25 29
33
37 41 45
Forly-five Different 5 Sellings
Figure 3: Empirical analysis of B-PAS on CIFAR- 10 with ResNet-18 across 45 5 settings. Smaller thresholds have less data savings with higher ac- curacy, larger thresholds save more data at the cost of accuracy, and intermediate values (e.g., 8 E [10-6,5 10 10-5)) provide the best trade-off.
of the potential training data is saved.
3.1 RESULT ANALYSIS
Saved (%) DSI (%) Accuracy (%)
6,=10
6.=5 x 10-*
78
6,=10-6
Empirical Analysis on CIFAR-10.
10-5
We first evaluate B- B-PAS using ResNet-18 18 CIFAR-10 10 across forty-five threshold (6) set- tings. As shown in Figure 3 we define five groups of starting thresholds (6.), each paired
76
6.=5 10-4
6.-5 x 10-7
6.=5 x 10-6
10-5
5.=5 x 10-5
with multiple end thresholds (6.). Specifically,
6.=5x100
for 5, = 5 x 10-S and 6. = 1 we con- sider three and six values of 6z, respectively, while for the remaining groups we test 12 values.
5,=10-5
6.=5 x 10
60
5,=10-5
5.-10-t
The &e values are {10 3,5 x 10 10-4,10-4.0 x 10-5,8 8 x 10-5 7 x 10-5,6 6 x 10-5.5
6.=10
40
6.=5 x 10-5
10-5 5015,4 3x 10-55 2x 22100-11000) For
the 6, = 10 group, only d, = (10-3
Eight Different 5 Settings
10-4,10 are included; for d, 10-5 we use 6, (10-3 (10-3,10) -,55x 10 10-4,11
155510
Figure 4: Empirical analysis of B-PAS on ImageNet-IK with ResNet-50 across eight set- tings. Smaller thresholds retain most data with higher accuracy but lower savings, larger thresh- olds prune aggressively and harm accuracy, while intermediate values (c.g.. 6 E [5 10 5, 10 10-5)) achieve the best trade-off.
10-5.3 x 10-5]. The results reveal a clear pat- tern in accuracy and DSI percentage. Larger threshold groups (e.g.. 8, 5 x 10-5) yield very high data savings but also sacrifice accu- racy. Conversely, smaller threshold groups (c.g.. 8, 10 7) retain nearly all data, resulting in high accuracy but reduced pruning benefits. Values in intermediate groups (e.g.. 6, = 10 provide balanced trf In particular, the 29th setting with E [10-6,5 5 x 10-0] achieves the highest accuracy while saving data by 25%. Thus, for CIFAR- 10 and similar 32 x 32 datasets, 8, = 10
and 6, = 5 x 10-5 represent a strong default setting, as further validated Table 3

---

Published as a conference paper at ICLR 2026
Empirical Analysis ImageNet-1K. Guided by the CIFAR-10 analysis, we eval- uate B-PAS with ResNet-50 on ImageNet- 1K over 200 epochs across eight threshold settings. As illustrated in Figure 4 we ob- serve the same trend: smaller thresholds pre- serve accuracy but maintain low DSI and GPU node-hours savings, while larger thresh- olds lead to aggressive pruning and accuracy degradation. For example, 5 E 110-;,3 5 10-5] yields the highest accuracy but also the lowest data and GPU node-hours savings. In contrast, 8 E [5 x 10-5, 5 x 10= leads to early, aggressive pruning, often terminat- ing training around epoch 100, resulting in very low accuracy. This behavior aligns with the pruning dynamics shown in Figure [ smaller thresholds delay pruning. retaining most batches until late epochs, while larger thresholds trigger sharp drops of batches ear- lier in training. The most balanced results are obtained for [5 10-6, 5 x 10-5) as supported by both Figure Figure
Table 1: Comparison of B-PAS against InfoBatch and random pruning on CIFAR-10/100 using ResNet- 18/50. B-PAS matches or excceds InfoBatch across different pruning ratios while maintaining the accu- racy of the full dataset, showing its competitiveness on small-scale datasets.
Approach
CIFAR-10
CIFAR-100
in % DSI
Acc
DSI
Acc
ResNet-18
0
95.60=0.2
0
0.3
+Random
25 94.60-0.3
24
7..
+InfoBatch (30%)
22
95.60-00 -01
19
78.20
+InfoBatch (50%)
37
95.10-0.3
32
7..10
+InfoBatch (70%)
53
94.70=0.4
47
76.50=0.4
+B-PAS
25
95.60-0.1
24
78.20=0.1
ResNet-50
0 95.66-0.1
0
80.60=0.5
+Random
33 94.50=0.3
30
75.77=0.4
+InfoBatch (30%)
21
95.66-0.1
18
80.60=0.1
+InfoBatch (50%)
36
95.20=0.3
32
80.05=0.1
+InfoBatch (70%)
52
94.99=0.4
45
79.37=0.4
+B-PAS
33 95.66
0.1 30
80.60
This setting achieves maximmm accuracy with DSI over 45% and saved GPU node-hours by 48%, indicating efficient pruning without compromising performance. Moreover, although the choice of the threshold hyperparameter d may appear to require extensive tuning for new datasets or model families, we find that full training is unnecessary for calibration. In practice, reliable 8 values can be obtained by training on only 10% of the data, enabling fast and efficient hyperparameter selection. Detailed analyses Node-hour savings follow the same trend across experiments, confirming that data usage is tightly coupled to training cost. Unlike node-hours, however, DSI provides more comprehensive and system-independent measure by capturing saved data across epochs.
provided in Appendix A.4 An additional observation is that DSI and GPU
Comparison with SOTA Method.
Table 2: Comparison of B-PAS and InfoBatch on ImageNet- 1K with ResNet-50. B- B-S achieves greater efficiency, re- ducing data usage by 57% and node-hours by 61, while preserving accuracy at the full dataset level. More conser- vative d values yield further accuracy gains (78.43%) with moderate savings, highlighting the scalability of activation stability signals in large-scale training.
We compare B-PAS with the state- of-the-art pruning approach Info- Batch (Qin et al.' (024) on CIFAR- 10, C0FAR-100, and ImageNet- 1K (Ta- bles and InfoBatch has previously demonstrated superiority over 14 static and three dynamic pruning baselines, establishing as a strong reference point (A Comprehensive tabular com- parison is provided in Appendix A..5). On CIFAR-10 and CIFAR-100 (Ta- ble T. B-PAS achieves accuracy com- parable to the full dataset and Info- Batch across multiple pruning ratios.
Approach
ImageNet-
in % + Saved hrs DSI
Acc
Re
0 78. 1
+InfoBatch (40%)
40
28 78.07=0.1
+B-PAS (de [10 11055100
40
10
61
57 1..
+B-PAS (ŌE [5 10 10-6 10 3D)
48
47
7.1
For example, in CIFAR-100 with ResNet-50, B-PAS saves 30% of the data while maintaining 80.6% accuracy, closely matching InfoBatch at the 30% pruning ratio (DSI = 18%). To further assess pruning behavior, we introduce random pruning baseline: we track the number of batches flagged by activation stability but prune the same number of batches chosen at random. Unlike B-PAS, random pruning consistently degrades accuracy, underscoring that activation stability identifies non- informative batches rather than simply reducing training data. These results demonstrate that B-PAS is competitive on small-scale datasets, with modest but reliable savings due to the limited learning utility in low-resolution tasks. In contrast, the advantage of B-PAS becomes more pronounced on ImageNet- K. As shown in Table 2 InfoBatch achieves 28% data savings and 40% GPU node-hour reduction while maintaining full dataset baseline accuracy (78.07)). By comparison, B-PAS delivers substantially larger gains: with 8 E [10-5, 10-4] it saves 57% of the data and 61 % of node-hours
7

---

Published as a conference paper at ICLR 2026
at the same accuracy, and with more conservative thresholds (6 [5 x 10-6. 10 10- it further improves accuracy to 78.43% while still achieving 47% data savings and 48% node-hour reduction. These results highlight that activation stability-driven pruning not only matches InfoBatch in accuracy preservation but also provides significantly greater efficiency on large-scale training.
This demonstrates that activation stability provides scalable pruning signal in large-scale training, where the learning utility of data is more significant. Finally, just as InfoBatch evaluates multiple pruning ratios (30%. 50%, 70%). B-PAS naturally supports different pruning regimes through the choice of 8 values. Smaller ō ranges yield conservative pruning with lower DSI, while larger values trigger more aggressive pruning and faster convergence. This flexibility allows B-PAS to adapt pruning aggressiveness without requiring handcrafted schedules or explicit loss tracking, underscoring its practical utility in both small- and large-scale settings.
Cross-Architecture, Cross- Cross-Task and Dataset Robustness. Table 3 evaluates the generalization of B-PAS across diverse architectures (ResNet-18. 18, ResNet-50, and CvT) and datasets (CIFAR-10 10, CIFAR-100, SVHN, and ImageNet-IK). Several consistent trends emerge.
First, across CNN archi- tectures, B-PAS preserves full dataset baseline accu- racy while significantly re- ducing data usage and train- ing cost. For example, on ImageNet-IK with ResNet- 50, B-PAS achieves 78.43% accuracy (slightly higher than full dataset) while sav- ing data usage by 47% and mode- by 48%. Simi-
Table 3: Cross-architecture and dataset robustness of B-PAS on CIFAR- 10/100, SVHN, and ImageNet-IK with ResNet-18/50 and CvT. B-PAS preserves accuracy across models while reducing data usage and GPU node-hours.
CIFAR-10 CIFAR-100
SVHN
ImageNet- IK
R-18
R-50 R-18 R-50 R-18 R-50 R-18 R-50 CvT
Full Dataset
95.60
95.66
78.2
80.6
95.90
96.27
70.05
78.07 79.65
B-PAS
95.60
95.66
78.2
80.6
95.97
96.27
71.5
78.43 79.6
DSI(%)
25
33
24
30
27
30
37
47
14
Saved hrs(%)
23
29
22
29
25
33
61
48
13
lar savings are observed on smaller datasets, with up to
33% data usage savings on CIFAR-I0 and 33% node-hours reduction on SVHN. These results demonstrate that activation stability provides a reliable pruning signal across scales and architectures within the CNN family.
Extending beyond CNNs, we evaluate B-PAS on CvT-13. Prun- ing is performed by tracking stage-wise activations: after each CvT stage, token sequences are projected back to spatial format to compute variance on the multi- scale feature maps. With a con- servative threshold range (6 E [1 x 10-5,5 5 x 10-"), pruning remains limited, achieving only 14% DSI and 13% GPU node-hour savings while preserving near- baseline accuracy. Using a more aggressive schedule (6, = 10 10 8 10-3), B-PAS attains substantially higher efficiency, reaching 35% DSI and GPU-hour savings with only a modest accuracy drop (~79%). This behavior reflects the slower and noisier activation dynamics of transformer- based models, where stability emerges later than in CNNs. Since our CvT training spans 200 epochs, shorter than the 300+ epochs typically required for full convergence on ImageNet- 1K stabilization is delayed, naturally limiting pruning under milder thresholds. These results indicate that effective pruning with activation stability in CvT requires longer training or more aggressive i schedules.
Table 4: Comparison of full-data training and B-PAS on loss, perplexity, training time, and pruning statistics for GPT-2 large.
Loss Perplexity Avg. Epoch
Total
Pruned DSI
Time (s)
Time (s) Batch (%) (%)
Full Data 0.2207
1.25
5359.88
54420.13
B-PAS 0.2201
1.25
5039.11
51211.29
23.00
Table reports the results of applying B-PAS during fine-tuning of GPT-2 large on the Alpaca instruction tuning dataset. Because this experiment involves short 10-epoch fine-tuning run on mansformer-based LLM, we adopt more aggressive pruning threshold range (ò, 10-3 3 d. = 10-2) to compensate for the smoother activation dynamics of transformers. As expected, activation stabilization occurs later in training, resulting in a lower Data Savings Index (DSI) compared to our large-scale vision experiments, nevertheless, B-PAS prunes 23% of batches. Crucially, pruning does not harm model performance: loss and perplexity remain unchanged or slightly improved. At the same
8

---

Published as a conference paper at ICLR 2026
time, total training time is reduced by about an hour 2XA100 GPÜs. These findings demonstrate that B-PAS is compatible with transformer- language models and provides measurable efficiency gains even in smaller fine-tuning settings. GPT-2 implementation details are in Appendix A.I A.I
3.2 ABLATION STUDIES
To better understand the behavior of B-PAS, we perform controlled ablation studies across learning rates, normalization layers, training epochs, pruning granularity, and optimizers. These studies highlight both the robustness of the method and the factors influencing its efficiency.
Effect of Learning Rate. Table 5 shows the impact of different learning rates on ImageNet-IK with ResNet-50. For batch size of 256, we consider learning rates of 0.2, 0.01, and 0.1. While all settings achieve comparable DSI 155%-47% accuracy varies significantly: 74.29% at LR 0.01, 77.27% at LR 0.2, and 78. 18.43% LR = 0.1. These results suggest that excessively small learning rates hinder convergence, while overly large ones reduce generalization. Importantly, the pruning dynamics of B-PAS remain stable across learning rates, showing that activation stability is largely agnostic to optimizer step size.
Table 5: B-PAS under dif- ferent learning rates (LR).
Table 6: Effect of Batch Normalization on B-PAS.
Table 7: Effect of train- ing epochs on B-PAS on ImageNet-Ik.
Table 8: B-PAS pruning granular- ity. Pruning Acc(%)
LR DSI(%) Acc(%)
DSI(%))cc(($)
Epochs DSI(%) Acc(%)
0.2
45
77.27
0.01
46
74.29
BN
19.72
89.87
90
12
78.07
Sample 70.87
0.1
47
78.43
+BN 25
95.60
200
78.43
Batch 78.43
Impact of Batch Normalization. Table 6 CIFAR-10 results with ResNet-18 with and without Batch Normalization (BN). Since BN normalizes feature statisties per batch, strongly affects B-PAS. Without BN, activation trajectories are unstable, requiring more aggressive thresholds (e.g.. 5, 5 10 8e 10-3) obtain reasonable DSI (19.72%); with the default 5 values, pruning is minimal (DSI = 2%). Also, removing BN results in an accuracy drop. In contrast, with BN, the default thresholds yield effective pruning (DSI 25%) while improving accuracy to 95.60%. These results show that BN not only stabilizes activations but also enhances the discriminative signal used by B-PAS, enabling more reliable identification of redundant batches. More analyses are provided in Appendix(A. A.7
Effeet of Training Epochs. Table 7 compares ImageNet- IK with ResNet-50 performance at 90 and 200 epochs. With 90 epochs, pruning remains limited (DSI 12%) due to insufficient stabilization, whereas at 200 epochs DSI rises to 47% with improved accuracy (78.43%). This highlights that longer training naturally allows greater pruning, where data savings become more critical and B-PAS proves most effective.
Pruning Granularity: Batch Sample Level. Ta- ble [ compares pruning at the sample and batch levels for ImageNet-IK on ResNet-50. For sample-level pruning, the accuracy reduces to 70.87%, compared to 78, 48.43% for batch- level pruning. This discrepancy arises because sample-level pruning may disproportionately eliminate certain classes, leading to class imbalance and degraded generalization. contrast, batch-level pruning preserves class diversity while still removing redundant information, confirming its superi- ority the granularity choice for B-PAS.
Table B-PAS preserves accu- racy while achieving similar DSI across optimizers on CIFAR-10 with ResNet-18, demonstrating robustness to optimization dynamics.
SGD Adam AdaGrad
Full Dataset95.66 93.36 92.93
B-PAS
95.60 93.34
92.93
DSI (%)
25
24
22
Optimizer Robustness. Finally, Table 9 explores different optimizers CIFAR-10 with ResNet-18. Across SGD, Adam, and AdaGrad Ruder 2016). B-PAS
maintains accuracy identical to baseline training. with DSI ranging from 22% to 25%. These results highlight that the pruning criterion is robust to different optimization dynamics, even when another first-order adaptive optimizer (Adam/AdaGrad) alters activation trajectories. This reinforces that B-PAS generalizes across diverse optimization regimes without requiring re-tuning.

---

Published as a conference paper at ICLR 2026
4 RELATED WORK
This work is related to three major research directions in efficient deep learning: static data pruning, dynamic data pruning. and dataset distillation. Static pruning methods select training subsets prior to training using criteria like geometric diversity (Sener & Savarese 2017] Agarwal et al. 20200. uncertainty (Coleman al. 2019 Gal et al. 2017), learning difficulty Toneva et al. 2018 Paul et al. 2021). Gradient-based approaches (e.g.. GraNd, EL2N (Paul et al. 2021)) and decision- boundary methods like DeepFool (Ducoffe & Precioso 2018) assess sample importance more directly. Other strategies include bilevel optimization (Killamsetry et al. (02T), submodular selection (Iyer
al. 2021), I. ensemble heuristics (Xia et al.) (022), and diversity-awarr methods (Welling [009]]
Zheng et al. 2023). These techniques often require full dataset access and heavy pre-computation,
limiting scalability. Dynamic pruning eliminates low-utility samples during training via online signals. Bandit-based methods (Raju et al. (021), soft pruning with gradient rescaling (InfoBatch
Qin et al. (2024)). and uncertainty-dritt -rrivveett--diven pruning (He et al.) (004)) have shown success but often involve complex heuristics and tuning. Dataset distillation synthesizes compact training sets via gradient (Zhao & Bilen 2023 Liu et al. 2023] Cui et al. 2023 Yang et al. [223), distribution (Wang
et al. [022 Sajedi et al., (203),, or trajectory matching (Cazenavette et al. 2022 [u et al. 2023
Guo et al. 2024). yet faces challenges in scaling to high-resolution data and large models.
5 DISCUSSION AND CONCLUSION
Recent work, such as InfoBatch [Qin et al., 2024). advances data prun- ing through temporary, sample-level pruning with gradient rescaling to preserve unbiased gradient estimates. While effective, this approach requires maintaining per-sample loss statis- ties and revisiting all data in subse- quent epochs, limiting efficiency in large-scale training. contrast, our proposed B-PAS performs permanent, batch-level pruning based on activa- tion stability, eliminating the need for gradient rescaling or repeated access to discarded data. This design signif- icantly reduces data usage, training time, and GPU node-hours by lever- aging internal activation dynamics in- stead of loss signals. A common con- cern in data pruning is the risk of introducing bias by disproportionntely removing informative samples or underrepresented classes. B-PAS operates at the batch level using activation stability and is agnostic to class labels and per-sample loss statistics; pruning decisions are therefore driven solely by model-internal dynamics rather than sample difficulty or data distribution. Across diverse experiments, B-PAS consistently preserves, and in some cases improves, classification accuracy compared to full-dataset training, indicating no measurable degradation in predictive performance. Furthermore, an analysis in Appendix AA shows that the pruning strategy does not favor either easy or hard examples and maintains class balance, resulting in a representative subset selected strictly according to the proposed criterion. Finally, our empirical analysis of pruning dynamics shows that batches gradually lose learning utility as training progresses and their activation variance stabilizes (Figure Specifically, the change in mean standard deviation (AX) is initially large and decreases monotonically across epochs, eventually saturating. When this change falls below the threshold 8, the corresponding batches are pruned. This mechanism enables B-PAS to eliminate redundant computation while prioritizing informative data. If activation stabilization were not indicative of reduced utility, pruning would significantly degrade performance; instead, our results demonstrate that activation stability serves reliable signal for scalable, efficient, and unbiased pruning. We conclude that B-PAS provides a practical, plug-and suug-nnd-play play approach to data-efficient deep learning, with particular promise for large-scale training where efficiency gains are most impactful.
Batch 0
Batch
Mcan Std. Deviation
0.8
Batch 3
Batch 4
Batch 5
0.6
Batch
Batch 8
0.4
Batch 9
0.2
0
25
50
75
100
125
150
175
200
Epoch
Figure 5: Declining mean standard deviation change (AX) across epochs for some selected batches on CIFAR-10 with ResNet-18, indicating diminishing learning utility as activa- tion variance stabilizes. Batches with AX <6 are pruned.
is acknowledged that a current limitation is the use of empirically chosen threshold schedules; developing data-driven, adaptive thresholding mechanisms is an important direction for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Research reported in this publication was supported by the Louisiana Board of Regents through the Board of Regents & Research Competitiveness Subprogram (RCS) Support Fund under Portions of this research were conducted with high-
Contract Number: LEQSSF(2023-2)))RD-2 performance computational resources provided by the Louisiana Optical Network Infrastructure (htp::/wwww.lom.orr))
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu), arXiv preprini arXiv: 1803.08375 2018.
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In Computer Vision- ECcvn 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XVI 16, pp. 137-153. Springer, 2020.
Sahan Ahmad, Gabriel Trahan, and Aminul Islam. When do convolutional neural networks stop learning? arXiv preprint arXiv:2403.02. 024773 2024.
Brian Bartoldson, Ari Morcos, Adrian Barbu, and Gordon Erlebacher. The generalization-staaiii tradeoff in neural network pruning. Advances in neural information processing systems, 33: 20852-20864, 2020.
David Bonet, Antonio Ortega, Javier Ruiz-Hidalgo, and Sarath Shekkizhar. Channel-wise early stopping without a validation set via nnk polytope interpolation. In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 351-358. IEEE, 2021.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4750-4759, June 2022.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: 11829, 2019.
Dominik Csiba and Peter Richtárik. Importance sampling for minibatches. Journal of Machine Learning Research, 19(27):1-21, 2018.
Justin Cui, Ruochen Wang. Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to ImageNet- 1K with constant memory. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 6565-6590, PMLR, 23-29 Jul 2023. URL https:: https://proceedingg mlr. wttpagsss...pprrr... www.pppresssv2222
cui23e html
Jiawei Du, Yidi Jiang, Vincent Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3749-3758, June 2023.
Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv: 000.0002229 2018.
David Duvenaud, Dougal Maclaurin, and Ryan Adams. Early stopping as nonparametric variational inference. In Artificial intelligence and statistics, pp. 1070-1077. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1183- 1192. PMLR, 06-11 11 Aug 2017. URL nattps://proceediinno mlr. wttp:pppssss/s///ps.11 htm

---

Published as a conference paper at ICLR 2026
Tushar Ganguli and Edwin KP Chong. Activation- pruning of neural networks. Algorithms, 17 (1):48, 2024.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-alignee trajectory matching, 2024, URL htttts://aaaat:t/
org/abs/233 2tabs//2310.055 05773
Sariel Har-Peled and Soham Mazumdar, On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing, pp. 291-300, 2004.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7713-7722, 2024.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato (eds.), Proceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 of Proceedings of Machine Learning Research, pp. 722-754, PMLR, 16-19 Mar 2021. URL https: http:///ooeeeecccceeeeeee
w.l..press/vee valsss/vi// 132 wewleerrlll... hhtml
Tyler B Johnson and Carlos Guestrin. Training deep models faster with robust, approximate impor- tance sampling. Advances in Neural Information Processing Systems, 31, 2018.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35()):8 10-8111,, 1118118, May 2021. doi: 2b.11609aaai.v3355 16988.
URL httphttps///oo //tps//jjj///// aaal. org/ hte///ndexxxx./ php http://AAAA/aatiiiiaea arwaaaattiiaeee wiww//wwwww/// 16988
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). a. URL htRLhhttp://wwwt Cs www.ttttttotoooooooooooooooeeeee
html
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). b. URL http:: wwwp://wwwww.oooom toronto. wdw.eduukkeeeee htri///iizzaafrr htt./.aaaaa.a.a.l........mm.....
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 2012.
Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European conference on computer vision (ECCV), pp. 734-750, 2018.
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17314-17324, October 2023.
Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig, Early stopping without a validation set. arXiv preprint arXiv:1703.099 09580, 2017.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
12

---

Published as a conference paper at ICLR 2026
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5198, 2021.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 16652-24663 2020.
Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active learning is a strong baseline for data subset selection. In Has Trained Yer? NeurlPS 2022 Workshop, 2022.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early training. Advances in neural information processing systems, 34: 20607, 2021.
Ziheng Qin, Kai Wang, Zangwei Zheng, Jianyang Gu, Xiangyu Peng, Daquan Zhou, Lei Shang, Baigui Sun, Xuansong Xie, Yang You, al. Infobatch: Lossless training speed up by unbiased dynamic data pruning, In The Twelfth International Conference on Learning Representations, 2024.
Ravi S Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning. arXiv preprint arXiv:2111.1262) 2021.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint
1407477609.04747, 2016.
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstantinos N. Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), pp. 17097-17107, October 2023.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1708.0048849 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05133
2018
Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12196-12205, 2022.
Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference Machine Learning, ICML '09, pp. 1121-1128. New York, NY, USA, 2009. Associ- ation for Computing Machinery. ISBN 9781605585161 doi: 10. 10.1145/155337415555177 URL
https: www.oo.oog.....oo
1ttpl/oog/...11///1111... 1et.ee1111111111 1553517
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions vision transformers. In Proceedings the IEEF/CVF international conference on computer vision, pp. 22-31, 2021.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022.
Beining Yang, Kai Wang, Qingyun Sun, Cheng Ji, Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 53201-53226. Curran Associates, Inc., 2023. URL hattps://proceedinne hwwp:eeedddinnneeueeeseeee..m htcpaaapeeerreeeeeeeeeeeeeeeeele wwwes//apeeeee/t/////// /223/
13

---

Published as a conference paper at ICLR 2026
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09339, 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In international conference on machine learning, pp. 1-9. PMLR, 2015.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates, 2023. URL https: / /ariiiiiv.. org htt///bb/222100 15800
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EXPERIMENTAL SETUP
A.1.1 MODEL SPECIFICATIONS
ResNet-18 18 and ResNet-50. For CNN backbones, we adopt standard ResNet architectures with residual connections following (He al. 2016). ResNet-18 constructed using BasicBlock units (expansion factor 1) with layer configuration 2. 2, 2, 2], while ResNet-50 uses Bottleneck units (expansion factor 4) with configuration [3, 4, 6, 3]. Both models begin with 7 7 convolution and max pooling, followed by four residual stages, global average pooling, dropout (0.2), and fully connected classifier. Batch Normalization is applied after each convolution, and ReLU serves as the activation. For activation tracking, we monitor outputs after each of the four residual stages (layerl-4).
Convolutional vision Transformer (CvT). We also evaluate B-PAS on CvT-13 (Wu al. 202 which integrates convolutional projections within transformer blocks. CvT-13 consists three stages (Table [OD: (i) 64-dim embedding with depth 1 and attention head; (ii) 192-dim embedding with depth 2 and heads; and (iii) a 384-dim embedding with depth 10 and 6 heads. Each stage applies convolutional embedding, convolutional multi-head self-attention with depthwise projections, and MI.P blocks with GELU activation. Activations are tracked at the stage outputs, where token sequences are reshaped back into spatial (B, C, H, W) format for variance computation. The network ends with layer normalization, global average pooling over tokens, and a linear classifier.
Table 10: Model specifications for architectures used in our experiments.
Model
Building Block
Depth
Embedding / Channels
ResNet-18
BasicBlock
[2, 2, 2, 2] [64, 128, 256, 512] [3, 4, 6, [][56, 512, 1024. 2048]
ResNet-50
Bottleneck
CvT-13
Conv-Attn + MLP [1,2, 10] [64, 192, 384]
B-PAS in GPT-2 Large. We apply our batch pruning mechanism to GPT-2 Large (774M parameters) during instruction tuning on the Alpaca dataset, demonstrating generalization to autoregressive language models. Activation Tracking: We track activations at every alternate transformer decoder block throughout the model. During each forward pass, the hidden-state output of each tracked layer is captured, and we compute the standard deviation of the activations across layers. These per-layer statistics are then averaged to produce a single scalar summary representing the activation profile of each batch. Pruning Decision: After the first epoch, we compare the standard deviation of each batch's 's mean activation between consecutive epochs. If the absolute change in a batch's 's activation profile falls below dynamically computed threshold, that batch is deemed converged its activation patterns have stabilized, indicating diminishing marginal contribution to further learning. The threshold follows an exponential schedule that starts small and increases over training, allowing more aggressive pruning in later epochs as the model converges. Pruned batches are permanently removed from subsequent epochs, and the data loader is rebuilt at each epoch reflect the reduced training set.
A.1.2 DATASET SPECIFICATIONS
Table 11 provides the number of training and validation samples used for each dataset. CIFAR-10 and CIFAR-ION 100 each include 50,000 training and 10,000 validation images. SVHN contains a larger validation set relative to its training size, with 73,257 training and 26,032 validation samples. ImageNet-1K: being significantly larger, includes over one million training images and 50,300 validation samples, reflecting its role as a large-scale benchmark. All images augmented with commonly adopted transformations, i.c., normalization, random crop, and horizontal flip not stated otherwise.
15

---

Published as a conference paper at ICLR 2026
Table 11: Dataset Splits for Training and Validation
Dataset
Training Samples Validation Samples
CIFARIO 10
50,000
10,000
CIFARI00
50,000
10,000
SVHN
73,257
26,032
ImageNet- 1K
1,230,867
50,300
A.1.3 HYPERPARAMETERS
Table 12
summarizes the hyperparameters used across datasets and architectures. Unless otherwise noted, all models are trained with SGD optimizer, using a momentum of 0.9, For ImageNet with ResNets, we adopt Multi tisteiRRRRR scheduling, while other CNN datasets use cosine annealing. CvT models are trained with Adam W. For ImageNet training, we use 4 GPUs parallel; hence, both the batch size and learning rate are scaled linearly by the number of GPUs (i.e., 256 x total batch size and base learning rate 0.1 x = 0.4). GPT-2 large is fine-tuned using the given hyperparameters.
Table 12: Training hyperparameters across datasets and architectures.
Dataset / Model
Epochs Batch Size
LR
Weight Decay
Scheduler
CIFAR-IO (ResNets)
200
128
0.05
x 10
ConannAnnealing
CIFAR-100 100 (ResNets)
200
128
0.10
x 10
Cosine
SVHN (ResNets)
200
128
0.10
5 x 10
ConealAnne
ImageNet (ResNets)
200
256x4
0.40
1 10
MultiStepLR
ImageNet (CvT) GPT-2 large (fine-tuning)
200
128x4
1 10-3
0.05
Cosine
10
2x2
5 X 10-5
0.01
Linear
A.1.4 HARDWARE SPECIHICATIOSS
Tables 13 and summarize the computational setups. ImageNet experiments were performed pegehperformance cluster with dual 32-core Intel Xeon Platinum CPUs and 4x A100 GPUs connected via NVLink. CIFAR-10/100 and SVHN experiments were conducted on local workstation with an AMD Ryzen 9 CPU and single Titan RTX GPU.
Table 13: Hardware specifications for CIFAR-10/100 and SVHN experiments.
ComponentSpeeclfcatioon
CPU
AMD Ryzen 7900X. 12 cores 24 threads, 4.7 GHz base
GPU
NVIDIA Titan RTX, 24 GB GDDR6
RAM
Corsair Vengeance, 128 GB DDR5, 6000 MHz
A.1.5 SOFTWARE SPECIFICATIONS
[5] the software environment used for all experiments. Python 3.8.18 served as the core programming language. Key libraries included Py'Torch and Torchvision for model development, along with Matplotlib, NumPy, Scikit-learn, Seaborn, Pandas, and Pillow (PIL) for data handling and visualization. CIFAR- 10/100 and SVHN experiments were conducted in Jupyter Notebook, facilitating interactive de- velopment and reproducibility. Additionally, Im- ageNet experiments were conducted Python scripts.
Table 15: Software Specifications
Component
Details
Python Version
Libraries
torch, torchvision. matplodib, numpy. scikit-learn, seaborn, pandas, pillow (PIL)
Platform
Jupyter Notebook
16

---

Published as a conference paper at ICLR 2026
Table 14: Hardware specifications for ImageNet experiments.
ComponentSpecifictiion
CPU
Dual Intel Xeon Platinum 8358 (Ice Lake), 32 cores each
GPU
4 NVIDIA A100 (Ampere) with NVLink interconnect 512 GB
RAM
Table 16: Results for Group of 6, = 5 10 10-5
8,
8
DSI (%) ACC (%)
5 X 10 5 x 10
75
89.65
x 10 55 x 10
75
90.24
x 10
10-4
68
91.65
A.2 MORE RELATED WORK
Bartoldson et al. (2020) analyzes weight pruning by defining stability as the accuracy drop after removing parameters, diagnostic notion operating entirely in parameter space while keeping the data fixed. In contrast, B-PAS functions in data space, tracking temporal activation variance across epochs online signal for permanently pruning batches, direction unexplored in prior pruning work. Similarly, Ganguli & Chong (2024) uses activation frequency to prune neurons in small fully connected networks, focusing on static model sparsification rather than data reduction. While activation patterns have been used to assess weight importance or characterize network behavior, no prior method leverages activation stability over time directly remove training data.
A.3 DETAILED TABULAR RESULTS
Detailed CIFAR-10 Results. Tables 16 20 provide the tabular counterpart of Figure reporting the full results of our CIFAR-10 10 analysis actoss forty-five (6,, 5e) configurations. Each table corresponds one starting threshold group, with multiple end thresholds. These results clearly illustrate the trade-off between pruning aggressiveness and accuracy:
Larger 8, values (Tables 16 17)
trigger early and aggressive pruning, yielding substantial data savings but lower accuracy.
Smaller 6, values (Table 20 retain most data, preserving accuracy at the cost of reduced pruning benefits.
Intermediate settings (Tables 18 19) achieve the most favorable balance, with the setting
E [10- 5x (Table 19 delivering the highest accuracy while saving 25% of training data.
Together, these tables complement Figure by providing a detailed numerical view of pruning dynam- ics, confirming that the thresholds govern an effective trade-off between efficiency and generalization.
Detailed ImageNet Results. Table 21 provides the tabular version of Figure reporting results of B-PAS with ResNet-50 on ImageNet- KK across eight threshold settings. The table includes accuracy, Data Savings Index (DSI), training time, node-hours, and early stopping behavior, offering a more granular perspective on pruning dynamics.
clear trade-off emerges between data savings and accuracy. Larger thresholds such as [5 x 10-5,5 5 x 10 40=4] trigger aggressive pruning, with training terminating around epoch 100 and accuracy dropping to 76. 7%.1%, despite saving 73% node-hours. Conversely, smaller thresholds (e.g., 5 E [10-6, 5 10 10) achieve slightly better accuracy compared full dataset (78.63%) but achieve only moderate efficiency gains (37% node-hours saved).
Intermediate thresholds provide the most balanced trade-off: for instance, å [5 10 10-9.5 9.5x 10-5 yields 78, 78.4%% accuracy with 47% DSI and 48% node-hours saved. GPU node-houe hours is calculated by (Training Time (in sec0nds)/3600)*4 (number of GPUs). The node-hours saved percentage is
17

---

Published as a conference paper at ICLR 2026
Table 18: Results for Group of 6, 5 10-6
6.
DSI(%)
ACC (%)
Table 17: Results for Group 8,
10-5
x 10-61 x 10-3
57
92.28
5 10-65 10-4
55
92.97
8,
8₂
DSI(%) (%)
5 10-61 10-4
45
94.46
1 x 10- x 10-3
63
91.28
5 10-099
45
94.20
10-55 x 10-1
59
92.28
5 1x100=8x100
94.28
1 10 10 100 10
52
93.61
5 xx=0=
40
94.81
1 1 10-55 10-5
47
94.31
5 9x100=61001
41
94.71
1 10 10-5
43
94.49
5 10-5
94.59
1x 10 10 10
43
94.58
5 10 6x10-64x10-1
37
94.79
5 5x10-63x10-
31
95.02
x 10-62x1005
95.05
5 x 10-
23
95.44
Table 19: Results for Group of 8, = x 10-6
Table 20: Results for Group of 8, = x 10-7
6.
5e
DSI(%) ACC (%)
6,
5.
DSI (%) ACC (%)
1 10- 10
47
93.24
1 10-71 10
34
94.47
1 1x 10-60-65. 65 10
42
94.13
1 1 10 10
31
94.94
1 1 101001100-1
31
95.18
1 1 10-7100 10-4
18
95.49
1 1010069x101
29
95.04
1 1 10-79 10-5
19
95.46
1 1x10-68 10-5
30
94.99
1 1X 10-78 10
19
95.50
1x 1x10-67x
29
95.07
1 X 10 10-77 10
18
95.33
1x 1x10-66x10"
26
95.44
1 10-76 10
15
95.52
1x x010655100-
25
95.60
1 1 10-75 10
14
95.43
1 1x10-64x10-6
22
95.47
1 1 10-74
12
95.57
1 1x10-63x100
22
95.28
1 10-73 10-5
95.55
1x 10- 10-5
16
95.54
1 1 10-22x 10-5
95.60
1 x 10-61 10
15
95.49
1 x 10- 10-5
95.60
calculated from the full dataset's node-hours (87.98). Importantly, DSI and node-hour savings follow consistent trends, reinforcing that pruning efficiency directly translates reduction in training costs. Early stopping occurs primarily under aggressive pruning settings, confirming that pruning not only reduces data usage but can also shorten training trajectories.
Table 21: Detailed ImageNet Results with node-hours, DSI, and Early Stopping.
6,
5a
DSI(%) Acc Early Stop Epoch Training
Node-hrs Saved %)
1 X 10
5 10-4
38
78.46
178
46409.41
51.57
41
1 x 10-6
6
x 10-5
33
78.63
49855.24 24
55.39
37
5 10-7
5 10
46
78.55
175
41460.80
46.07
48
x 10-6
5 10
47
78.43
40919.18
45.47
48
5 10-5
10-4
74
76.10
102
21067.8
23.41
73
1 10-5
5 10-4
62
77.85
143
29016.79
32.24
63
1 10
x 10
57
78.07
179
30841.92
34.27
61
1 x 10-5
5 x 10
57
78.07
179
30828.36
34.25
61
5 x 1
10-5
37
71.50
29638.31
32.93
62
Full Dataset
0
78.07
79183.84
87.98
18

---

Published as a conference paper at ICLR 2026
Table 22:
Pattern of 65-8, schedules on pruning (DSI) and accuracy under full-data vs. 10% of training data monitoring in CIFAR-10.
8 5
8e
DSI (Full) (Full) DSI (10% data) Acc (10 % data)
55
x 10
74.52
90.24
0.5655
79.0
10-6
x
10
46.83
94.31
0.2528
81.8
3x100-5
10-5
39.78
94.59
0.1683
82.2
10
5 10
25.06
95.60
0.1244
82.8
10-7
5 5 10
14.26
95.43
0.1030
82.1
Table 23: Patttern of ô5-6, schedules under full-data vs. 10% of training data monitoring in ImageNet- IK.
8
8e
DSI (Full) Ace (Full) Time (Full) DSI (10%
Acc (10%
Time (10%)
10 10-7 7
x 10
38.48
0.7846
46409.41 s
34.71
0,5057
5274.02
5 10
65
10
47.21
0.7843
40919.18 s
43.16
50.4
4793.08
10-6
x
10-5
57.34
0.7807
30828.36
51.78
50.14
4206.86
A.4 FAST AND RELIABLE SELECTION USING SMALL SUBSET TRAINING DATA
The tables 22 and 23 show that oy and & can be tuned quickly and reliably using only a small portion of the training set. Across both CIFAR-10 with ResNet- 18 and ImageNet-1K 1K with ResNet-50, the relative ordering of pruning strength and accuracy remains consistent between full data runs and ten percent subset runs. Larger thresholds such as 6, 10 & 5 x 10-5 consistently yield higher DSI, while smaller thresholds such as 6, 10-7, 6- = 5 10 produce more conservative pruning, precisely matching the full training patterns. These partial runs are extremely lightweight, requiring only few minutes for CIFAR-10 10 and roughly one hour for ImageNet- IK on four A- 100 GPUs. As result, selecting ó is fast, inexpensive, and does not diminish the overall efficiency gains of B-PAS.
Once a good 8 schedule is identified for dataset family, it transfers well to related settings. The values tuned on CIFAR-10 10 transfer directly to CIFAR- 100 and SVHN without further adjustment, preserving the expected DSI and accuracy behavior in low-resolution CNNs. Similarly, values tuned on ImageNet-IK IK with ResNet-50 generalize to ImageNet-like datasets and other CNN variants. Although transformers exhibit slower and noisier activation stabilization, they can also be handled with small subset data, as demonstrated by our experiments with CvT-13 and GPT-2 large. These findings show that 5 hyperparameters can be tuned rapidly on small data slices and reused across models, making B-PAS practical and scalable for new architectures and datasets.
A.5 EXTENDED COMPARISON WITH SOTA
Table 24 summarizes QQn et 2024) the performance of representative data selection and pruning techniques on CIFAR-I0 and CIFAR- Classical core-set and influence-based methods (e.g., Herding, Influence, K-Center) provide moderate gains, while more recent gradient- and uncertainty- based approaches (e.g., GraNd, EL2N, DP, UCB) achieve higher accuracy, especially on CIFAR-100. InfoBatch represents the prior state of the art, reaching 95.6% on CIFAR- 10 and 78.2% on CIFAR- B-PAS matches these best-reported results, maintaining a better DSI despite using fundamentally different criterion based on temporal activation stability rather than loss, gradient, or uncertainty- driven scoring. This highlights that activation variance dynamics provide an equally strong or complementary signal for identifying redundant training data.
In addition, we compare the results from recent large-scale pruning method: Large-scale Dataset Pruning with Dynamic Uncertainty (He et al. 2024). Their method achieves 25% lossless pruning ratio ImageNet- IK. By comparison, B-PAS removes up 57% of the data while matching baseline accuracy and reducing GPU-hours by %. indicating substantially greater pruning capacity and compute savings the same large-scale benchmark.
19

---

Published as a conference paper at ICLR 2026
Table 24: Comparison of pruning methods on CIFAR-IO and CIFAR-100.
Method
CIFAR-I0 Acc. (%) R-100 Acc.
Herding (Weliing
2009)
92.2
73.1
Influence Koh & Liang. 2017)
93.1
74.4
K-Center Sener & Savarese 2017
94.7
74.1
DeepFool (Ducolte & Precioso
2018
95.1
74.2
Forgetting (Toneva et al. 20188
94.7
75.3
EL2N-2 (Paul et al. 2021))
94.4
74.1
EL2N-20 (Paul et al.) 2021
95.3
77.2
Least Confidence (Coleman et al.
2019
95.0
74.2
Margin (Coleman et al. (019
94.9
74.0
CD (Agarwal al. (2020)
95.0
74.2
Craig (Mizzaoleemaa et al.
2020
94,8
74.4
GraNd-4 (Paul et al. 202T
95.3
74.6
Glister (Killamsetty et al.
2021
95.2
74.6
DP (Yang al. (2022
94.9
77.2
e-greedy (Raju et al. 2021
95.2
76.4
UCB (Raju et al.] (222001)
95.3
77.3
InfoBatch (Qun et al.
2024
95.6
78.2
B-PAS (Ours)
95.6
78.2
Table 25: Class distribution before training and after 100 epochs B-PAS. The differences are within :00. %, indicating class imbalance introduced by pruning.
Class (%)
plane
bird
cat deer
dog frog horse ship truck
Initial
10.00 10.00 10.00 10.00 10.00 10.00 10.00 10.00010.000000 10.00 After 100 Epochs 10.05 10.01 9.99 10.04 10.08 9.91 9.92 9.91 10.05 10.04
A.6 NATURE OF PRUNED BATCHES AND CLASS DISTRIBUTION
To examine whether B-PAS disproportionately removes "easy" examples, we conducted dif- ficulty analysis using two standard metrics from the pruning and curriculum learning literature: confidence (higher easier) and misclassifica- tion rate (lower = easier). Confidence is com- puted as the maximum softmax probability for each sample, with higher values indicating that the model already finds the example easy. Mis- classification rate reflects whether the model pre- dicted the sample incorrectly at an early epoch, with lower values corresponding to casier exam- ples. After pruning, these per-sample values are averaged over all samples contained within pruned batches and kept batches, so the reported means reflect the average difficulty of each group. As shown Table 26) pruned batches have lower confidence (0.60 vs. 0.65) and higher misclassification rates (0.40 vs. 0.35) compared to kept batches.
Table 26: Comparison of average difficulty metries between pruned and kept batches on CIFAR-I 10 (ResNet-18)) Pruned batches exhibit lower confi- dence and higher misclassification rates, indicating they are not disproportionately composed "casy" examples.
Metric
Pruned Mean Kept Mean
Confidence
0.60
0.65
Misclassification Rate
0.40
0.35
If B-PAS were pruning easy samples, we would expect the opposite behavior. Instead, both metrics indicate that pruned batches are not casier, and the means are very close, demonstrating that B-PAS does not introduce difficulty bias and prunes based solely activation stability rather than sample easiness.
Table 27: Effect of differ- ent normalization strategies on DSI and accuracy.
Normalizer
DSI (% Ace (%)
Furthermore, assess whether permanent pruning introduces class imbalance, we compared the class distribution of the dataset before training and after 100 epochs of B-PAS. As shown in Table 25 the
Layer
13.2
93.99
Layer Batch
21.9
94.97
None Batch
2.0
90.39
25.06
95.60
20

---

Published as a conference paper at ICLR 2026
proportions remain effectively unchanged, with all deviations within 0.1%. This stability occurs because B-PAS removes entire batches, and each batch contains a naturally mixed set of classes due to the initial random shufiling.
Consequently, pruning at the batch level does not preferentially remove any particular class. Combined with the difficulty analysis, these results demonstrate that B-PAS does not introduce class bias and does not prune based on sample easiness. Instead, pruning is driven solely by the activation-stability criterion, ensuring that the retained dataset remains representative of the original distribution with respect to both class balance and example difficulty.
Table 28:
Comparison of Full Data training and B-PAS on CIFAR-10 (LeNet-5).
Method DSI (%) Acc (%)
Full Data B-PAS 31.07
71.0
70.5
A.7 MORE ABLATION STUDIES
Extended Impact Analysis of Normalization.
Table 27 shows that CIFAR-10 with ResNet-18, BatchNorm yields the smoothest activation trajectories and therefore the high- est data savings (25.06% DSI,, but LayerNorm alone still enables meaningful pruning (13.2% DSI) with strong accuracy (93.99%), In contrast, removing normalization entirely severely destabilizes activation variance, leading to only 2% DSI and a substantial accu- racy drop, which aligns with the known behavior of unnormalized ResNets.
Table 29: Effect of batch
size on pruning effectiveness and accuracy on CIFAR-I0 (ResNet-18).
Batch Size DSI (% Acc (%)
32
20.25
94.86
64
23.26
94.93
The 6-range ablation further confirms this trend in Table 30 Without
128
25.06
95.60
BatchNorm, larger thresholds (e.g., 6, = 5 x 10 5e 10 ²) are
256
26.97
95.10
required to counteract noisier activation dynamies, enabling 19.72%
pruning at the cost of accuracy. More conservative thresholds (e.g..
5, 10-6, 5, = 5 x 10-5) yield minimal pruning (2%) reflecting the higher instability of unnormalized features. When BatchNorm is restored, the same thresholds (10-6 6 to 5 x 10-5) enable substantially higher DSI (25 06%) with top performance (95 60%), demonstrating that stable feature statistics directly expand the "prunable" region detected by B-PAS.
Table 30: Effect of pruming thresholds and BatchNorm on B-PAS.
6, 5 x 10
8,
BatchNorm DSI (%) Accuracy ( (%)
10
without BN
19.72
89,87
10-6
x 10-5
without BN
2.00
90.39
10-6
10-5
with BN
25.06
95.60
Finally, Table 28 shows experiments on LeNet-5, a normalization free architecture. shows that B-PAS is not dependent normalization layers. With tuned ā schedule (5, 10-1, & 10-3), the method prunes 31 % of the data and maintains accuracy within 0.5% of the baseline. This confirms that B-PAS is compatible with models lacking normalization, but the threshold schedule must reflect the architecture's 's intrinsic activation stability regime.
Effect of Batch Size. The batch size ablation in Table 29 shows that DSI increases consistently as batch size grows, reflecting the fact that larger batches yield smoother and more stable activation trajectories across epochs, which allows B-PAS to prune earlier and more aggressively. Accuracy remains nearly unchanged across all settings, indicating that the method is robust to batch-size variation and does not introduce batch-size- dependent bias. Although larger batches improve the magnitude of achievable data savings, the core behavior of activation stabi- lization and accuracy preservation remains consistent, demon- strating that B-PAS functions reliably under standard training configurations.
Table 31:
Comparison of differ- ent activation stability quantifica- tion for pruning.
Quantification DSI (% Ace (%
kurtosis entropy
22.22
95.00
74.89
89.64
max std
13.45
95.41
mean std
25.06
95.60
21

---

Published as a conference paper at ICLR 2026
Different Approaches to Quantify Activation Stability. The ablation in Table 31 compares several alternative activation stability metrics. Kurtosis yields moderate pruning but remains highly sensitive to early-epoch fluctuations, resulting in conservative DSI. Entropy exhibits the opposite behavior: its larger dynamic range causes overly aggressive pruning, leading to substantial accuracy degradation. Using the maximum standard deviation amplifies layer-wise noise and produces unstable pruning behavior with limited savings. In contrast, the proposed mean standard deviation provides stable and well-behaved signal, achieving a balanced trade-off between pruning strength and accuracy. These results indicate that the mean standard deviation is the most reliable activation stability quantifier among the tested alternatives.
A.8
VISUALIZATIONS
Figure 6 reports the number of batches pruned per epoch. Pruning does not occur in early epochs, when activation changes remain high, but becomes increasingly aggressive in later stages as changes stabilize.
Additionally, Figures 7 and 8 show the relationship between threshold growth and the number of remaining batches per epoch (CIFAR-10, ResNet- 18). As training progresses, the threshold value becomes larger, resulting in more batch pruning.
Number Batches Pruned
0
25
50
75
100
125
150
175
200
Epoch
Figure 6: Number of batches pruned per epoch. No pruming occurs in early epochs, while pruning accelerates in later epochs more batches stabilize, leading substantial reductions in training cost.
Number of Batches
Threshold
10-3
200
10.6
50
75
100 Epoch
125
150
175
200
25
50
75
100 Epoch
125
150
175
200
Figure 7: Exponential threshold evolution over epochs.
Figure 8: Remaining batches per epoch.
22

---

Published as a conference paper at ICLR 2026
A.9 LLM USAGE
In this research, large language models (LLMs) have been utilized to assist in verifying grammatical correctness. All contents were developed and verified by the authors.
23


Reviews:
[{"review_id": "fzsNxTP8R8", "reviewer": "Reviewer_7UER", "summary": "This paper proposes Batch Pruning by Activation Stability (B-PAS), a dynamic plug-in framework designed to accelerate deep neural network training by pruning low-utility data batches based on the stability of their internal activations.", "questions": "1) Can authors compared the proposed method with more SOTA pruning methods?\n2) Can the proposed method be applied to other domains like LLMs?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well-written and visually clear.\n2) The paper introduces a pruning method by exploiting activation stability to identify redundant data batches that contribute less to learning.", "weaknesses": "1) The conceptual originality is modest. Leveraging activation stability for pruning has been well studied in existing studies. The contribution appears more incremental than fundamental.\n2) The experimental comparison with existing baselines is insufficient. The evaluation primarily compares with InfoBatch, lacking a lot of related SOTA pruning methods.\n3) The paper proposes that activation stability reflects diminishing learning utility. But no direct evidence is provided to support this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rkBiCGu9IW", "reviewer": "Reviewer_BNgi", "summary": "This paper introduces \"Batch Pruning by Activation Stability\" (B-PAS), a novel, dynamic data pruning strategy to accelerate deep neural network training. The method operates as a lightweight, plug-and-play module that permanently removes batches from the training process if their contribution to learning has diminished. The core mechanism leverages a simple, model-internal signal: the stability of post-ReLU activation feature maps. By tracking the change in the mean standard deviation of activations for each batch across consecutive epochs, B-PAS prunes batches where this change falls below a dynamic threshold. The authors provide comprehensive empirical validation on various models (ResNet, CvT) and datasets (CIFAR, ImageNet), demonstrating up to 57% data savings and a 61% reduction in GPU node-hours on ImageNet without sacrificing accuracy, significantly outperforming prior art. The paper also introduces the Data Savings Index (DSI), a hardware-agnostic metric for data efficiency.", "questions": "On Fixed Batch Composition: Since using fixed batches deviates from standard shuffling, what is the impact on generalization? Does intra-batch shuffling fully mitigate potential overfitting to specific sample groupings?\n\nOn Batch Normalization Dependency: Given the method's strong reliance on Batch Normalization, does the stability signal depend on BN's per-batch standardization? Have you tested its effectiveness with other schemes like LayerNorm or in Normalizer-Free architectures?\n\nOn Transformer Performance: Could you elaborate on why B-PAS is less effective on the CvT architecture? Is the activation stability signal inherently weaker in transformers, and did you investigate tracking activations at different points within the transformer blocks?\n\nOn Hyperparameter Sensitivity: The $(\\delta_s, \\delta_e)$ schedule required extensive tuning. Are there any practical heuristics for setting these hyperparameters for new tasks without performing a costly grid search?\n\nOn the Nature of Pruned Batches: Could you provide a qualitative analysis of the pruned batches? For instance, are they primarily \"easy\" examples, and does the pruning process introduce any significant shift in the class distribution of the remaining data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Novelty and Simplicity: The core idea of using the stability of post-ReLU activation variance as a pruning signal is original, computationally cheap, and mechanistically sound. It provides an elegant, model-internal alternative to more complex methods that rely on loss or gradient statistics.   \n\nGreat Empirical Rigor: The method is thoroughly validated across diverse architectures (ResNet, CvT) and datasets (CIFAR, SVHN, ImageNet-1K). The comprehensive ablation studies and the significant outperformance of a strong baseline (InfoBatch) on ImageNet provide convincing evidence of the method's quality and effectiveness.   \n\nSignificant Practical Impact: The demonstrated efficiency gains are substantial, with up to a 61% reduction in GPU node-hours on ImageNet without accuracy loss. The introduction of the hardware-agnostic Data Savings Index (DSI) is also a valuable contribution to the community for standardizing efficiency reporting.", "weaknesses": "Critical Dependency on Batch Normalization: The method's effectiveness collapses without Batch Normalization, as shown in the ablation study (DSI drops from 25% to 2%). This severely limits its applicability to the growing number of modern architectures, such as many transformers, that do not use BN.   \n\nReduced Effectiveness on Transformers: B-PAS achieves significantly lower data savings on the Convolutional Vision Transformer (CvT) compared to CNNs (14% vs. 47% DSI on ImageNet). This suggests the pruning signal is architecture-dependent and questions the method's generalizability beyond CNNs.   \n\nPracticality Concerns of Hyperparameter Tuning: The method's performance is highly sensitive to the pruning threshold schedule ($\\delta_s, \\delta_e$), requiring extensive grid searches (e.g., 45 settings for CIFAR-10) to find the optimal configuration. This tuning overhead could offset the efficiency gains in practice.   \n\nDeviation from Standard Training Practices: The requirement to fix batch compositions at initialization to enable tracking is a departure from the standard practice of reshuffling the dataset each epoch. This could have unexamined negative effects on model generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qp4bF2sV5E", "reviewer": "Reviewer_FtHA", "summary": "To reduce the training costs of large-scale models, people have considered to reduce the number of training iterations via data pruning. This work proposes a data pruning method which dynamically prunes batches with small changes in activation variances across successive training epochs. Empirical results show that the proposed method can achieve better training efficiency without accuracy loss.", "questions": "The proposed method is only tested on image classification tasks. Can it be generalized to other tasks (e.g., text generation) or models with a larger scale?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work proposes to assess the contribution of batches using changes in their activation variances, which directly utilizes activation outputs from the forward pass, requiring no substantial additional computation, making it simple and efficient.\n\n2. The paper is well written and easy to follow, clearly describing the motivation of the work and the proposed algorithm.", "weaknesses": "1. In line 53, the paper criticizes the existing methods, saying that they often rely on complex heuristics. However, the proposed methods also introduce additional hyperparameters that need to be tuned.\n\n2. In the experiments, only one baseline (i.e., InfoBatch) is compared with the proposed method, which reduces the significance of the empirical results. Other recent data pruning algorithms, such as (He et al, 2024), are not compared.\n\n3. Intuitively, the performance of the proposed method and the selection of the hyperparameters are likely to be greatly influenced by the batch size. However, this important factor was not considered in the ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u5QucQ0YnE", "reviewer": "Reviewer_ZSux", "summary": "This paper introduces B-PAS (Batch Pruning by Activation Stability), a dynamic data pruning method designed to accelerate DNN training. The core idea is to permanently remove training batches whose internal activation statistics have stabilized across epochs, under the assumption that such batches contribute little to further learning. The method is presented as a lightweight, plug-and-play module. Experiments on various datasets and architectures show that B-PAS can reduce training data and GPU time while maintaining baseline accuracy.", "questions": "1.  The notation in Section 2.2 for the second-to-last batch is `B_{n_i}-1`. This could be misinterpreted. Is the intended notation `B_{n_i-1}`? Please clarify.\n\n2.  Given the high sensitivity to the δ schedule, have you explored any adaptive mechanisms for setting the threshold? For example, could δ be dynamically adjusted based on the validation set performance?\n\n3.  The method's core metric is the mean standard deviation of activations. Have you investigated other metrics to quantify activation stability? For instance, using different statistical moments (e.g., kurtosis), information-theoretic measures (e.g., entropy), or different aggregation methods across layers (e.g., maximum vs. mean)? An ablation on this choice would strengthen the justification for the proposed metric.\n\n4.  Regarding the non-monotonic accuracy curves in Figure 3: do you have a hypothesis for why a moderate increase in the pruning threshold can sometimes lead to slightly better generalization performance before it starts to degrade?\n\n5.  The method relies on permanent pruning. Did you analyze the characteristics of the pruned batches? Is there a risk that this process could introduce bias by disproportionately removing samples from certain classes or domains?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  **A Simple and Effective Pruning Heuristic:** The paper proposes using activation stability as a signal for data pruning. This is a straightforward idea that leverages the model's internal state rather than relying on external metrics like loss. The \"plug-and-play\" nature of the module makes it easy to integrate into existing training pipelines.\n\n2.  **Solid Efficiency Gains on Large-Scale Datasets:** The experiments demonstrate clear improvements in training efficiency, particularly on ImageNet. The method achieves a notable reduction in data usage and GPU hours compared to a full-dataset baseline, outperforming the InfoBatch method in terms of data savings while matching its accuracy.", "weaknesses": "1.  **High Sensitivity to Manually-Tuned Hyperparameters:** The method's performance is critically dependent on the `δ` threshold, which requires an extensive and costly grid search to tune (e.g., 45 configurations for CIFAR-10). The paper offers no automated way to set this parameter, which undermines its practical utility and the goal of saving computational resources.\n\n2.  **Lack of Theoretical Insight:** The work is entirely empirical and lacks a theoretical foundation. It does not explain *why* activation stability is a reliable proxy for a batch's training utility. Without this analysis, the method remains a heuristic without a clear understanding of its potential failure modes.\n\n3.  **Limited Baselines:** The comparison is mostly limited to InfoBatch. Other relevant data selection techniques like coreset selection are not discussed or compared against.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)