PruneFuse: Efficient Data Selection via Weight Pruning and Network Fusion
Authors: Humaira Kousar
Hasnain Irshad Bhatti
Jaekyun Moon
Venue: Accepted by TMLR ()

Abstract: Efficient data selection is crucial for enhancing the training efficiency of deep neural networks and minimizing annotation requirements. Traditional methods often face high computational costs, limiting their scalability and practical use. We introduce PruneFuse, a novel strategy that leverages pruned networks for data selection and later fuses them with the original network to optimize training. 
PruneFuse operates in two stages: First, it applies structured pruning to create a smaller pruned network that, due to its structural coherence with the original network, is well-suited for the data selection task. This small network is then trained and selects the most informative samples from the dataset. Second, the trained pruned network is seamlessly fused with the original network. This integration leverages the insights gained during the training of the pruned network to facilitate the learning process of the fused network while leaving room for the network to discover more robust solutions. Extensive experimentation on various datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.

Published in Transactions on Machine Learning Research (03/2026)
PruneFuse: Efficient Data Selection via Weight Pruning and
Network Fusion
Humaira Kousar" School of Electrical Engineering
humadinkkosar32@@aaist..cc
KAIST
Hasmain Irshad Bhatti"
hasnainn@kaist. ac..r
School of Electricol Engineering
KAIST
Jaekyun Moon
jmoon edu
School of Electricol Engineering and Kim Jaechul Graduate School of A1
KAIST
Reviewed on OpenReview: https: // openreviee. net/ forum? BunzenZugy
Abstract
Efficient data selection crucial for enhancing the training efficiency of deep neural networks and minimizing annotation requirements. Traditional methods often face high computational costs, limiting their scalability and practical use. We introduce PruneFuse, novel strategy that leverages pruned networks for data selection and later fuses them with the original network to optimizc training. PruncFuse opcrates in two stages: First, it applics structured pruming to create a smaller pruned network that, due to its structural coherence with the original network, well-suited for the data selection task. This small network then trained and selects the most informative samples from the dataset. Second, the trained pruned network is seamlessly fused with the original network. This integration leverages the insights gained during the training of the pruned network to facilitate the learning process of the fused network while leaving room for the network to discover more robust solutions. Extensive experimentation on various datasets demonstrates that PruneFuse siguificantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.
1 Introduction
Deep learning models have achieved remarkable success across various domains, ranging from image recognition to natural language processing (Ren al. 2015 Long et al. 2015 He et al. 2016). However, the performance of models heavily relies access to large amounts of labeled data. for training (Sun et al. 2017). In practical real-world applications, manually annotating massive datasets can be prohibitively expensive and time-consuming. Data selection techniques such as Active Learning (AL) Gal et al. 2017) offer promising solution to this challenge by iteratively selecting the most informative samples from the unlabeled dataset for annotation, thereby reducing labeling costs while approaching or even surpassing the performance fully supervised training. Even with the rapid scaling of large language models and multimodal foundation models, effective adaptation to downstream tasks continues to demand high-quality, domain-aligned labeled data. A growing body of work demonstrates that principled selection techniques, including AL, outperform simple scaling of in-domain data in both final performance and overall computational efficiency (Xie et al. 2023 Yu
et al. 2024 2025). Traditional AL methods, however, incur severe computational overhead, Each selection
"Equal contribution.

---

Published in Transactions on Machine Learning Research (03/2026)
cycle in AL typically requires extensive training or inference with large model on the entire unlabeled pool. As model and dataset sizes grow, this repeated training becomes a critical scalability bottleneck, especially in resource-constraingg
environments. In this paper, we propose novel strategy for efficient data selection in an AL setting that overcomes the limitations of traditional approaches. Our approach builds on the concept of model pruning, which selectively reduces the complexity of neural networks while preserving their accuracy. By utilizing small pruned networks as reusable data selectors, we climinate the need to train large models, specifically during the data selection phase, thus significantly reducing computational demands. By enabling swift identification of the most informative samples, our method not only enhances the efficiency of AL but also ensures its scalability and cost-effectiveness in resource-liiteed settings. Additionally, we employ these pruned networks to train the final model through a fusion process, effectively harnessing the insights from the trained networks to accelerate convergence and improve generalization.
Main Contribution. To summarize, our key contribution to introduce PruneFuse, an efficient and rapid data sclection technique that loverages pruned networks. This approach mitigates the need for continuous training of large model prior to data selection, which is inherent in conventional active learning methods. By employing prumed networks as data selectors, PruneFuse ensures computationally efficient selection of informative samples, which leads to overall superior gencralization. Furthermore, we propose the novel concept of fusing these pruned networks with the original untrained model, enhancing model initialization and accelerating convergence during training.
We demonstrate the broad applicability of PruneFuse across various network architectures, providing re- searchers and practitioners with flexible tool for efficient data selection in diverse deep learning settings. Extensive experimentation
CIFAR-10, CIFAR-100, Tiny-mmae ImageNet-1K, text datasets (Amazon Review Polarity and Amazon Review Full), as well as cittrf-Distribution (OOD) benchmarks, shows that PruneFuse achieves superior performance to state-of-the-art AL methods while significantly reducing computational costs.
2 Related Work
Data Selection. Recent studies have explored techniques to improve the efficiency of data selection in deep learning. Approaches such as coresct selection (Sener & Savarese 2018), BatchBALD (Kirsch et al.] 2019).
and Deep Bayesian Active Learning (Gal et al. (017) aim to select informative samples using techniques like diversity maximization and Bayesian uncertainty estimation. Parallelly, the domain of active learning
has unveiled strategies, such as uncertainty sampling (Shen et al. 2018 Sener & Savarese 2018 Kirsch
et al. 2019), expected model change-basee approach (Freytag et al.; 2014 Käding et al.) 2016). snapshot
ensembles Jung et al. (2023), and qnery-by-density (Sener & Savarese 2018). These techniques prioritize
samples that can maximize information gain, thereby enhancing model performance with minimal labeling effort. While these methods achieve efficient data selection, they stiill require training large models for the selection process, resulting in significant computational overhead. Other strategies, such as Gradient Matching (Killamsetty et al., 2021a] optimize this selection process by matching the gradients of subset with the training or validation set based on the orthogonal matching algorithm, and (Killamsetty et al.) 2021b) proposes meta-learning based approach for online data selection. SubSelNet (Jain et al. 2023) proposes to approximate a model that can be used to select the subset for various architectures without retraining the target model, hence reducing the overall overhead. However, it involves a pre-training routine, which is very costly and must. be repeated for any change in data or model distribution.
Proxy-based selection methods such as SVP (Coleman et al., 2020) train a smaller proxy model (e.g., ResNet-20) as data selector for a larger target model (e.g., ResNet-56). However, after selecting a subset, the proxy is discarded and the target is trained from scratch on the selected subset. Since the proxy and target architectures are typically different and not directly aligned, there is generally no canonical way to reuse trained proxy weights directly in the target, except indirectly, e.g., via distillation. PruneFuse differs in that the data selector model is obtained by structured pruning of the target model, yielding a channel- alignedd subnetwork of the target. This structural alignment enables woight-aligned fusion, where the weights of the trained pruned model are directly copied into the corresponding coordinates of the dense original model, while with a generic proxy like SVP's, there is no such one-to-one mapping of the proxy to the target model.
2

---

Published in Transactions on Machine Learning Research (03/2026)
Data Selection with Bp
Fusion 0p with Ø
Training of Fused Model 8p
Cyclic Pruning every Taynr
Fiull Labeled
Frieing
Tove
Fusion
Model (9)
(0p)
Trein
Ohiogy
Learning Modda (8f)
(9y)
(1
Otydde Pramue Aenatite)
Unlabeldd
Anmolate
Trained Learning Modile (9))
(updala)
Figure 1: Overview of the PruneFuse method: (1) An untrained neural network is initially pruned to form structured, pruned network 0p- (2) This pruned network Op queries the dataset select prime candidates for annotation, similar to active learning techniques. (3) Op is then trained on these labeled samples to form the trained pruned network 6-. (4) The trained prumed network 6z is fused with the base model 0, resulting in a fused model. (5) The fused model is further trained on a selected subset of the data, incorporating knowledge distillation from 07- At regular intervals Teyne; the fused model utilized dynamically update the pruned model for subsequent data selection.
In essence, fusion enables warm-start for the target model in PruneFuse by leveraging the training compute of the selection process and results in faster convergence and better accuracy than training the dense target from scratch on the same selected subset.
Efficient Deep Learning. Efficient deep learning has gained significant attention in recent years. Methods such as Neural Architecture Scarch (NAS) (Zoph & Le
2016
Wan et al., 2020 network pruning (Han et al, Zhou et al.
2016), quantization (Dong el al. 2020 Jacob et al.
2018
2016 and knowledge distillation
(inton et al. 2015 Yin et al.
2020) have been proposed to reduce model size and computational requirements. Neural network pruning has been extensively investigated as a technique to reduce the complexity of deep
2016). Pruning strategics can be broadly divided into Unstructured Pruning neural networks (Han et al.
(Dong et al.] 2017 Guo et al. 2016 Park et al., 2020) and Structured Pruning (Li et al. 2016 He al., 2017: You al. 2019 Ding et al. 2019) based on the gramularity and regularity of the pruning scheme. Unstructured pruning often yields a superior accuracy-size trade-off, whereas structured pruning offers practical speedup and compression without necessitating specialized hardware. While pruning literature suggests pruning after training (Renda et al., 2020) or during training (Zhu & Gupta 2017 Gale et al. 2019).
recent research explores the viability of pruning at initialization (Lee et al. 2019 Tanaka et al. 2020 rankle
et al. 2021 Wang et al.] 2020). In our work, we leverage the benefits of model pruning at initialization to create a small representative model for efficient data selection, allowing for the rapid identification of informative samples while minimizing computational requirements.
3 Background and Motivation
Efficient data selection is paramount in modern machine learning applications, especially when dealing with deep neural networks. We are given labeled dataset. D {* v;) drawn i.i.d. from an unknown distribution pz over X' x v, and a training procedure A that maps any labeled set s C D to model parameters 0, = A(s). The subset selection problem can be framed as the challenge of selecting a subset of fixed size such that the model trained on s has population risk close to that of the model trained on the full dataset:
arg min sCD, |s| ((zz)))pz (x., y; 0.)] E( Mx, w: 0p))
where 6, A(s) and 0D A(D) denote the parameters obtained by training on the subset and the full dataset D, respectively.
3

---

Published in Transactions on Machine Learning Research (03/2026)
3.1 Subset Selection Framework
Active Learning (AL) is a widely utilized iterative approach tailored for situations with abundant unlabeled data. Given a classification task with C' classes and a large pool of unlabeled samples U, AL revolves around selectively querying the most informative samples from U for labeling. The process commences with an initial set of randomly sampled data s from U, which is subsequently labeled. In subsequent rounds, AL augments the labeled set L by adding newly identified informative samples. This cycle repeats until a predefined number of labeled samples, denoted by b, has been selectod.
3.2 Network Pruning and Its Relevance
Network pruming emerges potent tool to reduce the complexily of neural networks. By eliminating redun- dant parameters, pruning preserves vital network functionalities while streamlining its architecture. Pruming strategics can be broadly categorized into Unstructured Pruning and Structured Pruning. Unstructured Pruning targets the removal of individual weights independent of their location. While it trims down the overall number of parameters, tangible computational gains on conventional hardware often demand very high pruning ratios (Park al., 2017]. On the other hand, Structured Pruning emphasizes the removal of larger constructs like kernels, channels, layers. Its strength lies in proserving locally dense computations, which not only yields a leaner network but also bestows immediate performance improvements (Liu al., 2017p. Given its computational benefits, particularly in expediting evaluations and aligning with hardware optimizations, we opted for Structured Pruning over its counterpart. Importantly, pruned networks maintain the architectural coherence of the original model. This coherence makes them inherently more suitable for tasks such as data selection. Unlike heavily modified or entirely different models that can be used for data selection Coleman et al. (2020); Jain et al. (2023), the pruned model echoes the original structure, particularly advantageous recognizing and prioritizing data samples that resonate with the patterns of the original network. The goal is clear: to develop data selection strategy that conserves computational resources, minimizes memory overhead, and potentially improves model generalization.
4 PruneFuse
In this section, we delineate the PruneFuse methodology. The procedure begins with network pruning at initialization, offering streamlined model for data selection. Upon attaining the desired data subset, the pruned model undergoes fusion process with the original network, leveraging the structural coherence between them. The fused model is subsequently refined through knowledge distillation, enhancing its performance. An overall view of our proposed methodology is illustrated in Fig. 1
Let 6p E O2 denote a pruned model (obtained by structured pruning of the target architecture 0), and let S,(D;pp) 8p) represents an acquisition operator that returns a subset of size b by scoring croring/ranking examples in D using 6p (e.g., least-confidence, entropy, or greedy k-centers). PruneFuse selects 8p = (D 8p), |spl = b, and then trains the target model on 8p, i.e. 0s,- = A(sp). This yields the following proxy-constrained variant of Eq.
* arg min
B(aa) [xx, y: 0,,)] ( 0:0)]]
2,D s.t. |spl = b, 30,- E O such that sp Sz(D;
(2)
where the subset can be defined as Sp = [( y/) D score 8p) 2 T} where 7 is chosen so that |sp| = Equivalently, for score-based acquisition (e.g., reast-confidence or entropy), ) 8p can be chosen by first computing score(æ; 8p) and then selecting the top-b as So(D; 8p) Top; (z,, y1) D by score(æ;; 0,)]. Whereas, for diversity-based acquisition (c.g., greedy k-centers), S₁ denotes the corresponding greedy selection routine applied to embeddings/ saaeeedddinn///aauureeree /fatuures produced by the pruned model.
Eq. 2 formalizes the goal of selecting &p so that training the target model on sp yields population risk close to training on the full datasct D, while performing selection using an efficient pruned model. Algorithm [ precisely describes the PruneFuse methodology, i.c. training the proxy on the current labeled pool, scoring the unlabeled pool, and querying the next batch for annotation. The key insight is that structural coherence
4

---

Published in Transactions on Machine Learning Research (03/2026)
between the pruned and target architectures makes this acquisition effective for the target while greatly reducing selection-time computation.
4.1 Pruning at Initialization
Pruning at initialization has been demonstrated to uncover superior solutions compared to the conventional approach of pruning an already trained network followed by line-tuning (Wang el al., 2020). Specifically, it shows potential in training time reduction and enhanced model generalization. In our methodology, we employ structured pruning due to its benefits, such as maintaining the architectural coherence of the network, enabling more predictable resource savings, and often leading to better-compreesed models in practice.
Consider an untrained neural network, represented as 0. Let each layer l of this network have feature maps channels denoted by é, with & E (1...., L). Channel pruning results in binary masks m' {0, 1]dd for every layer, where de represents the total number of channels in layer f. The pruned subnetwork, Op. retains channels described by cf @ m², where @ symbolizes the element-wise product. The sparsity E [0, 1] of the subnetwork illustrates the proportion of channels that are pruned: = 1= E, m²/ E, d'.
To reduce the model complexity, we employ the channel pruning procedure prune(0, p), which prunes the model f to a desired sparsity We first assign scores 2f E IR to every channel in the network according to their magnitude (using the 1.2 norm). The channels C are represented as (c* (ct,...,cc*) Then we take the magnitude scores Z = (z² ,...,z)) and translate them into masks m such that the cumulative sparsity of the network, in terms of channels, is p. We employ a one-shot channel pruning that scores all the channels simultaneously based on their magnitude and prunes the network to p sparsity. Although previous works suggest re-initializing the network to ensure proper variance (van Amersfoort et al. 20200) the performance gains are marginal; we therefore retain the weights of the pruned network before training.
4.2 Data Selection via Pruned Model
We begin by randomly selecting a small subset of data samples, denoted as s from the unlabeled pool = {ai)sej where [n] = .n.. These samples are then annotated. The pruned model 6p is trained on this labeled subset s0, resulting in the trained pruned model 8%. At cach subsequent round, Of scores U and proposes batch of k points for annotation.
Algorithm 1 PruncFuse
Notation: Labeled dataset L, prune model 0p. fuse model Op.
Input: Unlabeled dataset U, initial labeled data sa, original model 0, pruming ratio p. AL rounds R to achieve budget b, synchronization interval Tayne; and acquisition score.
1: Randomll initialize
2: 0p Prune(0. /structured pruning
We instantiate three widely used criteria for data
3: 8 + Train 8%)
selection, namely Least Confidence (LC) (Settles)
4: S
2012), Entropy (Shannon [948). and Greedy k- centers [Sener& & Savarese 2018).
5: for r = 1 to do
6:
D& + (1scoree) 07) x UJ)
7:
Query labels ye for selected samples De L LU ((De, nx)]
1. LEAST CONFIDENCE based selection lends to- ward samples where the pruned model exhibits the least confidence in its predictions. The confidence
8:
9:
if Tayne = 0 or mod Tayae != 0 then
OP Train L)
score is essentially the highest probability the model
11:
else if mod T Toyne 0 then
assigns to any class label. Thus, the uncertainty
12:
0:- + Fuse(0, 07) and Fine-tune (w/ KD) on L 0r + Prune(07 p) and Fine-tune on L
score for a given sample x; based on LC is defined score(z;; fr 1 max, P(i]x;;;7) 8:). 2. In EN- TROPY based selection, the entropy of the model's predictions is the focal point. Samples with high entropy indicate situations where 0z is ambivalent about the correct label. For each sample in U, the uncortainty based on entropy is computed as scoro( ()) Entropp = E, P(j x,; 07) log P()]x; 07). Subse- quently, we select the top-k samples exhibiting the highest uncertainty scores, proposing them as prime candidates for annotation. 3. The objective of the GREEDY K-CENTERS algorithm is to cherry-pick k centers
13:
14: 0g + Fuse(0, (5)))
15: 03- Fine-tune OF (w/ KD) on L
16: Return L.
5

---

Published in Transactions on Machine Learning Research (03/2026)
from the dataset such that the maximum distance of any sample from its nearest center is minimized. The algorithm proceeds in a greedy manner by selecting the first center arbitrarily and then iteratively selecting the next center as the point that is furthest from the current set of centers. The selection can be mathematically represented as I = arg ma mincecenters d(z, c) where centers is the current set of chosen centers and d(r,e) e) is the distance between point x and center c. Although various metrics can be used to compute this distance, we opt for the Euclidean distance since is widely used in this context.
Remark. These criteria are standard, and our contributions are orthogonal to the choice of acquisition score. Alternative learned scores can be seamlessly integrated into pipeline; see Supplementary Materials 9.5 9.13 for more details.
4.3 Training of Pruned Model
Once we have selected the samples from U, they are annotated to get their respective labels. These freshly labeled samples are assimilated into the labeled dataset L. At the start of each training cycle, a fresh pruned model 0p is generated. Training from scratch in every iteration is vital to prevent the model from developing spurious correlations overfitting to specific samples (Coleman et al 2020 This further ensures that the model learns genuine patterns in the updated labeled dataset without carrying over potential biases from previous rounds. The training process adheres to a typical deep learning paradigm. Given the dataset L with samples (I,, 34), the aim is to minimize the loss function: [(pp, L) | 1.=.p. I4),4, where C; denotes the individual loss for the sample z,. Training unfolds over multiple iterations (or epochs). each iteration, the weights of 8g are updated using backpropagation with an optimization algorithm such as stochastic gradient descent (SGD).
This process is inherently iterative, as in standard Active Learning, After each round of training, new samples are chosen, annotated, and the model is reinitialized and retrained from scratch. This cycle persists until certain stopping criteria, e.g., labeling budget or desired performance, are met. With the incorporation of new labeled samples at every stage, 8 progressively refines its performance, becoming better suited for the subsequent data selection phase.
4.4 Fusion with the Original Model
After achieving the predetermined budget, the next phase is to integrate the insights from the trained pruned model 07 into the untrained original model 6, This step is crucial, as it amalgamates the Icarned knowledge from the pruned model with the expansive architecture of the original model, aiming harness the best of both worlds.
0 FuseeE, 05,)
e
0 trajectory
Rationale for Fusion. Traditional pruning and fine-tuning methods often involve training a large model, pruning it down, and then fine-tuning the smaller model. While this is effective, it does not fully exploit the potential benefits of the larger, untrained model. The primary rcason that the pruning pro- cess might discard useful structures and connections within the original model that were not yet leveraged during initial training. By fusing the trained pruned model with the untrained original model, we aim to create a model that combines the learned knowledge by 0r with the broader, untrained model 0.
(b) Op trajectory
(c) Or with refined trajectory due to fusion
Figure 2: Evolution training trajectories. Con- ceptual illustration of how Pruning 0 to 0, tailors the loss landscape from 2a to 2b allowing 6, to converge on an effective configuration, denoted 07. This model, 0% is later fused with the original 0, which provides a better initialization and leads to an improved trajectory for Or follow, as depicted in 2c
The Fusion Process. Fusion transfers the trained parameters of the pruned selector 87, into the corresponding coordinates of the original (untrained) dense model 0, producing the fused model,
6

---

Published in Transactions on Machine Learning Research (03/2026)
Or F, 0,)
We view 0 as a sequence of layers j j= ,,...,, L. For each layer j, structured pruning at initialization selects a subset of output-channel indices 1 S (I...., Clil and, by architectural coherence, the pruned layer of 0p is isomorphic to the sub-tensor of the dense layer indexed by [L, 11-1] (with 4o C e.g., RGB channels). Instead of coordinate copy, one may spread the trained pruned weights across multiple dense coordinates via a layer-wise dispersion map Dj:
w
D2(W
+ M, +
W U Init о (1 (1-M),,
expanded onta (CL)) .(o)
where M, binary mask that indicates where the dispersed weights land. Further details on the exact implementation of Fusion for Convolutional and Linear layers are provided in Supplementary Materials 9.4
Advantages of Retaining Unaltered Weights. By copying the weights from the trained pruned model 07 into their corresponding locations within the untrained original model 0, and leaving the remaining weights of I yet to be trained, we create a unique blend. The weights from 6% encapsulate the knowledge acquired during training, providing a warm-start (better initialization). Meanwhile, the rest of the unt rained weights in offer an element of randomness. We consider various initialization strategies for the remaining weights after fusion, including retaining initial weights, zero initialization and random re-initialization (as implemented in PruneFuse). We provide this ablation in Table 18 of the Supplementary Materials. Fig. 2 illustrates the conceptual transformation in training trajectories resulting from the fusion process. We empirically validate this in Fig. 4 where this fused initialization in the dense model OF results in better performance than training the original model in isolation.
It is important to highlight that fusion is fundamentally different from Knowledge Distillation (KD), where the information is transferred iteratively through an auxiliary training objective on outputs/loog /logits from a teacher a student model. In contrast, fusion in PruneFuse is one-shot initialization technique that uniquely integrates the training compute spent on the selector into the larget via aligned weight reuse. Based on the strategy discussed until now, we show that PruneFuse already outperforms baseline AL (results provided in the Supplementary Materials 9.7p. However, since our trained pruned model can act as a teacher for the target model, natural to integrate KD PruneFuse. Hence, we use KD as an optional refinement Pruneluuse to further leverage the insights learned during the selection process.
Table 1: Components of PruneFuse.
Component
Required
Pruning at Initiaizzaionn
Fusion (weight trander)
Synchronization interval (I syno)
Optional
Knowledge Distillation
Optional
4.5 Refinement of the Fused Model
During the fine-tuning of the fused model 0F, we use: i) Cross-Entropy Loss, which quantifies the divergence between the predictions of 0r and the actual labels in dataset L, and ii) Distillation Loss (optional), which measures the difference in the softened logits of OF and 0z. These softened logits are derived by tempering logits of 0;, which in our case is the teacher model, with a temperature parameter before applying the softmax function. The composite loss for the finc-tuning phase is formulatod as a weighted average of both losses. The iterative enhancement of 0F is governed by: o((+1) = ( a ago (AL Entropy (0 L.) + (1
Cissstillttoon (o()) 02)). Here, a represents the learning rate, while À functions as coefficient to balance the contributions of the two losses. Incorporating KD in the fine-tuning phase aims to harness the insights of the pruned model 07. By doing so, our objective is to ensure that the fused model OF not only retains the trained weights of the pruned model but also reinforces this knowledge itcratively, optimizing the performance of 0F in subsequent tasks.
4.6 Iterative Pruning of Fused Model
PruneFuse introduces a strategy to dynamically update the pruned model, 0p. from the trained fused model 0% at predefined intervals Taync- In each AL cycle, the pruned model 0p, obtained by pruning randomly
7

---

9.2 of Supplementory Moterials, with probability at Published in Transactions on Machine Learning Research (03/2026)
initialized network, is trained the labeled dataset L and subsequently employed to score the unlabeled data U. At every Tayne cycle, the pruned model Op is obtained by pruning the trained fused model 0p, which is then fine-tuned with L to get 0f and later employed to score the U in the subsequent rounds. By periodically synchronizing the pruned model with the fused model at regular Tayne intervals, PruncFuse effectively balances compulational efficiency with data selection precision. This iterative refinement, process enables the pruned model to leverage the robust architecture of the fused model, allowing dynamically with each cycle and leading to continuous performance improvements. As result, PruneFuse achieves better trade-off between accuracy and efficiency, enhancing the AL process while maintaining computational viability. Table. D summarizes the core components of the PruneFuse pipeline, distinguishing essential elements from optional design choices. Pruning at initialization and weight-aligned fusion form the core of PruneFuse, while synchronization frequency and knowledge distillation are optional components that. trade off computation and performance in practice.
evolve
5 Error Decomposition for PruneFuse
We analyze a standard subset representativeness gap on the finite pool D (akin to coreset-style analyses in Active Learning (Sener & Savarese 2018)). Let or denote the dense target model at cycle t (i.e., the dense model after fusion/fine-tuning when synchronization is performed), let 0g denote the pruned selector used for acquisition cycle t, and let sp S D be the subset it selects, with |D] and |spl = m. We study the discrepancy between the average loss of f Sp and the full dataset D:
1
((a,)e,, (x, ys 0²) De
Σ
1(a, (1; y;;0))
(1 Mi 05)
(3)
m (EL
Theorem 5.1. Under the Assumptions stated in Sec. least 1 7
((a.,)ss UZ,, 00) - EE 5 2L.p.
(4)
The bound decomposes the representativeness error into two terms: an intrinsic acquistion error (6) and an additional proxy-target mismatch term (2Lpt). PruncFuse uses synchronization to promote proxy-target aligument and help control this mismatch, that the selection by the prumed model more closely resembles selection performed with the target model. Assumptions and proof of the bound, along with further discussion and empirical validation, are provided in the Supplementary Materials (Sec. 9.2 and Table 8
6 Experiments
6.1 Experimental Setup
Datasets. The effectiveness of approach is assessed different image classification datasets: CIFAR-I 10 Krizhevsky et al. 2009). 110 (Krizhevsky et al. 2009). TinTtnngoNet-200 (Le & Yang 2015). and ImageNet-IK (Deng et al. 2009). CIFAR-10 is partitioned into 50,000 training and 10,000 test samples, CIFAR-100 contains 100 classes and has 500 training and 100 testing samples per class, whereas TinyImageNet- 200 contains 200 classes with 500 training, 50 validation, and test samples per class. ImageNet-1K consists of 1,000 classes with approximately 1.2 million training images and 50,000 validation images, providing comprehensive benchmark for evaluating large-scale image classification models. We also extend our experiments to text datasets (Amazon Review Polarity and Amazon Review Full) (Zhang & LeCun 2015) Zhang et al. 2015) and to out-of-distribution (OOD) benchmark to assess generalization (results provided in Supplementary Materials 9.5
Implementation Details. We used various model architectures: ResNet (ResNet-50, ResNet-56, ResNet-110. and RosNet-1664), Wide-ResNet, VDCNN, and Vision Transformers (ViT) our experiments. We pruned these architectures using the Torch-Pruning library (Fang et al.) 2023) for different pruning ratios p 0.5, 0.6. 0.7, and 0.8 to get the pruned architectures. We ran these experiments for 181 epochs following the
8

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Performance comparison of Baseline and PruneFuse on CIFAR-10, CIFAR-100, Tiny ImageNet-22 and ImageNet-11.. This table summarizes the top-1 test accuracy of the final model (original in case of AL and Fused in PrunePuse) and compulational cost of the data selector (in terms of FLOPs) for various pruning ratios (p) and labeling bbdgels(b). Params corresponds to the number of parameters of the data selector model. All results use Srass-Conffdence sampling with Tayne= =0. ResNet-56 is utilized for CIFAR-10/100, while ResNet-50 is used for Iing-ImageNet and ImageNet-1K: Results better than the Baseline highlighted in Bold.
Accuracy (%)
Computation (x10'6)
Budget (b)
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
Method
Params
CIFAR-10
Baseline (AL)
0.85 M
80.53+0.20
87.74=0.11
00.85+00111
92.21=0.16
93.00=0.11
0.31
1.76
4.64
8.94
14.66
PruneFuse p-0.
0.21 M
80.92:10.41
88.35 0.333
01.44:0.15
92.77:0.03
93.6510.14
0.08
0.44
1.16
2.24
3.67
PruneFuse p-6
0.14
0035
87.79++0.20
90.9440.13
92.58-0.31
93.08++0.42
0.05
0.28
0.74
1.43
2.35
PrumeFuse p.0.7
80.19+0.45
87.88=+0.05
90.70:10.21
92.44=0.24
93.40+0.11
0.03
0.16
0.42
0.80
1.32
PruneFuse p=0.8
80.1110.28
87.58800.14
90.50:10.08
92.42:10.41
93.3210.14
0.01
0.07
0.36
Method
Params
CIFAR-100
Baseline (AL)
0.86
35.99:10.80
52.99.10.56
59.29:1:0.66
63.6810..3
66.72:10.33
0.31
1.77
4.67
9.00
14.76
PruneFuse 0-5
40.26:00.95
53.90++1.06
60.80+0.44
64.98:10.40
67.87+0.17
0.44
2.26
PruneFuse p=0.6
37.82:10883
52.6520044
60.0810.22
63.70:1..25
66.8910.446
0.28
1.44
2.36
PruneFuse 0.7
0.08 M
36.76:10.63
52.15:10.53
59.7
63.65
66.8410.43
0.03
0.16
0.42
0.81
1.34
PrimeFuse 0.0
0.04 M
18..9++0.20
50.98:40.54
58.53++0.50
62.87
05.85-+0.32
0.01
0.07
0.19
0.37
0.60
Method
Params
Tiny-ImageNNe--2
Boseline (AL.)
23.9
14.86:t0.11
03.62:40522
43.96-+0.22
49.86++0.56
54.65-+0.38
0.50
2.73
7.11
13.64
22.32
PruneFuse p=0.5
6.10 M
18.71:00.21
39.70=0.31
47.4110.20
51.8410.10
35.8911.21
0.13
0.70
1.81
3.48
5.69
PruneFuse 0.6
3.92 M
19.25
08.84
47.02
52.09:0.29
55.2910.25
0.08
0.45
1.16
2.23
3.66
PruneFuse p-0.7 77
2.24
0.99
39.2441.775
46.45
52.02:00655
55.63
0.05
0.26
0.67
1.28
2.09
PruneFuse p-0.8
1.02
18.34:0.93
37.86:1.42
47.15=0.31
61.77:10.40
55.1810.50
0.02
0.12
0.30
0.58
0.95
Method
Params
ImageNNet-1K
Baseline (AL)
25.5 M
52.9710.20
64.52:00.0 46
69.30:10.15
71.9820.11
73.56:00
6.88
37.34
97.28
186.77 70
305.60
PruncFuse p-0
6.91
55.03:1..33
65.12:0.33
6917
72.07:1.28
73.5610.55
1.86
10.10
26.30
50.47
82.62
PruneFuse p006
54.69 0.993
6..15:0.555
69.7440.38
02.48
74.00+0.68
6.71
17.47
33.53
PruneFuse p-0.7
2.74
53.73:10.7
61.4320.66
68.9510044
71.1120.31
73.8410.29
0.74
4.00
10.43
20.01
32.76
PruneFuse p.
1.35 M
53.08:10.22
64.00:10.17
69.00:10.90
71.9910.81
73.6410.52
0.36
1.97
5.14
9.86
16.14
setup in Coleman et al. ImageNet- IK. We used mini batii of 128 for CIFAR-10 10 and CIFAR-100 and 256 for Tiny InnllmaeeNet-2000 and ImageNet-1K. Further details are provided in Supplementary Materials 9.3 Initially, we start by randomly selecting 2% of the data. For the first round, we add 8% from the unlabeled set, then 10% in each subsequent round, until the required budget is met. After each round, we retrain the models from scratch, as described in the methodology. All experiments were carried out independently three times, and the mean is reported. Detailed experiments various model architectures, datasets, labeling budgets, and data selection metrics are provided in Supplementary Materials 9.5 We also provide detailed Complexity Analysis and Error Decomposition for PruncFuse in Supplementary Materials 9.1 and 9.2 respectively.
2020) for CIFAR-10 and CIFAR-100 and for 100 epochs for TinyImageNet-200 and
6.2 Results and Discussions
Main Experiments. Table 2 benchmarks PruneFuse against the standard AL pipeline across different. datasets. PruneFuse attains comparable higher top-1 accuracy while consuming only fraction of the computational resources measured in terms of FLOPs when computed for the whole training duration of the pruned network and the selection process for different label budgets. On CIFAR-10, with a pruned model (p=0.7), PruncFuse achieves similar or superior performance compared to the bascline, while reducing selector cost by more than 90% (e.g. 1.32x 1016 vs. 14.66x FLOPs at b-50%). Similarly, on CIFAR- 100 with (p = 0.5 at b=50%), PruneFuse outperforms baseline's accuracy by 1% while reducing 73% of the computational costs. The advantage bocomes even more pronouncod on the larger benchmarks. For Tiny-Imagel an aggressively pruned selector (p=0.8) lifts accuracy from 54.65% 55.18% while reducing selector computation by 96%. Similarly, ImageNet- with the same pruning ratio, PruneFuse attains
9

---

Published in Transactions on Machine Learning Research (03/2026)
60%
Ac-0%
Compuntio 110
1004
Compustoo Fubha)
(a) p 0.5
(b) p 0.6
(c) p = 0.7
(d) p= 0.8
Figure 3: Accuracy-cost trade-off for PruneFuse. This figure illustrates the total number of FLOPs utilized by PruneFuse for data selection, compared to the baseline Active Learning method, for Tayne=0,1 with labeling budgets 10%, 30%, 50%. The experiments are conducted on the CIFAR-10 dataset using the ResNet-56 architecture. Subfigures (a), (b), (c), and (d) correspond to different pruning ratios of 0.5, 0.6, 0.7, and 0.8, respectively.
Table 3: Comparison with baselines: Final top-1 test accuracy and cumulative selector cost for different label budgets compared with baselines for ResNet-56 on CIFAR-10.
Params
Accuracy (%)
Computation (x10' FLOPs)
Method
Budget (b)
10%
20%
30%
4.%
50%
10%
20%
30%
40%
50%
Baseline (AL)
0.85 M
80.53.10.20 20
87.74.0..15
90.85.10.11
92.2410.16
93.00.10.11
0.31
1.76
4.64
8.94
14.66
SVP PruneFuse (Teyn 00)
0.28 M
80.76-400.0
87.31
90.77440.45
92.59++0.25
92.95-+0.33
0.10
0.56
1.47
2.84
4.68
0.21 M
80.92:00.4
88.35:0.33
91.44:0.15
92.77=0003
98.65+0014
0.08
0.44
1.16
2.24
3.67
PruneFuse (Tym 2
0.21 M
80.90.0.22
88.40
91.5510.63
93.05
93.7410.44
1.17
1.89
5.14
6.58
PruneFuse (Tepce 1)
0.21 M
81.02::0.46
88.52:0.37
91.76:00.08
93.15-+0118
93.78++0.5
0.08
1.17
3.34
6.60
10.94
BALD
0.85 M
80.61#0024
88.11:0.41
91.21+0.56
92.98+0.811
93.36+00.2
0.34
1.81
4.71
9.03
14.77
PruneFuse (Tyye 1) + BALD
0.21 M
80.71 :0.41
88.38
91.44:0.55
98.16:0.21
93.58
0.08
1.18
3.36
6.62
10.97
ALSE
0.85 M
80.73++032
88. 13+0.41
90.9 00.99+0.56
92.58+0.63
93.13++0.1 75
0.41
1.96
4.92
9.29
15.08
PruneFuse (0 + ALSE
0.21 M
80.80=0.51
88.17=0.35
91.13:00.15
03.02:20.55
98.1910.61
0.10
0.19
1.23
2.32
3.70
73.64% (baseline 73.56%) yet requires 95% less computation. These results show that PruneFuse achieves a superior accuracy-cost trade-off compared to a typical AL pipeline.
We further investigated how the synchronization interval Tegne shapes the accuracy-cost trade-off. Fig. plots top-1 accuracy versus cumulative selector FLOPs for four pruning ratios using Tayne = =1 (the pruned sclector is updated from the finetuned fused model after every AL round). In all cases, PruneFuse lies on a superior accuracy cost curve compared with the AL baseline. Even the lightest variant (p=0.8) achieves baseline-level accuracy while spending only one-tenth of the computation, and the configuration with Tayne = delivers the best overall trade-off.
While PruneFuse introduces additional engineering components, such as structured pruning (at initialization synchronization) and fusion during training, their associated costs are explicitly bounded and amortized over multiple active learning rounds (Section 9.1]. In practice, this overhead is outweighed by substantial reductions in cumulative selector cost together with improved accuracy, as evidenced by the results in Table
2, Table 3 and Fig. 3
Comparison with Baselines. Table 3 compares PruneFuse with several prominent active learning baselines, including SVP, ALSE (Jung et al., 2023), and BALD. We evaluate all methods under a canonical protocol: they share the same target architecture (e.g. ResNet-56) for final training and evaluation, the same labeled budgets, and the same target training schedule. Baselines follow the acquisition procedures described in the original works and are adapted only to fit this shared pipeline. The methods therefore differ only in the selector used for acquisition, as defined by cach approach: SVP uses a smaller proxy network, whereas BALD and ALSE use the full larget model as the selector. Specifically, SVP employs ResNet-20 (0.26M parameters) as the closest (parameter-wirs wise)) standard proxy to PruneFuse, which uses a 50% pruned ResNet-56 (0.21M parameters) as its selector. ALSE utilizes 5 snapshots of the data selector model at various training steps for data selection, and BALD uses Bayesian uncertainty with the full target model for selection. Results demonstrate that. PruneFuse consistently outperforms SVP across all label budgets. For example, PruneFuse
10

---

Published in Transactions on Machine Learning Research (03/2026)
70-
70
80
Accuraro
8
only
Putanorrty
-D
only
Fusdonei)) -wo Futtoo-K
Furionkkt
-wh RustonKK
150
Epoch
Epoch
Epoch
(a) p 0.5
(b) = 0.6
(c) p = 0.7
Figure 4: Impact of Model Fusion on PruneFuse performance: This figure compares the accuracy over epochs for differrntt training variants within the PruneFuse framework on CIFAR-10 with ResNet-56. We compare fusion only, knowledge distillation (KD) only, fusion with KD, and training without fusion and KD. Subfigures (a), (b), and (c)
correspond to p = 0.5, 0.6, and 0.7, respectively, for b 30%.
Table 4: Evaluation on Coreset Selection task: Baseline PruneFuse (p = 0.5 and T Toyne 0) with Various
Selection Metrics including Forgetting Events (Toneva et al. 2019). Moderate
Xia et al. 2023 and CSS Zheng
et al. 2023 on CIFAR-10 dataset using ResNet-56 architecture.
Model Params
Selection Metric
Method
Data Selector
Target Model
Budget (6)
Least Couf.
Forgetting Events
Entropy
Moderate
CSS
Baseline
0.85M
0.85M
86. 36.13=0.41
86.50::0.21 21
86.0120.71
86,27=:0.65
87.21
25%
PruneFuse
0.21M
0.85M
86.71:00.4
86.6810.42
87 87.8410..09
87.6310.31
88.85:10.29
Baseline
0.85M
0.85M
91.41+#021
01.28-40.35
93.31 +0.76
90.97-40.55
90.68--0.35
50%
PruneFuse
0.21M
0.85M
92.24+00.4
92.75++0.65
93.40++0.19
91.08+00.99
90.79=0.55
peaks at 93.65% at b=50% compared to SVP with 92.95%, while having significantly lower computational costs (21% lower than SVP and 75% lower than the baseline). PruneFuse with iterative pruming of the fused model shows even better performance, reaching 93. 78% and 73% accuracy with Tayne and at 50% label budget, respectively, offering trade-off between computational efficiency and accuracy. BALD demonstrates competitive results at higher label budgets (e-g., 93.36% at b=50%). However, BALD be scamlessly integrated with PruneFuse. Capitalizing on the strengths of both methods, PruneFuse BALD yields improved performance of 93.58% al 50% label budget while consuming 26% less computation. Similarly, PruneFuse + ALSE also results in better performance while having 4x less computation compared to ALSE.
Table ! compares the performance of PruncFuse on the Corcsct Selection task against various recent works. In this setup, the work first trained on the entire dataset and then identifies a representative subset. of data (coreset) based on the selection metric. The accuracy of the target model trained on that selected coresct is reported. The results show that PruncFuse scamlessly integrates with these advanced selection metrics, achieving competitive superior performance compared to the baselines while being computationally inexpensive. This highlights the versatility of PruneFuse in enhancing existing coreset selection techniques.
Additional Experiments and Abla- tion Studies. Table demonstrates results for Vision Transformers (21M params). Results show that PruneFuse consistently outperforms AL baseline across all label budgets for both CIFAR- 10 and CIFAR-100, despite using small se- lector models. On CIFAR-10, PruneFuse with p=0.5 yields strong gains at low bud- gets (c.g., +5.69 points at b=10%) and maintains improvements even at higher budgets (e.g., +3.05 points at b-50%).
Table 5: Performance Comparison of Baseline and PruneFuse on CIFAR-10 and CIFAR-100 with Vision Transformers (ViT). This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (p) and labeling budgets (b).
CIFAR-10
CIFAR-100
Method
Label Burlget
Label Budget
10%
20%
30%
40%
50%
10%
20%
50%
40%
50%
Buseline (AL)
53.63
65.02
71.00
76.30
80.50
36.59
52.19
58.30
63.8
61.02
PrunePise (p - 0.5)
59.32
71.63
75.61
81.50
83.64
46.84
57.16
62.56
05.8
67.75
PronePuse
57.80
70.45
76.22
82.77
45.00
56.04
61.00
11.09
65.01
67.24
Promeluse (p- 0.7)
56.17
09.25
79.47
82.15
55.36
60.99
64.47
11

---

Published in Transactions on Machine Learning Research (03/2026)
The gains more visible on CIFAR-100 (+10.25 points at b=10%) and offers consistent 3-4 point improve- ments at larger budgets.
We also extend our evaluation beyond vi- experiments
Table 6: Performance comparison of Baseline and PruneFuse on Amazon Review Polarity Dataset and Amazon Review Full Dataset with VDCNN Architecture, This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (p) and labeling budgets (b).
sion tasks. on text classification using VDCNN for Amazon Review Polarity and Amazon Re- view Full. PruneFuse again improves the AL baseline in all pruning ratios and label budgets. In the case of Amazon Re- view Polarity, PruneFuse delivers consis-
Amazzon Review Polarity
Amazon Review Full
Method
Label Rurlget
Tabel Rudget
tent gains (e.g., 94.13 - 94.66% at b=10%, and 95.71 - 97% at b=50). On the other hand, the improvements are greater for the Amazon Full dataset: with y=0.5, PruneFuse improves the baseline by 0.8 1.0 points across all budgets, and even with p=0.8 it continues to match or surpass the unpruned model.
10%
20%
90%
40%
50%
10%
20%
50%
40%
50%
Baseline (AL)
94.13
90.00
90.54
90.73
90.71
58.46
00.03
661,30
02:20
62.41
PruneFuse (o - 0.5)
94.66
96.45
96.71
96.87
95.87
09.45
01.28
62.02
62.66
02.84
PronePuse
94.62
95.38
96.69
96.88
59.28
62.08
62.62
62.81
Promeluse 0.7)
94.47
95.43
96.71
96.83
95.84
09:42
01.05
01.98
02:48
02.86
PruneFuse 0.80
04.33
95.37
95.63
95.70
95.85
59.24
61.05
61.94
62.45
62.77
Fig. 4 provides a joint component ablation that disentangles the effects of weight- fusion and knowledge distillatinn (KD). Specifically,, for each pruning ratio, we train the same dense target model under an identical
training schedule on the data selected by the pruned model, and compare four variants: (i) No fusion / No KD, (ii) KD only, (iii) Fusion w/o KD, and (iv) Fusion + KD. Across pruning ratios, fusion (with without KD) consistently accelerates convergence relative to training from scratch on the same selected subset, indicating that the primary gain arises from reusing the trained selector parameters as a warm-start initialization. KD provides an additional, complementary improvement in several settings, but is not required on the same selected subset. Further implementation
for PruncFuse to outperform details and additional ablations are provided in the Supplementary Materials (Sec. 9.6).
Table 7 demonstrates the impact of differ- ent pruning techniques (c.g., static prun- ing, dynamic pruning) and pruning cri- teria (e.g., L2 norm, GroupNorm Im- portance, LAMP Importance Fang et al.] (203)) ) on the performance of PruneFuse. Static pruning involves pruning the entire network at once at the start of training, whereas dynamic pruning incrementally prunes the network in multiple steps dur- ing training. In our implementation of dynamic pruning, the network is pruned in five steps over the course of 20 epochs. Results indicate that PruneFuse exhibits only minor performance variations (generally within 1-2% across label budgets), demonstrating that PruncFuse is highly flexible to various pruning strategies and criteria while maintaining strong performance in data selection tasks.
Table 7: Effect of Pruning techniques and Pruning criteria on PruneFuse (p 0.5) on CIFAR-10 with ResNet-56.
Label Budget
Method
Pruning Criteria
10%
20%
30%
40%
50%
Baseline (AL)
50.53
87.74
90.85
92:21
93.00
PruneFuse (Tague = 0)
Magnitude Imp.
79.73
87.16
91.08
92.29
93.19
GroupNorm Imp.
50.10
88.25
91.01
92.25
93.74
(Dynamic Priming)
LAMP Imp.
81.51
87.45
90.64
92.41
93.25
PruncFuse (Tagno
Magnitade Imp.
80.92
88.35
91.44
92.77
93.65
GroupNorm Ima.
80.84
88.20
91.19
93.01
93.03
(Static Peuning)
LAMP Iop.
88.37
91.32
93.02
PruneFuse (Typne 1) (Static Pruning)
Magnitude Tmp-
81.23
88.52
91.76
93.15
93.78
GroupNorm Imp.
81.09
88.77
91.77
93.19
93.68
LAMP Imp.
81.86
88.51
92.10
93.02
93.63
7 Conclusion
In this work, we present PruneFuse, a novel strategy that integrates pruning with network fusion to optimize the data selection pipeline for deep learning. PruneFuse leverages a small pruned model for data selection, which then seamlessly fuses with the original model, providing faster training, better generalization, and significantly reducod computational costs. It consistently outperforms existing basclines while offering a scalable, practical, and flexible solution in resource-constrained settings.
12

---

Published in Transactions on Machine Learning Research (03/2026)
8 Acknowledgments
This work was supported by Center for Applied Rescarch in Artificial Intelligence (CARAT) grant funded by DAPA and ADD (UD2300I7TD), the National Research Foundation of Korea(NRF) grant funded by the Korea government (MSIT) (No. RS-2064-00339966 and No. RS-20024-004080003; and by the Institute for Information Communications Technology Promotion (ITTP) grant funded by the Korea government (MSIT) (No. HSS2024-004448822.
(MSII)
References
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. The Eighth International Conference on Learning Representations, 2020.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Databasc. In CVPR09, 2009.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruming very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVE conference on compuler vision and pallern recognition, pp. 4943 4953, 2019.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in neural information processing systems, 30, 2017.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace- weighted quantization of neural networks. Advances in neural information processing systems, 33:18518 18529, 2020.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16091-16101, 2023.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are missing the mark? The Ninth International Conference Learning Representations, 2021.
Alexander Freytag, Erik Rodner, and Joachim Denzler. Selecting influential examples: Active learning with expected model output changes. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, pp. 562-577. Springer, 2014.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1183-1192. JMLR. org, 2017.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. orXie preprint arXiu:1902.0 1ao:1992.0974, 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. The forth International Conference on Learning Represenlations, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Procetdings of the IEEE conference on compuler vision and pollern recognilion, pp. 778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruming for accelerating very deep neural networks. In Proceedings of the IEF. international conference on computer vision, 2017.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXie preprint NNN:1503.0255 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Malthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-rrithmetic-o inference. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2704-2713, 2018.
Eeshaan Jain, Tushar Nandy, Gaurav Aggarwal, Ashish Tendulkar, Rishabh Iyer, and Abir De. Efficient data subset selection to generalize training across models: Transductive and inductive networks. Advances in Neural Information Processing Systems, 36, 2023.
Seohyeon Jung, Sanghyun Kim, and Juho Lee. A simple yet powerful deep active learning with snapshots ensembles. In The Eleventh International Conference on Learning Representations, 2023.
Christoph Käding, Erik Rodner, Alexander Freytag, and Joachim Denzler. Active and contimuous exploration with deep neural networks and expected model outpul changes. arxiv preprint 1eXii:1612.011300 2016.
Krishnateja Killamsetty, Sivasubramauian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464 5474. PMLR, 2021a.
Krishnateja Killamsetty, Durga Subramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: A general- ization based data selection framework for efficient and robust lcarning. AAAI, 2021b.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. The Seventh International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint crXic:1608.0877 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. Tn Proceedings of the IEFE international conference computer vision, pp. 2736 2744, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 3431-3440, 2015.
Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. The Fifth International Conference on Learning Representations, 2017.
Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude- based pruning. The Bighth International Conference on Learning Representations, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fino-tuning in neural network pruning. The Eighth International Conference on Learning Representations, 2020.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Burr Settles, Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1-114, 2012.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technicol journal, 27 -23339-42,, 1948.
Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. In Proceedings of the 2nd Workshop on Representation Learning for NLP. Association for Computational Linguistics, 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning cra. In Proceedings of the IEEE international conference on computer vision, pp. 843 852, 2017.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33: 6377-6389, 2020.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning, arXiv preprint 2019.
Joost van Amersfoort, Milad Alizadeh, Sebastian Farquhar, Nicholas Lane, and Yarin Gal. Single shot structured pruning before training. arXiv preprint arXiv:2007.00388 2020.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Malthew Yu, Tao Xu, Kan Chen, al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognation, 2020.
Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, and Xiaolin Hu. Pruning from scratch. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderale coreset: A universal method of data selection for real-world data-efficient deep learning. In The Fleventh International Conference on Learning Representations, 2023.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201-333 -40022-27,, 2023.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 8715 8724, 2020.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances neural information processing systems, 32, 2019.
Yang Yu, Kai Han, Hang Zhou, Yehui Tang, Kaiqi Huang, Yunhe Wang, and Dacheng Tao. LLM data selection and utilization via dynamic bi-level optimization. In Forty-second International Conference Machine Tearning, 2025.
Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems, 37:108735-108559 2024.
Xiang Zhang and Yann LeCun. Text understanding from scratch. arXio preprint arXiv:1502.011 2015.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification.
A Aduacces in neural information processing systems, 28, 2015.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coresct selection for high pruning rates. aixiv preprint orXiv:2210. 15099, 2023.
Shuchang Zhou, Yuxin Wu, Zekum Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiu:1606. 002116666061000 06160, 2016.
Michael Zhu and Suyog Gupta, To prune, or not to prune: exploring the efficacy of pruning for model compression. arXie preprint 1XXi:1710.0180 2017.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. ar preprint arXiu:161.00157, 2016.
16

---

Published in Transactions on Machine Learning Research (03/2026)
9 Supplementary Materials
This Supplementary Materials provides additional details, analyses, and results to complement the main paper. The content is organized into the following subsections:
Complexity Analysis 9.1 0: A detailed breakdown of the computational complexity of PruneFuse
and its components.
2. Error Analysis for PruneFuse 9.2 proposed framework.
An error analysis outlining proxy-target mismatch in the
3. Implementation Details 9.3
Specific details about the experimental setup, hyperparameters, and configurations used in our experiments.
Details of the Fusion Process 9.4 Specific details about the fusion process of PruncFuse.
5. Performance Comparison with Different Datasets, Selection Metrics, and Architectures
995 : Results demonstrating PruneFusc's adaptability across datasets and architectures.
6. Ablation Study of Fusion 9.6
Analysis of the impact of the fusion process PruncFuse's
performance.
Ablation Study of Knowledge Distillation in PruneFusc 997 : An evaluation of the role of
knowledge distillation in improving performance.
8. Comparison with SVP
9.8 : A comparison highlighting differences and improvements over the
SVP baseline.
9. Ablation Study on the Number of Selected Data Points (k) 9.9 varying & affects PruneFuse's performance.
Investigation of how
10. Impact of Early Stopping on Performance 99.0 Evaluation of the utility of early stopping when integrated with PruncFuse.
11. Performance Comparison Across Architectures and Datasets 9.11 Additional results
comparing PruneFuse's performance on various architectures and datasets.
12. Performance at Lower Pruning Rates 9.12 Rosults demonstrating PruncFuse's effectiveness at lower pruning rates.
13. Comparison with Recent Coreset Selection Techniques 9.13): performance with recent coreset selection methods.
Evaluation of PruneFuse's
14. Effect Various Pruning Strategies and Criteria 9.14 Analysis of different pruning techniques and criteria on PruneFuse's performance.
15. Detailed Runtime Analysis of PruneFuse 9.15 A detailed runtime analysis of PruneFuse compared to baseline methods.
Each section provides additional insights, cvaluations, and experiments further validate and explain the effectiveness of the proposed approach.
17

---

Published in Transactions on Machine Learning Research (03/2026)
9.1
Complexity Analysis
Given P and N represent the total number of parameters in the pruned and dense model, where P& N, the computational costs can be summarized as follows:
Initial Training on Sy:
PruneFuse: O ( x x P log P) one time pruning cost.
Baseline AL: O (1sol N x T)
Data selection round with current labeled pool L:
PruneFuse: 0 x P + O P) selection
Baseline AL: O ( X N x xT) + 0 (1V x N) selection
Training of the final model on the final labeled set L:
PruncFuse: O(L x T) +P(P) one time fusion cost
Baseline AL: xxL)xNII)
Total training complexity:
PruneFuse:
O (1sol x P T) ( x lo Rx x P P)]
+ F Payme [0 (|L x x 7) + 0()) Px ( x log P)]
+ +0 (IL) x N x T) (PP)
Bascline AL: O (1sal N x T) R x O
x T) (IU) N)] ( N
Here T represents the total number of epochs for training round of AL which in is set to 181. U is the whole unlabeled dataset and R represents the total number of AL rounds. Fayne represents the frequency of iterative pruming based on the fused model.
We can see that the major training costs in Active Learning (AL) arise from the repeated use of a. large, dense model, which significantly increases computational expenses, especially across multiple rounds of data selection. By using smaller surrogate (pruned model) for these rounds, as implemented PruneFuse, the training cost and overall computation are reduced substantially. This approach leads more efficient and cost-cffective data selection process, allowing for better resource utilization while maintaining high performance.
9.2 Error Decomposition for PruneFuse
This section provides the proof and additional discussion for Theorem 5.1 in the main paper.
Setup. Let D = (xx, k be the finite pool and let S D be the subset of size m selected at cycle by the pruned selector 0g. Let 0f denote the dense target model at cycle t (after fusion desion/fine-tniing when synchronization is performed). We analyze the representativeness gap
B(en))e Ma,, y, 0²) (1x,,)
(5)
Assumption 9.1. The loss function Mx, 0) is Lipschitz continuous with respect to the model parameters 0,
d.e., there erists L 2 0 such that
[1x,, y: 0; 1(x, y: 02) & L| - 0211.
(6)
This regularity condition is assumed hold locally on the region of parameter space explored during training (e-g., between 0º and 03); and is used only to relate parameter mismalch to loss mismatch.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Assumption 9.2 (ô-coreset under the proxy selector). At cycle t, the subset 8p S D returned by the acquisilion rule using the prumed selector 02 is a å d-core-sel for D with respect to 0g (in terms of average loss), i.e. there exists 2 0 such that
((a,)),, [I),v:;]] ( [xx y: 02)] s 6,
with probability least - 1/ over any rendommess in the selection procedure.
Assumption 9.3 (Soft synchronization: celector-tareet proximity). After each synchronization step, the pruned selector 0g used for data acquisition is derived from the dense target model 0* such that:
10 - 0' Pr.
for some nonnegative quanlity Pt (which may depend on i, the pruning ralio, and the synchronization policy)- Here, we represent the pruned selector 0z in the same parameter space as the dense model 0e by embedding the pruned weights into the dense coordinates and setting the pruned coordinates to zero.
Theorem 9.1. Under Assumptions
9.1
with probability at least 1-1, 1),
E (z.3; -Err))e v.0%) < 2Lpr-
Proof. Fix a cycle Add and subtract expectations under the pruned selector Of and apply the triangle inequality:
E (sy)s, 1x,, (I, y: 8¹) E (( y: (x)))
(awse, (x, y; 0²) E (asse,, (;
+ B(a.))s (x, 0;) - E( v:
+ ( y: [(nn)enl(z,vr v:0")
By Assumption 9.2 the middle term is at most 5 with probability at least By Lipschitz continuity (Assumption 9.1 for every (r,y), v),
(a -1x,00))))) s L 0/11-
((x,v;0))
Taking expectation over (x,y) v) E 8p yields
Binaes, (xx 0*) - (I, s LIl0 -p21l-
Applying the same argument v) E yields
((,,)eD y 0,) - ((x,)EDl(z, y < 1110 0,11
Combining the three bounds, with probability at least 1 - 1-1,,
E( (x, y: 8 E y: y: < 2L| - 0/1l
Finally, Assumption 9.3 implics ||0 051l S Pt: which gives 2Lp:
Interpretation. The bound in Theorem 9.1 decomposes the representativeness error into two intuitive terms. The first term, 8, captures the intrinsic selection error of the acquisition rule itself: even when selection is performed using the target model, uncertainty- or diversity-based active learning methods typically incur a nonzero approximation gap relative to the full dataset. The second term, 2 is an additional penalty arising from performing selection with a proxy model rather than the target. This term quantifies the effect of proxy-target mismatch and bocomes small when the proxy well aligned with the target. (See soction 9.2.1 Fusion and synchronization in PruneFuse are therefore justified as mechanisms to help control Pt. ensuring that proxy-based selection behaves similarly selection performed directly with the target model.
19

---

Published in Transactions on Machine Learning Research (03/2026)
9.2.1 Empirical Proxy-Targer Alignment.
Assumption 9.3 states that the acquisition proxy remains reasonably aligned with the current target. To assess this aligmment empirically, at cycle we compare the syuchronized proxy 0g (obtained by pruning the current dense target 0* and fine-tuning) against fresh pruned proxy O2 trained from scratch at eycle t on the same labeled set.
Let V be a held-out validation set and let Accv (-) denote top-1 validation accuracy. We report the target gaps
ffresh |Accv (0*) Accy Accv(0))) Our results confirm that the synchronized proxy remains closer the target than the fresh proxy:
A (0') Accv Accv(0)))
AT < SAA.h. fresh
(10)
This provides an empirical sanity check supporting the proxy larget proximity premise and is consistent with PruneFuse outperforming other proxy-basee AL, where acquisition relies on a fresh proxy trained from scratch and not kept aligned with the evolving target.
Table 8: Empirical Proxy-Target Alignment This table summarizes performance comparison between the pruned selector and the fused dense model across label budgets, with and without synchronization on CIFAR-10 with ResNet-56 for pruning ratio p 0.5. The rest of experimental setup is held same for fair comparison.
Label Budget (6)
Method
10%
20%
30%
40%
50%
Data Selector (0p)
78.09
84.62
88.08
89.60
90.44
Data Selector (8p)
79.90
87.35
90.08
91.28
91.55
Target Model (9)
81.00
88.54
91.77
93.12
93.77
A Freah
2.61
3.27
3.34
3.24
3.04
Ayyuc
1.10
1.19
1.69
1.84
2.22
Table reports accuracics of the pruned selector and the fused dense model, along with their gap, across label budgets. It can be noted that the synchronization reduces the proxy-largee larget gap, providing empirical evidence that synchronization helps maintain proxy-target alignment as assumed in Assumption 9.3
9.3 Implementation Details
We used ResNet-50, ResNet-56, ResNet-110, ResNet-164, Wide-ResNet, VDCNN, and Vision transformers architectures in our experiments. We pruned these architoctures using the Torch-Pruning library Fang et al. 2023) for different pruning ratios p= 0.5, 0.6, 0.7, and 0.8 to get the pruned architectures. For CIFAR-10 and CIFAR-100, 100, the models were trained for 181 epochs, with an epoch schedule of [1, 90, 45, 45), and corresponding learning rates of [0.01, 0.1, 0.01, 0.001], using a momentum of 0.9 and weight decay of 0.0005. For TinylmageNet-200 and ImageNet-11K, the models were trained over an epoch schedule of [1, 1, 1, 1, 1, 25, 30, 20, 20), with learning rates of [0.0167, 0.0333, 0.05, 0.0667, 0.0833, 0.1, 0.01, 0.001, 0.0001], momentum of 0.9, and weight decay of 0.0001. We use the mini-batch of 128 for CIFAR-10 and CIFAR-100 and 256 for ImyyImageNet-
and ImageNet-IK. We also extend our experiments to text datasets: Amazon Review Polarity and Full (Zhang & LeCun 2015 Zhang et al., 2015). Amazon Review Polarity has 3.6 million reviews split evenly between positive and negative ratings, with an additional 400,000 reviews for testing. Amazon Review Full has million reviews split evenly between the 5 stars with an additional 650,000 reviews for testing. For Amazon Review Polarity and Full, the models were trained over an epoch schedule of [3, 3, 3, 3, 3), with learning rates of [0.01, 0.005, 0.0025, 0.00125, 0.000625], momentum of 0.9, weight decay of 0.0001, and a mini-batch size of 128. For all the experiments SGD is used as an optimizer. We set the knowledge distillation coefficient X to 0.3. We took Active Learning (AL) as a baseline for the proposed technique and initially, we started by randomly selecting 2% of the data. For the first round, we added 8% from the unlabeled set, then 10% in each subsequent round, until reaching the label budget, After cach round, we retrained the models from scratch, as described in the methodology. All experiments are carried out independently 3 times and then the average reported.
20

---

Published in Transactions on Machine Learning Research (03/2026)
9.4 Additional Details of the Fusion Process
Concolutional layers. Let W E I C x xky denote the dense weights of layer j, and W e 1 the trained pruned weights. The weight-aligned fusion copies the trained sub-tensor into the matching coordinates of the dense lensor and keeps all remaining entries at their initial values:
w (- + w
w [4 [4, w 1 1 1,
w w [ f W ;.::
with the same coordinate replacement for biases (if present) and normalization parameters (,,3, 14, o²) on the indices /--
Linear layers. For W6) R2 C x(( aligns on the input channels:
the output dimension (e.g., number of classes) is unchanged; fusion
t W W [. [ < W 7--)-
9.5 Performance Comparison with different Datasets, Selection Metrics, and Architectures
To comprehensively evaluate the effectiveness of PruneFuse, we conducted additional experiments comparing its performance with baseline using other data selection metrics such as Least Confidence, Entropy, and Greedy k-centers. Results are shown Tables 10 and for various architectures and labeling budgets. In all cases, our results demonstrate that PruneFuse mostly outperforms the baseline using these traditional metrics across various datascts and model architectures, highlighting the robustness of PruneFusc in selecting the most informative samples efficiently.
We further performed experiments on ViT, MobileNet for Vision task in Table 13 14 and VDCNN for NLP tasks in Table 15 16 to underscore PruncFusc's consistent efficiency and robust accuracy across different architectures and domains. Moreover, we demonstrated that PruneFuse does not degrade performance on OOD datasots in Table 17 reinforcing PruncFuse's stability.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: Performance Comparison of Bascline and PruneFuse CIFAR-10 and CIFAR-100 with ResNet-56 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (p); labeling budgets (b), and data selection metrics.
Label Budget (b)
Method
Selection Metric
10%
20%
30%
40%
50%
Least Conf
80.53
0.20
87.74 + 0.15
90.85
0.11
92.24 + 0.16
93.00
0.11
Baseline AL
Entropy
80.14 + 0.41
87.63 = 0.10
90.80
0.36
92.51 = 0.34
92.98 = 0.03
Random
78.55 1 0.38
85.26 1 0.21
88.13
1 0.35
89.81 1 0.15
91.20 1 0.05
Greedy k
79.63 + 0.83
86.46 = 0.27
0.20 90.09
91.9 + 0.08
92.80 = 0.08
Least Conf
80.92 0.41
88.35 = 0.33
91.44 0.15
92.77 + 0.03
93.65 0.14
PruneFuse p= 0.5
Entropy
81.08 0.16
88.74 + 0.10
91.33
0.04
92.78 + 0.04
93.48 0.04
Random
80.43 + 0.27
86.28 = U.37
88.75
0.17
90.36 = 0.02
91.42 = 0.12
Greedy k
79.85 +: 0.68
86.96 + 0.38
0.16 90.20
91.82 + 0.14
92.89 + 0.14
Least Conf
80.58 0.33
87.79 0.20
90.94 0.13
92.58 0.31
93.08 0.42
PrumeFuse p 0.6
Entropy
80.96 0.16
87.89 = 0.45
91.22
0,28
92.56 1 0.19
93.19 0.26
Random
79.19 0.57
85.65 = 0.29
88.27 0.18
90.13 + 0.24
91.01 0.28
Greedy k
79.54 + 0.48
86.16 = U.60
89.50 - 0.29
91.35 + 0.06
92.39 = 0.22
Least Conf
80.19 0.45
87.88 0.05
90.70 0.21
92.44 = 0.24
93.40 0.11
PruneFuse p= 0.7
Entropy
79.73 0.87
87.85 1 0.25
90.94 0.29
92.41 1 0.23
93.39 0.20
Random
78.76 0.23
85.50 = 0.11
88.31 0.19
89.94 = 0.24
90.87 0.17
Greedy k
78.93 0.15
85.85 1 0.41
88.96 0.07
90.93 1 0.19
92.23 1 0.08
Least Conf
80.11 + 0.28
87.58 + 0.14
90.50
0.08
92.42 + 0.41
93.32 0.14
PruneFuse 0.8
Entropy
79.83 1.13
87,50 0.54
90.52 0.24
92.24 = 0.13
93.15 0.10
Random
78.77
85.64 = 88.45 0.33
89.88 + 0.14
91.21 0.43
Greedy k
78.23 0.37
85.59 = 0.25
88.60 0.19
90.11 = 0.11
91.31 0.08
CIFAR-10 using ResNet-56 architecture.
Label Budget (b)
Method
Selection Metric
10%
20%
30%
40%
50%
Least Conf
35.99 + 0.80
52.99 + 0.56
59.29
+ 0.46
63.68 + 0.53
66.72 +: 0.33
Baseline AL
Entropy
37.57 + 0.51
52.64 = 0.76
58.87
+ 0.38
63.97 + 0.17
66.78 = 0.27
Raudom
37.06 1 0.64
51.62
1 0.21
58.77
1 0.65
62.05
1 0.02
64.63 0.16
Greedy
38.28 1.11
52.43 0.24
58.96
0.16
63.56 +
66.30 0.31
Least Conf
40.26 +: 0.95
53.90 +: 1.06
60.80 1: 0.44
64.98 1 0.4
67.87 + 0.17
PruneFuse p 0.5
Entropy
38.59 + 1.67
54.01 + 1.17
60.52 0.19
64.83 + 0.27
67.67 + 0.33
Random
39.43 + 0.99
54.60 + 0.64
60.13 + 0.96
63.91 + 0.39
66.02 + 0.3
Greedy k
39.83 2.44
54.35 + 0.41
60.40 1 0.23
64.22 +: 0.25
66.89 + 0.16
Least Conf
37.82 0.83
52.65 - 0.4
60.08 0.22
63.7 0.25
66.89 = 0.46
PruncFuse 0.6
Eutropy
38.01 1 0.79
51.91 1 0.56
50.18 1 0.31
63.53 1 0.25
66.88 1 0.18
Random
38.27 + 0.81
52.85 + 1.22
58.68 + 0.68
62.28 + 0.22
65.2 + 0.48
Greedy k
38.44 +: 0.98
52.86 + 0.74
50.36 + 0.57
63.36 1 0.75
66.12 + 0.38
Least Conf
36.76 0.63
52.15 = 0.53
59.33 0.17
63.65 + 0.36
66.84 = 0.43
PruncFuse p= 0.7
Entropy
36.95 1.03
50.64 1 0.33
58.45
0.36
62.27
1 0.27
65 88 1 0.28
Random
37.30 + 1.24
51.66 + 0.21
58.79 + 0.13
62.67
+ 0.29
65.08 = 0.08
Greedy k
38.88 1 2.18
52.02 1 0.77
58.66 1. 0.19
61.39 1 0.11
65.28 1 0.65
Least Conf
36.49 0.20
50.98
0.54
58.53
0.50
62.87
0.13
65.85 0.32
PruneFuse = 0.8
Entropy
36.02 1.30
51.23 - 0.23
57.44
0.11
62.65 + 0.46
65.76
0.30
Random
37.37 0.85
52.06
+ 0.47
58. 19
0.30
62.
+ 0.45
64.77 : 0.29
Greedy k
37.04 + 0.09
49.84 - 0.49
56.13 + 0.20
60.24 + 0.42
62.92 + 0.44
(b) CIFAR-100 using ResNet-56 architecture.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Table 10: Performance Comparison of Baseline and PruneFuse CIFAR-10 and CIFAR-100 with ResNet-110 architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (p); labeling budgets (b), and data selection metrics.
Label Budget (b)
Method
Selection Metric
10%
20%
30%
40%
50%
Least Conf.
80.74 0.04
87.80 + 0.09
91.50 + 0.09
93.19 + 0.14
93.68 +: 0.17
Baseline AL
Eutropy
79.81 + 0.18
88. 16 = 0.30
91.30 + 0.15
92.83
+0.30
93.47 = 0.31
Random
79.99 1 0.10
85.63 1 0.03
88.07
1 0.31
90.40
1 0.42
91.42 1 0.26
Greedy k
78.69 + 0.58
87.46 =0.20
90.72 + 0.14
92.55
+0.14
93.44 = 0.07
Least Couf.
81.24 0.43
88.70 + 0.15
92.02 0.10
93.32 1: 0.13
94.07 = 0.06
PrumeFuse 0.5
Entropy
81.45 0.39
88.90 + 0.11
92.13 0.15
93.49 + 0.16
94.07 0.05
Random
80.08 + 0.86
86.52 = 0.14
89.48 - 0.16
90.82 + 0.21
91.79 = 0.04
Greedy k
80.40 +: 0.09
87.77 +: 0.13
90.74 :: 0.09
92.48 +: 0.22
93.53 + 0.22
Least Conf.
81.12 0.34
88.33 0.31
91.57 0.03
93.25
0.21
93.90 0.17
PruncFuse p= 0.6
Entropy
80.02 0.41
88.49 1 0.18
91.51 0.14
93.03 1 0.11
93.94 0.12
Random
78.55 0.42
85.94 + 0.34
88.77 0.10
90.66 + 0.20
92.02 0.03
Greedy k
79.44 + 0.28
87.05 + 0.63
90.30 + 0.15
92.15 + 0.12
93.22 = 0.04
Least Couf.
79.93 0.06
88.04 = 0.23
91.51 0.34
92.90
93.82 0.09
PruncFuse p= 0.7
Entropy
80.16 0.27
87.78 1 0.52
91.21
0.13
92.99
93.81 0.12
Random
79.11 0.36
86.14 = 0.44
88.86 - 0.11
90.35 + 0.08
91.35 = 0.24
Greedy k
78.58 1 0.91
86.37 1 0.36
89.70 1 0.33
91.71 1 0.18
92.97 T 0.10
Least Conf.
80.34 + 0.39
88.00 + 0.13
91.22 + 0.07
92.89 + 0.23
93.80 + 0.23
PruneFuse p 0.8
Entropy
79.61 0.35
88.12 + 0.00
90.94
0.13
92.76 + 0.14
93.54 0.24
Random
78.94 0.49
86.20 + 0.10
0.34
90.50
91.42 0.23
Greedy
78.41 0.76
85.90 + 0.73
89.57 0.51
91.38 + 0.32
92.21= 0.22
(a) CIFAR-10 using ResNet-110 architecture.
Label Budget (b)
Method
Selection Metric
10%
20%
30%
40%
50%
Least Couf.
38.61 +0.32
54.47
+0.56
61.46
+0.25
65.96
+00.88
68.91 +: 0.40
Baseline AL
Entropy
38,00 + 0.99
54.71
20.83
60.82
10.15
66.19 + 0.31
68.79 = 0.50
Raudom
37.88 1 1.03
52.84
10.11
59.41
10.34
64.11 1 0.11
67.22
0.36
Greedy
37.41 0.98
53.86
=0.55
61.44
+0.26
65.73 + 0.50
68.17 0.46
Least Couf.
41.42 +:: 0.51
55.91 + 0.36
62.43 +: 0.32
66.95 + 0.20
69.79 + 0.26
PruneFuse p 0.5
Entropy
40.83 + 0.59
56.29 + 0.83
62.62 + 0.45
66.91 + 0.02
69.96 + 0.39
Random
40.36 + 0.74
58.48 + 0.25
61.14 + 0.68
66.03 + 0.42
67 85 = 0.53
Greedy k
41.22 1 0.46
55.70 + 0.54
62.27 1: 0.02
66.20 + 0.14
68.86 + 0.14
Least Conf.
38.52 1.49
54.90 + 0.32
61.50 0.77
66.14 + 0.68
69.03 = 0.24
PruncFuse = 0.6
Eutropy
38.78 1 1.35
53.13 1 0.30
61.42 1 0.14
65.62 1 0.43
68 89 1 0.09
Random
40.24 + 0.90
53.38 + 0.68
59.93 + 0.12
64.70 + 0.15
66.62 + 0.24
Greedy k
39.90 + 1.56
54.91 1 2.23
61.04 + 0.25
64.69 1 0.63
67 60 + U 0.08
Least Couf.
37.83 1.02
53.08 = 0.25
61.41 0.21
65.77 + 0.43
68.03 = 0.14
PruncFuse p= 0.7
Entropy
36.53 0.97
52.97 1 0.76
59.82 1 0.63
64.97 1 0.13
68 64 1 0.54
Random
39.46 + 0.59
52.89 + 0.77
59.92 + 0.55
63.69 + 0.25
66.30 = 0.15
Greedy k
40.44 1 0.13
52.56 1 0.28
50.83 1 0.45
64.50 1 0.29
66.99 T 0.50
Least Conf.
38.33 0.58
52.89
0.49
60.08
0.32
65.12
0.60
68.06 0.56
PruneFuse = 0.8
Eutropy
35.34 0.98
51.88 - 0.74
59.80
0.82
64.58 + 0.43
68.02 = 0.17
Random
38.22 0.39
53.37
+ 0.72
59.84
1: 0.43
64.31
4: 0.33
67. 27 : 0.25
Greedy k
37.72 + 0.70
50.55 - 1.79
57.39 + 0.93
61.79 + 0.53
65.21 - 0.24
(b) CIEAR-100 using ResNet-110 architecture.
23

---

Published in Transactions on Machine Learning Research (03/2026)
Table 11: Performance Comparison of Baseline and PruneFuse CIFAR-10 and CIFAR-100 with ResNet-164 architecture. This table summarizes the test accuracy of final models (original case of AL and Fused in PruneFuse) for various pruning ratios (p); labeling budgets (b), and data selection metrics.
Label Budget (b)
Method
Selection Metric
10%
20%
30%
40%
50%
Least Conf.
81.15 0.52
89.4
0.27
92.72
0.10
94.09 + 0.14
94.63 +: 0.18
Baseline AL
Eutropy
80.99 + 0.44
89.54 = 0.18
92.45 + 0.16
94.06 + 0.05
94.49 = 0.09
Random
80.27 1 0.18
87.00 1 0.08
89.94 1 0.13
91.57 1 0.09
92.78 1 0.04
Greedy k
80.02 + 0.42
88.33 + 0.47
91.76 + 0.24
93.39 + 0.22
94.40 = 0.18
Least Couf.
83.03 0.09
90.30 + 0.06
93.00
0.15
94.41 + 0.08
94.63 = 0.13
PrumeFuse 0.5
Entropy
82.64 0.22
89.88 + 0.27
93.08
0.25
94.32 + 0.12
94.90 0.13
Random
81.52 + 0.54
87.84 = 0.15
90.14 0.08
91.94 + 0.18
92.81 = 0.12
Greedy k
81.70 +: 0.13
88.75 + 0.33
91.92 + 0.07
93.64 +: 0.04
94.22 + 0.09
Least Conf.
82.86 0.38
90.22 0.18
93.05 0.10
94.27
0.06
94.66 0.08
PruncFuse p= 0.6
Entropy
82.23 0.39
90.18 1 0.11
92.11 0.15
0.14
94.66 + 0.14
Random
81.14 0.26
87.51 + 0.26
90.05 0.20
91.82 + 0.22
92.43 0.20
Greedy k
81.11 + 0.10
88.41 + 0.18
91.66 0.18
92.94 + 0.12
94.17 = 0.02
Least Couf.
82.76 0.29
89.89 0.17
0.08
94.10
94.69 0.13
PruncFuse p= 0.7
Entropy
82.59 0.69
0.24
92.77
0.07
94.74 0.02
Random
80.88 0.38
87.54 = 0.26
90.09
0.08
91.57 + 0.26
92.64 0.10
Greedy k
81.68 1 0.40
88.36 1 0.56
91.64 1 0.40
93.02 1 0.42
93.97 T 0.51
Least Conf.
82.66 0.09
89.78 + 0.27
92.64
0.14
94.08 + 0.10
94.69 + 0.17
PruneFuse p 0.8
Entropy
82.01 0.88
89.77 = 0.44
92.65
0.09
94.02 + 0.17
94.60 0.18
Random
0.49
87.43 + 0.44
90.08
0.12
91.40
92.53 0.18
Greedy
79.66 0.60
87.56 + 0.12
90.79 0.07
92.30 + 0.12
93.17 = 0.14
(a) CIFAR-10 using ResNet-161 architecture.
Label Budget (b)
Method
Selection Metric
10%
20%
30%
50%
Least Conf
38.41 + 0.73
51.39 + 0.30
65.53 + 0.31
70.07 + 0.17
73.05 +: 0.11
Baseline AL
Entropy
36.65 + 0.76
57.58 + 0.63
64.98
+ 0.30
69.99
+ 0.17
72.90
0.15
Random
39.31 1.22
57.53
1 0.26
63.84
0.14
67.75
1 0.14
70.79
0.07
Greedy
39.76 0.58
57.40
0.20
65.20
0.31
69.25
72.91
0.29
Least Conf
42.88 1.11
59.31 + 0.70
66.95 1: 0.30
71.45 +: 0.42
74.32 +: 0.58
PruneFuse p 0.5
Entropy
42.99 0.18
59.32 + 1.25
66.83 + 0.29
71.18 + 0.40
74.43 + 0.34
Random
43.72 + 1.05
58.58 = 0.61
64.93 + 0.43
68.75 + 0.57
71.63 = 0.40
Greedy k
43.61 0.91
58.38 + 0.24
66.04 +: 0.21
69.83 +: 0.16
73.10 0.39
Least Conf
41.86 0.70
58.97 + 0.50
66.61
0.39
70.59 + 0.11
73.60 = 0.10
PruncFuse 0.6
Eutropy
42.43 1 0.95
58.74 1 0.80
65.97 1 0.39
70.90 1 0.48
73.70 1 0.09
Random
42.53 + 0.46
58.33 + 0.42
65.00 + 0.26
08.53 + 0.30
71.46 + 0.32
Greedy k
42.71 + 0.91
58.41 + 0.18
65.43 + 0.69
69.37 1 0.14
72.49 = 0.25
Least Conf
42.00 0,20
57.08 = 0.36
66.41 0.30
70.68 + 0.29
73.63 = 0.29
PruncFuse p= 0.7
Entropy
41.01 1.66
57.45 1 0.50
65.99 1 0.10
70.07 1 0.54
73.45 1 0.04
Random
42,76 + 1.00
57.31 + 0.07
64.12 + 0.57
68.07 + 0.24
70.58 + 01.25
Greedy k
42.42 1. 0.32
57.58 1 0.52
65.18 1. 0.51
68.55 1 0.10
71.89 T 0.16
Least Conf
41.19 1.07
57.98
0.70
65.22
0.44
70.38 0.22
73.17 0.26
PruneFuse = 0.8
Entropy
39.78
1.16
57.30
+ 0.41
65.19
0.63
69.40 - 0.34
72.82 - 0.03
Random
42.08
1.55
57.23
+ 0.47
64.05
0.40
67.85
1 0.19
70.62 + 0.06
Greedy k
42,20
1.21
57.42 - 0.50
64.53 - 0.21
68.01 - 0.40
71.29 = 0.14
(b)
CIEAR-100 using ResNet-164 architecture.
24

---

Published in Transactions on Machine Learning Research (03/2026)
Table 12: Performance Comparison of Baseline and PruneFuse on Tiny ImageNet-200 with ResNet-50 architecture, including test accuracy and corresponding standard deviations. This table summarizes the test accuracy of final models (original in case of AL and Fused in Prunefuse) for various pruming ratios (p) and labeling budgels (b).
Label Budget (b)
Method
10%
20%
30%
40%
50%
Baseline (AI.)
14.86 + 0.11
33.62 + 0.52
43.96
0.22
49.86
0.56
34.65
0.38
PrumePuse (p 0.5)
18:71 + 0.21
39.70 1 0.31
47.41 = 0.20
51.84 1: 0.10
55.89
1.21
PruneFuse 0.6)
19.25 + 0.72
38.84 1 0.70
47.02
0.30
52.09
0.29
55.29
0.28
PruneFuse (p 0.7)
18.32 = 0.95
39.24 + 0.75
46.45
0.58
52.02
0.65
55.63
0.55
PrumeFuse (p 0.8)
18.34 = 0.93
37.86 + 0.42
47.15 = 0.31
51.77 + 0.40
55.18 + 0.50
Table 13: Performance Comparison of Baseline and PruneFuse CTFAR-10 with MobileNett Architecture This table summarizes the test accuracy of final models (original in case of AT and Fused in PruneFuse) for varions pruning ratios (p) and labeling budgets (b).
Label Budget (b)
Method
10%
20%
30%
40%
50%
Baseline (AL)
81.63
89.47
91.25
91.54
91.85
PruncFuse 0.5)
84.49
90.07
92.63
93.49
93.55
PruneFuse (p 0.6)
84.16
90.22
92.56
98.34
93.43
PruncFuse (p 0.7)
84.10
90.21
92.46
93.29
93.22
Table 14: Performance Comparison of Baseline and PruneFuse CIFAR-10 with Vision Transformers (VIT). This table summarizes the test accuracy of final models (original in case AL and Fused in PruneFuse) for various pruning ratios (p) and labeling budgets (b).
Label Budget (6)
Method
10%
20%
30%
40%
50%
Baseline (AL)
53.63
65.02
71.09
76.86
80.59
PrumeFuse = 0.5)
59.32
75.61
PrumeFuse (p = 0.6]
57.80
70.45
75.22
80.22
82.77
PruneFuse 0.7)
56.17
69.25
73.87
79.47
82.15
Table 15: Performance Comparison of Baseline and PruneFuse on Amazon Review Polarity Dataset with VDCNN Architecture. This table summarizes the test accuracy of final models (original in case of AL and Fused in PruneFuse) for various pruning ratios (p) and labeling budgets (b).
Label Budget (6)
Method
10%
20%
30%
40%
50%
Baseline (AL)
94.13
95.06
95.54
95.73
95.71
PruneFuse (p = 0.5)
94.66
95.45
95.71
95.87
95.87
PruneFuse 0.6)
94.62
95.38
95.69
95.82
95.88
PruneFuse
94.47
95.43
95.71
95.83
95.84
PrumeFuse = 0.8)
94.33
95.63
9.6 Ablation Study of Fusion
The fusion process is a critical component of the PruneFuse methodology, designed to integrate the knowledge gained by the pruned model into the original network. To isolate the effect of fusion on optimization dynamics, evaluate fusion vs. no fusson and also against knowledge distillation under controlled, matched training
25

---

Published in Transactions on Machine Learning Research (03/2026)
Table 16: Performance Comparison of Baseline and PruneFuse on Amazon Review Full Dataset with VDCNN Architecture. This table summarizes the lest accuracy of final models (original in of AL and Fused in Prunelusee for various pruning ratios (p) and labeling budgets (b).
Label Budget (6)
Method
10%
20%
30%
40%
50%
Baseline (AL)
58.46
60.65
61.50
62.20
62.43
PruneFuse (p = 0.5)
59.45
61.28
62.02
62.66
62.84
PruneFuse 0.6)
59.28
61.14
62.08
62.62
62.81
PrumeFuse = 0.7)
59.42
61.05
61.98
62.48
62.85
PruneFuse (p = 0.8)
59.24
61.05
61.94
62.45
62.77
Table 17: Results of CIFAR-10 (in-distribution, ID) and CIFAR-10-C (OOD corruptions) using a
ResNet-56 backbone.
Label Budget (b)
Method
10%
20%
30%
40%
50%
Baseline (AL) (ID)
48.67
58.95
65.23
72.30
73.23
PrumeFuse (p = 0.5) (ID)
51.20
64.68
67.52
73.28
77.71
Baseline (AL) (OOD)
42.95
48.56
53.62
57.27
58.32
PruncFuse (p = 0.5) (OOD)
44.58
52.44
54.60
58.36
63.37
Table 18: Initialization schemes for Fused Model. Performance comparison of initializing the remaining weights of the Fused model after fusion with pruned model via 1) retaining weights from first initialization during pruned model initialization, 2) zero weights and 3) random re-initialization (PruneFuse) on ResNet-56 with CIFAR-10 dataset.
Label Budget (b)
PruneFuse Method
10%
20%
30%
40%
50%
Zero initialization
79.96
87.01
90.32
91.80
92.52
Retained Initial Weights
80.61
88.22
91.19
92.75
93.66
Random ro- re-Initialized Weights
80.92
88.35
91.44
92.77
93.65
protocol. For each pruming ratio and labeling budget b, we first run PruncFuse to obtain a selected labeled subset L. We then train the same dense larget architecture (ResNet-56) on L under two conditions: (i) No fusion / No KD: randomly initialized target trained with cross-entropy, (ii) KD only: no fusion; randomly initialized target trained with KD from 07, (iit) Fusion w/o KD: target initialized via Op = Fuse(0, 0p) and trained with cross-entropy, and (iv) Fusion + KD: fused initialization trained with cross-entropy and KD (full PruneFuse). Importantly, the architecture, subset size, optimizer, learning-rate schedule, batch size, and total number of epochs are identical across all the conditions. In all fusion ablations, we train for 180 epochs with learning rate 0.01. Therefore, per-epoch compute is matched and using epoch as the x-axis provides a fair comparison of convergence behavior.
Figs. and 7 show the resulting lcarning curves for different pruning ratios. In the reported scttings (budgels b E (10% 30%, 50%) and pruning ratios p E (0.5, 0.6, 0.73), fusion (with or without KD) consistently converges faster and reaches higher final accuracy than training from scratch the same selected subset L. These results provide direct empirical evidence that fusion improves both convergence and final performance under a matched training budget.
Additionally, in Table 18 we study how the initialization the remaining (previously untrained) weights after fusion affects performance. We compare three schemes: (i) retaining the original denso-model initialization for the non-copied weights, (ii) zero initialization, and (iii) random re-initialization. Results show that while fusion provides the primary performance gains, zeroing the remaining weights is suboptimal, and simple
26

---

Published in Transactions on Machine Learning Research (03/2026)
randomized initializations, either retained ro-initialized, sufficient to support effective post-fusion training.
60
70
70
Accurstoy
Accuraty
-00
88
50
18
16
40
Rusion
KD only
only
KD
Rusion-KD
Ruslon-KD
30
RusienKDD
wis FuskinkK
wis KusionKK
we FusionrKD
100
100
30
100
160
50
100
150
Epoch
Epoch
Epoch
(a) 0.5, b 10%
(b) p 0.5, b 30%
(c) 0.5, 50%
Figure 5: Ablation Study of Fusion on PruneFuse (p 0.5). Experiments are performed ResNet-56 architecture with CIFAR-10.
90
90
80
Actorric P%)
13
18
Fasion only
Fusion only
Fusion only
only
KD only
only
RuslonHHD
30
30
10
20
Epoch
Epoch
Epoch
(a) p = 0.G, 10%
(b) p 0.6, 30%
(c) p= 0.6, 50%
Figure Ablation Study of Fusion on PruneFuse (p = 0.6). Experiments are performed on ResNet-56 architecture with CIFAR-10.
00
70
70
Anaiaacy
88
8
86
only
wis PurloneK
Fusione
20
RusionKO
wis Ruslon-DD
FusionKO
wit Rusion-KD
10
100
150
50
100
150
50
100
150
Epoch
Epoch
Epoch
p 0.7, 10%
(b) p 0.7, 30%
(c) p 0.7, 50%
Figure 7: Ablation Study of Fusion PruneFuse (p = 0.7). Experiments performed ResNet-56 architecture with CIFAR-10.
27

---

Published in Transactions on Machine Learning Research (03/2026)
9.7 Ablation Study of Knowledge Distillation in PruneFuse
Table 19 demonstrates the effect of Knowledge Distillation on the PruneFuse technique relative to the baseline Active Learning (AL) method across various experimental configurations and label budgets on CIFAR-10 and CIFAR-100 datasets, using different ResNet architectures. The results indicate that PruneFuse consistently outperforms the baseline method, both with and without incorporating Knowledge Distillation (KD) from a trained pruned model. This superior performance is attributed to the innovative fusion strategy inherent to PruneFuse, where the original model is initialized using weights from previously trained pruned model. The proposed approach gives the fused model an optimized starting point, enhancing its ability to learn more efficiently and generalize better. The impact of this strategy is evident across different label budgets and architectures, demonstrating its effectiveness and robustness.
9.8 Comparison with SVP
Table 21
delineates a performance comparison of PruneFuse with SVP techniques, across varions labeling budgets b for the efficient training of Target Model (ResNet-56). SVP employs a ResNet-220 as its data selector, with a model size of 0.26 M. In contrast, PruneFuse uses a 50% pruned ResNet-56, reducing its data selector size to 0.21 M. Performance metrics show that as the label budget increases from 10% to 50%, the PruncFuse consistently outperforms SVP across all label budgets. Specifically on the target model, PruneFuse initiates at an accuracy of 82.68% with a 10% label budget and peaks at 93.69 93.69% accuracy at a 50% budget, whereas SVP achieves 80.76% at 10% label budget and achieves 92.95% accuracy at 50%. Notably, while the data selector of PruncFuse achieves lower accuracy of 90.31% at b 50% compared to SVP's 91.61%, the target model utilizing PruneFuse-sclected data attains a superior accuracy of 93.69%, relative to 92.95% for the SVP-selected data. This disparity underscores the distinct operational focus of the data selectors: PruneFuse's selector is optimized for enhancing the target model's performance, rather than its own accuracy. Fig. a) and (b) show that target models ResNet-14 and ResNet-20, when trained with data selectors of Prune achieve significantly higher accuracy while using significantly less number of parameters compared to SVP. These results indicate that the proposed approach does not require an additional architecture for designing the data selector; it solely needs the target model (e.g. ResNet-144. In contrast, SVP necessitates both the target model (ResNet-14) and a smaller model (ResNet-8) that functions data selector.
Table 20 demonstrates the performance comparison of PruneFuse and SVP for small model architecture ResNet-20 on CIFAR-10. SVP achieves 91.88% performance accuracy by utilizing the data selector having 0.074 M parameters whereas PruneFuse outperforms SVP by achieving 92.29% accuracy with a data selector of 0.066 M parameters.
Accumny vs Model (Pannerenn)
dccomey ve Model Sime (aannetes))
91.5
92.4
$2.3
Accunsy 1S
$1.3
Accumcyyy 166
52.2
$1.2
A2011-00
21.1
4::4ppmd
52
$1.9
$0.9
93.6
50.8
Numberof of Parameters of Data elector
X10
Number of Parameters Data Selector
X10*
(a) Target Model ResNet-14
(b) Target Model
ResNet-20
Figure 8: Comparison of PruneFuse with SVP. Scatter plot shows final accuracy on target model against the
model size for different ResNet models CIFAR-10, b = 50%. (a) shows ResNet-14 (with 0.5 and p == 0.6) and ResNet-8 models are used as data selectors for PruneFuse and SVP, respectively. While (b), PruneFuse utilizes 2esNet20 (L.e. p = 0.5 and = 0.6) and SVP utilizes ResNet-8 models.
28

---

Published in Transactions on Machine Learning Research (03/2026)
Table 19: Ablation Study Knowledge Distillation on PruneFuse presented in a and datasets.
b and C
with different architectures
Label Budget (6)
Method
Selection Metric
20%
30%
40%
50%
Least Couf
80.53
87.74
90.85
92.24
93.00
Baseline
Eutropy
80.14
90.80
92.51
92.98
Random
85.26
88.13
89.81
91.20
Greedy
79.63
86. 16
90.09
91.90
92.80
PrumeFuse p 0.5 (without KD)
Conf
88.71
91.24
92.68
93.46
Entropy
88.08
90.98
92.74
93.43
Ramdom
80.11
85.78
88.81
90,20
91.10
Greedy k
80.07
86.70
89.93
91.72
92.67
Least Conf
80.92
88.35
91.44
92.77
93.65
PruneFuse
Entropy
81.08
88.74
91.33
92.78
93.48
p
Random
80.43
86.28
88.75
90.36
91.42
(with KD)
Greedy k
79.85
86.96
90.20
91.82
92.89
(a) CIFAR-10 using ResNet-56 architecture.
Label Budget
Method
Selection Metrie
10%
20%
30%
40%
50%
Least Conf
92.72
94.09
94.63
Baseline
Entropy
80.99
89.54
92.45
91.06
91.49
Random
80.27
87.00
89.94
91.57
92.78
Greedy k
80.02
88.33
91.76
93.39
94.40
Least Conf
83.82
90.26
93.15
94.34
94.90
PrumeFuse p=
Entropy
82.72
90.42
93.18
94.68
95.00
Random
81.94
88.04
90.37
91.93
92.67
(without KD)
Greedy k
81.99
89.04
92.14
93.40
94.44
Least Couf.
83.03
90.30
93.00
94.41
94.63
PruncFuse
Entropy
82.64
89.88
93.08
94.32
91.90
p= (with KD)
Random
81.52
87.84
14
91.94
92.81
Greedy
81.70
88.75
91.92
93.64
94.22
(b) CIFAR-10 using ResNet-164 architecture.
Label Budget
Method
Selection Metrie
10%
20%
30%
40%
50%
Least Couf
59.29
63.68
66.72
Baseline AT.
Entropy
37.57
52.6 6.
58.87
63.97
66.78
Random
37.06
51.62
58.77
62.05
64.63
Greedy k
38.28
52.43
58.96
63.56
06.30
Least Conf
39.27
54.25
60.6
64.17
67.49
PruncFuse p= (without KD)
Entropy
37.43
52.57
60.57
64.44
67.31
Raudom
40.07
52.83
59.93
63.06
65.41
Greedy k
39.25
52.43
59.94
63.94
66.56
PruncFuse
Least Conf
40.26
53.90
60.80
64.98
67.87
Entropy
38.59
54.01
60.52
64.83
67.67
(with KD)
p=
Random
39.43
54.60
60.13
63.91
66.02
Greedy
39.83
54.35
60.40
64.22
66.89
(c) CIFAR-100 using ResNet-56 architecture.
9.9 Ablation Study on the Number of Selected Data Points (k)
Table 22 and 23 present ablation studics analyzing the effect of varying i on the performance of PruncFuse with T Tayne- 0 and Tayne = 1, respectively, on CIFAR-10 using the ResNet-56 architect and least coufidence as the selection metric. The results demonstrate that the choice & significantly impacts the quality of data selection and the final performance of the model. As incrcases, the selected subset quality diminishes as can be seen by comparing performance of the target network in both tables. This study highlights the importance of tuning & to achieve optimal trade-off between computational efficiency and model accuracy.
29

---

Published in Transactions on Machine Learning Research (03/2026)
Table 20: Comparison of SVP and PruncFuse on Small Models.
Label Budget (b)
Techniques
Model
Architecture
Params (Million)
10%
20%
30%
40%
50%
Data Selector
RosNet-8
0.074
77.85
83.35
85.43
86.83
86.90
SVP
Target
ResNet-20
0.26
80.18
86.31
89.22
90.75
91.88
Data Selector
ResNet-20 (p 0.5)
0.066
76.58
83.41
85.83
87.07
PruneFuse
Target
ResNet-20
0.26
80.25
87.57
90.20
91.70
92.29
Table 21: Comparison with SVP.
Label Budget (b)
Method
Model
Architecture
Params (Million)
10%
20%
30%
10%
50%
Data Selector
RosNet-20
0.26
81.07
91.08
SVP
Tarpot
ResNet-56
0.85
80.76
87.31
90.77
92.59
92.95
PruncFuse
Data Selector
ResNet-56 (p 0.5)
0.21
78.62
84.92
88.17
89.93
90.31
Target
ResNet-56
0.86
82.68
88.97
91.63
93.24
93.69
Table 22: Ablation study of on CIEAR-10 using RosNet-56 architecture and least confidence as sclection metric.
(4) l 7.5K.
(b) h 5K.
Label Budget
Label Budget
Method
Method
15%
30%
45%
60%
75%
10%
20%
30%
40%
50%
Buseline (AZ)
84.63
90.59
92.77
93.12
93.94
Baseline (AL)
81.53
87.74
90.85
92.24
93.00
PruneFuse (o 0.5)
93.72
PrumeFuse (p = 0.5)
80.92
92.77
Table 23:
Ablation study of & on Cifar- 10 using ResNet-56 with p 0.5 and Tayee
Selection
Label Budget (b)
Selection
Label Budget (b)
Method
(k)
20%
40%
60%
Size (lk)
20%
40%
60%
Baseline (AL)
5,000
88.51
93.04
93.83
10,000
86.92
92.51
93.81
PruneFuse
5,000
88.52
93.15
93.90
10,000
87.19
93.11
91.01
9.10 Impact of Early Stopping Performance
Table 24 explores the effect of utilizing early stopping strategy alongside PruneFuse (p 0.5) CIFAR-10 10 with the ResNet-56 architecture. The results indicate that carly stopping not only reduces training time of the fused model but also maintains comparable performance to fully trained models. This highlights the compatibility of PruneFuse with training efficiency techniques such as early stopping and showcases how the expedited convergence enabled by the fusion process further enhances its practicality, particularly in resource-onnsrraneed environments.
Table 24: Performance Comparison when Early Stopping strategy is utilized alongside PruneFuse (p 0.5).
Experiments are performed with Resnet-56 on CIFAR-10.
Label Budget (6)
Method
Epochs
10%
20%
30%
40%
50%
80.9244./M
88. 3..55.03322
91.41=00.188
92.77=0.026
93.66:t0.. 141
Least Conf.
110
60.51++0.375
87.64+0.222
90.70+0.052
12.11+0.1554
93:00++0.005
181
81.08220.15 135
88. 71:27.1:11000 103
01:33:000115
92,78:000055
98.1820:012
Entropy
110
80.51:11.000
17.46:::416
90.17:00.116
12.2200108
82.8810.261
60.43
0.366
88.75-10.17
90.36
91.42.0.11
Randam
110
79.29110.355
84.9920.115
07.86:20.323
89.00110.000
10..5.10.012
181
79.85-+0.6676
86.96 -0.385
90.20 0..10
01.82-0..18 136
92.89 +0.80+-0.144 144
Greedy
110
79.36-+-.274
86. 36------446
80. 90.67
19+00-203302
91.91+0.021
30

---

Published in Transactions on Machine Learning Research (03/2026)
9.11 Performance Comparison Across Architectures and Datasets
In Table 25 we present the performance comparison of Baseline and PruneFuse across various architectures and datasets. These results demonstrate the adaptability of PruneFuse different network architectures, including ResNet-18, ResNet-50, and Wide-ResNet (W-28-10), well as datasets such as CIFAR-10, CIFAR- 100, and ImageNet. The experiments confirm that PruneFuse consistently improves performance over the baseline, highlighting its generalizability and robustness across diverse scenarios.
Table 25: Performance Comparison of Baseline and PruneFuse presented in b and with different archilectures and dalasets.
Label Budget (0)
Method
10%
20%
30%
40%
50%
Bascline (AG)
83.12
90.07
92.71
94.07
94.81
PrumeFuse 1 0.5)
83.29
90.56
93.17
94.56
95.08
(a) ResNet-18 architecture CIFAR-10.
Label Budget (6)
Method
10%
20%
30%
40%
50%
Baseline (AZ)
81.71
91.48
91.17
95.21
96.75
PrunePuse (p= 0.5)
85.65
92.27
94.65
95.73
96.24
(b) Wede-ResNet architecture CIFAR-10.
Label Budget (6)
Method
10%
20%
30%
40%
50%
Baseline (AL)
52.97
64.52
69.30
71.98
73.56
PruneFuse (p=
55.03
65.12
72.07
73.86
(c) ResNet-50 architecture on ImageNet-Ike
9.12 Performance at Lower Pruning Rates
Table 26 provides a performance comparison of Bascline and PruneFusc with a lower pruning rate pf 0.4 CIFAR-10 and CIFAR-100 using the ResNet-56 architecture. Least Confidence and Entropy were used selection metrics for these experiments. The results show that even at lower pruning rate, PruneFuse effectively selects high-quality data subsets, maintaining strong performance in both datascts. These findings validate the method's effectiveness across different pruning rates.
9.13
Comparison with Recent Coreset Selection Techniques
Table 27 compares the performance of Baseline (Coreset Selection) and PruneFuse (p= 0.5) using various recent selection metrics, including Forgetting Events (Toneva et al., 2019), Moderate (Xia al., 2022), and CSS (Zheng et al., 2022) on the CIFAR-10 dataset with the ResNet-56 architecture.
To incorporate these recent score metrics, which are specifically designed for coreset-based selection, we utilized the coresct task sctup. In this setup, the network is first trained on the entire datasct to identify a representative subset of data (coreset) based on the selection metric. The accuracy of the target model trained on the selected coreset then reported. The results demonstrate that PruneFuse seamlessly integrates with these advanced selection metries, achieving competitive or superior performance compared the bascline while maintaining computational efficiency. This highlights the versatility of PruneFuse in adapting and enhancing existing coreset. selection techniques.
31

---

Published in Transactions on Machine Learning Research (03/2026)
Table 26: Performance Comparison of Baseline and PruneFuse(p 0.4) on Cifar-10 10 and Cifar-100 using ResNet-56 architecture.
Label Budget (t)
Method
Selection Metric
10%
20%
30%
40%
50%
Baseline (AL)
Confidence
80.53
57.74
90.85
92.24
93.00
Entropy
57.63
92.51
92.98
Leust Coufideuce
PrumeFuse (y = 0.4]
Entropy
80.94
88.27
91.09
92.73
93.38
(a) CIFAR-10
Label Budget (6)
Method
Selection Metric
10%
20%
30%
40%
50%
Least Confidence
35.99
52.99
59.29
63.68
66.72
Baseline (AL)
Entropy
37.57
52.61
58.87
63.97
66.78
PrumeFuse (p= 0:4)
Least Coufidence
38.73
54.35
60.75
64.80
67.08
Eutzupy
38.35
54.19
60.79
65.00
67.47
(b) CIFAR-100
Table 27: Performance Comparison of Baseline (Coreset) and PruneFuse (p=00 0.5) for Various selection
metrics including Forgetting Events Toneva et al. 2019). Moderate (Xia et al. 2023 and CSS (Zeng et al. 2023 on Cifar- 10 dataset using ResNet-56 architecture.
Data Selector's Params
Target Model's
Accuracy (b 25%)
Method
Selection Metric
Params
Eatropy
86.13
Leust Confidurce
86.50
Baseline
Forgetting Events
0.85 Million
0.85 Million
86.01
Moderate
86.27
CSS
87.
Eatropy Leust Confiennee Forgetting Events
86.71
86.68
PruneFuse Fuse
0.21 Million
01.85 Million
87.84
Moderate
CSS
88,85
9.14 Effect of Various Pruning Strategies and Criteria
In Table 28 we evaluate the impact of different pruning techniques (e.g., static pruning, dynamic pruning) and pruming criteria (e.g.; L2 norm, GroupNorm Importance, LAMP Importance (Fang al. 2023 on the performance of PruneFuse (p = 0.5) on CIFAR-10 using the ResNet-56 architecture.
Static pruning involves pruning the entire network at once at the start of training, whereas dynamic pruning incrementally prunes the network in multiple steps during training. In our implementation of dynamic pruning, the network pruned in five steps over the course of 20 epochs.
The results demonstrate that PruncFuse highly adaptable various pruning strategics, consistently maintaining strong performance in data selection tasks. This flexibility underscores the robustness of the framework across different pruning approaches and criteria.
9.15
Runtime Comparison of Data Selector Networks and Detailed Breakdown of the Training Runtime for each Component of PruneFuse
Table 29
compares the training runtimes of the data selector network (pruned network for PruneFuse and dense network for the baseline) across various network architectures. The reported times correspond to the training phase of the data selector network prior the final selection of the subset (at b 50%, label budget). Note that the variation in runtimes across different datasets is due to the experiments being conducted on different servers, each equipped with specific GPUs (e.g., 2080Ti, 3090, A100). The results show that.
32

---

Published in Transactions on Machine Learning Research (03/2026)
Table 28: Effect of Pruning techniques and Pruning criteria on PruneFuse (p= 0.5) on CIFAR-10 dalasel with ResNet-56.
Label Budget (0)
Method
Pruning Technique
Pruning Criteria
10%
20%
30%
40%
50%
Baseline (AL)
80.53
87.74
90.85
92.24
93.00
Magnitude Tmp.
79.73
87.16
91.08
92.29
93.19
PruneFuse
Dynamic Pruning
GroupNorm Imp.
80.10
88.25
91.01
92.25
93.74
(Tayee =0) 0)
LAMP Imp.
81.51
87.45
90.64
92.41
93.25
Magnitude Imp.
80.92
88.35
91.44
92.77
93.65
PruneFuse
Static Pruning
GroupNorm Imp.
80.84
88.20
91.19
93.01
93.03
(Tague 0)
LAMP imp.
81.10
88.37
91.32
93.02
93.08
Magnitude Imp.
81.23
88.52
91.76
93.15
93.78
PruneFuse
Static Pruning
GroupNortu Imp.
81.09
88.77
91.77
93.19
93.68
(Tape 1)
LAMP Imp.
81.86
88.51
92.10
93.02
93.63
PruneFuse significantly reduces training time due to the efficiency of the pruned network as compared baseline, making it well suited for cesotrce-consrrained environments.
Table 30 provides a detailed breakdown of the training time for each component of PruneFuse, including the data selector training time, the selection time, and the larget network training time. These measurements offer a comprehensive view of the computational requirements of PruneFuse, demonstrating its efficiency compared to the baseline methods. The breakdown highlights that the pruned network and the fusion process contribute to significant computational savings without compromising performance.
Table 29: Training Runtime of data selector network i.c. pruned network in the case of PruneFuse and dense network for baseline, for various network architectures. The reported time training time when the network is trained before selecting final subset of the data (b 50%).
Data Selectors (Selection Models)
Training Runime (Minutes)
Datasets
ResNet-56 (Baseline)
127.67
CIFAR-10
ResNet-56 (PruneFuse 0.5))
72.55
56 (PruneFuse 0.8))
67.23
ResNet-18 18 (Baseline)
85.68
ResNet- 18 (PruneFuse (p 0.5))
61.15
Wide ResNet. (Baseline)
122.43
Wide ReaNet (PruneFuse (p 0.5))
75.48
ResNet- 164 (Baseline)
129.23
CIFAR-100
ResNet 164
(PruneFuse (p 0.5))
83.52
ReaNet-161 (Pruneuuse (p 0.5))
78.55
ResNet-110 (Baseline)
95.80
ResNet- 110 (PruneFuse (p 0.51)
80.42
ResNet-110 (PruneFuse (p 0.8))
69.50
ResNet-50 (Baseline)
248.48
TinyImagenet
ResNet-50 (PruneFuse (p 0.5))
147.47
ResNet-50 50 (PruneFuse (p 0.8))
94.42
Resuct-50 (Baseline)
2081.3
ImageNet-1K
ResNet-50 (PruneFuse (p 0.5))
951.17
33

---

Published in Transactions on Machine Learning Research (03/2026)
Table 30: Detailed Training time Baseline and PruneFuse(P = 0.5) for TinylmageNet-200 for Resnet-50
using Least Confidence as selection metric.
Label Budget.
Data Selectors [Triinnng Time
Data Selection Time
Target: Model
(i)
[Minntes]
(Triining Time) (Minnttss)
[Minutess
Baseline (AL)
10%
207
90.23
3.50
90.22
30%
145.32
3.15
155.32
40%
195.38
2.72
30%
248.48
2.38
248 48
PruneFuse
10%
32.17
20%
61.70
1.67
90.39
40%
117.10
1.37
196. 28
50%
Table 31: Results with ResNet20 and ResNet56 CIFAR-10 (Least Confidence Entropy) using ALSE Jung et al. 2023).
Model Selection Metrie
Method with ALSE
10%
20%
30%
40%
50%
Baseline PrinePuse (p 0.4)
ResNot20 Least Conf.
81.05
86.82
90.02
90.89
91.54
Pruneluse
91.51
PruncPusse (p 0.6)
80.53
86.68
89.56
90.81
91.32
Baseline
40.73
88.13
92.08
93.13
ResNee56 Least Conf.
PruncFuse (p 0.4)
80.86
$8.28
91.36
93.15
93.44
PrumeFuse
80.80
68.17
91.43
93.02
93.19
Prunelas 0.6)
80.76
91.29
92.77
93..20
Baseline
$0.12
86.00
85. 96
90.68
91 :11
RosNor:00 Entropy
PrumeFase (p 0.1)
80.64
86.99
89.65
91.27
91.16
Prinelase 0.5)
80.39
$6.59
89.49
90.97
91.42
PrinePusc 0.6)
91.24
Baseline
$0.50
88.11
90.98
92.52
93.06
PruncFuse
ResNee556 Eucropy
PruneFuse 0.5)
80.97
88.37
91.31
92.87
93.32
0.8)
34


Reviews:
[{"review_id": "4gVlk1yxlb", "reviewer": "Reviewer_V76W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose PruneFuse, a framework for efficient data selection in Active Learning (AL). To address the computational scalability bottleneck of traditional AL , the method utilizes structurally pruned networks as lightweight data selectors. A key contribution is the network fusion mechanism, which transfers learned weights from the pruned selector back to the full model, providing an optimized initialization that accelerates convergence and improves generalization. The approach also integrates knowledge distillation and iterative synchronization. Extensive experiments across CIFAR, ImageNet, and NLP datasets demonstrate that PruneFuse significantly reduces computational costs  while outperforming state-of-the-art baselines like SVP and ALSE.\n\n**Key Strengths:**\n\n- Novelty: The \"prune-then-fuse\" paradigm effectively addresses the limitations of prior \"proxy\" methods (e.g., SVP), which discard the proxy model after selection, by reusing learned parameters to aid the target model.\n\n- Efficiency: The method demonstrates substantial reductions in FLOPs and training time across various benchmarks while maintaining or exceeding baseline accuracy.\n\n- Rigorous Evaluation: The submission includes comprehensive experiments across diverse architectures (ResNet, ViT, VDCNN) and tasks, supported by theoretical error bound analysis.\n\n**Weaknesses:**\n\n- Pipeline Complexity: The proposed workflow involves multiple stages which introduces higher implementation complexity compared to standard AL pipelines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well-supported by empirical and theoretical evidence:\n\n- Efficiency & Accuracy: Tables 1 and 2 clearly show that PruneFuse achieves comparable or superior accuracy to baselines with a fraction of the computational cost (e.g., 95% less computation on ImageNet-1K).\n\n\n- Efficacy of Fusion: The ablation studies (Figs. 4, 5, 6) provide convincing visual evidence that the fusion mechanism significantly accelerates convergence compared to training without fusion.\n\n- Theoretical Grounding: Theorem 5.1 provides a formal error bound, theoretically justifying the synchronization strategy.", "audience_yn": "Yes", "audience_explanation": "The paper addresses the critical trade-off between data efficiency and training costs in Deep Learning. The findings are highly relevant to researchers focusing on Efficient AI, Active Learning, and Model Compression. The proposed method offers a practical solution for training large models in resource-constrained environments, which is of broad interest to the TMLR community", "requested_changes": "I recommend the following adjustments to further strengthen the submission:\n\n- Implementation Complexity Trade-off: While the computational savings are clear, the engineering overhead of the PruneFuse pipeline is non-trivial. I suggest adding a concise discussion in the main text weighing this implementation complexity against the performance gains. \n- Justification for Unaltered Weights: Section 4.4 states that unaltered weights in the fused model serve as gateways to unexplored loss landscape regions. Please provide a more rigorous justification or specific ablation evidence (referencing Section 8.5 if applicable) comparing this strategy against alternative initializations (e.g., zero or random re-initialization) to empirically validate the claim that retaining initial weights specifically aids \"exploration.\"\n- Sensitivity to Pruning Criteria: The method relies on structured pruning. While Section 8.13 in the supplementary material covers this, a brief summary in the main text regarding the method's robustness to different pruning criteria would reinforce the claim of flexibility.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YpLLaFnFkN", "reviewer": "Reviewer_Kwk8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors propose a special scheme of active learning that can be specified roughly as:\n1. Select and initialise architecture suitable for the downstream task\n2. Perform structural pruning to select a small subnetwork\n3. Train subnetwork with several rounds of active learning\n4. Fuse subnetwork to the original unpruned network\n5. Finetune the resulting network on the dataset selected in 3.\n6. If the computation budget is not exhausted go to 2.\n\nAuthors evaluate their scheme on several datasets and compare accuracy and computation load with various active learning baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "To an extent, authors support their claims by experimental evidence. To me, debatable claims are:\n1. The importance of distillation.\n2. The benefits of fuse steps.\n3. The usefulness of theoretical analysis.", "audience_yn": "Yes", "audience_explanation": "Active learning is a classical widely applicable direction in machine learning and statistics. Given that, present contributions can be of interest for many researchers.", "requested_changes": "**Unclear parts**\n\nSeveral parts of the main text are unclear:\n\n1. Problem setting given in equations (1) and (2)\n\n   Equation (1) formulate a problem as finding a subset of data sufficient for training an accurate model. The result of optimization problem (1) is a subset of data. Next, authors \"modify\" problem (1) and formulate it as problem (2). I find two problematic parts in this framing.\n   \n   First, in equation (2) authors use $E_{(x, y)\\in s_{p}}\\left[l(x, y; \\theta, \\theta_p)\\right]$ and call this expression \"expected loss on subset $s_p$ (selected using $\\theta_{p}$)\". If $s_p$ is selected based on $\\theta_p$ then the resulting expectation is not a function of $s_p$. Given that, it is not clear how to understand the minimization problem $\\arg\\min_{s_p}$.\n   \n   Second, from the rest of the article it becomes clear that authors do not modify problem (1). In place of that they suggest a specific way to approximately solve problem (1). Their solution is to train progressively a set of reduced models obtained with pruning from a large base model (see Algorithm 1).\n   \n   I suggest authors clarify the relation between problems (2) and (1) and their setup overall.\n   \n2. Algorithm 1\n\n   I find several unclear parts in Algorithm 1:\n\n   a. Among inputs to the algorithm authors list: (i) prune model $\\theta_p$, (ii) fuse model $\\theta_{F}$, (iii) scored data samples $D_{j}$. All these objects are later computed in the body of the algorithm. Besides that, they all depend on the training dynamics, evaluation of trained models, initialisation of the model, etc. What do authors mean when they claim the user should provide all that as an input to the algorithm?\n\n   b. Algorithm 1 does not reference distillation that should be added to the line with fine tuning of the base model.\n\n   c. On page 7 authors write \"Meanwhile, the rest of the untrained weights in $\\theta$ still have their initial values, offering an element of randomness. This duality fosters a richer exploration of the loss landscape during subsequent training.\" Does it mean that we still need to keep original parameters from the initialisation even when we already perform finetuning of the based model for the first time? If randomness is important, why not generate random weights during the fuse step?\n   \n3. Figure 2: Evolution of training trajectories.\n\n   As I understand Figure 2 provides a schematic of convergence that does not correspond to any real optimisation process. In light of the relative unimportance of fuse step (see below) it is unclear how authors can back up this illustration. Can the authors comment why they believe this illustration is important and how it would be possible to test it experimentally?\n\n**Importance of fuse step**\n\nFuse step is a primal innovation of the research. The ablation of the fuse step is given in Figure 3, Table 2 and Section 8.5.\n\nResults from Figure 3 indicate that the training without fuse step ($T_{\\text{syn}} = 0$) requires less FLOPs for all pruning ratios and leads to the same or better accuracy for pruning ratios $0.5, 0.7, 0.8$. Only for $p=0.6$ the performance of training without fuse steps is noticeably worse.\n\nSimilarly, from Table 2 one can conclude that the fuse step is not important since best results are observed for $T_{\\text{syn}} = 0$.\n\nIn Section 8.5. authors perform experiments that compare training dynamics with and without fusion. The precise setup of the experiment is not available in Section 8.5. Can the authors explain what kind of version of Algorithm 1 they use to obtain data presented on Figure 5? It is also not clear whether one can use epoch as $x$ axis, because methods with and without fusion will require different numbers of resources for training per epoch. Can the author take this into account in their analysis?\n\n**Importance of distillation**\n\nAblation on distillation is discussed in Section 8.6. The results presented in Table 15 indicate that knowledge distillation is not important: sometimes it results in better performance, sometimes in worse performance. Do authors agree with this conclusion? If the answer is positive, can the authors clarify why description of knowledge distillation in the main text (Section 4.5) is needed?\n\n**Theoretical results**\n\nTheoretical results are covered in Section 5. The main result is a straightforward consequence of three assumptions: assumption 5.1, assumption 5.2, assumption 5.3.\n\n1. The first assumption 5.1. requires the neural network to be Lipschitz with respect to model parameters, which is an unusual but in principle enforceable and testable assumption. Did the authors try to estimate Lipschitz constant for neural networks they trained? What is a typical Lipschitz constant (wrt parameters) for ResNet-110?\n\n2. Assumption 5.2. is quite strong and completely unjustified. In plain English this assumption states that: (i) a reduced subset of a full dataset exists, (ii) scores of pruned networks are enough to find that subset. Can the authors explain why they make such an assumption? Is it possible to test this assumption experimentally?\n\n3. Similarly, Assumption 5.3. is way too strong. It implies that synchronisation steps lead to linear convergence to the parameters of the full model. In principle such an assumption could be tested, with a caveat that the same function can be represented by neural networks with parameters that are far apart in the $L_2$ norm. For example, for neural network with two layers\n   $$\n   W^{2}\\text{ReLU}\\left(W^{1}x + b^{1}\\right) + b^{2} = sW^{2}\\text{ReLU}\\left(\\frac{1}{s}W^{1}x + \\frac{1}{s}b^{1}\\right) + b^{2},\n   $$\n   for arbitrary positive $s$. The distance between parameters of these two networks increases with the increase of $s$ (for sufficiently large $s$) if weights $W^{2}$ are nonzero. Besides that, the statement should take into account somehow that the loss landscape can have a large number of distinct global minima.\n\nCan the authors comment on the naturalness of assumption and on the purpose of given theoretical results? Why are these results useful and what they explain?\n\n---\n\nMy main takeaways after reading the paper are:\n\n1. Fuse step is not important.\n\n2. Distillation is not important.\n\n3. Samples selected with active learning using small models transfer reasonably well to larger structurally similar models.\n\nI find point 3. to be the main contribution of the present research. I kindly ask authors to discuss this (agree/disagree, provide arguments) in the rebuttal.", "broader_impact_concerns": "na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8JgDENJCvk", "reviewer": "Reviewer_9M11", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes PruneFuse, a framework for efficient data selection in active learning settings by leveraging structured pruning and weight-aligned network fusion. The core idea is to use a pruned network at initialization as a lightweight proxy for data selection, thereby significantly reducing the computational cost of active learning, and then to fuse the trained pruned model back into the original dense network instead of discarding it. This fusion provides a warm-start initialization for the final model and can be optionally combined with knowledge distillation and periodic synchronization to further improve convergence and performance.\n\nThe paper demonstrates the effectiveness of PruneFuse across a wide range of vision and text benchmarks, including CIFAR, Tiny-ImageNet, ImageNet-1K, and Amazon Reviews, as well as across multiple architectures such as ResNets, Vision Transformers, and VDCNN. Empirically, the method consistently achieves comparable or improved accuracy relative to standard active learning baselines while reducing the computational cost of the data selection stage by a large margin.\n\nKey strengths of the work include its practical relevance, broad experimental validation, and a clear systems-level motivation addressing the scalability limitations of conventional active learning pipelines. Key weaknesses include limited causal isolation of the individual components (e.g., fusion, synchronization, and distillation), and a theoretical analysis that relies on strong assumptions which are not fully justified or empirically validated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the empirical claims of the paper are supported by extensive experimental evidence. The authors evaluate PruneFuse on multiple datasets, model architectures, pruning ratios, and labeling budgets, and consistently report both accuracy and computational cost metrics. The experimental results clearly demonstrate that the proposed approach can substantially reduce the cost of data selection while maintaining or improving downstream performance relative to standard active learning baselines.\n\nThat said, while the evidence is sufficient to support the main empirical claims, some of the stronger interpretive claims—particularly regarding the specific role of model fusion in improving optimization trajectories and exploration of the loss landscape—are not fully substantiated by targeted ablation or diagnostic analyses. These claims would benefit from more controlled experiments or from being stated in a more measured manner. Nonetheless, the core performance and efficiency claims are convincingly supported.", "audience_yn": "Yes", "audience_explanation": "Yes. The problem of computationally efficient data selection is of broad interest to the machine learning community, particularly for researchers and practitioners working on active learning, data-efficient training, and scalable learning systems. The idea of reusing pruned models not only as proxies for data selection but also as informative initializations for the final model is likely to be of interest to TMLR’s audience, especially given the growing emphasis on efficiency and sustainability in large-scale machine learning.\n\nMoreover, the framework is general and compatible with existing acquisition functions and model architectures, which further increases its relevance beyond a narrow application domain.", "requested_changes": "Below I list the requested changes, indicating which are critical for acceptance versus those that would strengthen the work.\n\nCritical (required for acceptance):\n\t1.\tClarify the conceptual novelty relative to prior proxy-based active learning methods.\nThe relationship between PruneFuse and prior approaches such as Selection via Proxy (SVP) should be articulated more sharply. In particular, the authors should clearly state what is fundamentally enabled by weight-aligned fusion that cannot be achieved by proxy-based selection followed by standard training or distillation.\n\t2.\tProvide stronger causal evidence for the role of model fusion.\nWhile empirical results show that fusion improves performance, the current experiments do not sufficiently isolate fusion from other factors such as training budget, distillation, or initialization effects. Additional ablations or more controlled comparisons (e.g., fusion vs. no-fusion under matched training cost) are necessary to support claims about the benefits of fusion.\n\t3.\tRevisit and clarify the theoretical analysis in Section 5.\nThe error bound relies on strong assumptions—particularly the contraction assumption after synchronization—that are not justified by the method’s mechanics. The authors should either empirically validate these assumptions, substantially weaken the claims of the theory, or clearly position the analysis as an intuitive or qualitative justification rather than a formal guarantee.\n\t4.\tAddress fairness of baselines in computational cost comparisons.\nIt should be made explicit whether the active learning baselines are FLOPs-matched or represent canonical implementations. Providing a FLOPs-matched baseline or clarifying the comparison protocol would reduce potential concerns about bias in efficiency comparisons.\n\nNon-critical (would strengthen the paper):\n\t1.\tImprove the clarity of the method description by separating core components from optional extensions (e.g., synchronization and knowledge distillation).\n\t2.\tMove detailed fusion equations and layer-wise implementation specifics to the appendix, focusing the main text on intuition and design rationale.\n\t3.\tSlightly tone down speculative language regarding loss landscape exploration unless supported by additional analysis.\n\t4.\tAdd a concise summary figure or table highlighting which components are essential versus optional in the PruneFuse pipeline.", "broader_impact_concerns": "I do not identify any significant ethical concerns specific to this work. The proposed method focuses on improving the computational efficiency of data selection and model training, which may have positive environmental and resource-related implications. The approach does not introduce new risks related to data privacy, misuse, or social harm beyond those already associated with standard supervised learning and active learning pipelines. A separate Broader Impact Statement does not appear to be necessary.", "additional_comments": "This is a well-executed and practically motivated paper that addresses a real bottleneck in active learning. With clearer positioning relative to prior work, stronger causal analysis of its core components, and a more cautious treatment of the theoretical section, the work has the potential to become a solid contribution suitable for TMLR.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)