ModHiFi: Identifying High Fidelity predictive components for Model Modification
Authors: Dhruva Kashyap
Chaitanya Murti
Pranav K Nayak
Tanay Narshana
Chiranjib Bhattacharyya
Venue: NeurIPS 2025 spotlight ()
Keywords: Pruning
Machine Unlearning
Primary area: deep_learning

Abstract: Open weight models, which are ubiquitous, rarely provide access to their training data or loss function. This makes modifying such models for tasks such as pruning or unlearning, which are constrained by this unavailability, an active area of research. Existing techniques typically require gradients or ground-truth labels, rendering them infeasible in settings with limited computational resources.
In this work, we investigate the fundamental question of identifying components that are critical to the model's predictive performance, without access to either gradients or the loss function, and with only distributional access such as synthetic data. 
We theoretically demonstrate that the global error is linearly bounded by local reconstruction errors for Lipschitz-continuous networks such as CNNs and well-trained Transformers (which, contrary to existing literature, we find exhibit Lipschitz continuity).
This motivates using the locally reconstructive behavior of component subsets to quantify their global importance, via a metric that we term *Subset Fidelity*. 
In the uncorrelated features setting, selecting individual components based on their Subset Fidelity scores is optimal, which we utilize to propose **ModHiFi**, an algorithm for model modification that requires neither training data nor access to a loss function. 
**ModHiFi-P**, for structured pruning, achieves an 11\% speedup over the current state of the art on ImageNet models and competitive performance on language models. **ModHiFi-U**, for classwise unlearning, achieves complete unlearning on CIFAR-10 without fine-tuning and demonstrates competitive performance on Swin Transformers.

ModHiFi: Identifying High Fidelity predictive components for Model Modification
Dhruva Kashyap
Chaitanya Murti HP Inc. AI Lab chaitanya.murriihpp.ccom com
Pranav Nayak CSA, IISc pranavk@iisc.
CSA, IISc
idhruva@iisc.ac.in
Tanay Narshana
Chiranjib Bhattacharyya CSA, IISe chiruBiisc. ac.ac..cc.a
Google
tamayparrhaaa@googgee.cco
Abstract
Open weight models, which are ubiquitous, rarely provide access to their training data or loss function. This makes modifying such models for tasks such as prun- ing unlearning, which constrained by this unavailability, research. Existing techniques typically require gradients or ground-truth labels, rendering them infeasible settings with limited computational resources. In this work, we investigate the fundamental question of identifying components that are critical the model's predictive performance, without access to either gradients or the loss function, and with only distributional access such as synthetic data. We theoretically demonstrate that the global error is linearly bounded by local reconstruction errors for Lipschitz cusschitz-ontinuuuss networks such as CNNs and well-trained Transformers (which, contrary to existing literature, we find exhibit Lipschitz continuity). This motivates using the locally reconstructive behavior of component subsets to quantify their global importance, via a metric that we term Subset Fidelity. In the uncorrelated features setting, selecting individual components based on their Subset Fidelity scores is optimal, which we utilize propose ModHiFi, an algorithm for model modification that requires neither training data nor access to loss function. ModHiFi-P, for structured pruning, achieves 1 11% speedup over the current state of the and competitive performance on language models. ModHiFi-U, for classwise unlearning, achieves complete unlearning CIFAR-10 10 without fine-tuning and demonstrates competitive performance on Swin Transformers."
active of
ImageNet models
1 Introduction
Modern deep learning has made significant strides in a wide variety of tasks, such as classification [35, 36), image generation [22], and natural language processing [49]; moreover, well-trained open weight models for such tasks are easily accessible. However, significant challenges remain in their deployment, such as inference in resource-constraineet settings [59, 68], inference with unbalanced or biased data [28, 29], and interpretable inference [93]. These challenges have increased interest in methods that modify the parameters of well-trained models to alter their behavior [61, 63, 65]. These methods include pruning [27], classwise unlcarning [31, 34, 65], and debiasing [50, 65], among other model modifications. Moreover, recent work has studied model modification in the setting where the original training data and loss function are unavailable [53]; this is motivated by concerns
Author primarily contributed to this work before joining Google.
Our code is available at https:// //tpa://////t////// cama/Daruuakaaaayppooohht.t.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

related to privacy and security [88], and also the use of synthetic data, which has become critical in variety of language modeling settings [10. 721. Thus. we address the challenging problem of altering well-trained models without training data or the loss function, and only with distributional access to the original training distribution the form of synthetic data, focusing specifically on structured pruning and classwise unlearning.
Modifying open weight models without the loss function and only synthetic data requires answering a fundamental question: which components in model contribute significantly to its predictive performance However, most methods that identify critical components for specific modifications ge.g., pruning) cannot be applied to others (e.g., unlearning) [53], often require expensive fine-tuning tuning. Moreover, most methods utilize gradients to assess the impact of a
and are wrchiteturee-specife component on the loss objective, which is not feasible in the absence of the loss function and the training data. While the I.I. pruning literature uses calibration datasets to mitigate the problem of the absence of datasets [2, 46], the problem of achieving sparsity in vision models without original training data is hard and unsolved [27]. Moreover, the issue of performing classwise unlearning without access to the original training data has not been addressed [32, 53].
Towards enabling the modification of well-trained open weight models amidst these challenges, we make the following contributions:
(C1) Prral-to-Global with Lipschitzness. An open question is the extent to which local model modifications impact the predictive performance of the model. In the absence of loss functions and training sets, estimating the impact of component modification by using gradients (as done in [32, 43, 46]) is infeasible. To address this, in Theorem 3.6, we show that for Lipschitz continuous networks, the reconstruction error at the final layer is at most linear in the local reconstruction errors. Moreover, contrary to the assertion that transformers are not Lipschitz continuous [60], Corollary B.4, we show that this not the case for well-trained transformers, allowing us to apply Theorem 3.6 to not just CNNs, but well-trained ViTs and LLMs as well.
(C2) Identifying Subsets of Important Components. Contrary to prior work, which usually infers saliencies for single components, we propose measuring the importance of sets of components to understand the cumulative effects of groups components on a model's predictive performance. Leveraging Theorem 3.6, we propose Subset Fidelity, which quantifies the extent to which a subset of components can reconstruct the output after modifying their weights. However, computing optimal subsets is NP-complete, motivating to compute Subset Fidelity scores for singleton sets. Thcorem 3.9 establishes that selecting singletons with the highest subset fidelity scores is optimal when the features are uncorrelated.
(C3) Modifying Models with ModHiFi-XX Motivated by Theorem 3.9, we propose the ModHiFi algorithm, which uses the subset fidelity of singletons to modify models for pruning and class- wise unlearning, the algorithm identifies important components using the singleton scores, and removes them (for classwise unlearning, ModHifi-UU or retains them (for structured pruning, ModHiFi-P). We demonstrate that ModHiFi-P achieves state-of- totte-of-thearrr the-art speedup for ImageNet models and consistently competes with current baselines for language models. For classwise unlearning. ModHiFi- achieves complete unlearning on all CIFAR- 10 classes without fine- tuning and competitive with baselines on Swin Transfornnsss that require fine-tuning. When allowing for similar fine-tuning budget as said baselines, ModHiFi-U outperforms, partic- ularly when given access to training data. These empirical results demonstrate the practical effectiveness of Subset Fidelity.
2 Background, Setup, and Related Work
In this section, we review the background relevant to our study, establish the notation, and formalize the model modification problem. We also unify Convolutional Networks (CNNs) and Transformers under single abstraction that underpins our theoretical results in Section 3.
'We measure predictive performance using accuracy for classification tasks, and perplexity and other measures for language modeling tasks.

---

2.1 Background and Notation
Notation Let [p] = (I,...,.1l .......pp. for E N. We denote vectors by E Ik" with entries Vi, and matrices by E 12xx with rows by and columns B. The vectors 1, and 0₂ denote the all-ones and all-zeros vectors in Rd, respectively. We use ||l|| for the Euclidean norm. For matrices C, D, the inner product is (C, D) = Tr(CTDD the Frohenius norm is ||C|| = VCC
(C,C), C)2 and the spectral norm ||C||2 the largest singular value. For index sets A [ [n] and B [m], C[A, B) denotes the submatrix of C' defined by these indices. Expectations of random variable X are written [x omitting the subscript when clear from context.
2D Convolution Consider the (-th layer of a convolutional network. It transforms an input (the output of preceding layers) o' (X) E R xxf=x xwf=1 into output (x) R xh' xwe The
layer is parameterized by a weight tensor W' E Re xel, xk x Each output channel [coue] is
computed as the sum of convolved input channels:
Y: (X) =
(X) * W
(X),
(CONV)
where * denotes the standard 2D convolution. We define (X) (X) * W (E h
as the imput contribution from channel to output channel c. For notational simplicity, we omit explicit bias terms and stride/padding specifications, as our analysis generalizes these standard configurations without loss of generality.
Transformers Transformer blocks consist of Multi-Head Attention (MHA) and Ferd-Forward Networks (FFN), with pre-normalization (LayerNorm or RMSNorm) [2, 3]. Our analysis focuses on the FFN: we leave attention-speciie analysis for future work. Let the input to the I-th layer be ø' dx(X) E Td'xd, two linear transformations, W E ddxd and W E Rd and an elementwise nonlinearity (.)) and it's output, FFN' (X)) = a(d (X)Wt) w Defining the intermediate activation ('(X) = ((0((X) E R Xdxee, the contribution from intermediate neuron E [dR] to output coordinate [d] is:
xd, where I' is sequence length and d is model dimension. The FFN comprises
A' (X) (X)) W..t
(LIN)
Unified Notation To unify these architectures, we define common abstraction used in our theoretical results. Let N2 f o... f be a network composed of L layers. Each layer maps an input t' (xX) to an output Y (X). Crucially, for both CNNs and Transformers, the output channel
e can be decomposed as a sum of atomic input contributions: Y!(X) Σ A A (X); where A!, represents either the spatial convolution (Equation (CONV)) or the token-wist linear projection (Equation (LIN)). This decomposition is central to our analysis of component importance. This additive structure allows us analyze component fidelity an architecture-aat aanitecturr--anostii manner.
2.2 Modifying Open Weight Models without Training Data the Loss Function via Distributional Access
We formally define model modification as the process of selectively altering parameters of a pre- trained model, without retraining from scratch, to satisfy constraints such efficiency, privacy, or safety [27, 32, 63, 65]. This includes tasks including structured pruning, unlcarning [37], debiasing [31], continual or life-long learning [21, 61]. A major impediment in real-world modification is the unavailability of the original training data and loss function [27, 53]. To address this, we operate under the constraint of distributional access, specifically utilizing synthetic data [10, 72]. to proxy the underlying data distribution without requiring the original corpus.
These considerations motivate the central question addressed in this work: Can effectively modify trained models, for tasks such as structured pruning or unlearning, using only distributional access provided through synthetic data?
3

---

Formulating Model Modification Let 8* E RD be the parameters of a well-trained model. We seek modification mask m* E M (where M defines permissible modifications, e-g., binary masks for pruning) to produce modified parameters 0 = 0* @ m'. Given data distributions (D:) and weights a E IR the optimal modification mask is defined as:
arg min
Σ
a, EXwD, [C(N (mN))
(MODIFY)
mEM
We instantiate this framework for two distinct tasks:
Structured Pruning The goal is to maximize performance subject to sparsity. Let the parameters be partitioned into G disjoint structured groups { (e.g., filters, channels, rows, or columns), with
0" = (0g, :..,,,..,,000 (ga)) The admissible set enforces a sparsity budget B, Msp = {m E PRP 3z E (0. 8 E IR s.t. m, 2,0; Vj 9g: Σg= 2g S B). Structured pruning recovered from (MODIFY) by setting K D1 D (original task distribution), a1 1, and M MMsp. yielding (STRUCT-PRUNE)
m arg min Ep [C(No (X))].
mE.spP
Classwise Unlearning The goal degrade performance on a forger distribution D, while preserv- ing performance on retain distribution D,. We impose no additional structural constraints on the modification and set MMr = RDD Classwise unlearning obtained from (MODIFY) by setting K (D1, D2) = (Dr, Dj), (01, 02) = (1, 11), and M = MMu yielding
m arg min Ex~D, [C(N cm cm (X)] Ex-D, [C(No om (X))].
(UNLEARN)
M
Our core challenge is to solve (MODIFY) using only synthetic samples, without access ground-truth labels or the original loss.
2.3
Related Work
We briefly situate our work within the literature on vision and language model modification. A comprehensive survey is provided in Appendix A.
Vision Model Modification While structured pruning is well-established for CNNs and ViTs [14, 27, 89, 91], and classwise unlearning has seen recent progress [12, 32], these tasks are typically treated in isolation. Crucially, prior methods for jointly addressing these problems rely heavily on access to labeled data [53]. Our work presents the first unified framework for both pruning and unlearning, which operates effectively using only unlabeled synthetic data.
LLM Modification Efficiency in LLMs is primarily addressed via structured pruning [46, 47] or sparsification [2]. However, these methods are often architecture--peccffe and do not extend to unlearning. By validating our method on both LLMs and vision models, we demonstrate generalized approach to model modification that bridges the gap between these distinct domains.
3 Which Components Are Important for Modifying Well-Trained Models?
We now address the problem of identifying model components critical to predictive performance. We introduce Subset Fidelity, metric that quantifies the local reconstructive capacity of component groups and High-Fidelin (HiFi) components. We show theoretically that maximizing local fidelity minimizes linear upper bound the global predictive error.
3.1 High-Fidelity Components and the Subset Fidelity Score
Our objective is to estimate the impact of removing subset of input contributions on the model's output, after optimally compensating for this removal. Directly quantifying this effect is difficult, so we introduce the Subset Fidelity, measure of how well a subset of components can locally approximate the layer's output.
Definition 3.1 (Subset Fidelity). The fidelity of subset of components C C [ in layer I for output channel c is defined as
(1- E [I
- Seec A (x)11]
FS) max
(I)
E Y! (X)11
8 gERRin

---

hareo Noper coW2
Nara www..rreeeee..
layer ble ceed
(a) OPT-125M (WikiText) (b) ResNet50 (CIFAR-10) (c) ResNet50 (CIFAR-ION) 100) (d) ResNet50 (ImageNet)
Figure 1: Monte Carlo estimation of Equation (K-MFS) across selected layers of various models.
The x-axis indicates subset size k, and the y-axis the maximum fidelity found across random samples.
where 8 is the compensation term.
The following properties (proved in Appendix B.2) justify its use as an importance measure.
Lemma 3.2 (Properties of Subset Fidelity). For any subset C S [c,m] in layer 1, (Boundedness)
0 S (C) S and (Monotonicity) If D S C, then (D) S (C).
larger Subset Fidelity indicates that the subset more effectively reconstructs the output, thereby re- ducing the еπrоr of approximating the sum of components with components from a subset. Lemma 3.2 implies two key insights: (1) Fidelity serves as a principled measure of component importance, and (2) Monotonicity suggests that greedy selection strategies may be effective.
Remark 3.3. Equation (1) is generalizes the formulation of El Halabi et al. [11]. In this work, we focus only on the case where the subset fidelities are measured with the expected squared difference. We leave to future work an exploration of other possible measures of distributional similarity.
To capture the tradeoff between the size of a subset and its fidelity, we define HiFI Sets. Definition 3.4 ((k, - Set). Given a target subset size & and a fidelity threshold 17 E (0, 1). the (k, ())FF Set S for output channel is any subset in [chn] satisfying
(St( 2 T)
|S < k.
(HIFf)
Thus, attributing predictive performance to components reduces to finding the HiFI set for given (k:, 17). We can reduce the identification HIFI sets solving an optimization problem, the solution of which yields the Maximum Fidelity Subset, which contains the components that best recover the layer's 's output.
Definition 3.5 (k-Maximum Fidelity Subset). Given target subset size & for layer I, the Maximum Fidelity Subset S for channel c is defined as
S arg FS'((S). 1CIE..11S
(K-MFS)
simple algorithm for identifying a (k, 77HFFFI set to solve Equation (K-MFS) for the given l and check whether its fidelity exceeds 77. If it does not, such (k, 7)-HFFt set exists. Before proceeding to our theoretical analysis, we empirically verify whether small HiFi sets actually exist in standard models. Our experiments in Section 5.2 empirically establish the existence of small subset of components that can achieve high fidelity. Moreover, in Section 5.3, we validate the effectiveness of Hilii components with the model's predictive performance. Figure I indicates a sample of the results indicating that fewer than 20% of components can achieve high fidelity (2 0.8).
3.2 Local Distributional Measures of Component Importance
Finding HiFi subsets corresponds to finding subsets that minimize the le reconstruction error while ac- counting for weight compensation. Additionally, enables the derivation of closed-fomn form expression for weight compensation, allowing for accuracy recovery without requiring fine-tuning.
Bounding Global Error via Local Modification We now show that the influence of a component on its immediate layer output provides a tractable proxy for its overall effect on model predictions. The global error is the expectation of the squared difference in the predictions of a network under modification.
5

---

Theorem 3.6 (Local to Global), Consider network No as defined in Section 2.1. Let M' be mask modifying parameters at layer I, and let m be the mask vector for output channel c. Assume there exist scalars y > O for all layers l L such that [| (x)]IF re almost surely. Then,
B (X) Noa (X)II OO
E
(
Em' (
(2)
IEC
Sketch. The proof relies on the propagation of error through Lipschitz-continuon layers. See Appendix B.1.
M
Theorem 3.6 upper-bounds the global error, given by the left-hand side, by a linear function of the local reconstruction errors for each channel in layer This implies that global error grows at most linearly with local error, making local fidelity a practical, influence. The theorem requires that the networks discussed in this work are Lipschitz continuous under suitable conditions. While CNNs are known to be Lipschitz continuous [90], transformers are not [60]. In Corollary B.4, we show that this is not the case for well-trained transformers.
proxy for component
Remark 3.7. The leading constant in the order notation quantifies the amplification of local errors through subsequent layers and activations, and is independent of the data distribution, depending only on the model's architecture. Empirical estimates of the constant reported in Appendix C.2 demonstrate the practicality of these constants.
Subset Fidelity for Individual Components Next, we show that both the compensation term and the singleton fidelity scores admit closed-form expressions, thus motivating their use in this work. A derivation is provided in Appendix B.3.
Proposition 3.8 (Compensation and Singleton Fidelity). For the l2 reconstruction error, the optimal compensation term 6% which is the value at which the fidelity score is computed according to Equation (1), for subset C, is given by,
( + ((2)C. 0
JiEC
8' (C)
(FS)
if
where Q¹ E RR xcin is the component similarity matrix (CSM) for channel with entries (Q") = A (X), A/ (X)). The singleton fidelity scores are:
8/4 FS) = E (X) a' A (1) EE
B(Y (X)) E A 02 (X) [2)
d
(3)
(X
Note that solving Equation (K-MFS) exactly is still equivalent to a constrained binary quadratic optimization problem, known be NP-hard [1]. Viewing Q the adjacency matrix of a weighted graph, maximizing Equation (K-MFS) corresponds to identifying a clique of size k, the decision version of the MAXIMUM CLIQUE problem. Intuitively, such cliques correspond to groups of components whose joint removal maximally increases the reconstruction error.
Computing the K-MFS Since fidelity is monotonic, natural heuristic selects the & components with the highest singleton fidelities s we call this strategy: NAIVE. To compute the set of highest fidelity, the K-MFS, identify conditions under which the NAIVE selection strategy is optimal.
Theorem 3.9. Consider output channel in the thh layer of a network described in Section 2. Let the 5'4 be defined according to Equation (3) and Si* be defined according to Definition 3.5. Let
S = {i 2 2 {(K)} where 8(k) is the kth largest value of s Assuming that there are no ties, 1 k. (A (X), Al (X) vi # then St. =!!
Sketch. Under the assumptions, the objective simplifies from quadratic to linear. See Appendix B.4.
Remark 3.10. Theorem 3.9 connects statistical property of the representations to the efficient discovery of HiFI components. states that when the input contributions are pairwise uncorrelated, the optimal subset is the set of components with the highest fidelity score.

---

Although the assumption of uncorrelated features rarely holds exactly in practice, it offers a sound theoretical justification for NAIVE HiFi selection. We demonstrate the practical effectiveness of NAIVE HiFT selection through our experiments in Section 5.
4 Modifying Model Behavior using HiFi Sets
We now propose MODHIFI, unified algorithmic framework for model modification using only distributional access. We apply this framework to two distinct tasks: structured pruning (MODHIFI- P) and classwise unlearning (MODHiFI-U). The central idea is to identify high-fidelity (HIFi) components and then modify them in a targeted manner using unified algorithmic procedure, as shown in Algorithm 1. The two tasks operate duals: pruning retains the high-fidelity components necessary for general performance, while unlearning removes the high-fidelity components most discriminative for specific target class. Additional details, including complexity and implementation specifics, are provided in Appendix D.
Structured Pruning To address Equation (STRUCT- where the objective is to remove entire input channels (or features) that contribute minimally to the model's predictive performance. In convolutional architectures, we identify and remove input channels across all layers that do not appear in the HIFI sets of any output channel of the residual-coupled layers. For CNNs, pruning is applied to the input channels of convolutional layers. For LLMs, we target the input features of the MLP down- projjppjecction matrices (WD). After pruning, we compute the optimal compensation term 8* (derived in Proposition 3.8) using the remaining weights. This step restores the fidelity of the layer output without requiring gradient-based fine-tuning.
Class Unlearning The goal of Equa- tion (UNLEARN) is to erase the influence specific forget class. To perform unlearn- ing, we first compute HIFI sets using only samples from the class wish forget. The components in these sets are then ze- roed out, effectively erasing the influence of that class. This causes the model's pre- dictive performance the forgotten class to degrade, without significantly impacting the performance of other classes.
Algorithm 1 ModHiFi-X
Require: Model parameters 6 layer I, k components, thresh-
old data D
Ensure: Modified parameters 0
Estimate Fidelity: Compute singleton scores s' 'D via
Equation (3).
2: Select HiFi Set: H, <- Top-k indices of s'.
3: if X Prune then
4: for i E [ch,] (i (c, Hi) do
5:
W + 0 Wc E [Cindd]
6: Apply compensation 8" to remaining weights.
7:
else if X Unlearn then
Fidelity Estimation For vision models,
for (c, E H, do
the singleton fidelity score (-) can be estimated efficiently using distributional ac- 10: return 0
9:
w +
cess to the input data, i.c., synthetic samples. In practice, for vision models, we estimate the scalar
coefficients a's directly via batched forward passes synthetic samples. A large a'a indicates high-fidelity component. For LLMs, we develop a tractable Cholesky- based heuristic to estimate the score, providing details in Appendix D.2.
5 Experiments
We empirically validate our framework by addressing four central questions: (QI) Existence of HiFi components. Do small subset of components exist that can achieve high fidelity?
(Q2) Effectiveness of HIFI components. Do HiFT components accurately represent those compo- nents important for the predictive performance?
(Q3) Effectiveness of using HIF1 components for pruning using ModHiFi-P. Does ModHiFi-P result in better eecuracy-sparsig tradeoff compared to structured pruning algorithms for vision tasks and language modeling tasks?
(Q4) Effectiveness of using HIFT components for machine unlearning using ModHiFi-U. Is it possible to perform machine unlearning, as posed by Jia et al. [32], without finetuning? If so, how does ModHiFi-U compare to their method?
7

---

0.6
bear reporekle
rntuert accidd vittara ingaserd
wheed woorld
012
0.-
065-
00
000
1030 1156x
1igut: Sarrel
cta Indox
Vigat charnel
Figure 2: Fidelity score of selected layers of a ResNet-50 model on CIFARIO and the effect of noise on the fidelity score.
5.1 Details of the experimental setup
Models, Datasets, and Evaluation We conduct experiments on ResNet-50/101 [25], VGG19 [69], Swnn-rranrerrmrr [44] and Llama-2-7B [76], benchmarking against relevant experiments from related literature [2, 46]. For vision tasks, we measure the classification accuracy, and for NLP tasks, [19].
use EleutherAI's
Distributional Access in Appendix C.3. We use Alpaca [74] (a synthetic dataset) and WikiTexx-22 [48] as calibration data for NLP tasks following related literature [2, 46]. We provide ablations to measure the impact of synthetic data quality in Appendix C.3.3.
For CIFAR10/100 [35], we use synthetically generated images as detailed
Compute platform and implementation details We discuss the compute platform, implementation details, and hyperparameters used for our experiments in Appendix C.6.
5.2 Existence of HIFI components: Exploring (Q1)
To empirically assess whether small subsets can achieve high fidelity, we estimate S by sampling random subsets of size k across different architectures and selecting the subset with the highest fidelity. Detailed results are presented in Appendix C.I.
Observation 1. Across all evaluated models, each layer typically contains a small subset of input channels (fewer than 20%) that achieves high subset fidelity (2 0.8).
This empirical observation suggests that in trained models, only a small subset of components in each layer responsible for the model's prediction. This observation aligns with the success of structured pruning algorithms in constructing small subnetworks with high statistical performance.
5.3 Effectiveness of HIFI components:
Exploring (Q2)
To answer (Q2), and verify whether HIFI components are the components that matter for the final predictive performance, we measure the effects of the fidelity of a component getting destroyed by noising. For ResNet-50 on CIFAR-10, when 20% of the HIFI components are perturbed with zero mean Gaussian noise with standard deviation of 0.01, the accuracy of the model drops by around 12%. In contrast, perturbing 80% of the non- non-HIFI components identically results in an accuracy drop of only 1%. At 50% of components with a noise of standard deviation 0.02, the accuracy drops by 85% when HIFT components are noised compared to only around 1.4% when non-HiFt components are noised. In Appendix C.1.3, we make similar observations across various models and tasks. In Appendix C.1.2, we additionally performed experiments where we compare the removal of HIF1, non HIFi, and random sets of the same size and make similar observations.
5.4
Structured Pruning Experiments: (Q3)
5.4.1
Vision Models
Baselines We compare against the state-of-the-art structured pruning algorithms specialized for pruning vision models [8, 45, 56, 79], and present additional results on other architectures and datasets in Appendix C.4 where we make similar observations. Following [16], we update the batch norm statistics using the data from distributional access.
8

---

Table Comparison of pruning methods on ResNet50 evaluated on ImageNet.
Algorithm
Accuracy
FLOP Reduction
Param Reduction
CPU Speedup
GPU Speedup
Unpruned
76.1
1x
1x
1x
1x
2 [79]
73.9
3.02x
2.31x
1.36x
1.53x
OTO [8]
74.7
2.86x
2.81x
1.25x
1.45x
DepGRAPH [13]]
75.83
2.07x
ThiNet [45]
71.6
3.46x
2.95x
1.38x
1.50x
DFPC (30) [56]
75.9
1.98x
1.84x
1,42x
1.53x
DFPC (54) [56]
73.80
3,46x
2.65x
2.37x
2.38x
Ours
76.70
2.17x
1.47x
1.69x
1.70x
Ours
73.82
3.66x
3.05x
2.42x
2.38x
Table 2: Comparison of pruning methods on ResNe150 with CIFARIO (ST: Synthetic Tuning).
Algorithm
Accuracy
FLOP Reduction Param Reduction
Unpruned
94,99
1x
1x
DFPC [56]
90.25
1,46x
2.07x
L2 [39]
15.91
4.07x
4.71x
Lg w/ ST [39]
90.12
4.07x
4.71x
Ours
91.02
4.07x
5,36x
Observations We find that our method yields a better to other algorithms across various datasets. We also train model obtained with Lg norm-based structured pruning using the synthetic set based on CIFAR10 10 for comparison. In Table 2, we observe that for the same FLOP sparsity, our method obtains higher accuracy than the model finetuned on synthetic samples, indicating that our method can outperform finetuning in some cases using synthetic samples for the same sparsity. For the ImageNet dataset, we compare our approach against various state-of-the-art structured pruning algorithms for networks with complex interconnections, including those trained on the ImageNet training set. Table we observe that for models similar accuracy, our algorithm obtains the best accuracy-speedup tradeoff with fewer epochs of finetuning. Details of pre-traines networks and post-training are given in Appendix C.7.2. Our study of the effect of the algorithm in Appendix C.3.3 indicates that the pparsity-accuree
tradeoff compared
quality of synthetic samples tradcoff our algorithm degrades with lower quality samples, but it does not degrade as much as L2 pruning finetuning synthetic samples.
5.4.2 Large Language Models
Baselines We evaluate ModHiFi on Llama-2-7B. comparing it against state-of-the-an algorithms for structured pruning [2, 47]. The of calibration datasets compute statistics aligns with framing of distributional access to data, as LLMs do not make their training data openly accessible. Unless otherwise specified, the algorithms 1024* 4, None of the algorithms performs post-pruning recovery finetuning. Additional details about choice of baselines can be found in Appendix C.4.3.
WikiTexx-2 for calibration, with 128 samples of length
Evaluation We also measure the performance of the model via its zero-shot accuracy on a suite of standard NLP tasks [5, 9, 62, 92] and WikiText perplexity. In Table 3, we observe that method competitive, with consistently high average and task-specific performance, and outperforms at moderate sparsity levels. We find that the quality of the calibration set plays crucial role, with the performance of ModHiFi-P-Alpace outperforming that of ModHiFi-P -WikiTexxt This indicates that retaining only HIFi components provides model-agnostic approach to structured pruning, with its application to LLMs requiring no modifications beyond its application to vision models.
5.5 Class Unlearning Experiments: (Q4)
Baselines and Metrics We report the forget and retain accuracy averaged across 10 classes of the CIFAR10 dataset on ResNet-50 and Swin-T models. We benchmark against Gradient Ascent and Jia et al. [32], which are both retraining- techniques for Unlearning.
'ShortGPT's calibration data is not publicly available.

---

Table Comparison of pruning methods on -lama-2-77 measured with PPL and task accuracy
Sparsity
Algorithm
WikiText PPL & ARC-e ARC-c1
PIQA
WinoG.
HellaS. t Average
0%
Dense
5.12
74.58
46.25
79.11
69.06
75.99
69.00
SliceGPT [2]
6.46
56.14
35.33
69.53
64.80
$9.02
59.96
BoadHiFF-P-WikiToee (ours)
5,97
68.1
41.89
75.89
65.43
69.92
64.23
10%
ModHiFi-P-- PaapHii-P-Alppca (ours)
6,36
71.42
42.06
76.44
68.19
71.67
65.96
ShorGPT [47]
14.32
58.33
38.05
72.58
65.51
65.27
59.95
SliceGPT [2]
8.13
50.08
31,14
64.85
62.04
48.84
51.39
20%
ModHiFF-P-Wikiiees (ours)
7.91
34.89
70.62
61.48
58.7
57.16
ModHifi-P-Alaace (ours)
9.38
64.73
38.22
72.79
64.64
62.7
60.62
ShonGPT [47]
33.21
48.65
32.85
64.31
64.33
56.13
53.25
SliceGPT [2]
10.96
44.19
27.47
58.71
57.46
41.27
45.82
30%
ModHiiii--WikiiTce (ours)
11.53
48,98
28.07
64.03
55.88
46.19
48.63
ModHiFi-P-Alpen (ours)
14.78
53.15
32.5
66.59
59.35
50.61
52.44
Table 4: Comparison of class unlearning methods CIFARIO.
Model
Algorithm
Forget Acc.
Remain Ace.
Time (sec)
Base
94.99
94.99
Gradient Ascent
6.59
93.44
30
ResNet-50
Jia et al. [32]
3.54
94.14
363
Ours
0.2
92.98
Base
92.31
92.31
Swin-T [44] et al. [32]
1.20
90.69
235
Ours
8.83
73.57
Unlearning Results We report the results of our algorithm Table answer (Q4), we observe that it is possible to perform unlearning without finetuning in a general editing framework 10x faster than our baseline. In Appendix C.5, we compare results with finetuning using synthetic and training data. We note that the results for Tran-Traasformer without finetuning fail achieve the state the art. However, as reported in Appendix C.5, we observe drastic improvement with only three epochs of finetuning find that our forget accuracy is superior to that of [32] (who use full training) when using synthetic samples. Both forget and remain accuracy superior when using training samples. Experiments with VGG-19 are present in Appendix C.5 where we make similar observations.
synthetic samples. After 10 epochs of finetuning with our algorithm,
6 Discussion and Conclusion
We have addressed the challenge of modifying well-trained deep networks without access to gradients, loss functions, or original training data. By theoretically connecting local layer-wise reconstruction to global predictive error, we established Subser Fidelity rigorous proxy for component importance. Our empirical analysis reveals a fundamental property of modern networks: predictive performance is concentrated in sparse HIFI substructures that are robust noise and identifiable via synthetic data. Leveraging this insight, we proposed MODHIF1, a unified framework for model modification. Unlike prior architecture-ppeifee heuristics, MODHIF1 is domain-agnostic effectively handling both structured pruning and classwise unlcarning across CNNs and Transformers. Crucially, our method designed for the regime of distributional access, making it uniquely suited for modern deployments where privacy or scale necessitates the use of synthetic data.
Limitations and Future Work Our theoretical bounds in Theorem 3.6 rely on the local Lipschitz continuity of the network. While we demonstrate that this property holds for well-trained models (including Transformers on bounded domains), it is not guaranteed at initialization. This suggests that the emergence of High-Fidelity components a consequence of the training dynamics. In this work, we use the expected square loss as a measure of distributional similarity, and we leave for future work the exploration of other metrics of distributional similarity, like the TV distance or Wasserstein metric.
10

---

Acknowledgments and Disclosure of Funding
We, the authors, gratefully acknowledge AMD for its support. The authors thank Ramaswamy Govindarajan (IISc) and Prakash Raghavendra (AMD) for their insight and assistance in this work. The authors are also grateful the reviewers of this work for their valuable feedback, which has significantly improved the content.
References
[1] Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Approach. Cambridge University Press, 2009. (Cited on page 3.2.)
[2] Saleh Ashkboos, Maximilian Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, ed- itors. International Conference on Learning Representations, volume 2024, pages 1 11682- 11701, 2024. URL https: / httpp:s:/pp.oceeiingss... iclr. ct/cccc/pp/. hccpppppr_ssee paper_files__llllppppppppplpeeee files/ wilesspappprre ppppp///22//fille
14166880004411000101555000888880 Paper- Parpeecoo--Pappe--oeeeeeeeeeee pdf. pdf. (Cited on pages 1, 2.1, 2.3, 5.1, 5.1, 5.4.2, 3, A, and C.4.3.)
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxxe www./bgg//aa.oeeooooo.oo.ee absaabs//aa /167/ 06450. (Cited on page 2.1.)
[4] Cenk Baykal, Lucas Liehenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Data- dependent coresets for compressing neural networks with applications to generalization bounds. In International Conference Learning Representations, 2019. https:/ U /ttp////epeeeeeeee/eeee/e. http/forummmddHfffwJJ/ll/.
(Cited on page A.)
[5] Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning
about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-7443) Apr. 2020. doi: 10.1609/aaai w34.v3.4433222 6239.. URL attps://ojss aaai org/ index.nexx./ php/ htAl//rticl///e htt///vew////// (Cited page 5.4.2.)
[6] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In I. Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems, volume 2, pages 129- 146, 2020. URL httpst://procceedinnn mlsys. www.ooggppa..oooo..ooooooo...eee ffrerrffie files hellee/pal hatss/aprr//000/ paaer/2020/file/ 6c44dc7301 0011130000000000044884880000-0
pdf. (Cited on page A.)
[7] Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159, IEEE, 2021. doi: 10.1109/ 2021.00019, (Cited on page A.)
[8] Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and prun- ing framework. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19637- 19651. Curran Associates, Inc., 2021. URL https: htwp::/weeeeeeinnnnneeeeeeeeeeee neurips. cc/paper_ Paper .df.. (Cited
on pages 5.4.1 and 1.)
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. https:/ httpp:s//aaxxv..rr// httgg/ab/18800 1045./55770.0.. 05457. (Cited page 5.4.2.)
[10] Bosheng Ding. Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wen- han Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. Data augmentation using LLMs: Data perspectives. learning paradigms and challenges. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 1679-1705, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10. 00.188533vv/2022 http///...n.inns-a... findings-acl.997. URL. hhttps://aclaaate http://aaaaaaatnaaaaaeea www.aaattoooooooooo////o org/2024. findings acl. 97/. (Cited on pages and 2.2.)

---

[11] Marwa El Halabi, Suraj Srinivas, and Simon Lacoste-Julien. Data-efficient struc- tured pruning via submodular optimization. In Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Informa- tion Processing Systems, volume 35, pages 36613-36626. Curran Associates, Inc., 2022. URL. https://proceeddinggg neurpp ccccccc/pp..co htpppppprfllleeee www.eeeteeeeeeeeeeeeeeeeaeeeeeee ppaper/202/fflle 0a0864c4566e13aaffaaaaaeeeeeeee Papeeer-------------- Paper-Cone Conference pdf. (Cited on pages 3.3 and A.)
[12] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, 2024. (Cited on page 2.3.)
URL. httpps://pperreiieeeeeee
[13] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang- Depgraph: Towards any structural pruning. In 2023 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16091-16101, 2023. doi: 10.1 (0.11099CCPPS222999000000 on pages and A.)
(Cited
[14] Gongfan Fang, Xinyin Ma, Michael Bi Mi, and Xinchao Wang. Isomorphic pruning for vision models. In Aleš Leonardis. Elisa Ricci, Stefan Roth, Olga Russakovsky. Torsten Sattler, and Gül Varol, editors, Computer Vision European Conference on Computer Vision (ECCV) 2024, pages 232-250, Cham, 2025. Springer Nature Switzerland. ISBN 978 -37----03--73404-5 (Cited on page 2.3.)
[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019. URL wwtpeevvvwieneeefeeeeeeee mettfforummddrrl11 b3RcF7. (Cited on page A.)
[16] Elias Frantar and Dan Alistarh. Optimal brain compression:
A framework for ac- curate post-training quantization and pruning. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Informa- tion Processing Systems, volume 35, pages 4475-4488. Curran Associates, Inc., 2022. URL https: https://pproceedings www.:ooceedingg..eeireepeee neurips.co. wccpappe_ffilleecccce www.sf/wwweee/eeeeeeeeaaaaeeeeee 1 (acf00cffee660000000077777777777 http://222700--aae----e- Craper-Conference pdf. (Cited on pages 5.4.1 and A.)
[17] Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10323- 10337, PMLR, 23-29 Jul 2023. URL attps://proceedingn mlr. press/v202/ frantar236 23a html. (Cited on page A.)
[18] Rohit Gandikota, Joanna Materzynska, Jaden Frotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2426-2436, October 2023. (Cited on page A.)
[19] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https:://ennt wtteloooorrrrreeeeeeeeoeooeooeoe arg/records/ 10256836. (Cited on page 5.1.)
[20] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9304-93311 June 2020. (Cited on page A.)
[21] Siavash Golkar, Micheal Kagan, and Kyunghyun Cho. Continual learning via neural pruning. In Real Neurons & Hidden Units: Future directions at the intersection of neuroscience and artificial intelligence NeurlPS 2019, 2019. URL httpt::///ppeeeeeeeeeeee netiieewweeeeeeeeeeee ceet/foouumiii Hyl_XX XXYLIB. (Cited on page 2.2.)
12

---

[22] Ian J. Goodfellow, Jean Pouget-Abadicg Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In 7. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. http:: mew.reeeedddinneeeneiiree..ee cc/ cc/paper f files/ wiles/papeete pttpp/ppap/e//ieeeeee 09344886880224474461119770000603 Paperr pdf. (Cited on page 1.)
[23] Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 4218-4233, Curran Associates, Inc., 2021. URL https: //pooe/ee////ooooeennnnegg/ mew.eeedddiinnnnnininneeeeeeeees neurips. cc/ cc/paper_ffee wwarrrffleeeppre///eee http///ppa/p//ile/eeeee 21 2146600f222544b5336cc006666663 Paper pdf. (Cited on page C.3.2.)
[24] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13):1151--1152 May 2021. doi: 10. 10.16099aaii.v3511.3 1737.. URL https://ojss aaai. org/index. ind/x.ndxxxx.. php/AMAT/arttclle view/1737/ (Cited on page A.)
[25] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), doi: 10. N0.1109/CVPR.2016..00 (Cited on pages 5.1 and D.2.)
pages 770-778, 2016.
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / httt:////cccoccc///eeocecccos itwp:/eeeedddineeeneeeeeeeeeeeee neurips. hacc/pppe__aee www.f//pppeee//e/eee htl.ee//apper/21///i2ee. 8a116447771ee0fffee68811666606/0
Paper. pdf. (Cited on page C.3.1.)
[27] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(1), January 2021, ISSN 1532-4435. (Cited on pages 1, 2.2, 2.3, and A.)
[28] Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characteris- ing bias compressed models, 2020. URL https: http:::/arrd..orr/aa///0...... org/abs/2010 03058. (Cited on page 1.)
[29] Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressed deep neural networks forget?, 2021. URL battps://arxtee www.:orgggaoo arg/abs/1911. 05248. (Cited pages and A.)
[30] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statisties, volume 130 of Proceedings of Machine Learning Research, pages 2008-20166 PMLR, 13- 15 Apr 2021. URL sttps://proceeteee page A.)
mww.mrr..seeeeee wttppp/eess/////////// html. (Cited on
[31] Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. In The Eleventh International Conference on Learning Representations, 2023. URL http::::///peneveee...o on pages and 2.2.)
(Cited
[32] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, PRANAY SHARMA, and Sijia Liu. Model sparsity can simplify machine unlearning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 51584-51605. Curran Associates, files/ peess/ppprr/000 22////2222ffff Paperee---Paee----eeeeeeeeeee Conference pdf. (Cited on pages 1, (C1),
Inc., 2023. https: pttttpp://prooceedings www.eeiinnnnnnnnnnneeeeeeeeeeeee neurips. meurips.cc/papess 220.a66aa4477eeeccccfflllc...a Paapeeddddde--aa-----e-eeee- 2.2, 2.3, (Q4), 5.5, 4, A, C.5, and 12.)
13

---

[33] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the de- sign space of diffusion-bused generative models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 26565-26577. Curran Associates, Inc., 2022. URL. https: httpps://proceeddiggs neurips.c. cc/ http/ppper__ieeeee saars_filespppppee https/ppppr/////e/eeee 4BB4GG9scoo11c177e00044446666 Paper- pdf. (Cited on page C.3.3.)
[34] Sangamesh Kodge, Gobinda Saha, and Kaushik Roy. Deep unlearning: Fast and efficient gradient-free class forgetting. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https::/openrreiiieeeeee http://eeeeoooooonnnnnnnaoeeeee
(Cited on page 1.)
[35] Alex Krizhevsky. Learning multiple layers of features from tiny images. Techni- cal report. University of Toronto, 2009. URL https://www cs toronto
Saaninn--features
TR. pdf (Cited on pages and 5.1.)
[36] Alex Krizhevsky, Πyy Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https: httpttpe:///prrooeeddngggg neerre cc/paper files files/papeeteee /ppper/2012//tlle/ pdf. (Cited on page 1.)
[37] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. To- wards unbounded machine unlearning. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Infor- mation Processing Systems, volume 36, pages 1957-1987. Curran Associates, Inc., 2023. URL https: httpss:/prroceednnggg and A.)
wwwpiipe.cpppppr..ppp wttees/pppr//0ee/ (0aa111667772222225444466666666 Paper- pdf. (Cited on pages 2.2
[38] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touret- zky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1989. URL https::/pproceeediggg wwwppps.ccppppp........ httpppppr_fileeee wil.feeeepppprrre /pwpsp//1p1/////// fil1//c88220a1170088088811788812
Paper pdf. (Cited on page A.)
[39] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017. URL
(Cited on pages 2 and A.)
[40] Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations, 2020. URL https: /ttp////pee/eiieeeeeeeeeee/e net:foorumm.dd-BJxk0l....
(Cited page A.)
[41] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition (CVPR), June 2020. (Cited on page A.)
[42] Jing Liu, Bohan Zhuang, Zhuangwei Zhuang, Yong Guo, Junzhou Huang, Jinhui Zhu, and Mingkui Tan. Discrimination-awarr network pruning for deep model compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4035-4051, 2022. doi: 00.119//PAMM.2201.30066 3006646...... (Cited on page A.)
[43] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7021-7032, PMLR, 18-24 Jul 2021. mlr. www.peesvsviii liu21ab. html. (Cited pages (C1) and A.)
https: htRp:https:///oooe httpas:////ccceeeee htt.///drrceeee.....o.mp
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CV International Conference on Computer Vision (ICCV), pages 10012-10022, October 1.21. (Cited pages 5.1, 4, and D.2.)
14

---

[45] Jian-Hao Luo. Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017. (Cited on pages 5.4.1 and 1.)
[46] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM--Puner: On the struc- tural pruning of large language models. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Informa- tion Processing Systems, volume 36, pages 21702-21720. Curran Associates, Inc., 2023. URL https://prooceeddngg.. neurips..errrrr.c........ cc http/ppper_ffte faless/flllleeleeeer pappr//pp/////// Seaper/2023/faee 2023/fi1e/ ftt.es//aaeeeer//////eeaepaapeel
44965513119997774447777888888881 horpra/eeeeaaaaeee--oeoo----eeee Conference pdf. (Cited on pages I, (CI), 2.3, 5.1, and A.)
[47] Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShorGGT: Layers in large language models are more redundant than you expect. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics; ACL 2025, pages 20192-20204, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89176-28 https:// http:///aaaatthllooooooo org/2025 A, and C.4.3.)
doi: 10. 10.18653/v1/20 /2253///1/225.... htt.://.....innssa..... findings-acl... 1035. URL w.r.fffnninnge www.iinnnnngiie acl. 1035/. (Cited pages 2.3, 5.4.2,
[48] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https:// openreview. atst/foruumiiddyyyyaddat
(Cited on pages 5.1 and C.3.)
[49] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56 (2), September 2023. ISSN 0360-0300. doi: 10.1145/3605943. URL https://doi. http::///ooo.rg10..... 1145/3605943. (Cited on page 1.)
[50] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. Fast model editing at scale. In International Conference Learning Representations, 2022. URL
ddd-or/eeodddoo0eeeeed (Cited on page 1.)
[51] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017. URL /ttpsr///penrrviee page A.)
(Cited on
[52] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrn Recognition (CVPR), June 2019. (Cited on page A.)
[53] Chaitanya Murti and Chiranjib Bhattacharyya. DisCEdit: Model editing by identify- ing discriminative components. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Pro- cessing Systems, volume 37, pages 47261-47296. Curran Associates, Inc., 2024. doi: neurips cc/ htte//ppeefflle/pppeeeeeeeeeeeee filesspaper/
10. 10.5220221799017-1499 URL htttps://proceedinng 2024/fi1e/54801 waa.ee/000000111200eeeeeeaaeeea. Paper- PwwePaaapfeeeeeeeeeeee pdf. (Cited on pages 1, 2.2, 2.3, A, C.5, C.5, 9, and 10.)
[54] Chaitanya Murti, Tanay Narshana, and Chiranjib Bhattacharyya. TVSPrune pruning non- discriminative filters via total variation separability of intermediate representations without fine tuning. In The Eleventh International Conference on Learning Representations, 2023. URL nttps://one
(Cited page A.)
[55] Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In International Conference on Learning Representations, 2021. URL https //tpa////opppee////eeee//ee met//forum?id=guerr/r on page C.3.1.)
(Cited
15

---

[56] Tanay Narshana, Chaitanya Murti, and Chiranjib Bhattacharyya. DFPC: Data flow driven pruning of coupled channels without data. In The Eleventh International Conference on Learning Representations, 2023. URL. https: //wpe/eeee/eeeeeee/eeeeeeeeeeeee wwwpanreviwweeeeeeeeeeeeeeeeeeee http///rrunidmmnnngiaeee ddmhndaqqejyuu (Cited on pages 5.4.1, 2, A, C.4.2, C.6, and D.3.)
[57] Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chune Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. ACM Trans. Intell. Syst. Technol. 16(5). September 2025. ISSN 2157-6904. doi: 10.1 10.1145/3749987, URL attps://don wrgeii.rgg1...
(Cited on page A.)
[58]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https: http////99/////// http:///w//../oocc..////////////
neurips cc cc/paper_ Paper pdf. (Cited
on pages C.6, C.6, and C.7.2.)
[59] Prafull Prakash, Chaitanya Murti, Saketha Nath, and Chiranjib Bhattacharyya. Optimizing dnn architectures for high speed autonomous navigation in gps denied environments edge devices. In PRICAI 2019: Trends in Artificial Intelligence: 16th Pacific Rim International Conference on Artificial Intelligence, Cuvu, Yanuca Island, Fiji, August 26-30, 2019, Proceedings, Part II, page 668--81) Berlin, Heidelberg, 2019. Springer- Verlag. ISBN 978-3-030-29910-11 doi: 10. 101199988--03--9999 1-8_36. 36. URL //tii. ora.oi..org// www.oo.g/.00... 1007 00t.00000000070------000-9------ (Cited on pages and A.)
[60] Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi, and Lei Zhang. Lipsformer: Introducing lip- schitz continuity to vision transformers. In The Eleventh International Conference on Learning Representations, 2023. URL cectpa://openreviii net/forummtii d=cc 1DcCwcH3, (Cited on pages (C1) and 3.2.)
[61] Sabyasachi Sahoo, Mostafa ElAraby, Jonas Ngnawe, Yann Pequignot, Frederic Precioso, and Christian Gagne. A layer selection approach to test time adaptation, 2025. URL https: argg/b//////eee abs/ /240. 03784. (Cited pages I and 2.2.)
/ llaxii.orrg////
[62] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL htwph:t/tpsaata http::://axxvvvvi//eee www.aaiww..../////////// 10641. (Cited on page 5.4.2.)
[63] Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing a classifier by rewriting its prediction rules. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 23359-23373, Curran Associates, neurips. cc/ hecp/pper_steeeee files//lllllell files/paper/ pttpp//ppp////////p/pp/ppaaaaaaa 2221//f12e 0465992225aaaeccccc0001700022222 Paper pdf. (Cited on pages and 2.2.)
Inc., 2021. URL
[64] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Re- member what you want to forget: Algorithms for machine unlearning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 18076. Curran Associates, Inc., 2021. URL htttpss:/pprocceednnggeoo neurips. cc/ seeppape___ files/ walles/papent ppppp//222///i/// Se271655f44440011333222226668883
Paper, pdf. (Cited on page A.)
[65] Harshay Shah, Andrew Ilyas, and Aleksander Madry. Decomposing and editing predictions by modeling model computation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41si International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 44244-44292. PMIR, 21-27 Jul 2024. URI. https: / l/proceedings.o mlr, press/ h2t///23////h// html. (Cited on pages and 2.2.)
16

---

[66] Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M. Alvarez. Structural pruning via latency-saliency knapsack. In S. Koyejo. S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 12894-12908, Curran Associates, Inc., 2022. https: wtt.:...ttttssss///p//oooooo ptttpp:://rroceednggg neurips cc/ cw/paper_rpppepeeeeeeeeeeeee ffffffllee files/ httss/pppppp/2////// pdf.
#4441e448826443770scccccaffaaaa and C.6.)
(Cited on pages A
[67] Xuan Shen, Zhao Song. Yufa Zhou, Bo Chen, Jing Liu, Ruiyi Zhang, Ryan A. Rossi, Hao Tan, Tong Yu, Xiang Chen, Yufan Zhou, Tong Sun, Pu Zhao, Yanzhi Wang, and Jiuxiang Gu. Numerical pruning for efficient autoregressive models. Proceedings of the AAAI Conference on Artificial Intelligence, 39(19):20418-20 20426, Apr. 2025. doi: 10. 1609/6aai v39ii9 19,4499,, URL betps://oj]. aaai. org/ index. php/ AAAI/ /AA.AAAAAT//aaaaa/aaa att.ilettccceeeeeeeeeeeeee /vee/vviiiee/eeeee/eeeeeeeeeev 34249. (Cited on page A.)
[68] Md. Maruf Hossain Shuvo, Syed Kamrul Islam, Jianlin Cheng, and Bashir I. Morshed. Effi- cient acceleration deep learning inference on resource-constraiined edge devices: A review. Proceedings of the IEEE, 111(1):42-91, 2023. doi: 10.1 10.1100/PROC.22222222666 (Cited on
page 1.)
[69] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL https:///rxivv wwwpaii..rg//b///// 1etp//149..... 1556. (Cited on pages 5.1 and D.2.)
[70] Yang Sui, Miao Yin. Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. CHIP: CHannel Independence-based pruning for compact neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14604-24616. Curran Associates, Inc., 2021. htt.htttss:/pproeeeddiee neurips cc/ aecppper__. f files/ /paper/221//Elee/ tebbab000aa$$cc111777790/acaaaoe Paper pdf. (Cited on page A.)
[71] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki. M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 4942-4964, 2024. URL https::t//pteeeeee https:/prroceediing iclr. h00.://eee.eeeeaaa1100ooooocoeee Paper- Conf--ooofeee conference..o pdf. (Cited on page A.)
[72] Zhen Tan, Dawei Li, Song Wang. Alimohammad Beigi, Bohan Jiang, Amrita Bhattachar- jee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 930 957, Miami, Florida, USA, November 2024. As- sociation for Computational Linguistics. doi: 10.18653/v 10/533/11/222/ 1/2014. men.eemminl-mnnn- 12203//.2/2....mpmmaaiiia
URL and 2.2.)
ttttp::/aaclnntho//
org/ 2224.
54/. (Cited on pages
[73] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neu- ral networks without any data by iteratively conserving synaptic flow. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors. Advances in Neural In- formation Processing Systems, volume 33, pages 6377-6389. Curran Associates, Inc., 2020. URL htttps://prte p/ttoeee///roooeiinggs neurips cw/paper...p.....pp files Matesspaper/2022/ /paper/2000/tile/ (5447788854600000570666622222222 Paper. pdf. (Cited on page A.)
[74] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followii llama model. https://gtt thub. thuubbb...b........m//g. com/ www.totoomottttee aatsuu-labsstaae httss//ffodd_aaaeeeosso alpaca, 2023. (Cited on pages 5.1 and C.3.)
[75] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (FuroS&P)) P), pages 303-319, 2022. doi: 10. 10110F/rrrSSS3334222222 00027. (Cited on page A.)
17

---

[76] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Rohert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: //axxiv../.ivv.///aa org/aaggg/aa page 5.
09288. (Cited on
[77] Murad Tukan, Loay Mualem, and Alaa Maalouf. Pruning neural networks via core- sets and convex geometry: Towards no assumptions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 38003
98003-38019. 38019. Curran Associates, Inc., 2022. URL https: neurips cc/ pecepppe__ffeeeeeeeeeeeeee files fiwes/paperr /tter///2//ff/////////// f7f TTEC3314055511664711119955358853 Paper Conference pdf. (Cited page A.)
[78] Enayat Ullah and Raman Arora. From adaptive query release to machine unlearning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett, editors. Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 34642-34667. PMLR, 23- 29 Jul 2023. URL https: http://pooceedng...oo mlr. pwwps//2ee/eea/p w202/22122/222.222 html. (Cited on page A.)
[79] Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations, 2021. URL https://openreval net/forum? id=o966_ Is_nPA. (Cited on pages 5.4.1 and 1.)
[80] Junxiao Wang. Song Guo, Xin Xie, and Heng Qi. Federated unlearning via class-discriminative pruning. In Proceedings of the ACM Web Conference 2022, WWW '22, page 622-632, New York, NY. USA. 2022. Association for Computing Machinery. ISBN 9781450390965. doi: 10. 1..144//388544735122222 URL /ttpp:://oi.... 20g/10..10 on page A.)
3512222. (Cited
[81] Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-5383, 2024. doi: 10. 00.1100/TAMMI.224.3333355
(Cited on page A.)
[82] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. In Proc. of the 30th Network and Distributed System Security (NDSS), 2023. (Cited page A.)
[83] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-an art natural language processing, 2020. URL https: http://axxvvv... hwtp:/ivw.rr/a///////oaaa 1910. 03771. (Cited on page C.6.)
[84] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng. and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Sun, editors, In- lernational Conference on Learning Representations. volume 2024, pages 5385- 5409, 2024. URL. http:::/prrcceeddnngg iclr. cc/ wccppappr_fiiiiieceeee files// filess/ppprerr/////eeeeerraaaaee paper/2024/ERa 1 2a2aa221188822000884444422222222 Cone
Conference, pdf. (Cited on page A.)
18

---

[85] Zuobin Xiong, Wei Li, Yingshu Li, and Zhipeng Cai. Exact-fun: An exact and efficient federated unlearning approach. In 2023 IEEE International Conference on Data Mining (ICDM). pages 1439-1444, 2023. doi: (..1099/CDM588222200 2023.00000 00188. (Cited on page A.)
[86] Jie Xu, Zihan Wu, Cong Wang. and Xiaohua Jia. Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence, 8(3):1550-216. 2024. doi: 2.1109/TTCC12222.337777
(Cited page A.)
[87] Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient architecture for exact machine unlearning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Join! Conference on Artificial Intelligence, IJCA1-22, pages 4006-4013. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: 10.24963/ijcai. 2022/556.. 2022///55 URL hww.h:tp///ooo org/ 10. 14963/ijjcat 2022/556. Main Track. (Cited on page A.)
[88] Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. (Cited on page 1.)
[89] Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width depth pruning for vision transformers. Proceedings of the AAAI Conference Artificial Intelligence, 36 (3):3143-315A) Jun. 2022. doi: 10.1609/aaii, wttp://ai..i63..220 202222 URL https: aaai org/ index php/
(Cited page 2.3.)
[90] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S. Davis. Nisp: Pruning networks using neuron importance score propagation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9194-9203, 2018. and B.1.1.)
doi: 10. (009CCVPR.20880009 (Cited on pages 3.2, A,
[91] Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. In International Confer- ence on Learning Representations, 2022. URL https://openrevier wttp:/rrmeeeeetteeeeeeeeee 9jsZiUgkCZP. (Cited on page 2.3.)
[92] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pages 4791 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1..18653//1/P11-4447 URL https: / petps://aclantho/// awlaaaholllcoooocaoa wwtp:19- page 5.4.2.)
(Cited on
[93] Yu Zhang, Peter Tiño, Aleš Leonardis, and Ke Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence, 5(5):726-742, 2021. doi: 10.1109/TTCC2222111000066
(Cited on page 1.)
[94] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556-1577, 2024. doi: 10.1 162/tacl_a_00704 URI. https: http:////caaathlog//oee org/ 2024.. facl-1.85/ (Cited on page A.)
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We link each contribution in the paper's contents when we theoretically or empirically justify the claims.
19

---

Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, for each theoretical result, we provide a complete set of assumptions and correct proof. Proofs attached in the appendix, Appendix B. We link the relevant appendices in the main body for the reader.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
20

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we fully disclose all the information to reproduce the main experimental results Section 5 and the appendices mentioned within that section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and the instructions reproduce the experiments provided the GitHub: https:/ //gi //gitttuu com/Dhruve hat//Drruaaassyaagg hatrap/mmmoii...... Moreover, the data sets used open-sourced, and details on how obtain them are provided our GitHub.
Guidelines:
21

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
public/ wwtpc//ggge///eeeeeeeeeeee haaia//ggiiiee//eeummiiii
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwwp/pullc//eeeecccceccecee wwtpcc/ggg//e/eeee Swt.d/eudieissssteeoeeeeaae coi.iissssiiyyyyyyocc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details to understand the results and reproduce the results are provided in Section 5 and the appendices mentioned therein.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: For experiments where error bars are relevant and computationally feasible, report them.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We sufficiently deseribe the compute resources used for experiments in Section and the appendices referred to within the section, specifically, Appendix C.6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https pottps://nete httpp::////uuis....// httpp/pbll.c/hicssiiieeeeeeeeeee
Answer: [Yes]
Justification: We go through the NeurIPS Code of Ethics and confirm that adhere to them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We do not discuss the societal impact of the work performed in this manuscript since this is foundational research and not tied to any particular applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release generative models or data in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit original owners of assets used in this work appropriately through citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
25

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We do not use LLMs to develop any methods presented in this work. We clarify further in Appendix E.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : http://eess/./eeeeeeeeeee httpe//rips..ccoooeeeeeeeeeeeeee / for what should or should be described.
26

---

APPENDIX
This appendix and their takeaways are summarized below for ease of navigation:
1. Appendix A contains additional related work and description of the gaps in existing literature addressed in our work.
2. Appendix B contains proofs and discussions not presented in the main body. In particular, we present the following:
In Appendix B.I, we provide the proof for Theorem 3.6. We also state and justify the assumption used towards proving the Theorem.
In Appendix B.2, prove the properties of Subset Fidelity stated in Lemma 3.2.
In Appendix B.4, we prove the optimality of the naive algorithm in selecting the k-MFS Optimal set. While the assumption of uncorrelated features might not hold under practical scenarios, this result provides an indication that the method could result in effective identification of critical model components in practical settings. Our experiments in Section 5 and Appendix C practically demonstrate the empirical efficacy of the methodology.
3. Appendix contains additional experimental validation that make the following points:
In Appendix C.I validate the existence of HiFi sets.
In Appendix C.I.I, show the results of the full Monte-Cart experiments
more models and datasets to strengthen our answer for Question (Q1).
In Appendix C.1.2, we conduct counterfactual experiments to validate if the sets computed by proposed method are disproportionately responsible for predictive performance. This emphasizes the effectiveness of Subset Fidelity addition to our theoretical results in Theorem 3.9.
In Appendix C.1.3, we empirically discuss the sensitivity of the estimation of Q" This ablation study shows that the Subset Fidelity score robust to the number of samples used for estimation. Among the datasets used, we see that we require at most 200 synthetic samples per class for accurate estimation.
In Appendix C.3, we study the effect of quality of synthetic samples on our proposed method. We find that higher quality data leads to an improved sparsity accuracy tradeoff.
In Appendix C.4 we provide pruning results on additional datasets strengthen our validation of Question (Q3).
In Appendix C.4.1 we show that each sub-module of our model modification is critical towards successful model modification.
Appendix C.4.2 we compare the pruned ImageNet models of ModHiFi-P against SoTA data-free structured pruning DFPC [56] to compare layer-wise sparsity to see where is improved speedup coming from. In Appendix C.4.3, we justify the appropriateness of our baselines for the LLM pruning baselines.
In Appendix C.5 we provide additional unlearning experiments with different models to strengthen our validation of Question (Q4) and discuss unlearning with finetuning. We validate that ModHifi-U performs competitively without finetuning against baselines. Moreover, with very few epochs of finetuning over synthetically generated samples, we achieve complete unlearning, as opposed to our baselines who fully- filet-fe on entire training set.
In Appendix C.6, we discuss implementation and compute platform details and addi- tional timing measurements, to improve replicability of our empirical results.
In Appendix C.7 we discuss hyperparameters used for training, to improve replicability of our empirical results.
4. Appendix D contains additional algorithmic details including
Appendix D. clarifies of the role of Lipschitz Constants in our algorithms since they are critical to Theorem 3.6.
Appendix D.2 presents practical details as to how we implement fidelity estimation in our model modification algorithms.
27

---

Appendix D.3 presents the computational complexity of estimating fidelity. The fidelity
linear in number of layers as opposed to the component identification module of SoTA in data-free structured pruning, which is quadratic in the number of layers.
A
Related Work
We present short literature review in Section 2. In this section, we discuss recent related work on structured pruning and unlearning not discussed in the main body.
Structured Pruning Structured pruning has been widely researched, with a wide variety of methods proposed for it [27]. Unlike unstructured pruning, which sparsifies the weight matrices without changing the architecture of the model [6, 15, 16, 38, 73J, structured pruning enables immediate improvements in real-world performance measures such as inference time and memory footprint without requiring specialized hardware or software [27, 51, 59]. variety of methods have been proposed for structured pruning of convolutional networks, including using norms of weight tensors [39], directional derivative scores [51, 52, 66]. feature map ranks [41, 70]. coresets [4, 40, 77], discriminative ability of filters [42, 54], and reconstruction error [11, 67, 90]. However, modern neural networks possess complex interconnections, making them difficult to structurally compress [13, 43, 56], for which some recent algorithms have been proposed that use gradient information [43] or bounds on the reconstruction error [56, 90]. Moreover, pruning without access to either the training data the loss function is an increasingly important area of research, for which some works have been proposed that use the discriminative ability of filters as a saliency [42, 54]. However, none of these works address the problem of pruning large language models.
Pruning of Large Language Models (LLMs) has garnered significant interest in recent years [94]. A variety of unstructured pruning methods have been proposed, such as [17, 71J. However, these methods do not provide direct improvements on inference time and memory footprint. Thus, the problem of pruning models with structural interconnections has naturally been applied to pruning LLMs as well, in works such as [2, 46, 47, 84]. A key drawback of these works is that most are not applicable to CNNs or other kinds of models. Our work proposes a unified framework for both pruning models with complex interconnections, including transformers and ResNets, as well as classwise unlearning.
Classwise Unlearning Machine unlearning has gained significant interest in recent years, both for data privacy concerns as well as connections to continual learning [7, 29, 57, 81]. Machine unlearning typically categorized into exact and approximate unlearning [86]. Exact unlearning involves training models from scratch without the forget data (the data to be forgotten), or by training modules or experts on subsets of data [85, 87]. Approximate unlearning, on the other hand, refers to techniques that approximate exact unlearning via various approaches [30, 86]. Machine unlearning can be further classified into sample unlearning (wherein individual samples or random subsets of samples are (6learned))44, 78] or classwise unlearning (where classes or concepts are unlearned) [18, 24]. In this work, we focus on classwise unlearning.
A variety of approaches have been proposed for classwise unlearning [24]. Popular methods include fine-tuning the model without data from the forget class [20, 82], gradient ascent on the forget set
[24, 75], iitillatioonbbasee approaches [37], and influence function based methods [30]. More recent work studies using sparsity for machine unlearning, such as [32], which first sparsifies the model, and then applies a fine-tuning-bart based unlearning algorithm, or [53, 80]. which identify dlassdissrimminttiee filters in CNNs, and removes them for unlearning. Two key drawbacks of prior art, however, are: first they exclusively address classwise unlearning, and do not address wider problems of model modification. Second, all prior art assumes access to the original training data. Our proposed approach for classwise unlearning differs from prior art because it only requires synthetic class data, uses a variety of granularities for sparsity in unlcarning, and is part of a unified approach to model modification.
B Proofs
In this section, we restate the formal statements made the main body of the paper and present the proof's omitted in the main body. We follow the notation defined in Section 2.
28

---

B.1 Proof of Theorem 3.6
We now provide a proof of Theorem 3.6. We first state the properties of the normalization layer and provide empirical evidence to justify their validity, followed by restatement of the theorem and its proof.
Definition B.1 (RMSNorm and LayerNorm). Consider the (-th layer parameterized by Y. R For an input ((x) E R2, the output y E R is defined as:
NM(Ø(z)) =
+ B',
where Mob(æ).
|z|||
Here, O denotes the Hadamard product. For RMSNorm, M Lg. For LayerNorm, M = I 11.11 (the centering matrix).
Normalization layers are not globally Lipschitz continuous due to the singularity at zero. However, they locally Lipschitz domains bounded away from the origin.
Definition B.2. A function f I + R is Lipschitz continuous in its domain if there exists positive scalar constant L such that
fa f(y)ll2 s L|| ulla Var, y R.M
for all æ, in the domain of f.
Lemma B.3. Let X, = {x E IR ||x|| 2 < 0J. Define the map f X, - Sd-1 as f(x) [x] Then is Lipschitz continuous on X, with constant T.g = 1/r. That is,
[f(x) - f(w)ll2 S - || yll
Proof. For any x,y E Xy,
[I((x)) - /())) = [12]]
//
2 2 2c
3/1
Simultaneously,
x y112 = ||x|| + |yy|
2x T
||x||
|
x
= =||x|||y||
2
yy
x
x
|
Using the AM-GM inequality + 1/a 2 2 for > O, and noting that ||x|| ||>|| 2 r:
2 2- x
X 113 ||
||
f()) -l1f(x) s(x)
|x|| 1yll
Rearranging terms completes the proof.
Using Lemma B.3, we can show that the operation performed by normalization layers is Lipschitz continuous in Corollary B.4.
Corollary B.4. Let the input to the 1-th normalization layer satisfy [M 2 0 all æ. Then, the normalization layer satisfies
[NM(6(z)) NM(6 < | |M]l2lb((x) xb())1
Note that [|M]2 for both RMSNorm and LayerNorm.
Proof. Let u(x) = Mx. Then |JN((x) - NM(y))la = l @
u(x) |
u(y) T
lla
bb0 [M(x -2)l22 by applying Lemma B.3 to complete the proof.

---

While the lower bound assumption |z| 2 r is technically not guaranteed for all x E R we empirically verify that for trained networks, activation norms are strictly bounded away from zero. This validates the local Lipschitz property in the region of interest. We show the layer-wise minimum norm of the pre-LayerNorm representations in Figure 3, estimated on 100 samples from the Alpaca dataset. For various models, we observe the lower bound to be between 0.2 and 60. For clarity of exposition, we only show the layers with the largest and smallest values, along with 5 randomly selected layers. Code for generating these plots can be found in Appendix C. We also observe that this value tends to increase for layers deeper in the network, and leave the utilization of this observation to future work.
B.1.1 Main Proof
We first state well-known fact about Lipschitz functions. We then restate and prove Theorem 3.6. Fact 1. A function = 1 fL- ....o I where each I is Lipschitz continuous with Lipschitz o constant L', is Lipschitz continuous with Lipschitz constant I L'.
Theorem 3.6 (Local to Global), Consider network Ng as defined in Section 2.1. Let M' be a mask modifying parameters at layer I, and let m be the mask vector for output channel c. Assume there exist scalars ? > 0 for all layers E. >> 1 such that ( (X)]l 2 rf almost surely. Then,
Cont
E (X) NoG (X)13) SO
ΣE
Y
Σ 4
(X)11)
(2)
Proof. Consider a network as defined in Section 2.1. Let No fL ... f f(11:1 where fl-lil = fl-1 o o.... fr. Under standard assumptions on the smoothness of activations [90], each layer fl Lipschitz continuous with Lipschitz constant Lj. From Fact 1,
(X)) NooM (X)1I
s
1 E>1
Σ E Y'()
Σ mei (X)]I
By taking an upper bound on the Lipschitz constants of each layer in the composition, we see that the subnetwork after layer I has Lipschitz constant of least C" Lg. Where, for convolution-bant networks,
hil of
C
max 0
nd
I
max
and for transformer models,
hill max
C - I
pe
F>1
The expected squared error the final output Is:
(X) NN NXoM(X)]11 ( -
We decompose the layer output by channels e [cont]. The masked output for channel is Y! E, A. where mes E (0, 1} are entries of M'.
Y 8'X)
Σ
E
Σ
X
Σ
Ma A (X)
i=1
E
m
A
30

---

Let v. = 1 m. be the indicator vector of removed components, Expanding the squared norm:
E
Σ u
(
=
Σ Ves (X), A2)(
Vot (Qo) v Qive
Substituting this back completes the proof.
B.2 Proof of Lemma 3.2
In this section, we prove the properties Subset Fidelity stated in Lemma 3.2. We restate the definition of fidelity score and state a proposition. We then restate the proposition and provide proof.
Definition 3.1 (Subset Fidelity). The fidelity subset of components S output channel e is defined as
[cn]] layer / for
(1 1 (X Eiec 4 (x))1 E (X)
(C) = max
where 6% is the compensation term.
Lemma 3.2 (Properties of Subset Fidelity). For any subset C S [c, in layer 1, (Boundedness)
< (C) < and (Monotonicity) If D C, then FS FS (C).
Proof. Let L(,; C* = [X - EiEc (X)11) The fidelity is FC) = (
L(6:C)
1. Boundedness: Since the norm is non-negative, C),;C) = FS < 1. Selecting 5 = 0 yields ((; C) |[[ Y Since the minimum is bounded by this value, the ratio is < 1. so FS 2
2. Monotonicity: Let D C C. The optimization for equivalent to optimizing over C with the constraint 6, = 0 C D. Since D C. the feasible for D is subset of the feasible set for C. Therefore, ming (8,; C* S ming L(8'; D). A lower minimum error implies a higher fidelity score. Thus. (C) 2 (D).
B.3 Proof of Proposition 3.8
Proposition 3.8 (Compensation and Singleton Fidelity). For the la reconstruction the optimal compensation term 8. which is the value at which the fidelity score is computed according to Equation (1) for a subset C, given by. (C2.CC.0)--)))CCCC....
(1 1
iiEE
(c) -
(FS)
ii
where Q¹ E Re xe,n is the component similarity matrix (CSM) for channel c, with entries (Q)));
EK (X), A (X))). The singleton fidelity scores are:
$ (())()))) = E YY
(X) - a A (1
d EXY A
(3)
E
Y!
(X)
E A (X)
Proof. Define the error e(X) Y'(x) Eiec ( We minimize (()) E[]le(X)
8, where u is supported on the full set of indices but we constrain 6, = 0 (so 1) for & C. The objective is:
Recall that (X) Den A! (X). Let
((u) =E E
Σ
u (X)
= uu TQu..
31

---

We partition indices into C and C. Decompose as [uct uz]. The constraint 6; for i C implies u = lz. We optimize with respect to ug:
J (uc) = [uc 1 co
Qcc Qce
uc 7t.
uuQQuou Qcouc Qcc +
const.
This is convex quadratic function, whose optima can be computed by taking the gradient wirt ug and setting to zero:
2Qceuc 20clo = 0 = u 2cole
Recalling 8c 1c - uc, we obtain:
8c 1c + Qcc acc
This matches Equation (3).
B.4 Proof of Theorem 3.9
In this section, we prove the optimality of the naive algorithm in selecting the k-MFS Optimal set. Theorem 3.9. Consider oulput channel in the thh layer of a network described in Section 2. I. Let the s'as be defined according to Equation (3) and St be defined according to Definition 3.5. Let St. = =i s 2 {(K)} where S(k) is the hth largest value of s'c- Assuming that there are no ties, 18 k. If (X) Al (X) vi # j. then S Stt
Proof. The assumption [A ( = 0 for # implies that the Component Similarity Matrix Q'. is diagonal. Let matrix is diagonal. For any subset S, the optimal compensation 8* for diagonal Q simplifies. The reconstruction error for subset $ is minimized when we perfectly reconstruct the components in S (since they are orthogonal to components in S). Thus, the residual error comes purely from the removed components S:
(Q') )N = B|| 0. This implies that the component similarity
min E
Y((X) Σ A,A! (X) |
=E Σ A (x) JES
= Σ SQS
9s5-
iES
The Subset Fidelity is:
Σ Ejgs qn
Eies lii Tr(QQ))
(S) =
Σx qkk
Similarly, the singleton fidelity score for component i is d'3s
The optimization problem:
TTQ
S
FS arg max 1
qu-
is
This linear objective is trivially maximized by selecting the & indices with the largest qu values. Since S/ X lis this is equivalent to selecting the top-k singleton fidelity scores.
Remark B.5. While the assumption of uncorrelated features might not hold under practical scenarios, this result provides an indication that the method could result in effective identification of critical model components in practical settings. Our experiments in Section 5 and Appendix C practically demonstrate the empirical efficacy of the methodology.
C Additional Experiments
In this appendix we detail additional results and ablations.
1. We elaborate on the Monte Carlo simulations of Equation (K-MFS) across multiple models, as well as the efficiency with which Subset Fidelity estimates this while heing robust to data samples. We present noising and counterfactual results to demonstrate this.
32

---

Selected Layer Norm Distributions
Selected Layar Distributions
23.54
1omms
10"
2.23
1.54
10
10
11 Layer
15
16
19
27
Laye
(a) OPT-125M
(b) OPT-350M
Selected Layer Norm Distributions
Selected Layer Norm Distributions
10
3.04
Nome
21.39
Norm
30
10
10
10
27
20
35
Layer
Laye
(c) OPT-1.3B
(d) Llama-3.2 2-1B
Selected Layer Norm Distributions
Selected Layar Distributions
10
25.50
10a
Noms
Norms
30
107
27
29
51
57
96
27
51.
Layer
Laye-
(e) Llama-2-7B
(I) Llama-3.1
Figure 3: Boxplots for the distribution of norms of inputs to normalization layer. Minimum value indicated in Red, showing that is at most 5. Y-axis is log scale.
2. We discuss the synthetic samples used our vision experiments and the effect of their quality the algorithm.
3. We provide additional pruning and unlearning experiments for variety of architectures.
4. We provide details our compute platform, hyperparameters, and training procedure for our experiments.
Our code is available attp::///ggtabb..om/bbrvvvaaaaeg
33

---

combinatorial C.1 Validating Subset Fidelity and HiFi Sets
C.1.1
Monte Carlo Experiments
In this section, we provide additional details regarding Figure 1 and demonstrate this behaviour across architectures.
Ideally, one would solve Equation (K-MFS) exactly to compute those sets that have optimal recon- structive ability at different sizes. However, since enumerating across subsets problem, we instead approximate solution by randomly sampling 1000 sets of the given size and compute the maximum across these samples. This will always provide a lower bound for the "true" curve.
Solving Equation (K-MFS) allows us to compute the optimal (k: ())-))HHH sets, since this captures the relation between 77 and k; i.c the tradeoff between sparsity and accuracy. We observe that in many layers (at least 50% of the model), across multiple models, there are sets which contain at most 20% of components but have subset fidelity of around 0.8. We also observe that the difficulty of the task increases (CIFARIO to ImageNet), fewer layers exhibit this sparsity, validating the assertion that networks trained harder problems less overparameterized.
show this for ResNet-50 trained on CIFARIO, CIFARIO0. and ImageNet. Figure shows this for OPT-125m model using 128 samples of WikiText and 100 random subsets instead of a 1000. Due to the expensive nature of this experiment, we are forced to use only 100 random subsets for each size, leading to noisier curve. However, it is clear to see that the general trend continues to hold for several layers, especially for the "down-projection weight matrices, which are the focus of pruning algorithm for LLMs.
Figures 4
C.1.2 Counterfactual study of HiFi sets
We compare the effect of removing HiFi components from a layer with the effect of removing a random subset of the same size. For ResNet-50 model trained on CIFAR-IO, when around 22% of HiFi components are removed, the accuracy drops by around 70%, whereas removing a random subset of the same size decreases the accuracy by 32%. Note that there is roughly 1% decrease in accuracy when only 22% of the non-Hit non-HiFi components are removed. This indicates that components classed as "High Fidelity" have significantly higher impact on the model's 's predictive performance than those with lower fidelity scores.
C.1.3 Robustness of the Fidelity Score
In this section, we perform ablations on the number of samples required for estimating the fidelity score and show how it reacts to additive noise on the model's weights.
In Figures 8 and 9 we show how different data sizes affect different layers ResNet50 model trained on CIFAR1O and ImageNet, respectively. Each data size is selected over random seeds, with error bars shown. For clarity, we show only subset of layers and provide plots and code generate them.
We observe that the values remain stable for 0.2%, 0.5%, and 2% of the data selected, indicating that the model is robust to the number of samples selected.
We also observe the effect of training in these graphs. In untrained models, almost all components have very small fidelity scores with a sharp increase for some values. This indicates that HiFi components are function of training, with the well-trainedness of the network being prerequisite for their presence
When investigating the effect of adding noise to the weights and its effect on accuracy and the fidelity score, we observe that adding zero mean noise of larger standard deviations, starting from 0.005 to 0.05, decreases the fidelity of components, with noisier weights behaving more like untrained models. We test this on ResNet-50 trained on CIFAR1O and present the results in Figure 10. Again, we present only random subset of the layers for clarity.
34

---

Table 5: Effect of data quality when pruning ResNet-55 on CIFARIO
FID
Diffusion Steps
Accuracy
FLOP Reduction
Param Reduction
85.80
86.27
2.63x
2.66x
35.58
5
90.75
2.78x
2. 2.78x
14.42
90.39
3.50x
3.60x
C.2 Constants in Theorem 3.6
In this section, we provided worst case and average case estimates of the constant C₁ in Theorem 3.6. In Figure 1aa we plot the constants obtained in the proof of Theorem 3.6 Appendix B.1 for a ResNet-50 trained on ImageNet, and observe that the values can be very large (38 orders of magnitude). However, it is important to note that these worst case guarantees, and that these constants are much smaller in practice. In Figure 11b, we compute the ratio between the global
| Eieo m A (x)]l" and observe that these values are indeed much smaller (10-50) for random values of M for the expected square loss.
error, E [X) xX;
[[?] and the local error, >ee E
C.3 Discussion on the synthetic samples used in the experiments
We describe the synthetic datasets used in our vision experiments simulate distributional access. Randomly selected example images are provided in Figure 12. For NLP tasks, we use WikiTeet and Alpaca datasets [48, 74] which are standard in this field.
C.3.1 CIFARSM
For experiments with the CIFARIO dataset, CIFARSM, dataset containing 6 million synthetic CIFAR- 10-like images sampled from Diffusion model and labeled by Big-Transfer model [55], which we randomly sample 10,000 samples from each of the 10 classes create dataset. This dataset has an FID [26] of 15.95 with respect to the CIFAR10 training set. This dataset is obtained from here.
C.3.2 CIFAR100-DDPM
For experiments with the CIFAR100 dataset, downsample to contain 1,000 samples from each of the 100 classes. This dataset has an FID of 4.74 with respect the CIFAR 100 training set. We randomly sample 1,000 samples from each of the 100 classes to create our dataset. This dataset is obtained from here.
[23], which randomly
C.3.3 Effect of Data Quality
To study the effect of data quality on the performance of our algorithm in vision tasks, we apply the pruning algorithm using synthetic datasets based CIFAR10 generated with different FIDs. We use a diffusion model [33] to generate 3 datasets of differing quality by changing the number of diffusion steps (4,5, and 6). We report the results of pruning algorithm with different quality datasets in Table 5. We observe that higher quality data leads to an improved sparsity accuracy tradeoff.
C.4 Additional Pruning Experiments
We present additional pruning experiments Tables and 7.
C.4.1 Ablation of weight compensation and BatchNorm correction
In this section, we perform ablations for each component of our pruning algorithm, simple pruning, correcting batch norm statisties and weight compensation. We report our results for pruning ResNet- 50 on CIFAR in Table We observe that each component allows for a better accuracy sparsity trade-off.
35

---

Table Comparison of ResNet-50 pruning for CIFARIO and CIFAR100. ST Synthetic Training. Lec. training using synthetic samples.
Dataset
Algorithm
Accuracy
FLOP Reduction
Param Reduction
Unpruned
94.99
1x
1x
DFPC
90.25
1.46x
2.07x
L2 La w/ ST
15.91
4.07x
4.71x
CIFAR10
90.12
4,07x
4.71x
Ours
91.02
4.07x
5,36x
Unpruned
78.85
1x
1x
DFPC
70.31
1,27x
1.22x
L2
16.77
1.93x
1.40x
CIFAR100
L2
73.83
1.93x
1.40x
Ours
70.93
1.93x
1.38x
Table 7: Comparison of ResNet- 101//GG-1//G9 pruning on CIFARIO and CIFAR100. ST = Synthetic Training, i.e. training using synthetic samples.
Datasel
Model
Algorillun
Accuracy
FLOP Reducctoo
Param Reduction
Unpruned DFPC
72,02
1x
1x
1,26x
1,50x
CIFAR-100
VGG19
La
56.46
1.50x
2,40x
La we ST
72.42
1.50x
2.40x
Ours
70.26
1,51x
2.31x
Unprined
ReaNet-101
DFPC
89.80
1.53x
1.84x
La w/ ST
90.49
4.20
5.29x
CIFARIO
Ours
91,20
4.21x
4.79
Unpruned
93.50
1x
1x
DFPC
1.46x
VGG19
Lg w/ ST
89.23
2.39x
9.19k
Ours
2.39x
5.52x
C.4.2 Final ImageNet Pruned Model
In Figures 13 and 14 we compare the final pruned models for ResNet-50 on ImageNet with DFPC [56]. We observe that our pruning algorithm removes more channels in later coupled channels than DFPC leading to higher gains in sparsity.
C.4.3 Baseline selection for LLM Pruning
We choose ShortGPT [47] and SliceGPT [2] as baselines against which we compare ModHiFi. We do so for broad reasons: all three methods together represent three different granularities for conducting structured pruning for LLLSs, and both ShortGPT and SliceGPT are the state-of-the-art within their respective lanes.
The three different granularities are
Layer pruning: Entire layers (i.e. transformer decoder blocks) are removed from the network. This is viable since transformers are constant width networks, i.e., there are no architectural restrictions to the ordering or number layers. ShorGPT falls within this granularity.
Table 8: Ablation of different components
BatchNorm
Compensation
Accuracy
FLOP Reduction
Param Reduction
No
93.37
1.61x
1.53x
No
Yes
93.49
2.21x
2.17x
Yes
No
93.17
2,53x
2.39x
Yes
Yes
93.76
3.22x
3.30x
36

---

2. Embedding pruning: The width of the network (i.e. the embedding dimension) is pruned at uniform rate across the entire network. This entails form of feature selection: along with weight matrix pruning, one also has to prune the corresponding dimensions from the feature matrix being fed into every layer. SliceGPT falls within this granularity.
3. Hidden dimension pruning: Here, the number of layers and the width of the embedding are left unchanged. Instead, one prunes the hidden dimensions within the modules that constitute a transformer decoder block. ModHiFi falls within this granularity
We would like to emphasize that both SliceGPT and ShortGPT are designed to operate on Transformer models, and as such are able to leverage specifics of the architecture their advantage. In return for this specificity, however, they trade off the ability to generalize to CNNs, something that ModHiFi does with ease due to its archilecture-agnosiic nature; the only assumption made by the Fidelity Score is that the components being scored belong to linear layers.
C.5 Additional Unlearning Experiments
We report additional experiments in Table on class unlearning on different architectures. For VGG-19 networks, we remove the HiFi channels for the forget class of the last 12 convolution layers. We also compare our work with DisCEdit-U from [53] wherein we remove discriminative components from the last 8 convolutional layers. We use a custom implementation of the algorithm for our VGG19 and ResNet50 models for CIFARIO, as those models are unavailable in the codebase of [53]. We also compare our work with DisCEdit-U U on ResNet50 trained on CIFARIO as well, which present in Table 10
We show that unlearning method achieves similar superior performance that of [53] without fine-tuning. Moreover, unlike [53], our approach uses only synthetic samples, showing the efficacy of work in classwise unlearning, in the absence of training data.
Unlearning with finetuning Here we compare our method with 3 additional epochs of finetuning on synthetic samples of the remaining class data. Although this setup does not fall into the setup of the work since we do not assume access to the loss function, we provide these results to indicate that using very few synthetic samples can perform perfect unlearning. We present these results in Table 11, where we observe almost perfect unlearning for both ResNet-50 and Swin-Transformert
Unlearning with baseline budgets In this section, we compare our method when allowing for the same amount of finetuning as [32], with both synthetic data and training data access. While this violates our assumptions about loss function and training data access, we present these results to provide a fair comparison of our algorithm when run within the same constraints as our baselines. Our results can be found in Table 12 for the Swin Transformer.
C.6 Compute Platform
Implementation Details We implement our proposed methods in PyTorch [58] and use Hugging- face's transformers [83] for LLM implementations.
Inference time measurements We follow the inference time measurement setting of [56, 66). Inference time is the time taken for model compute the forward pass for an input and does account for loading data into memory. We compute the inference time for a batch of 640 random tensors for GPU and 64 for CPU. 100 iterations are used for warm up, after which the inference
Table 9: Class unlearning CIFARIO for VGG19
Model
Algorithm
Forget Accuracy
Remain Accuracy
93.50
93.50
VGG19
DisCEdit-U [53]
2.39
84.2
Ours
0.86
77.85
37

---

Table 10: Class unlearning on CIFARIO for ResNet50
Model
Algorithm
Forget Accuracy
Remain Accuracy
94.99
94.99
ResNet50
DisCEdit-U [53]
3.2
91.6
Ours
0.2
92.98
Table 11: Class unlearning with 3 epochs of finetuning synthetic samples
Model
Remain Accuracy
Forget Accuracy
ResNet-50
93.1
Swin-T
83.6
0.1
time is averaged over the next 1000 forward passes. We compute CPU and GPU measurements on machine whose specifications can be found in Appendix C.6.
JIT Compilation
We present inference time numbers with JIT compilation on Pytorch [58].
Hardware Table 13 details the hardware we use to conduct our experiments. Values in (*)
indicate reported values obtained from https://wwet amd. cww/en/prodeeeeeccooccom cctpe/eaccceeerrrss///
instinct/mi httmnn/////////mm//eme/ 200/mi210 html. This machine runs Ubuntu 22.04.3 LTS with kernel 6.8.0-40- generic with the hardware in Table 13. Our software stack comprises of Python 3.12.8, PyTorch 2.5.1 built for ROCm 6.2, and torchvision version 0.20.1 built for ROCm 6.2.
Inference times measured generic on the hardware specified in Table 14. The software stack used for inference consists of Python 3.12.8, PyTorch 2.5.1, and Torchvision 0.20.1 for CUDA 12.3.
machine rumning Ubuntu 20.04.1 LTS with kernel 5.15.0-91-
C.6.1 Module-level Time Consumption
In this section, we break down the time each component of our algorithm takes. For 2000 samples batched into batches of size 64, when running the algorithm a ResNet-50:
Computation of fidelity scores takes between 32GB to 51GB of VRAM, and between minutes to minutes, GPU of machine 13, across data from CIFAR1O, CIFAR100, and ImageNet.
Computing 8 across 4 GPUs using average of 60GB per GPU takes 60 minutes for CIFARIO/100, and 90 minutes for ImageNet, averaging to roughly minute per layer.
C.7 Hyperparameters and Training Procedure
C.7.1 Hyperparameters for Experiments
typically set the percentile of removed components to be between 0.01 to 0.2. We randomly select 2% of our synthetic samples to select data for vision tasks and select 128 samples for NLP tasks.
Table 12:
Class unlearning with 10 epochs of finetuning
Approach
Dataset
Forget Accuracy
Remain Accuracy
Jia al. [32]
CIFAR10 Train
1.20
90.69
Ours
CIFARIO Synthetic
0.37
84.63
Ours
CIFAR10 Train
0.00
91.1
38

---

Table 13: Specifications of GPU hardware used for computation
CPU Model Name CPU(s)
AMD EPYC 9654 96-Core Processor
192
Thread(s) per core
1
Core(s) per socket Socket(s)
96
2
NUMA node(s)
2
CPU MHz(MMx)
3707.8120
Lid & Lli cache
6 MiB
L2 cache
192 MiB
L3 cache
768 MiB
RAM
1.48 TiB (DDR5, 4800 MT/s)
GPU Model name
Instinct MI210
GPU(s)
GPU Architecture
AMD Aldebaran
Dedicated Memory Size(per GPU)
64 GB
ROCm Version
6.0.2
Peak FP32 Performance*
22.6 TFLOPs
Peak FP64 Performance*
22.6 TFLOPs
Memory Clock
1.6 GHz
Peak Memory Bandwidth
1.6 TB/s
Table 14: Specifications GPU and CPU hardware used for computing inference time
CPU Model Name CPU(s)
Intel(R) Xeon(R) Silver 4216 CPU 2.10GHz
64
Thread(s) per core Core(s) per socket Socket(s)
2
16
2
NUMA node(s)
2
CPU MHz(Max)
3200
Lid & Lli cache
I MiB
L2 cache
32 MiB
L3 cache
44 MiB
RAM
62.53 GiB (DDR4, 2666 MT/s)
GPU Model name
NVIDIA GeForce RTX 2080 Ti
CUDA version
12.3
GPU(s)
8
GPU Architecture
NVIDIA Turing
Dedicated Memory Size(per GPU)
11.81 GB
C.7.2 Training procedure
Pretraining procedure: For CIFAR10 and CIFAR100, train models using SGD with momen- tum factor of 0.9 and weight decay 5f x 10 4, for 200 epochs using Cosine Annealing step sizes with initial learning rate of 0.
ImageNet post training: For ImageNet, we use off-the-shelf pretrained models from Torchvision
[58]. We train the model for 3 epochs after each iteration of pruning with learning rates of 0.1, 0.01, 0.001. After the pruning ends, we finally train the network for 160 epochs with batch size of 512. We use the SGD Optimizer with momentum factor 0.9 and weight decay of 1 10 and start with an LR warm-up for 10 epochs, followed by Cosine Annealed step sizes with an initial learning rate 0.1 with Cutmix and Mixup augmentations.
Le Post training procedure: For the synthetic training experiments mentioned in Section 5, we first prune the model using T22 norm as the grouped saliency to a similar sparsity as our algorithm. We then train the model using 50000 samples from the synthetic dataset for 100 epochs with a batch
39

---

size of 128 using SGD optimizer with momentum factor of 0.9 with initial learning rate of 0.01 and MultiStepLR learning rate scheduler with milestones 60 and 80 epochs.
D
Additional Algorithm details
In this section, we discuss algorithmic nuances not discussed in the main body of the paper.
D.1 Clarification on Lipschitz Bounds and Their Role
In Section introduced local-to-global error bound (Theorem 3.6) that connects intermediate- layer deviations to changes in the final-layer output, assuming the model is composed of Lipschitz- continuous layers with constants C. This result serves to theoretically motivate the use of local reconstruction error - what we formalize as Subset Fidelity - as a proxy for reconstruction error at the output.
Importantly, we do not compute or estimate Lipschitz constants in any part of our algorithm. Our pruning and unlearning algorithms do not depend knowledge of the values of C.. The bound in Theorem 3.6 is used qualitatively to support the intuition that preserving high-fidelity intermediate representations leads stability in the final model predictions.
Empirically, we find that Subset Fidelity correlates strongly with the effect of component removal on prediction quality (see Figure 1 and Appendix C), even in the absence of explicit Lipschitz bound estimation. This supports our design choice to treat Theorem 3.6 as motivating principle, not an operational tool.
We believe this distinction is important to clarify: while our framework draws conceptual inspiration from Lipschitz continuity, it remains loss-free, and hyperparameter-drre driven in practice, with no reliance on any difficult-to-sstimee estimate constants.
D.2 Additional Algorithmic details on Fidelity Estimation
To efficiently estimate the fidelity of each component at a given layer, we use saliency measure to approximate the fidelity score. This is based the component's contribution to reconstructing the layer's 's output, computed via the inner product between the layer output and the component-spen speciifc activation contribution. This can be written as
Rr = E(Y((,, (x)) (Q), 1) =0', a,, Am:
In networks that include BatchNorm (e.g.. ResNets [25], VGG [69]), refine this reconstruction by centering the activations using the BatchNorm's 's stored running mean. This leads to a modified formulation of the component similarity matrix:
OG = =E - (BIA
These quantities are computed efficiently using modern GPU architectures. The forward activations from a calibration set are batched and evaluated across multiple GPUs in parallel. In sequence-based architectures such as Transformers [44], compute the expectation over all elements in the sequence dimension.
Numerical Stability and Regularization. To compute the optimal linear compensation for modi- fying components, we solve a least-squares system involving the component similarity matrix Q However, this matrix may be ill-conditioned rank-deficient practice. To ensure numerical stability and avoid inversion errors, we add small & regularization term (A = 10 10-4) f to the diagonal before solving.
Behavior HiFi Components During Editing. editing task:
The role of HiFi components depends on the
40

---

We retain HiFi components and discard the rest. While we cannot guarantee fixed sparsity level in the output model (since HiFi components may span all inputs), we observe in practice that reasonable sparsity emerges naturally. For more aggressive pruning, the algorithm is applied iteratively.
Structured Pruning:
Simply discarding low-fidelity components is insufficient. Instead, aim to remove or disrupt the influence of HiFi components that are specific to the forget class. The editing strategy depends on the network type:
Class Unlearning:
In BatchNorm networks, zero out the weights of HiFi components computed per the forget class samples.
In LayerNorm- based networks with residual connections (e.g., Swin-T), we negate the weights of HiFi components. This rotates the forget-class representation in the opposite direction due to the residual path.
The unlearming strategy for Transformer-ban architectures is captured in the following procedure:
Algorithm 2 ViT-Edit-X: Structured Editing for Transformers
Require: Model parameters 0, HiFi components H, coupled channels CC
Ensure: Edited model parameters
if X Prune then
2: for E [c/a] (i (c. E U, do
3:
W c: 0
Ve E [ I E CC
4:
else if Unlearn then
5:
for each layer E CC do
6: W! - 43
V(c, H
7: Return:
Fidelity Estimation in LLMs estimation of scores becomes more challenging. We estimate the the fidelity scores by computing the row norms the regularized Cholesky decomposition Q. The scores are estimated as
Due to the large scale of LLMs and the range of floating point values,
FS({i)) ||
where Q = LL the Cholesky decomposition of Q
We use the Cholesky decomposition since it is efficient to compute.
D.3 Computational cost
Let N be the number of data points used to estimate the saliency and M be the complexity of computing the input contribution layer / for single sample in of coupled channels with m layers. The complexity to compute the set of retained channels for an output channel of a layer is, Haal ON M C² d'). To select the components for the coupled channels, the top p elements for each layer and output channel in them are collected, this costs O Clna(C", log C²,, + tam))). The algorithm shows linear dependence on the number of layers in the network, compared with the BGSC algorithm [56] which has quadratic dependence.
E
Full LLM Disclosure
We employed Large Language Models (LLMs) to refine the text for grammar and clarity. Additionally, LLMs were used to generate auxiliary scripts for data visualization (plots). We confirm that LLMs were not used to implement any of the core algorithms or methodologies proposed in this work.
41

---

Masheon??
Nenno Syyn
Sare wiwpetiiikkkiiee
Mry
12
20
50
100
130
Medi
100
02-
350
20
80
16
25
20
Mettzew
100
30
25
155
24
200
38
18
300
200
40
1.0
750
hdbg
Figure 4: Estimates of Optimal subset fidelity for ResNet-50 on CIFAR1O.
42

---

Masheon??
Nenno Syyn
Sare wiwpetiiikkkiiee
Mry
12
20
50
100
130
Medi
100
02-
350
20
80
16
25
20
Mettzew
100
30
25
155
24
200
38
18
300
200
40
1.0
750
hdbg
Figure 5: Estimates of Optimal subset fidelity for 5e on CIFAR100.
43

---

Narno wontliiitiiioo
Sare ia SigertiiakDoon
13
20
50
100
130
Musi
Nomna
10
0.-
300
20
256
80
166
25
20
Mettarw
10
30
25
155
24
200
18
300
200
40
102
5
1.2
730
hugge
D4/get
0u0ge
Figure 6: Estimates of Optimal subset fidelity for ResNet-50 on ImageNet.
44

---

100-2/m
Figure 7: Estimates of Optimal subset fidelity for OPT-125M.
45

---

Figure 8: Fidelity scores for select layers of ResNet50 trained on ImageNet showing the effect of training and data set size (ds).
030
Figure 9: Fidelity scores for select layers of ResNet50 trained CIFAR10 showing the effect of training and data set size.
3.014
Adkint
3000-
1105
wibee mn coo02o015
aibeel ov sca-a 0915
2053
3003
Inuit Chuana Incks
1134 derie 1nd0
1n2at derice Inde
1:ul Cueree Inder
Sator
ES
2.3 Pqgan.
0.2
(11
400
100
153
20
Figure 10: Fidelity scores for select layers of ResNet50 trained on CIFARIO showing the effect of adding noise
46

---

Wordt esiimates
Empirical estimates of
10a
10
1036
14
109
10
Layer
(a) Worst case
(b) Empirical
Figure 11: Estimates of constants Cj across layers for a ResNet-50 trained on ImageNet.
Automobile (5M)
(b) Deer (5M)
(c) Beaver (DDPM)
(d) Forest (DDPM)
Figure 12: Randomly selected images from the synthetic sets
Figure 13: Number of remaining channels of pruned ImageNet model compared with DFPC (30)
47

---

Figure 14: Number of remaining channels of pruned ImageNet model compared with DFPC (54)
48


Reviews:
[{"review_id": "Id0ifSsGsh", "reviewer": "Reviewer_z8ND", "summary": "This paper presents a new theoretical result bounding the global impact of removing a component from the neural network (in terms of how it affects the loss) with respect to its local impact (the change to the output of that specific component, such as a convolutional filter). This result depends on the neural network being Lipschitz continuous to start with. When that is the case, this result can be used to make it more manageable to perform structured pruning or classwise unlearning. In the first case, we would look for components with little effect if removed. In the second case, we would look for components that mostly affect performance in the class to be unlearned if removed.", "questions": "1) How do you characterize a well-trained model?\n\n2) Where is the evidence claimed in Lines 49-51 that transformers are Lipschitz continuous?\n\n3) There are many other approaches to structured pruning without access to the data or loss function that also measure the impact of components, such as [1-3]. Can you please contextualize your work with those and broaden your literature review accordingly?\n\n[1] https://proceedings.neurips.cc/paper/2020/file/46a4378f835dc8040c8057beb6a2da52-Paper.pdf\n\n[2] https://proceedings.neurips.cc/paper_files/paper/2021/file/e35d7a5768c4b85b4780384d55dc3620-Paper.pdf\n\n[3] https://ieeexplore.ieee.org/document/9878423\n\n4) How large can be the constants from Theorem 3.1? I am wondering if, either in theory or practice, these constants are sufficiently small to validate the approach.\n\n5) There is a certain confusion about complexity classes in the paper: you talk about computing optimal subsets being NP-complete in Line 59, then say that this is a binary quadratic optimization problem in Line 168, mentioned to be NP-complete in Line 169, and then seem to refer again to this same problem being NP hard in Line 193. To be clear, optimization problems are typically NP hard. So my impression is that all other mentions are wrong. Are all of these mentions of the same problem? What is the complexity class of this problem?\n\n6) Related to the item above, can you please rephrase footnote 2 in a clear and more detailed way?\n\n7) What is the intuition for Theorem 3.7?\n\n8) What do you mean by \"the compensation term\" in Line 228? Can you please rephrase that whole sentence for clarity?\n\n9) When you talk about greedy selection in Lines 235-237, do you mean sorting the components from smallest to largest fidelity? That should be made explicit.\n\n10) In Table 1 (which is actually two tables), what is the difference between the two rows for DFPC and Ours on ImageNet? Please make that clear in the paper.\n\n11) For the same table as before, what exactly do you mean with columns CPU and GPU?\n\n12) Across all tables, are the results in each row based on a single model or are they a summary of the results over multiple models with different random seeds?\n\n13) I cannot understand Table 2. Can you please explain it?\n\n14) Please consider these minor comments on the writing:\n- Use \"Lipschitz continuous\" rather than \"Lipschitz\" alone, as that is not very formal and may confuse the reader if read out of context.\n- Replace the word \"tasks\" with something like \"purposes\" in the abstract. Task may sound like the application of the neural network, such as classification for a specific dataset.\n- Separate text from the reference citations with a space. If you do not want the citations in a separate line, use something like \"x~\\cite{y}\".\n- What is \"F\" in Line 259?", "limitations": "There is a brief and reasonable acknowledgement about limitations at the end.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The authors make an interesting case by first presenting general theoretical results, refining them in a usable way, and then applying those results to practical problems. That makes the approach sound and reasonable. \n\nWith that said, I would have appreciated a little more intuition about the results, since I am not in a position to fully vouch for the claims and proofs. Moreover, some parts of the development are not entirely clear, and some of the experimental results are difficult to understand. I go in more detail about those issues in Questions.", "was_revised": true, "final_justification": "I cannot vouch for the correctness of the proofs, but the idea seems sound and I am satisfied with how the authors addressed the points raised.", "rebuttal": "We thank the reviewer for their appraisal of our work and their insightful questions. We particularly appreciate their note on how we refine our theoretical results to apply them to problems of practical importance. \n\nWe also want to take this opportunity to make an important clarification, that pruning and unlearning are consequences of our main finding: **the presence of a small number of statistically significant, high-fidelity components**. While we focus our experiments on the two applications listed, they are simply present to demonstrate that subset fidelity serves as a versatile tool for measuring component importance, and that it arises from the statistical properties of the model’s output alone, as opposed to the specifics of the application for which the importance is being computed.\n\nWe now discuss the reviewer’s questions about our work, and hope that these address their concerns. If not, we are eager to further engage in a constructive discussion towards improving our work and scores.\n\n## Questions\n**1. How do you characterize a well-trained model?**\n\nFor our work, we make no formal characterizations of well-trainedness, and it suffices to treat well-trainedness to mean sufficiently high task performance, as is the case for all the models we experiment on. For our theoretical results, this allows us to leverage Lipschitz continuity in ResNets (for which it is established) and for pre-trained Transformers (for which we provide evidence, cf. our answer to Question 2 below). We will add a clarifying remark in the updated version of our manuscript to avoid confusion for the reader. \n\n\n**2. Where is the evidence claimed in Lines 49-51 that transformers are Lipschitz continuous?**\n\nAssumption B.2 (line 740) allows us to lower-bound the activations of normalization layers, which we then use to prove (in Lemma B.3, line 753) that said layers are Lipschitz continuous. We also provide empirical evidence in Figure 3 of Appendix C (on page 22) for the Lipschitz continuity of these layers across a selection of standard transformer-based models. We will add a reference to these experiments in the main body to enhance clarity.\n\n**3. There are many other approaches to structured pruning ... Can you please contextualize your work with those...**\n\nWe appreciate the reviewer’s note of these relevant works, contextualizing against which has strengthened the position of our work, and we will add them to our Related Work section. For completeness, we provide said contextualization below.\nWe emphasize that our method is not explicitly tailored for pruning, but presents a more fundamental approach to scoring components. However, in our applications to pruning, we differ from the cited works as follows: \n\n*Synflow* [1] targets **unstructured pruning at initialization**, and scores parameters to assess their “trainability”. Our work performs **structured pruning of well-trained** models and scores components according to their contribution towards the model’s output.\n\n*Leo++* [2], an exact compression method, can only be applied to **small ReLU-only feedforward neural networks** and is not generalizable to architectures like CNNs and Transformers, unlike our work. Additionally, they determine “prunability” alone instead of providing a relative scoring of components, with their algorithm being explicitly dependent on training data, while ours only requires distributional access.\n\n*CDG* [3] prunes CNNs while training, unlike our work, which is post-training pruning. Moreover, they score channels through gating the activations of a subset of channels within a layer, which requires access to training data. Our method does not require access to training data and is general enough to be immediately applicable to Transformers trained on language data.\n\nImportantly, **none of these works are directly applicable to unlearning.**\n\n**4. How large can be the constants from Theorem 3.1...**\n\nTheorem 3.1 only posits the existence of constants $C_l$. Empirically, we find that these constants are typically between 10-100 for ResNet50 trained on Imagenet. We will update the manuscript with exact details about these constants and how they are computed. In the proof of Theorem 3.1, we use an upper bound on the Lipschitz constants, which is very conservative. We emphasize that this is a conservative bound, and **reveals the non-trivial fact that even in the worst case, approximating global reconstruction error with the local one only incurs an error that is a linear function of the local reconstruction error.**\n\n**5, 6. There is a certain confusion about complexity classes in the paper: … Related to the item above, can you please rephrase footnote 2...**\n\nThe complexities refer to two different problems: solving k-MFS and minimizing local reconstruction error.\nk-MFS (equation k-MFS above line 197), being an optimization problem, is indeed NP Hard. Minimizing local reconstruction error (below line 151) is exactly equivalent to the optimization variant of Max-Cut, whose decision problem (determining the existence of a cut of size k) is NP Complete. However, Max-Cut itself is NP Hard, and we will make this distinction clear in the manuscript.\n\n**7. What is the intuition for Theorem 3.7?**\n\nIn our work, we make distributional access to data on which the model has been trained. Our work focuses on finding components of a model that are responsible for the model’s predictive power. We show that the problem of finding these components can be reduced to solving the k-MFS problem as stated in Definition 3.6. However, as noted in line 199, this is computationally infeasible. Theorem 3.7 makes the connection between the statistical properties of the data and the solvability of the k-MFS problem. In particular, in the scenario where the input representations of a layer are uncorrelated (a statistical property about the input representations of the layer), we can efficiently solve the k-MFS problem for that layer.\n\n**8. What do you mean by \"the compensation term\" in Line 228...**\n\nThank you for pointing this out. We will update the manuscript with our clarification, which is as follows:\n\n‘Compensation’ refers to the modification in the parameters by the application of the mask, aimed towards recovering performance. In our case, we perform linear compensation, scaling the parameters by the term specified in the second case of Equation FS, namely $1+((\\mathbf{Q}^c_{C,C})^{-1})_{i}^{T} (\\mathbf{Q}^{c} _{C, \\bar{C}}) \\mathbf{1}\\_{n-k}$. In the case of pruning, this corresponds to scaling the retained weights within a layer according to the formula above. Compensation differs from recovery fine-tuning in that it is a closed-form update to the weights. \n\n**9. When you talk about greedy selection in Lines 235-237...**\n\nThank you for bringing this up. Yes, by greedy selection, we mean the process of computing the set containing the **k-largest** singleton fidelity scores. This set represents the set of components that are either retained to perform pruning or discarded to perform unlearning. We will make this nomenclature explicit in Theorem 3.7 (on line 211, where the procedure is introduced) in our updated manuscript.\n\n**10, 11. In Table 1 (which is actually two tables), what is the difference ...? For the same table as before...**\n\nThe two DFPC rows indicate the results after performing pruning for 30 and 54 iterations, respectively, as specified in their paper. The two “Our” rows correspond to the results when running ModHiFi-Prune with different sparsities, which result in different reductions in FLOPS and parameters in each row. The columns CPU and GPU refer to the relative speedups when running the model for inference on CPUs and GPUs, respectively. Specifications of the compute platform are presented in Appendix C.\n\n**12. Across all tables, are the results...**\n\nFollowing standard practices in this literature [4,5,6,7,8,9], the results are computed on a single trained model. We have provided ablation studies on the stability of subset fidelity to noise, data size, and over a variety of trained architectures, which can be found in Appendix C.\n\n**13. I cannot understand Table 2...**\n\nTable 2 lists the performance of Llama-2-7B (in the zero-shot regime) when pruned via the listed algorithms to the specified parameter sparsities. ModHiFi-P-WikiText and ModHiFi-P-Alpaca are two variants of ModHiFi-Prune that differ only in the dataset used for calibration, as described in lines 313-317. Performance is reported for WikiText perplexity (for which lower is better) and for accuracy on standard language modeling tasks [9] (for which higher is better).\n\n**14. Please consider these minor comments on the writing…**\n\nWe thank the reviewer for pointing these out and will incorporate the suggested changes into the manuscript.\n\n**We thank the reviewer for their comments and their thorough review, as they have helped us strengthen our work. We hope to consequently increase our scores and are eager for any further discussion with them.**\n\n--- \n\n## References\n[4] Shah et al. Decomposing and editing predictions by modeling model computation. ICML 2024\n\n[5] Jia et al. Model sparsity can simplify machine unlearning. NeurIPS 2023.\n\n[6] Narshana et al. Dfpc: Data flow driven pruning of coupled channels without data. ICLR 2023.\n\n[7] Ma et al. Llm-pruner: On the structural pruning of large language models. NeurIPS 2023.\n\n[8] Men et al. Shortgpt: Layers in large language models are more redundant than you expect. Preprint - arXiv:2403.03853, 2024.\n\n[9] Ashkboos et al.Slicegpt: Compress large language models by deleting rows and columns. ICLR 2024"}, {"review_id": "raMOlUJipl", "reviewer": "Reviewer_qfdv", "summary": "The paper introduces ModHiFi, a novel framework for modifying well-trained models without access to the original training data or loss function. The authors address two key tasks: structured pruning and classwise unlearning. The core idea revolves around identifying High-Fidelity (HiFi) components which are small subsets of model components critical to predictive performance, using a new metric called Subset Fidelity, which quantifies how well a subset of components can reconstruct model outputs after weight adjustments.", "questions": "See concerns above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n+ Architecture Generalization: The proposed algorithm performs well on both vision models (CNNs/ViTs) and language models (LLMs), demonstrating versatility across different architectures.\n+ Task Generalization: The modification based on the HiFi objective unifies two tasks, classwise unlearning and structured pruning. Specifically: The pruning method (ModHiFi-P) achieves an 11% speedup on ImageNet while maintaining superior accuracy over baselines. The unlearning method (ModHiFi-U) completely erases target classes without fine-tuning and is 10x faster than existing approaches.\n+ Comprehensive Experimental Setup: The experiments thoroughly address the four key questions posed in the paper.\n\n**Weakness:**\n- As shown in Table 2, while the method maintains strong average performance under high pruning ratios, its individual task metrics are not particularly outstanding.\n- The method’s effectiveness relies on the assumption that the target model is well-trained; its performance on poorly initialized or underfitted models remains unexplored and is theoretically expected to degrade.\n- The classwise unlearning performance on Swin-Transformers is subpar, possibly due to insufficient model training.\n- Computing Subset Fidelity requires high-dimensional matrix inversion, which may be computationally prohibitive for extremely large models (e.g., LLMs).\n- The experimental scope is somewhat limited:\n1) No exploration of the impact of k and η (key hyperparameters in HiFi selection).\n2) No discussion on the choice of B (pruning budget) for structured pruning or the sizes of forget set and retain set for classwise unlearning.", "was_revised": true, "final_justification": "The rebuttal addressed my questions. But after checking other reviews, I feel the paper is not good enough for an accept. I will keep my rating.", "rebuttal": "We thank the reviewer for the positive appraisal of our work, in particular, your note about the generalisability of our work to different architectures and model modification tasks.  We hope the additional results and clarifications provided here address the concerns and further demonstrate the significance and robustness of our work. We are eager to engage further with the reviewer during the discussion period regarding our work and, in the process, improve our scores too.\n\n## Response to Questions and Weaknesses \n\n\n**1. As shown in Table 2, while the method maintains strong average performance under high pruning ratios, its individual task metrics are not particularly outstanding.**\n\nWe would like to gently point out that on lower sparsity ratios (10%, 20%), our approach achieves the best performance on 80% of tasks, and is better than or competitive with the nearest baselines at higher sparsity as well. The variable performance on specific tasks can be seen across the baselines as well as shown in Table 2 of our main paper. \n\n**2. The method’s effectiveness relies on the assumption that the target model is well-trained; its performance on poorly initialized or underfitted models remains unexplored and is theoretically expected to degrade.**\n\nWe gently point out that only well-trained models are relevant to this work, as common to the structured pruning literature for both vision and language models, under-trained/underfitted models are not considered for pruning [1, 2]. Moreover, unlearning cannot be defined for a poorly trained model, as such models haven’t learned sufficiently to make unlearning meaningful. \n\n**3. The classwise unlearning performance on Swin-Transformers is subpar, possibly due to insufficient model training.**\n\nWe point out that the unlearning performance results for Swin-Transformers presented in the main paper are **in the more restricted setting without any fine-tuning and using synthetic data only**, whereas the nearest baselines all perform 10 epochs of fine-tuning using the original training data. Moreover, we show in Appendix C that performing only 3 epochs of fine-tuning achieves just 0.1% accuracy on the forget class. \n\nWe also present an experiment comparing the efficacy of our unlearning methods with 10 epochs of fine-tuning, matching those of the baseline [3], both with synthetic data as well as the training data in the table below. This experiment highlights the efficacy of our unlearning method by **demonstrating *perfect unlearning* when using the CIFAR10 train set and superior remain class accuracy as well, outperforming the baseline from [3] when the same extensive retraining and the training set are available to us.**\n\n---\n| Approach | Dataset | Forget Acc. | Remain Acc. |\n|--|--|--|--|\n| Jia et al | CIFAR10 Train | 1.20% | 90.69% |\n| Ours | CIFAR10 Synthetic |0 .37% | 84.63% |\n|Ours | CIFAR10 Train | 0.00% | 91.1% |\n---\n\n**4. Computing Subset Fidelity requires high-dimensional matrix inversion, which may be computationally prohibitive for extremely large models (e.g., LLMs).**\n\nWe clarify that **no actual matrix inversion is required in our work**. The least squares problems are solved by solving linear equations. For language models, we use a Cholesky decomposition of the $Q$ matrices to further improve numerical performance, as discussed in Appendix D. As we have already discussed in Appendix C.5.1, our approach allows us to **compute fidelity scores for all layers in a ResNet in approximately 5 minutes for ResNet-50 trained on Imagenet, and around 1 minute per layer to compute the optimal compensated mask $\\delta_c^{\\star}$.** Similarly, for Llama-2 models, **it takes approximately 30 minutes to compute Q matrices for all of the MLP layers, and approximately 2 minutes to compute $\\delta_c^{\\star}$.**\n\n**5. The experimental scope is somewhat limited:**\n\nWe politely disagree that our experimental scope is limited. Commensurate with recent work [4], we have showcased the effectiveness of our method on two distinct tasks, on both vision and language models. Moreover, in Appendix C, we have provided thorough ablation studies supporting the validity of our hypothesis, highlighting that our key hypothesis holds across data splits and random seeds.\n\n**6. No exploration of the impact of k and η (key hyperparameters in HiFi selection).**\n\nWe have already provided a discussion on the tradeoff between $k$ and $\\eta$ in Appendix C.1.1. In particular, the Monte Carlo simulations presented in Figures 4-7 of the appendix illustrate the tradeoff between $k$ and $\\eta$. Specifically, we plot $k$ vs $\\eta$ for a variety of models, showing how different layers have smaller or larger subsets (of size $k$) that achieve high fidelity scores ($\\eta$).\n\n**7. No discussion on the choice of B (pruning budget) for structured pruning or the sizes of forget set and retain set for classwise unlearning.**\n\nWe point out that we have already provided a discussion on the impact of pruning budgets in Observation 1 (Section 3.2). Furthermore, in Appendix C,  we show, via Monte Carlo sampling, the impact of pruning on different layers in a variety of models. Note that the pruning budget $B$ can be thought of as the same as the size of the high-fidelity set $k$. For classwise unlearning, we only prune a small fraction of parameters, which varies from model to model.\n\n---\n\n**Your valuable review has helped us strengthen our work, especially to show our strong empirical performance on SwinTransformers. We hope to consequently increase our scores, and are eager to discuss any further questions.**\n\n\n## References\n\n[1] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1–124, 2021.\n\n[2] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129–146, 2020.\n\n[3] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.\n\n[4] Chaitanya Murti and Chiranjib Bhattacharyya. DisCEdit: Model editing by identifying discriminative components. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024."}, {"review_id": "FAlB6m0nSE", "reviewer": "Reviewer_Ze6v", "summary": "This paper presents ModHiFi, a layer-wise reconstruction-based method for estimating feature importance in neural networks using synthetic data, avoiding the need for original training data. It introduces the High Fidelity (HiFi) metric, which quantifies the contribution of each component to local reconstruction. This importance score is used for both structured pruning (retaining high-HiFi components) and classwise unlearning (removing high-HiFi components). The method leverages the empirical observation that pretrained Transformers exhibit approximate Lipschitz continuity, and uses this to theoretically justify that local reconstruction errors can linearly bound global prediction degradation. ModHiFi is evaluated on CNNs and Transformers across CIFAR-10 and ImageNet, showing competitive performance without access to the original training data.", "questions": "1. **Mitigating theoretical limitations in simultaneous pruning**: Theorem 3.1 assumes that inputs to each layer remain fixed, yet ModHiFi prunes multiple layers simultaneously. Could you clarify how the theoretical guarantees could hold under this setup? It might be worth considering including an empirical comparison between layerwise iterative and simultaneous pruning to assess the bound’s practical validity.\n\n2. **Addressing data sensitivity more explicitly**: The reliance on synthetic or calibration data is mentioned in the appendix but is central to the method’s performance. The paper would benefit from moving this discussion into the main text and expanding it with quantitative evaluations showing how fidelity scores and performance vary with data size, diversity, or distribution mismatch / dataset sfhit.\n\n3. **Quantifying computational overhead**: ModHiFi requires activation logging and per-layer evaluation for each candidate component. It could be worth providing runtime, memory, or efficiency comparisons with standard pruning baselines (e.g., L1-norm). Furthermore frontier plots of sparsity-accuracy tradeoffs would be helpful to position ModHiFi against related work.", "limitations": "The authors acknowledge certain limitations, such as the method’s reliance on pretrained models and the intractability of full subset selection, which are briefly discussed in the main text. However, additional limitations - particularly the method’s dependence on data quality for synthetic inputs should be more explicitly addressed, and ideally consolidated.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- **Works for both pruning and unlearning**: ModHiFi applies the same reconstruction-based importance score to both structured pruning and classwise unlearning, demonstrating that the metric captures feature relevance across different model editing tasks.\n- No reliance on training data: The method does not require access to the original training set or supervised loss function. Instead, it operates using either synthetic inputs or a calibration set, making it significant for pre-trained models and their applications.\n- **Theoretical support via Lipschitz continuity**: By leveraging the Lipschitz property of trained networks, the paper provides a theoretical guarantee that local reconstruction errors upper-bound global output degradation. This justifies the use of local fidelity scores for importance estimation, and makes the approach more practical by avoiding the need for end-to-end evaluations.\n- **Empirical validation of the importance metric**: The paper includes several supporting experiments and ablations, such as noise injection that empirically demonstrate the HiFi score correlates with true functional importance.\n- **Empirical effectiveness**: Despite operating without training data, gradients, or loss access, the method achieves competitive performance across pruning and unlearning on CIFAR-10 and ImageNet.\n\nWeaknesses\n- **Limited novelty and insufficient comparison to prior reconstruction-based pruning methods**: While ModHiFi is framed as novel, the core idea of using reconstruction error to guide pruning, has been well explored in prior literature [1,2,3,4]. The paper does not adequately position its contributions relative to this body of work or clarify what distinguishes ModHiFi’s reconstruction-based formulation beyond its application to multiple tasks. Moreover, the practical implementation relies on singleton importance scores due to the infeasibility of true subset selection, which diminishes the claimed departure from per-component heuristics.\n\n- **Dependence on synthetic or calibration data**: While ModHiFi avoids training data, it still relies on synthetic or curated datasets to compute fidelity scores and fit compensation terms. The quality, representativeness, and diversity of this data can significantly impact performance, yet the main paper offers limited discussion or quantification of this sensitivity. Additionally, fitting compensation weights via linear regression constitutes a lightweight form of fine-tuning, which somewhat undermines the “no fine-tuning” framing.\n\n- **Theoretical guarantees rely on unrealistic assumptions**: Theorems 3.1 and 3.7 rely on assumptions that rarely hold in practice. The uncorrelated feature assumption underlying singleton optimality is implausible in deep networks, where strong interdependencies exist. Similarly, Theorem 3.1 assumes that each layer’s inputs remain fixed during analysis, yet ModHiFi prunes multiple layers simultaneously, violating this condition. \n\n- **Lack of computational cost analysis**: Although the method is presented as more efficient than fine-tuning, the paper does not quantify its computational demands. Evaluating fidelity scores requires full forward passes and activation logging across a dataset for each candidate component, which may scale poorly in large networks. The absence of runtime comparisons or memory profiling against baseline methods (e.g., magnitude-based pruning) leaves open questions about ModHiFi’s practical efficiency.\n\n\n[1] Jiang, Chunhui, et al. Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error. IJCAI, 2018.  \n[2] Kamma, Koji, and Toshikazu Wada. Reconstruction Error Aware Pruning for Accelerating Neural Networks. ISVC, 2019.  \n[3] Zhuang, Zhuangwei, et al. Discrimination-aware Channel Pruning for Deep Neural Networks. NeurIPS, 2018.  \n[4] Molchanov, Pavlo, et al. Importance Estimation for Neural Network Pruning. CVPR, 2019.", "was_revised": true, "final_justification": "The strong rebuttal clarified and addressed my mentioned concerns.", "rebuttal": "We thank the reviewer for providing a comprehensive review of our work. We are grateful that you find value in our theory and empirical performance, and wish to provide clarification to the questions raised by the reviewer.\n\n## Summary and Strengths\n\nWe are happy to note that the reviewer finds several strengths in our work. Specifically, its broad applicability, theoretical support, and empirical effectiveness. However, we would like to highlight some points in our work that the reviewer might have missed, particularly that **we also show the broad applicability of our results to LLMs.**\n\n---\n## Questions and Weaknesses\n\n\n**1. Mitigating theoretical limitations in simultaneous pruning: Theorem 3.1 assumes that inputs to each layer remain fixed, yet ModHiFi prunes multiple layers simultaneously. Could you clarify how the theoretical guarantees could hold under this setup?**\n\nWe would like to clarify that **Theorem 3.1 does not assume inputs are fixed**, only the weights of layers after the layer being investigated are fixed (say layer $l$). All layers before layer $l$ can be modified, and the reconstruction error at the output would still be bounded by that of layer $l$. \n\nWe also want to clarify that **Algorithm 1 on page 6 is applied to a single layer**. In practice, we find that the best performance is achieved when layers are modified iteratively, starting from the initial layer for pruning. For unlearning, we find we only need to iteratively modify the last few layers, as specified in Appendix C.4.\n\n\n**2. Addressing data sensitivity more explicitly/Dependence on synthetic or calibration data: The reliance on synthetic or calibration data is mentioned in the appendix, but is central to the method’s performance.**\n \nWe first clarify that **our method itself does not require synthetic data**. Rather, we address the problems of pruning and unlearning in the restrictive setting where the training data and loss function used to train the model are unavailable, but with distributional access. Our method addresses that problem and is effective even when only synthetic data can be used.\n\n**3. The paper would benefit from moving this discussion into the main text…the main paper offers limited discussion or quantification of this sensitivity.**\n\nAs we have **already discussed in Appendix C.2**, we show that **higher quality synthetic data leads to better estimation and identification of high-fidelity components and performance for pruning**, as illustrated in Table 4 of the Appendix. Furthermore, as we show in [our response to reviewer qfdv](https://openreview.net/forum?id=lClK4uBxSG&noteId=4eCrWRgDBD), **using the original training data for identifying HiFi components for unlearning tasks results in superior unlearning performance than with synthetic data.** We have also provided **experiments highlighting the efficacy of fine-tuning a pruned model (using $L_2$ norms of weights) with synthetic data instead of the original training set in Appendix C.3, Table 5**, as well as Table 1 of the main document. We show that low-quality synthetic data is insufficient for fine-tuning heavily pruned models. If accepted, we will move this discussion from the Appendix to the main document.\n\n**4. Quantifying computational overhead/ Lack of computational cost analysis… Although the method is presented as more efficient than fine-tuning, the paper does not quantify its computational demands...absence of runtime comparisons or memory profiling…**\n\nWe kindly refer the reviewer to Appendix C.5, where we have **already discussed the theoretical and practical computational cost of the ModHiFi algorithm.** In particular, we showcase that **fidelity scores can be computed in roughly 5 minutes for ResNet-50 models trained on Imagenet**, and **compensation terms can be computed in roughly 1 minute per layer on average.** We also point out that while using weight norms for pruning would undoubtedly be faster than ModHiFi, purely weight-based techniques cannot be broadly applied to other tasks, such as unlearning. Additionally, norm-based techniques are not as performant in retaining accuracy, as noted in Table 1.  \n\n**5.  Furthermore frontier plots of sparsity-accuracy tradeoffs would be helpful to position ModHiFi against related work.**\n\nWe will provide such plots in our revised manuscript. However, we regretfully remind the reviewer that **we are prohibited from providing any plots during the rebuttal period**. \n\n**6. Limited novelty and insufficient comparison to prior reconstruction-based pruning methods: While ModHiFi is framed as novel, the core idea of using reconstruction error to guide pruning, has been well explored in prior literature [1,2,3,4]. The paper does not adequately position its contributions relative to this body of work or clarify what distinguishes ModHiFi’s reconstruction-based formulation beyond its application to multiple tasks.**\n\nWe politely disagree that the novelty of our work is limited. As stated by reviewer qfdv, the applicability of our approach to a variety of tasks is a strength of our work and is novel. Specifically, the applicability of **reconstruction error to Unlearning has not been addressed in prior art.** Moreover, unlike prior art, our proposed method identifies **groups of components in connected layers** (such as layers connected by residual connections in ResNet and transformer models) **as noted in Section 4 and Appendix D.2**, which has not been addressed in prior art. Moreover, reconstruction error is a metric used to form a basis for our model modification algorithms, which are novel and have novel consequences, as also noted by other reviewers.\n\n**7. Moreover, the practical implementation relies on singleton importance scores due to the infeasibility of true subset selection, which diminishes the claimed departure from per-component heuristics.**\n\nOur work provides singleton importance scores as a heuristic for solving an otherwise NP-complete problem, which can then be applied to a variety of model modification tasks, such as pruning and unlearning\n\n**8. Additionally, fitting compensation weights via linear regression constitutes a lightweight form of fine-tuning, which somewhat undermines the “no fine-tuning” framing.**\n\nFollowing recent work such as [5,6], fitting compensation weights does *not* constitute lightweight fine-tuning and is considered to be part of the pruning methods in general. However, if the reviewer insists, we will modify the manuscript to make this distinction by using the term “gradient-based fine-tuning.” \n\n**9. Theoretical guarantees rely on unrealistic assumptions…**\n\nTheorem 3.7 serves to highlight that there are cases when the greedy pruning strategy is optimal, even if it is not the case generally, **motivating its use as a useful heuristic** for solving an NP-hard problem. For a practical setting, we rely on our empirical results to justify the merit of our model modification algorithms. As already mentioned in Appendix C.1.2 and Appendix C.1.3, we conduct several experiments to show that the subsets generated by our algorithm are correlated to the final prediction by showing that these components are disproportionately responsible for model’s prediction.\n\n**10. Similarly, Theorem 3.1 assumes that each layer’s inputs remain fixed during analysis, yet ModHiFi prunes multiple layers simultaneously, violating this condition.**\n\nThe purpose of Theorem 3.1 is to highlight the fact that the error in the output layer is at worst linear in the error at any intermediate layer, and serves to provide a principled grounding for using local reconstruction for identifying important components for pruning and unlearning.\n\n---\n**We thank the reviewer for their comprehensive review, which has helped us improve our submission. We would like to invite them to engage with the discussion period to discuss and clarify any other concerns preventing them from increasing their score and accepting this work.**\n\n---\n\n## References\n\n[1] Jiang, Chunhui, et al. Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error. IJCAI, 2018.\n\n[2] Kamma, Koji, and Toshikazu Wada. Reconstruction Error Aware Pruning for Accelerating Neural Networks. ISVC, 2019.\n\n[3] Zhuang, Zhuangwei, et al. Discrimination-aware Channel Pruning for Deep Neural Networks. NeurIPS, 2018.\n\n[4] Molchanov, Pavlo, et al. Importance Estimation for Neural Network Pruning. CVPR, 2019.\n\n[5] Shah et al. Decomposing and Editing Model Predictions by Modeling Model Computations. ICML 2024.\n\n[6] Hoefler et al. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. JMLR, 2021."}, {"review_id": "oURz3YkqBe", "reviewer": "Reviewer_17dD", "summary": "This paper proposes ModHiFi, a pre-training data- and loss-free algorithm for model modification tasks such as pruning and unlearning. It introduces a theoretical method to identify critical model components and demonstrates competitive performance on both vision and language benchmarks.", "questions": "Q1: The term \"model modification\" may encompass diverse tasks beyond pruning/unlearning (e.g., model editing in LLM and some vision tasks in [2]). In addition, the paper exclusively evaluates zeroing out as the modification strategy. How might the identified components be adapted to other modification strategies (e.g., fine-tuning, structural reparameterization), and would their effectiveness generalize across different model modification tasks? Alternatively, it may be more appropriate to refine the terminology and scope description to better reflect the range of tasks actually validated in the experiments.\n\nQ2. Under the same setting that uses synthetic image data for component identification in vision models, how does the proposed method compare to [2], which estimates component importances across the entire model rather than treating each layer separately? \n\nQ3. In Table 3, the reported time cost for the proposed methods appears to exclude the Monte Carlo sampling-based identification step, which may make the comparison unfair.  Additionally, what is the time cost when applying the method to large language models? \n\nQ4. Could the paper include comparisons with more recent methods, such as [3] (which appeared online before March 1st, 2025), to better position the proposed approach within the state-of-the-art?\n\n[2] Decomposing and Editing Predictions by Modeling Model Computation, ICML 2024\n\n[3] Numerical Pruning for Efficient Autoregressive Models, AAAI 2025", "limitations": "The method computes the fidelity of each layer independently, potentially overlooking inter-layer connections. Additionally, the computational cost of the Monte Carlo–based subset identification may become high when applied to large foundation models.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths \n- The paper provides a theoretical analysis showing that, under Lipschitz continuity, the reconstruction error of the model is linearly bounded by the local reconstruction error.\n- It introduces a Monte Carlo-based sampling strategy to identify subsets of important components using the proposed Subset Fidelity metric.\n-The method is extensively validated on CNN and Transformer architectures, across vision and language tasks, demonstrating its generality and effectiveness.\n\n### Weaknesses\n-  Theorem 3.1 appears to be an extension or adaptation of prior work [1], which is not cited in the main paper.\n- The method computes fidelity scores and prunes components independently for each layer, which potentially overlooks inter-layer dependencies among components during pruning.\n- The paper would benefit from comparison with more recent baselines, where applicable, to ensure a fair and up-to-date evaluation.\n\n[1] NISP: Pruning Networks using Neuron Importance Score Propagation, CVPR 2018", "was_revised": true, "final_justification": "The rebuttal clarifies the Monte Carlo sampling and the model modification strategy, and discusses prior work. Therefore, I keep the current positive rating.", "rebuttal": "We thank the reviewer for the positive evaluation of our work. We hope the additional clarifications provided here address the concerns and further demonstrate the significance and robustness of our work. We are eager to engage further with the reviewer during the discussion period to improve our work and scores in the process. \n\nWe respond to their questions in the order that we believe addresses their most pressing concerns first.\n\n## Questions\n\n\n**1. In Table 3, the reported time cost for the proposed methods appears to exclude the Monte Carlo sampling-based identification step, which may make the comparison unfair.**\n\nWe would like to clarify to the reviewer that **Monte Carlo sampling of subsets is not a part of our pruning/unlearning algorithm**. The Monte Carlo experiment is an ablation study shown in Figure 1, as stated in line 203, to demonstrate the existence of small sets of high-fidelity components and provide context for Observation 1. Instead, as mentioned in lines 201-202, we approximate the solution by constructing the subset of a required size with the largest individual fidelity scores. We further clarify that we perform random sampling of a set of synthetic data to estimate the statistics of the intermediate solution. The experiment described in Figure 1 adds empirical validation to our observations.\n\n**2. [Continuation to your question on Monte-Carlo Simulation experiments] Additionally, what is the time cost when applying the method to large language models?**\n\nTo perform the Monte Carlo Simulations on LLMs is expensive. It took us up to 20 hours to conduct this experiment on the OPT-125m, as shown in Figure 1 on the hardware described in Appendix C.5. However, we reiterate that this Monte-Carlo step is not a part of our model-modification algorithm and is only used to demonstrate the existence of small sets of high-fidelity components in various architectures.\n\n\n**3. The paper exclusively evaluates zeroing out as the modification strategy. How might the identified components be adapted to other modification strategies (e.g., fine-tuning, structural reparameterization), and would their effectiveness generalize across different model modification tasks?**\n\nIn this work, we focus on the identification of components important for predictions without access to gradients or loss functions. We choose pruning and unlearning as they are two tasks that directly benefit from identification in this setting. We hope the strong empirical results from this work encourage further interest in the community in applying these techniques to additional tasks. We would like to point out, per our definition of model modification as stated in Equation MODIFY (line 109), structural reparametrization is not in our scope of work. Moreover, for finetuning, we precisely aim to eliminate it to reduce the computational burden for model modification, which is the goal of this work.\n\n**4. The term \"model modification\" may encompass diverse tasks beyond pruning/unlearning (e.g., model editing in LLM and some vision tasks in [2])...Alternatively, it may be more appropriate to refine the terminology and scope description to better reflect the range of tasks validated in the experiments.**\n\nWe thank the reviewer for the suggestion and will refine the terminology to contextualize the experimental evaluation.\n\n**5. Under the same setting that uses synthetic image data for component identification in vision models, how does the proposed method compare to [2], which estimates component importances across the entire model rather than treating each layer separately?**\n\nWe would like to note that [2] requires access to the loss function and access to training data to compute a brute force estimate of component importance across the entire network. In our work, we assume the setting wherein the loss function is not available, and we propose a technique that does not require access to either of these. Moreover, their work is not architecture-agnostic. Our results indicate that even treating each layer separately is sufficient to provide strong performance on pruning and unlearning tasks. This assumption has become standard in literature due to the large size of the models [4, 5, 6,7, 8, 9]. However, equipped with Theorem 3.1, we can bound the error in the last layer linearly in terms of the local error, indicating that for such networks, the propagation of local error across layers is not severe, and can be upper-bounded by a linear function of the local error in the worst case.\n\n**6. Could the paper include comparisons with more recent methods, such as [3] (which appeared online before March 1st, 2025), to better position the proposed approach within the state-of-the-art?**\n\nWe thank the reviewer for drawing our attention to this work. We will include this in our related work and provide context for recent pruning methods. As for direct comparisons, [3] does not report results on the models considered in this work and does not make its code available. In terms of differences in pruning methodology, they propose an iterative scheme for computing pruning-specific scores, while we propose a single-shot scheme for computing general-purpose importance scores that can be applied to a variety of tasks. Additionally, it is not clear how this would apply to other modification tasks or convolution architectures.\n\n---\n\n\n## Summary and Strength\n\nWe thank the reviewer for appreciating the theoretical analysis and extensive empirical validation of our work. While we appreciate the summary and the strengths, we would like to bring to the reviewer’s notice a clarification. Our proposed method is also gradient-free and does not require computing gradient-based updates to identify components or compensate for lost statistical performance.\n\n---\n\n\n## Weaknesses and Limitations\n\n**Lack of mention of related work**\n\nWe thank the reviewer for bringing [1] to our notice. We will reference this work in the paper and highlight how our work differs from it in our related work section. In this work, we specifically consider the statistics of the intermediate distributions to inform our algorithm by upper-bounding the expected error, whereas [1] considers assigning importance scores without taking the distribution of data into account, providing a more conservative bound. Moreover, their theorem only holds for feed-forward layers with Lipschitz non-linear functions and doesn’t generalize to other architectures such as transformers. Our work also addresses networks with complex interconnections. Additionally, [1] focuses only on pruning, whereas this work focuses on the identification of components essential for prediction.\n\n**Expensive Monte Carlo Sampling**\n\nWe refer the reviewer to our response to Question 1.\n\n**Inter-layer dependency**\n\nWe refer the reviewer to our response to Question 5.\n\n\n**We thank the reviewer for their thorough review, which has enabled us to clarify the key claims of our work. In particular, the reviewer helped us contextualize the Monte Carlo experiments showing the existence of small subsets of high-fidelity components. We hope this discussion results in an increase in our scores, and we are eager for further discussion on the merits of our work.**\n\n## References\n\n[1] NISP: Pruning Networks using Neuron Importance Score Propagation, CVPR 2018\n\n[2] Decomposing and Editing Predictions by Modeling Model Computation, ICML 2024\n\n[3] Numerical Pruning for Efficient Autoregressive Models, AAAI 2025\n\n[4] Elias Frantar, Sidak Pal Singh, and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning, 2022. \n\n[5] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.\n\n[6] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 5058–5066, 2017.430\n\n[7] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720, 2023.\n\n[8] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024.\n\n[9] Elias Frantar and Dan Alistarh. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, 2023."}]

Author rebuttal:

Meta-review: The paper proposes a synthetic data–based method (named ModHiFi) for estimating feature importance in neural networks *without* access to training data. It introduces the HiFi metric for local reconstruction, applying it to structured pruning and classwise unlearning. Leveraging approximate Lipschitz continuity in pretrained Transformers, the method theoretically links reconstruction error to prediction degradation. Experiments on CNNs and Transformers (CIFAR-10, ImageNet) show ModHiFi achieves competitive performance without original data.

This is novel idea, solving a critical problem in large model training and fine-tuning. The authors have well addressed the questions and concerns by reviewers during discussion period. We decide to accept this paper. The authors are also required to appropriately polish the paper by incorporating the reviewers' comments (e.g., the additional experiments and context).
Decision: Accept (spotlight)