DisCEdit: Model Editing by Identifying Discriminative Components
Authors: Chaitanya Murti
Chiranjib Bhattacharyya
Venue: NeurIPS 2024 poster (2024)
Keywords: model editing
selective forgetting
structured pruning
total variation distance
Primary area: deep_learning_architectures
TL;DR: We use new lower bounds on the TV distance to identify discriminative network components for structured pruning and selective forgetting.

Abstract: Model editing is a growing area of research that is particularly valuable in contexts where modifying key model components, like neurons or filters, can significantly impact the model’s performance. The key challenge lies in identifying important components useful to the model’s predictions. We apply model editing to address two active areas of research, Structured Pruning, and Selective Class Forgetting. In this work, we adopt a distributional approach to the problem of identifying important components, leveraging the recently proposed discriminative filters hypothesis, which states that well-trained (convolutional) models possess discriminative filters that are essential to prediction. To do so, we define discriminative ability in terms of the Bayes error rate associated with the feature distributions, which is equivalent to computing the Total Variation (TV) distance between the distributions. However, computing the TV distance is intractable, motivating us to derive novel witness function-based lower bounds on the TV distance that require no assumptions on the underlying distributions; using this bound generalizes prior work such as Murti et al. [39] that relied on unrealistic Gaussianity assumptions on the feature distributions. With these bounds, we are able to discover critical subnetworks responsible for classwise predictions, and derive DISCEDIT-SP and DISCEDIT-U , algorithms for structured pruning requiring no access to the training data and loss function, and selective forgetting respectively. We apply DISCEDIT-U to selective class forgetting on models trained on CIFAR10 and CIFAR100, and we show that on average, we can reduce accuracy on a single class by over 80% with a minimal reduction in test accuracy on the remaining classes. Similarly, on Structured pruning problems, we obtain 40.8% sparsity on ResNet50 on Imagenet, with only a 2.6% drop in accuracy with minimal fine-tuning.

DISCEDIT: Model Editing by Identifying Discriminative Components
Chaitanya Murti
Robert Bosch Centre for Cyberphysical Systems Indian Institute of Science
mchaitanyaßiisse aasaanaa@iiss..mm ac.in
Chiranjib Bhattacharyya
Computer Science and Automation
Robert Bosch Centre for Cyberphysical Systems Indian Institute of Science chiru@iisc ac.in
Abstract
Model editing growing area research that particularly valuable in contexts where modifying key model components, like neurons or filters, can significantly impact the model's performance. The key challenge lies in identifying important components useful to the model's predictions. We apply model editing to address two active areas research. Structured Pruning, and Selective Class Forgetting. this work, we adopt a distributional approach to the problem of identifying impor- tant components, leveraging the recently proposed discriminative filters hypothexis, which states that well-trained (convolutional) models possess discriminative filters that are essential prediction. To do so, we define discriminative ability terms of the Bayes error rate associated with the feature distributions, which is equivalent to computing the Total Variation (TV) distance between the distributions. However, computing the TV distance is intractable, motivating us to derive novel witness tunction-based lower bounds on the TV distance that require no assumptions on the underlying distributions; using this bound generalizes prior work such Murti al. [39] that relied on unrealistic Gaussianity assumptions on the feature distributions. With these bounds, we are able discover critical subnetworks responsible for classwise predictions, and derive DISCEDIT-SP and DISCEDIT-U algorithms for structured pruning requiring access the training data and loss function, and selective forgetting respectively. We apply DISCEDIT-U to selective class forgetting models trained on CIFAR 10 and CIFARI00, and show that on average, we can reduce accuracy on single class by over 80% with a minimal re- duction test accuracy on the remaining classes. Similarly, Structured pruning problems, we obtain 40.8% sparsity on ResNet50 on Imagenet, with only a 2.6% drop in accuracy with minimal fine-tuning.
Introduction
The black-box nature of neural networks makes understanding the precise mechanism by which neural network makes a prediction (in the classification or regression settings), or generates a sample (in the generative settings) active area of research 4F and relevant several related problems,
Our code is available at: https:///itthu: https/a.uu.occaaaaaaaaaaaaaaaaaa httpa///./..o.caammt////saaaaaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Idintiffy Discriminative Components lower bounds contribute to casswlse predictions
Classwise Unlearming Pruncinask discrininativ for the chosen closs
Structured Pruning discrininative for the no classes
Prunelnask
Figure I: Identifying Discriminative Components for Model Unlearning and Structured Pruning
such as robustness, sparsity, and memorization. In particular, recent work aims to address this problem by identifying which components such as neurons, convolutional filters, or attention heads in the model contribute most significantly to predictions [4]. Prior work addressing this problem includes 12. which identified important filters by using human-defined concepts, and aligning them with filter responses to those concepts, whereas more recent work [88.. uses an exhaustive regression- bassdd approach. The field of structured pruning [19 4 also provides a variety of heuristics for component
importance, including first- and recond-order derivative information [38 37 43). feature map ranks
[35 (2). and layerwise reconstruction errors [59 41. Component attribution has also found attention in the fields of machine unlearning (58) where class-wise important components can be identified and edited to remove information about a given class or group 23 33).
In this work, we address this problem from a distributional milicu: that is, by analyzing the distribu- tions of feature maps for models trained on datasets with distinct classes or subgroups. Specifically, we use the discriminative filters hypothesis (DFH) stated in [39], which states that well-trained models for classification possess a mix of discriminative filters- filters that yield feature maps with distinct class-conditions distributions and non- con-dd-rriiinatiee filters and that the discriminative filters are important to model predictions. Discriminative filters can be used to identify which filters contribute to the prediction for samples from given class. Under restrictive assumptions of Gaussianity, the DFH was used [39] to derive a structured pruning algorithm that required only some form of distributional access, and no access to the loss function and training set. However, several key challenges remain. First, is it possible to identify discriminative filters without restrictive assumptions on the filter outputs? Second, can discriminative filters be used in settings other than structured pruning? Last, can the performance of algorithms leveraging the DFH be improved upon with better identification of discriminative filters? In this work, answer each of the questions affirmatively.
First, to identify discriminative filters without unrealistic assumptions on the feature map distributions, we derive tractable, witness function- lower bounds on the Total Variation (TV) distance, with which we also reveal hitherto unknown ties between the TV distance and classical, discrimination- based classifiers. We then use our proposed methodology to identify discriminative filters for classwise model unlearning and structured pruning, which we call DISCEDIT: Discriminative Component identification for model Editing. Our methodology is simple- we first identify discriminative filters for a given class, and prune them (for class forgetting). called DISCEDIT-U, or in the case of structured pruning, prune filters that are non-discriminative for all classes, called DISCEDIT-SP. Note that our method for identifying discriminative filters requires only distributional access, and neither the training data nor the loss function. We illustrate our approach in Figure I and formally state our contributions below.
Discriminative ability of a filter: We quantify the discriminative ability of a filter in a neural network, as the worst possible Bayes error rate of binary classifiers trained upon the features it generates (See Section 3). Since, in general, computing the Bayes" error rate is intractable, we seek to approximate it using lower bounds on the TV distance.
Witness function based Distribution Agnostic Lower bound on TVV distance: In order to identify discriminative filters without distributional assumptions, we propose novel witness function- based lower bound on the TV distance between distributions to address this gap in Theorem T We propose a lower bound that relies only on knowledge of finitely many moments( Thenorrm 2 and derive another lower bound accounting for moment measurement errors in Theorem 3 These bounds do not require distribution-sseecife assumptions and hence enable this work to generalize previous work [39, which requires the distributions to be Gaussian( which is an unrealistic assumption). To be noted that these bounds are of independent interest as they are broadly applicable. Moreover, using a careful choice of witness function, our bounds reveal new connections between discriminant-brrt based classifiers like

---

the Fisher linear discriminant and the Minimax Probability Machine, and the TV distance, which we state in Corollary ]
Model Editing using Lower Bounds: We apply our lower bounds to the problem of identifying which components in model contribute to predictions from certain classes.
1. Class Unlearning: We identify components capable of discriminating each class, which we then prune in order to unlearn that class, requiring no access to the loss function, called DISCEDIT-U.
2. Pruning without the training data or loss function: Next, we derive family of algorithms for structured pruning requiring no access to the training data loss function by identifying non-discriminative filters with our proposed lower bounds, called DISCEDIT-SP.
Experimental Validation: We produce a slate of experiments to illustrate the efficacy of identifying discriminative components for structured pruning and machine unleaming/class forgetting. We compare the efficacy of DISCEDIT-ST SP for pruning VGG nets and ResNets trained on CIFARIO, and show that achieve up to 76% sparsity with .12% reduction in accuracy on VGG19, and on Imagenet, we achieve 40.8% sparsity with a 2.58% drop in accuracy with minimal fine-tuning. Similarly, for machine unlearning, we show that method enables 83% drop in accuracy the class to be forgotten, with 1.2% increase in accuracy on the remaining classes, without any finetuning.
1.1 Related Work
In this section, we introduce brief summary of related literature. We provide more detailed discussion of the literature Appendix A
Structured pruning Structured pruning reduces the inference time and power consumption of neural networks by removing entire filters or neurons, as opposed to unstructured pruning that removes individual weights (29 [8]. A variety of structured pruning methods require the loss function, and identify import components using gradient information 38 30 or approximations of the Hessian of
29 18 51 36).
variety of structured pruning methods that do not rely on the loss function have been proposed, which identify important components by norms of the weights [3] 32 bounds on the reconstruction error incurred by pruning a filter [41 [59,, the rank the feature maps [35 (2). and coresets of feature maps 34 More recent approaches identify components based on the discriminative ability of the corresponding featuremaps, using metrics like the Hellinger distance between class-conditional feature maps (39), or Fisher discriminant based methods [20, 21). In this work, formalize the notion of discriminative ability in terms of the Bayes risk and derive novel lower bounds on the total variation distance approximate it effectively. Unlike previous works, make assumptions about the class-conditional distributions, and our pruning algorithms require no access to the training data or loss function. For comprehensive discussion structured pruning, refer readers to Appendix or the surveys 19 4.
Machine unlearning Machine Unlearning has recently received significant attention due to con- cerns like data privacy and security 5 42 A variety of works propose methods to forget data points while maintaining model accuracy, even adaptive settings where classes groups are forgotten, connects machine unlearning to continual learning [58] 57]. Model editing for unlearning. however, remains an underexplored area of research. Recent studies machine unlearning can be enhanced by sparsifying models (23 46). and discrimination-ane
16 22
14]. Selective forgetting,
prun- ing has been used in federated settings (56) and for forgetting specific classes [48). Our work differs by directly utilizing the discriminative ability of model components to identify and remove those responsible for a given class, enabling effective unlearning without access to the original training data. For more detailed discussion, refer readers to Appendix A
2 Background and Notation
In this section, we introduce notation and provide basic background definitions.
3

---

Notation For an integer N > O, let [ : f1. f..... ... Let O2 be vector of zeros of dimension N. Let sort sfrt am3) )) be the set of the largest elements of {aI.... oar). Suppose P. Q are two distributions supported on X, with densities given by p(a) and g(x)). For a function
::X- + R. let F E [=]]] and let fp E [f()]]()))] Let be data distribution. Suppose the dataset has C' classes, then let Dr be the class-conditional distribution of the e-th class, and let DD the distribution of the complement of e (that is, samples are drawn from all classes other than c).
Suppose we have neural network W (W1,- .. WL). Each layer yields (flattened) representations
('()) = [Y(()). ...) (x)]
(1)
where N₁ is the number of filters in layer I. Since Y is dependent on X, we assume that ('(X) ~ D', and (x) ~ Djj Furthermore, let Dj,c and Dj,, be the class-conditional distributions and class- complement distributions of Yj(X) respectively.
Background In this section, we provide relevant background for this work. First, let P. and Q be distributions supported on X, with moments Up, Ep and Mgg 2g. Then, recall that
(u (up - ubg) 2 Fish Q3 u (S,p + Mng) u u)
| (Up Ma) Vu En Vu V
and MPM(PP Q;
(2)
where Fish denotes the Fisher discriminant
and MPM denotes the Minimax probability machine 27 28 If we choose the optimal u. denoted by u', write max,, FIsh Q; u)
FIs Q: ut" Fish" (IP, Q) and max,, MPM(P, Q; = MPM(P (3n")= = MPM Q)"
We define the TV and Hellinger distances as follows.
Definition 1. Let P and Q be two probability measures supported on X, and let p and be the corresponding densities. Then, we define the Total Variation Distance
TV (P,Q)) sup ((4)-Q(4)) 2 lp(z) - ((z)) dz
ACX
The Bayes Optimal classifier, as given in 9 8] associated with distributions and Q with labels -1 and 1 respectively is given by
f(x) arg pr((()) cE(-1,1)
and has the error rate (P, Q), called the Bayes Error Rate. Next, we relate the Bayes classifier and the Bayes error rate (as described in, say, Devroye et al. (8) of a classifier to the TV distance with the following identity. Consider binary classification problem with instance æ and labels c {-1, 1}. with class conditional distributions given by P and Q. The Bayes error rate satisfies the identity
= TV(P, Q) 2R (P.
(3)
Editing Models by Identifying Discriminative Components
There three central questions addressed in this work:
1. Model Unlearning:
How do we edit components in order to reduce accuracy on certain groups or classes only?
2. Structured pruning:
How do we remove components ensure that the accuracy of all
classes is minimally affected?
3. Determining the Discrimination Ability of a Model Component: How do we assess the Discrimination ability of filters without access to the training data or loss function, and without making any assumptions about the class-conditional feature distributions?
Inspired by Murti et al. 39 the key idea in this work for addressing each of the questions raised above to identify discriminative components, that yield feature maps upon which accurate classifiers can be trained, and thus with distinct class-conditional distributions. A heuristic to address this

---

problem would be to train a classifier upon the feature map; those features upon which accurate classifiers can be trained are generated by discriminative filters; however, this highly impractical. Thus, as with (39, we address this problem by identifying filters for which the class-conditional distributions of the feature maps are distinct, which are identified based on estimates of the total variation distance between the dlasscoodittionss distributions of the associated feature maps. In this work, we formalize discriminative ability of a component in terms of the best possible classifier that can be trained on the features generated by it. In the sequel, we illustrate similar but distinct notions of discriminative ability important for classwise unlearning, and structured pruning. We first formally define discriminative ability and the class-c discriminative ability as follows.
Definition 2 (Discriminative Ability). Consider a CNN with L layers trained on dataset with C classes, and consider the jth filter in the lth layer. The class-c discriminative ability nh and the discriminative ability Ti of the filter are given by
nhg = RR (D).c.Di.. Pi,))
Mh = max TE = mmx R (De) cE(C]
(4)
cE|C]
Classwise Unlearning For classwise unlearning, we aim to identify those components that are responsible for predictions of a selected class group within the dataset. Thus, we aim identify those components that are able to discriminate between the given class, say with label c, and others. say D. is the Forget Set, and Da is the Remain Set. As noted in Shah et al. [48] Jia et al. [23]. our aim is to minimize the test accuracy of the model on De, while maintaining the test accuracy on Dg. Thus, unlearn class c, we edit those components for which the class-c discriminative ability, ni is low. An interesting point to note is that some components may be discriminative only for class c, which we call class-selective components, whereas others may be discriminative for multiple classes. Note that when the dataset has large number of classes, our experiments indicate that class-specific components generally can't be found.
As with Wang et al. [58
Identifying discriminative components (specifically filters in convolutional networks) for structured pruning was first introduced in Murti et al. [39]. Structured pruning involves removing components for network while maintaining the accuracy of the classifier. Following Murti et al. 39], we aim to retain components that are discriminative for multiple classes. Our goal is to remove components from the model while maintaining the model's 's accuracy each class conditional De. Thus, Definition 2 gives us a means by which we can identify discriminative components based on the worst Bayes error rate for discriminating the class conditional distributions of the given feature map. Filters for which M). is small are considered discriminative, as the best possible classifier that can be trained on those features will be highly accurate.
Structured Pruning:
Assessing the Diseriminative Ability: In general, the Bayes error rate cannot be computed. However, since the TV distance and the Bayes error rate are connected by the identity 3 which states that for two distributions IP, Q the Bayes risk is given by 1 - TV Q)), we can reformulate our distributional pruning strategy as identifying those filters that generate features for which class- conditional distributions are well-separated in the Total Variation sense, and prune them. In (39, this was achieved by making the strong assumption that the distributions of the conditionai feature maps were spherical Gaussian. In the sequel, we propose novel lower bounds on the total variation distance that require no restrictive Gaussianity assumptions, thereby enabling to effectively identify discriminative components.
4 Witness Function-Based Lower Bounds for the Total Variation Distance
In this section, we derive lower bounds on the Total Variation Distance that rely on the moments of a wimess function, a scalar-valued function whose moments can be used to derive bounds on divergences between distributions. More formally, we write ff : X + R, where X' is the support of the distribution(s) in question. We then adapt our lower bound to variety of scenarios, depending on the extent of the information about the distributions available to us. When access to only the first two moments is available, we derive lower bounds on the total variation distance based the Fisher linear discriminant and the minimax probability machine.
Estimating the Total Variation distance is known to be #P complete 3 Estimating lower bounds on the TV distance is an active area of research (see Davies et al. 7 and the references within), with variety of applications from clustering [7] to analyzing neural networks (59). However, most
5

---

bounds such as those presented in Davies et al. 7 require prior knowledge about the distributions, and tractable estimation of lower bounds given access to collections of moments or samples, without assumptions on the distributions themselves, remains an open problem.
4.1 Witness Function-basse Lower Bounds on the TV Distance
In this section, we propose lower bounds on the TV distance that only requires access to the moments of witness function, as described in Gretton al. (15), Kübler et al. [25]
Theorem 1. Let P. Q be two probability measures supported on X S R. and let p and be the corresponding densities. Let F be the set of functions with bounded first and second moments defined on X. Then,
(Jp (f--Jo) Ia)
[V(IP, Q) 2 sup FEF
(5)
f02) ((2)
Proof Sketch. We provide a sketch of the proof. We express the quantity f- Lg in terms of the densities (XX and (X). We then isolate the integral of ppx)) q(x)]. After rearranging terms, we obtain the result. For the full proof, we refer readers to Appendix B
L
4.2 Moment-based Lower Bounds the TV distance
Motivated by the need to identify discriminative filters when we only have access to the moments of feature map distributions, propose worst-case lower bounds on the TV distance given access finitely many moments of the distributions P and Q.
Let S& (P) == (P- : Ex-p[Xp .... x = = Payidda E, di S k3
be the set of probability measures
whose moments are given by P. where Ex-7 X ... ....dd. Xd= n
i similarly, let Sy ( (Q) be
the set of measures whose moments are given by Q. For any random variable X E Rd supported on X, suppose yp: _Rd R" for which there exist functions, g and G such that Ex [x{ = g(P)
and IE (x)) measure the worst-case TV separation between all distributions possessing the moments given in P
G(P). Given two collections of moments of the same order, want to
and Q. We define this as
min (Tv().(P), (ES
TV(PP, Q)
(6)
For the sake of brevity, we write (8x = DTV(P,
Theorem 2. Suppose P and Q are sets of moments of two probability measures supported on X. Let (X) be vector of polynomials such that Ex(p(X)] =P(P), Rx(+((X)) = gQ),,
(x(p(X))((x))1 = G(P), and (x))(X))(x)]]] = G(QQ), and let f= (x(X) ((P)+ )
be witness function. Then, for any P (P), Q E (Q). supported set X C Rd, we have
Drv 2 Siv Q) (2 STy (P, Q)) 2 (P Q) (Vz+ + (H Q) where:
In (g(P) (QQ))
Siv(P, Q) (G())((())) - ((Q)) ( (Ag) and SH (P. Q) max
Vu a G(P) Vu G(Q)a)
and Ag - g(() (Q) and ((P) - (P))
Sketch. Sketch. terms of (iy Q). Since the bound holds for any distributions that yield the given moments of the witness function, the statement holds. The full proof is provided in Appendix
We apply Theorem T with the given witness function, and obtain an expression in
Theorem 2 is worst-case lower bound on the TV distance between distributions with given truncated moment sequences. While we focus our results on the case where vector of monomials, this bound is valid for any choice of ff with bounded first and second moments.
(x) = u ya(r), where yp(x) is

---

4.3 Computing TV(P, Q) from the Lower Bound
In general, the bounds proposed in Theorem are not tight. However, an interesting observation that the lower bound proposed in Theorem can, in certain cases, be used to compute the Bayes optimal classifier, and thus the true TV distance. Specifically, if the Bayes optimal classifier lies in given set of functions F. the bound can be used to compute the Bayes classifier. We state one such case below in Corollary [
Corollary 1. Suppose IP = (upp E) and Q = ( Σ) Let I (a;; u) u (x 211ppp 1yy) be witness function. Then,
(E [f u) Br- u)))
E, En (f; u)21 En [f (a; 2] = E (14p p May
arg max
16
the weight vector of the Bayes optimal classifier (ogee sign (ut TutTz x bb). and TV(P, Q)
24h
- (4n
Proof Sketch. We find the u* that maximizes the the TV lower bound given in Theorem I and then apply the same the exact expression of the TV distance between Gaussian measures with the same covariance. The full proof is provided in Appendix B
Remark 4.1. This result illustrates the case where the Bayes' classifier lies in the set of functions F: {(x) (xx u² ( for given function (xx).. In this case, if (p(x) = 1 (Mp pg)).
and IP and Q are Gaussian with the same variance, the Bayes classifier equivalent to the Fisher discriminant.
4.4 Connections to Discriminant Based Classifiers
In this section, we exploit the bound stated in Theorem to reveal extensive connections between the total variation distance and discriminant-based linear classifiers, specifically the Fisher Linear Discriminant and the Minimax Probability Machine, that are of independent interest to readers.
Specifically, we show that the TV distance is lower-bounded by monotonic functions of the Fisher Discriminant and the Minimax Probability Machine. We state this result formally in Corollary 2.
Corollary 2. Let P, Q be two probability measures supported on X S R2, let and q be the corre- sponding densities, and let Up My and Ep. Mg be the means and variances of P and Q respectively. Then,
Fish v (P,
MPM (IP, Q
TV(P,Q)
+ Fish (P,Q)
V2 MPM' (P.
Proof. Choose (xx) x. Thus, g(P) Mps and G(P) Exp + upp Substitute these into Theorem D complete the proof.
This lower bound can be improved upon by selecting witness function of the form f(z; u) u (ax) where (rr) is a vector of basis functions (such as monomials, if f(r; u) is a polynomial). Moreover, lower bounds that are robust estimation error, based on the Fisher Discriminant and Minimax Probability Machines, can be derived by directly applying the techniques proposed in (24) (for the Fisher discriminant case) and (28 (for the Minimax Probability Machine case). We discuss this in Section 6.2.
4.5
Lower bounds Robust to Estimation Errors
The lower bounds derived Theorems and 2 are functions of moments of the distributions, which must typically be estimated from samples. In practice, we use plug-in estimators for g(P), ((Q). G(P) and (()). Since these estimators are computed using samples, errors in estimation arise, which in turn creates emrrr in computing the lower bound.
7

---

This estimation error is particularly challenging issue in high dimensions, where 'high dimensions' refers the setting where the number of samples n significantly less than the dimension of the data, d. However, in typical neural networks such as VGG-nets and ResNets, the feature maps, particularly of the layers close to the output that can be effectively pruned (see, for instance, [44 (. have low dimensional feature maps. For instance, on VGG16 trained on CIFAR1O, the feature maps generated by the 5th layer are of dimension 64; thus, for a relatively small number of samples n, the dimension is less than
In this section, we present robust lower bounds for the case when f(z) = u a, and the moments being estimated Hy EE [x] and Gp Ep(axx "] using plug-in estimators of the form
N
1 N
and Cp- =
2
N
where 24 are drawn iid from P.. We assume that we can quantify the estimation error for the above moments, and can apply lower bounds as proposed in Kim et al. (24) accordingly.
Theorem 3. Suppose P, Q be nwo probability measures supported on X S R with densities p and and let Mp = [x)x (hg = Ee (r) and Cp = Ep [apxxx) J, Cg = F T. Suppose we have plug-in estimates Fy. Cp. Pa. Cg as defined in (7). that satisfy
1|pp -ppl2 S 6p and 1Mq PAll2 S 8q. 11Cp Cp S Pp and 11C -Cyll= S Pr-
Then, with witness function ohe
the form f(x) = u Jx
DTV 2) 2 min
(Api) (C, + C. (Ap))
(8)
where M = ((pp: Mg) 11pp - Ppllz S 6p, 1lu - Bg l2 S 1gg, Ap = Mp Hag and p = Pp + Pg
Proof Sketch. The proof is similar to the derivation of (15) in Kim et al. [2 A full proof is provided
in Appendix B
This result can also be applied to the estimation error of functions of æ. such as a vector of monomials pp(),, provided the estimation error of each moment can be bounded.
DISCEDIT: Distributional Algorithms for Model Editing
In this section, we leverage the lower bounds proposed in Theorems 1 and 2 and Corollary 2 to develop one-shot algorithms for model editing that require no access to the training data or loss function, but only access to the data distributions. We propose two algorithms, DISCEDIT-SP and DISCEDIT-US U, that identify discriminative components (filters in CNNs), and prunes them to either unlearn class (DISCEDIT-U) U) to sparsify the model with minimal loss of accuracy (DISCEDIT- SP). A variety of variants of these algorithms based on diffcrent witness functions are provided in Appendix D
5.1 DISCEDIT-SP: A Distributional Approach to Structured Pruning
In this section, we propose DISCEDIT-SP, an algorithm for structured pruning that identifies filters (in convolutional networks) that are discriminative for multiple classes, and retains them. Unlike the upproach proposed in Murti et al. (3, no restrictive assumptions on the Gaussianity of class- conditional feature distributions is needed. Furthermore, by assuming the distributions are Gaussian and using the closed-for form Hellinger lower bound, (2) pairwise TV distances need to be computed for each filter. We now state the DisCEDrT-SPalgone
Let ('()) be the features generated by layer / of a neural network as defined in (I. We choose a witness function f = u (x)) = (X), and let fi.,e(u) Ex-D, [u v/ (X),, fuga(u) = I(u"
Ex<D,[u [u (x)), fi (w) = Ex=D,[(" (X))) and ((2) (l) = Ex~Dallu T (o)(())) Next, define r to be the saliency score for the jthh filter in the Ith layer
r = min max (Frs.(l)) F.so(u))² (Flld()) + R2 + Rldd(s))
(9)
CE(O) "
8

---

We use the lower bound established in Theorem on the TV distances between the class conditional distributions to measure the saliency or importance given filter. With this, we formally state DISCEDIT-SP in Algorithm
The DISCEDIT-SP algorithm has several advantages. First, as compared to TVSPrune, it requires that only CV distances be computed each step. Second, by varying the choice of witness function, we obtain new algorithms for structured pruning; we can choose different witness functions for each class as well.
Algorithm 1: DISCEDIT-X
Input: Class conditional distributions D. and
class-complmments D3 for all E (C], Pretrained CNN parameters
W = (Wi... ..., W layerwise sparsity budgets B', witness function f
E [7.]
5.2 DISCEDIT-U:
Sct S [sf..... .xN,) 0%₁
A Distributional Approach to Machine Unlearning
x is SP then
each compute of using
We now state an algorithm for classwise unlearning based model editing, called DISCEDIT-U. Motivated by works such as Shah et al. 48 our algorithm re- quires identifying and editing (specifically pruning) class- selective components for the class which is to be unlearned. DISCEDIT- Uuses the same setup DISCEDIT-SP. How-
sortn, ((r) 1y=1) then
Sct
Forget
For each compute ri using TT)
if E sort ((r) N
3 then
ever, DISCEDIT- Uonly requires identifying discrimina- tive filters for single class, we have
Oulput: Binary masks 8 ,. 8
return
3 max (Fij,e(u) - fis,el)) (R) St..e()) (10)
We formally state DISCEDIT-U in Algorithml As we show in our experiments in Appendix witness functions.
The DISCEDIT-U algorithm has several advantages.
few samples are required to effectively compute the
Empirical Evaluations
In this section, we empirically study the effectiveness of identifying discriminative filters for model editing tasks, specifically structured pruning and class unlearning. Additional experimental details are given in Appendix [E. Experiments showing that class-conditional feature map distributions are non-gaussian, the effectiveness of variants of witness functions, the effectiveness of sparsity in class forgetting, and other ablations, are provided in Appendix E Our experiment setup is provided in Appendix along with baseline accuracies of all models, shown in Table
6.1 Identifying Discriminative Subnetworks and Class Unlearning with DISCEDIT-U
In this section, we investigate the utility of the lower bounds given in Theorems 13 in identifying discriminative subnetworks and for class unlearning.
Experiment Setup: We investigate VGG16, Resnet56, ResNet20, and custom ViT (details given in Appendix (F) trained on CIFAR10, and VGG16 and ResNet56 models trained on CIFAR100,
Table 1: summary results uniearning using DISCEDIT-U. after applying DisCEorr- Uto each class. Note that FA mocurnncy drop on forget class, RA RAmaccurcyy drop remain set, Values avernged over 10 trials. NoFT refers using DISCEDII-U without fine-tuning, and with praning only 5.4% weights VGGI6, 1.8% of weights for ResNet56, 1.8% of weights for ResNe120, whereas FT refers using DisCEDIT-U U with epoch of line- tine-tnning and with pruning ratios 18.4%, 22%, 16.6%, and 10.2% for VGG16, ResNei56, ResNet20, and our VIT respectively..
the average the Forget
accuracies (FA RA)
Work (NoFT)
Baselimes
Dutusel
Model
FA (NoFT)
(FT)
(FT)
FA (GA) MA
RA (GA)
(TU) [3
RA (TU) MA
VGG-16
8.7
8%
11.5
89.8%
CIFAR-10
ResNer56
16.35
85.0%
9.7%
89.4%
20
9.4%
83.9%
6.0%
86.6%
11.52%
85.46%
VGGG6
11.3%
68.05
10.7%
72.9%
CIFAR 100
ResNec56
31.1%
60.4%
13.9%
68.7%
VIT
13.1%
44.2%
14.6%
61.0%

---

and identify subnetworks responsible for predictions from each class. We use the witness function f()) = =xx]([==]]²) in our experiments, using 256 samples from the Forget class, and 1024 samples for the Remain class, from the training sets of 2IFAR10 and CIFAR100's s, for computing ri, for each filter. For VGG-16, we only consider the last 8 layers, and for ResNet56, we only consider the final layerblock. We then select the sparsity budgets B;, and prune the most discriminative filters for that layer. We also fine-tune the models for 1 epoch. We measure the accuracy on the class test set, and the class complement test sets both before and after fine-tuning, and we compare against models trained from scratch on the retain set (the class complement).
Results: We present a summary of our results in [ In particular, we highlight that on CIFARIO models, particularly VGG16, the performance is comparable to or outperforms baselines with minimal fine-tuning. There are two interesting observations: first, as the number of classes exceeds the width of the network (as was the case with ResNet56 trained on CIFART (OF), the efficacy of our method is drastically affected. Second, fine-tuning models on the remain set still raises the accuracy on the forget set unless dramatically more filters are pruned. The reasons for this will be the focus of future work.
This set of experiments highlights the fact that for classifier models, it is possible to identify sub- networks responsible for predictions for each class. Identifying these subnetworks then facilitates classwise unlearning.
6.2 Structured Pruning with DISCEDIT-SP with Fine- Tin-TTnning
In this section, we investigate the ability for DISCEDIT-SP to sparsify models effectively both with and without fine-tuning.
Table 2: DISCEDIT-SP performance CIFARIO ImaggNett models with fine tuning. TVSPrune refers to (39) and CHIP refers to 52 "Sparsity' refers to parametric sparsity.
Experiment Setup: We prune VGG16, VGG19, and ResNet56 models trained on CIFARIO with two sets of fixed sparsity budgets, and then fine- tune them for 50 epochs. We repeat the ex- periment for ResNet50 trained on Imagenet, and fine-tune the pruned models for 100 epochs. We choose f(x)= u ((x), where sp(x) =
CIFAR-10
Model
Sparsity
work
TVSPrune
CHPP
LI
VGG16
61.25
-1.37%
-0.8%%
-0.33%
1.26%
75.05%
-1.32%
-1.54%
-162
-2.31
VGGB9
72.4%
-0.12%
40.6%
-2.1%
76.1%
.0.96%
-1.13%
NI
-3.00%
[1 x, (1 z)²iT as our witness function in each of the experiments. For details about the pre- trained models used, refer to Appendix FF)
ResNec56
60.7%
-1.21%
-1.92%
-1.77%
6.11%
ImageNet
Model
Sparsity
our work
TVSPrune
CHIP
L.1
Results: We show that models pruned with DISCEDIT-SP without fine-tuning retain high accuracies, particularly on the CIFARIO dataset. Moreover, after fine-tuning, DISCEDIT-SPis able to almost fully recover the accuracy of the original models. Our results are summarized in Table 2 which shows the drop in accuracy achieved by the different pruning algorithms compared after fine-tuning.
ResNas0
20%
+1.12% -2.8%%
-0.4%
+0.10%
-1 1.08%
403%
-2 -274%
-276%
-445%
Conclusions
In this work, we address the problem of model editing by analyzing discriminative properties of the feature maps. We leverage the notion of discriminative components to derive algorithms for two rele- vant tasks: structured pruning and class unlearning. Additionally, in order to identify discriminative components, we derive new lower bounds on the TV distance. These lower bounds also elucidate previously unknown connections between the Total Variation distance and discriminative classifiers, specifically the Fisher discriminant and the Minimax Probability Machine. Our experiments show that the model editing algorithms derived by our methods are highly effective, and match or outperform current state of the art in structured pruning, and can reduce accuracy almost completely class, while maintaining accuracy on the remaining classes. This work, however, currently analyzes discriminative components (and thus, subnetworks responsible for classwise predictions) in classifier models. Current avenues of research include extending these results to unlearning and pruning of generative models well. Lastly, the techniques proposed in this work can be extended to other editing tasks as well, such as debiasing.
a given
10

---

Acknowledgments
The authors gratefully thank Shell India Markets Pyt Ltd, for their generous support and contribu- tions to this work.
References
[1] A. Bakshi and P. Kothari, Outlier-robust clustering of non-spherical mixtures. arXiv preprint M.Xiv:0005002970 2020.
[2] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference
computer vision and pattern recognition, pages 6541-6549, 2017.
[3] A. Bhattacharyya, S. Gayen, K. Meel, Myrisiotis, Pavan, and N. Vinodchandran. On approximating total variation distance. arXiv preprint 220.::::200.722009 2022.
[4] D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag. What is the state of neural network pruming? Proceedings of machine learning and systems, 2:129-144 146, 2020.
[5] L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot. Machine unlearning. In 202/ IEEE Symposium Security and Privacy (SP),
pages 141-159. IEEE, 2021.
[6] H. Cheng, M. Zhang, and J. Q. Shi. A survey on deep neural network gruning-taxonomy, comparison, analysis, and recommendations. arXiv preprint urXiv:2308.06760 2023.
[7] S. Davies, A. Mazumdar, S. Pal, and C. Rashtchian. Lower bounds on the total variation distance between mixtures of two gaussians. In International Conference on Algorithmic Learning Theory, pages 319-341. PMLR, 2022.
[8] L. Devroye, L. Györfi, and Lugosi probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013.
[9] R. O. Duda, P. E. Hart, et al. Pattern classification. John Wiley & Sons, 2006.
[10] R. Eldan and M. Russinovich. Who's harry potter? approximate unlearning in 1lms. arXiv preprint arXiv:2310.022358 2023.
[11] Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural
networks. In International Conference Learning Representations, 2018.
[12] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https //openreview
[13] R. Gandikota, J. Materzynska,
and D. Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2426-2436, 2023.
[14] A. Golatkar, A. Achille, and S. Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304-9312, 2020.
[15] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
[16] V. Gupta, C. Jung, S. Neel, A. Roth, S. Sharifi-Malvjjeddi and C. Waites. Adaptive machine unlearning. Advances in Neural Information P Processing Systems, 34: 14:16619-16330, 2021.
[17] M. Hardt and Price. Tight bounds for learning a mixture of two gaussians. In Proceedings of the ferty-seventh annual ACM symposium on Theory of computing, pages 753-760, 2015.
[18] B. Hassibi and D. Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
[19] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training neural networks. Journal of Machine Learning Research, 22(241): 1-124, 2021.
[20] 7. Hou and S. Kung. feature-map discriminant perspective for pruning deep neural networks. arXiv preprint arX1v:2005.13796, 2020.

---

[21] Z. Hou and S.-Y. Kung. A discriminant information approach to deep neural network pruning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9553-9560. IEEE, 2021.
[22] Z. Izzo, M. A. Smart, K. Chaudhuri. and J. Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages PMLR, 2021.
[23] J. Jia. J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu, Model sparsity can simplify machine unlearning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httLtttts///peeeee http://pppnreeeeeeeeeeeeooooo peet//orummid==02HBBB ZH883134
[24] S.-J. Kim, A. Magnani, and S. Boyd. Robust fisher discriminant analysis. Advances in neural information processing systems, 18, 2005.
[25] J. M. Kübler, W. Jitkrittum, B. Schölkopf, and K. Muandet. A witness two-sample test. In International Conference on Artificial Intelligence and Statistics, pages 1403-1419. PMLR, 2022.
[26] M. Kurmanji, P. Triantafillou, J. Hayes, and E. Triantafillou. Towards unbounded machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
[27] G. Lanckriet, L. Ghaoui, C. Bhattacharyya, and M. Jordan. Minimax probability machine. Advances in neural information processing systems, 14, 2001.
[28] G. R. Lanckriet, L. E. Ghaoui, C. Bhattacharyya, and M. Jordan. A robust minimax approach to classification. Journal of Machine Learning Research, 3(Dc)):555- 582, 2002.
[29] Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. Advances in neural information processing systems, 2. 1989.
[30] B. Li, B. Wu, J. Su, and G. Wang. Eagleeye: Fast sub-net evaluation for efficient neural network
pruning. In European conference on computer vision, pages 639-654. Springer, 2020.
[31] H. Li, A. Kadav, I. Durdanovic. H. Samet, and H. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608 1608.08000 2016.
[32] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets.
In International Conference on Learning Representations, 2017. /ttp:tttp://ooperrvvie
net/forum?id-r_gpee..
[33] J. Li, R. Wang, Y. Lai, C. Shui, S. Sahoo, C. X. Ling, S. Yang, B. Wang, C. Gagné, and Zhou. Hessian aware low-rank weight perturbation for continual learning. arXiv preprint arX1v:2311.1516,, 2023.
[34] I. Lebbenweii, C. Baykal, H. Lang, D. Feldman, and D. Rus. Provable filter pruning for efficient neural networks. In International Conference Learning Representations, 2019.
[35] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEFCVF conference on computer vision and pattern recognition, pages 1529-1538. 2020.
[36] L. Liu, S. Zhang. Z. Kuang, A. Zhou, J.-H. Xue, X. Wang. Y. Chen, W. Yang, Q. Liao, and W. Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021-7032. PMLR, 2021.
[37] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11264- 1264-11272, 2019.
[38] P. Moichanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Leurning Representations, ICLR 2017 2017-Conference Track Proceedings, 2019.
[39] C. Murti, T. Narshana, and C. Bhattacharyya. Tvsprune-prunin non-discriminative filters via total variation separability of intermediate representations without fine tuning. The Eleventh International Conference Learning Representations, 2022.
[40] B. Mussay, D. Feldman, S. Zhou, V. Braverman, and M. Osadchy. Data-independent structured pruning of neural networks via coresets. IEEE Transactions on Neural Networks and Learning Systems, 2021.
12

---

[41] T. Narshana, C. Murti, and C. Bhattacharyya. Dipe: Data flow driven pruning of coupled channels without data, In The Eleventh International Conference on Learning Representations. 2022.
[42] T. T. Nguyen, T. T. Huynh, P. L. Nguyen, A. W.-C. Liew, H. Yin, and Q. V. H. Nguyen. A survey of machine unlearning. arXiv preprini arXiv:2209.02999, 2022.
[43] P. Prakash, C. Murti, S. Nath, and C. Bhattacharyya. Optimizing dnn architectures for high speed autonomous navigation in gps denied environments on edge devices. In Pacific Rim International Conference on Artificial Intelligence, pages 468-481. Springer, 2019.
[44] T. Räuker, A. Ho, S. Casper, and D. Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTTM.)) pages 464-48% IEEE, 2023.
[45] B. D. Ripley. Pattern recognition and neural networks. Cambridge university press, 2007.
[46] S. Sahoo, M. Elaraby, J. Ngnawe, Y. Pequignot, F. Precioso, and C. Gagné. Layerwise early stopping for test time adaptation. arXiv preprint arXiv:2404.03784 2024.
[47] A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34: 18075-18086 2021.
[48] H. Shah, A. Ilyas, and A. Madry. Decomposing and editing predictions by modeling model computation. arXiv preprint arXiv:2404.11552 2024.
[49] S. S. Shapiro and M. B. Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(31)):511-611, 1965.
[50] M. Shen, H. Yin, Molchanov, L.. Mao, J. Liu, and J. M. Alvarez. Structural pruning via latency-
saliency knapsack. Advances in Neural Information Processing Systems, 35:12894-12908 2022.
[51] Singh, V. K. Verma, Rai, and V. P. Namboodiri. Play and prune: Adaptive filter pruning for deep model compression. arXiv preprint 1eXiv:1905.044
2019.
[52] Y. Sui, M. Yin, Y. Xie, H. Phan, S. Aliari Zonouz, and B. Yuan. Chip: Channel independence-
based pruning for compact neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[53] M. Tukan, L. Mualem, and A. Maalouf. Pruning neural networks via coresets and convex geometry: Towards no assumptions. Advances in Neural Information Processing Systems, 35: 38003-38019, 2022.
[54] C. Wang, Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow, arXiv preprint arXiv:2002.07376 2020.
[55] H. Wang, C. Qin, Y. Zhang, and Y. Fu. Neural pruning via growing regularization. arXiv preprint arXiv:2012.099 09243,, 2020.
[56] J. Wang, S. Guo, X. Xie, and H. Qi. Federated unlcarning via clas-diisriminnttiee pruning, In Proceedings of the ACM Web Conference 2022, pages 622-632, 2022.
[57] L. Wang, Zhang, H. Su, and J. Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[58] Z. Wang, E. Yang, L. Shen, and H. Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09216 2023.
[59] R. Yu, A. Li, C.-F. Chen, J. Lai, V. L Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2018
[60] S. Yu, Z. Yao, A. Gholami, Z. Dong, Kim, M. W. Mahoney, and K. Keutzer. Hessian-aware pruning and optimal neural implant. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 3880-3891, 2022.
13

---

Appendix
In this appendix, we provide the following material.
In Appendix
we provide brief survey of the literature that addresses both structured
pruning.
In Appendix
B
we provide proofs for the theoretical results proposed in this work.
In Appendix
provide additional analysis of the practical performance of our algorithm.
In Appendix we provide details for variants of the DISCEDIT-SP algorithm, such the DISCEDIT-SP-FF DISCEDIT-SP-M, DISCEDIT-SP-R, and DISCEDIT-SP-E variants. We also discuss the use of the BatchNorm random variables as opposed the entire feature map, in order to reduce memory storage, well as deriving TVSPrune (39 using our results.
In Appendix B provide additional experiments, including more detailed results from experiments conducted in Section
In Appendix F we provide additional details, such as hyperparameters and dataset splits, used in this work.
A Related Work
A.1 Structured Pruning
In this section, we detail related work in the space structured pruning Pruning is a long-standing strategy for reducing the inference time and power consumption of neural networks [2]]8]] [8] Pruning algorithms can be divided into structured wherein entire filters or neurons are removed or
unstructured wherein individual weights removed techniques [19 4 We refer readers to and the references therein, for a detailed discussion on unstructured pruning methods.
19
A.1.1 Structured Pruning that Requires the Loss Function
Structured pruning methods are of interest as they enable the reduction of inference times without Pruning algorithms require saliency measures that quantify the importance of filters or neurons, and prune the unimportant ones [19]. Pruning
requiring specialized hardware or software 19
38 41
algorithms where access to the loss function is not only assumed, but is necessary, generate saliencies using gradient information 38 37 43 50], and the loss function Hessians, or approximations thereof
29
36
55 54
For more similar works, we refer readers the surveys [19
A.1.2
Structured Pruning without the Loss Function
In this section, we discuss structured pruning methods that do not require derivatives of the loss function, aside from fine-tuning. There variety of approaches to solve the problem of identifying which filters to prune without using the loss function. Works such as BI 32
use the norms of the weights identify filters to prune. Other works, such as [41 559] identify which filters to prune by bounding the reconstruction error incurred by pruning a given filter. Works such as 35| 52 analyze
the rank of feature maps measure filter importance. Another body of work maps to identify subsets important filters, as described in works such as (34 some works propose identifying important filters by their discriminative ability. Recently, in 20 21).
coresets of feature
53
40). Recently,
discrimination-awwrr metrics were used to identify filters important to classification accuracy. In 39
the class conditional distributions were assumed to be Gaussian, and the Hellinger distance between the class conditional feature maps was used to identify important filters. In this work, we formalize the notion of discriminative ability in terms of the Bayes risk, and derive novel lower bounds on the TV distance to effectively approximate it. Furthermore, we emphasize that this work is clearly differentiated from prior art this area in these critical respects: first, no assumptions are required of the class conditional distributions; second, the discriminative ability of a filter is formalized in terms of the Bayes error of a classifier trained on the feature maps it yields; third, the pruning algorithms derive using method require access to the training data loss function.
14

---

A.2
Machine Unlearning
In this section, we provide detailed literature survey on machine unlearning, both with and without model editing.
A.2.1 Machine Unlearning without Model Editing
Machine unlearning has gained importance in recent years owing to data privacy and security concerns
S 42.. A wide variety of works exist address this problem. Several works aim to forget data points, even in the adaptive setting, while maintaining the accuracy of the model, such as [6] [2] [4]. The work in [47] also provides bounds on the number of samples that model can be allowed to forget before accuracy degradation. Machine unlearning is also a significant area of research in the space of large language models, as noted in [26 10,, and generative models
Another aspect of machine unlearning is selective forgetting, wherein classes, groups, or sets of samples forgotten from the network, as described in (58) and the references therein. This connects machine unlearning to the continual learning setting as well, as described in [57 and the references cited there.
A.2.2 Machine Unlearning with Model Editing
While there has been significant research into selective forgetting and model unlearning, facilitating selective forgetting and machine unlearning editing models remains underexplored field. Recent works such as [23] 46) explore the effect of sparsity on machine unlearning and continual learning; in particular, [23 shows that sparsifying models prior to unlearning can increase the effectiveness of
However, recent work such as [56] investigates using pruning for model unlearning in the federated setting. More recently, 48 uses pruning selectively forget single class from ResNet50 imagenet model with minimal loss in accuracy. Our work differs from prior work in this space since we directly use the discriminative ability of model components to identify which components remove to forget given class.
B Proofs of Main Results
In this section, we provide the proofs for the main theoretical results proposed in the paper. Specifi- cally, we provide proofs for Theorems 1-3 and Corollary
B.1 Proof of Theorem 1
In this section, we provide the proofs for Theorem T
Theorem. Let P. Q be probability measures supported X Bd and let p and be the corresponding densities. Let be the set of functions with bounded first and second moments defined on X. Then,
(Fp - Fa)²
TV(P.Q) > sup fEF
(11)
((2) + ((2)
15

---

Proof. Choose an arbitrary f E F. Then, we have
(Js -a) (Ls (f(x)) ((x)) (x))dx)
= -(x (v (p(x) (Vle(z)
g(0)) (p(z) g(xx) f f(z))z))
(V. // (o(x)) ((<))dx)) (VV (p(x)) (x)))
(by Cauchy- (auchy-Schwarz))
2TV !)) (/, (p(x) (x))
(by Definition 1
S 2TV Q) (/ (p(x) (())) fz
2TV Q
(() f(2))
Thus, for any arbitrary f E F, we have
(Jp - fo)
2TV Q
T00 + [(22)
from which it follows that
(fp - fa)²
(TV Q)
sup fet
(12)
((2) + fq
L
The proof of Theorem follows from the fact that choose ((x) = u p(x), and apply the formula for the expectation. Then, maximizing over FF is equivalent to maximizing over u. Thus, the proof is completed.
B.2 Proof of Theorem 2
First, we restate the Theorem for clarity.
Theorem. Suppose and Q are sets of moments of two probability measures supported on X. Let (x)) be vector of polynomials such that Exly((X) g(P). Be(p(X)] = 9()), G(Q), and ler f = u (x(X) -
(x)) G(P), and (eelp(XX)((X))
2
be a witeess function.. Then, for any P E Sz (P), Q E (Q). supported on a set C Rd, we have
DTv(P.Q:K) > (P
(13)
2 STy (P,Q)
where
(Py Q) = (Ag) (G(() + G(Q))
(14)
and =-= gPP) g)Q) and -
Proof. First, recall that Theorem I provides lower bounds for all distributions for which the moments of the witness function are given by fp. ((2) Sg f(2) Begin by choosing
f=ul T (((x) g(P) +(P)+((Q)
2
where d is constant. Then,
1 u (g(Q) g(P))
F, n" (g(P) (PP))(QQ)) and f,
2
16

---

and
Bp (((x) g(P) + =0()) ) (p(() g(P) + =(Q))
fp = T
u
2
2
2 (x(xx) - - =))) a (setting ー= 9(P) + ((Q)
T
- 4)
2
u
T
E
(p(xx) - (P) + (P) A) (f((x) (P) A) =) u
T (((P)- - (P)) + (o(P) ( -QQ) -(Q))) u.
Similarly, we have
f - =() +
u.
Substituting this into we get
TV(P,, Q 2 sup aERd a (G(P) - ((P))()) + G(Q) (QQ])()) + (u (g(P)
1
2 (QQ))
sup ue 2 2u (G(P) (PP)g(P) + -9())(())))))))))))))))))))))))) (
(u (v())))))(
" (G(P)+(()))
sup
2 + (g(P)))1111111
uER4
(G(P)+G(())
(u (g(()(P))(QQ))
(G())+((())
sup
(a (x())-((11))
2 +
a (()))
Let
STv(P, Q: (T((P)))(Q)) -
([GG())+((O)) G(Q))²
Then,
(g ,(Q)))²
Siv max u: uT (T((P)) + G(Q))u
Thus, we have
TV(P, Q) > SPP.QQ)
(15)
2+ (P,
To show the second inequality, we first state the following Lemma.
Lemma 1. Suppose and Q are sets of moments of two probability measures supported on
X. Let (x( be vector of polynomials such that Ep Bp(s((X)] g(P), Relp(X)) = 9{Q), Ex(((())x(X)) G(P), and Bg (x))))))X))) G(QQ), and ler
|u TuT(on (g(P) 9(Q))
(u](((()) (Q)))
SH(P, u)
and (P,(P) u (G(P) G(Q))
Vu G(P)u Vu G(Q)u)
and let
SH Q) = arg max arg
(H PP Q u) and ST Q) = arg max STV (P, Q
Then,
2
Q)(PQQ)
Siy (P Q)
<
(16)
V2+ SH
2+ siv (P, QQ
17

---

Proof. To prove this statement, we first show that [ Q) I Siv (P, Q). Fix
? arg max S Q: u).
From this, we see that
Si (á -
(a - (())))²
V + / G(Q)a)
a & + + a
G(P)av/ 0
((Q)
$ (
(G(P)u+ G(Q)) S STv(P,
Next, let
Sty (P. Q
SH Q
2
di
TV PP and 2d2
2 +
V2+ (2
We have
(PP.QQ) Q)
(P,2Q)
d2 2 2+ S Q) 2 2
SP,
Since
is monotonically increasing for positive I, we prove the statement by the fact that
2+3
STV
With these results, the Theorem is proved.
B.3
Proof of Corollary 1 Computing the Bayes Classifier and TV(P, Q) from the Lower Bound
In this section, we prove Corollary ! Recall that the lower bound proposed in Theorem i not tight, as the Cauchy-Schwarz inequality used in the derivation of the bound is only not strict when the witness function is a constant. However, there are cases where the bound can be used to compute the true TV distance. We show this case in Corollary [ which we restate and prove in the sequel.
Corollary. Suppose P = N (Hpp Σ) and Q = (Ng. Σ) Let f(æ; u) = T (x 2pp - Hg)) be witess function. Then,
TV(P, Q) =24 24 (V(ut)T(HH (pp) 1n)12) 1,
where
(E u) Exwal [ (x; u)))
u' = arg max
Ex- If (z: u + E E [z; u
Proof. First, following the proof of Corollary 2 we have u' = Σー (14p Mg). Note that this is identical to the weights of the Gaussian discriminant classifier discussed in, say, 9. Substi- tuting u* into the expression TV(P, Q) 2Φ (V(u*)TTrr (Hp 12)/2) 1, we get TV(P, !
24 (Hp Mq) E (Mp m2)/2) 1. The risk of the Bayes classifier, as given in 45 is
(1p 1gy (1p - m2)/2), where ((x) is the Gaussian edf. Using the fact that (xx) = 1 (--r), and the identity 2 (P. Q) = TV(P, Q), we get TV(IP, Q)
R (P, Q) Φ0 Φ
24 (ee, Ma) Σ " - -my5/2) 1. Note that with this choice of u=, the square root term remains well-defined. This matches the well-known result for the TV distance between Gaussian measures with the same variance. Thus, we prove the statement.
Remark: This result also illustrates the case where the Bayes' classifier lies in the set of functions F: {f(x) (xx) n. ((r) for given function (zx)). In this case, if (p(x) = [1pp pg)) and P and Q are Gaussian with the same variant, the Bayes classifier is equivalent to the Fisher discriminant.
18

---

B.4 Proof of Theorem 3
In this section, we prove Theorem thereafter.
We restate the result for convenience, and prove the Theorem
Theorem. Suppose P, Q be two probability measures supported on X S Rd, with densities p and and let Up Ep (2], My = Eggat [x] and Cp = Eglazz J, Cg Egg [ax Suppose we have plug-in estimates Hp, Cp. Pag Cg as defined in (7). that satisfy
1pp Mpll2 S 6p and l|اg Rull2 S 6g
|C,p - Cplle S Pp and |10g Cg |F S Pq.
Then, with witness function of the form [(x) = Txx
DTv(P, Q)
min MeMidEMM
(Ap) + Cy + pl) pA)-)
(17)
where M = {(pp: Pa) 111b Ppll2 I 6p, l1 Bol2 I 6,}. Ap = Mp Myp and p = Pp + Pr
Proof. First, note that f(x) uuTaa Thus,
f = u T Up and Fn = u Pu
f/2) Opu and f(n) u Cgu.
Thus, for exact values, we have
TV(P, Q) 2 DTv Q:) (u (up pa)) (1p- (C, (mp -Ma). (Cp + Cy)u
However, there are errors that arise from using plug-in estimators (defined in 7 To handle the estimation error for Cp and Cg. we rely on (23) Lanckriet et al. 228. That is,
u CpCpu = u (Cp + pT)u.
max Cp:1cc-Cppless
We substitute these values back into the lower bound, and impose the constraints on the μpp and Pa Thus, we prove the theorem.
Remark B.I. The proof follows a similar logic that of the derivation of (15) in Kim et al. 24
C Additional Results: Computational Complexity of Fıg scores with Different Witness functions
In this section, we detail the computational cost of computing the rı, scores needed for DISCEDIT- SP and DISCEDIT-U, We tabulate our results in Table 3
Explanation for Storage Complexity For a fixed witness function, for each filter, we only need
to store 4 real numbers per class (that is, for distribution P, we store fc, f!!2) fe, and f( For Fisher and MPM type witness functions, we need to store the class-conditional and class- complemnp means and covariances of ((X) for each class, in total requiring the storage of O(n²) values. For DISCEDIT-U, since we only need to compute scores for single class, the dependence on C vanishes.
Explanation for Computation Complexity The computational complexity of one-shot pruning of a given layer using DisCEDr-SPdepende on the number of filters L, the number of classes in the dataset C, and the witness function itself. If the Witness function is fixed priori, the cost of computing each TV distance is 0(1). Thus, in this case, the complexity is O(LC)) to compute all the pairwise TV distance lower bounds, and O(LC) Fisher discriminant-type scores (such as are used in DISCEDIT-SPA Section 7.3), given that (p(X) is vector of length n, this requires the inversion of matrix, which requires (nz²) ) iterations. Thus, in this case, the cost is O(LCn²) iterations (this subsumes the cost of finding the minimum). For MPM based witness functions (such as those used in DISCEDIT-SP-Q), this requires solving an SOCP (Ó(n²) complexity, Ô(-) suppresses c accuracy terms). Thus, the cost is Ó(TCCn)). For DISCEDIT-U, since we only need to compute scores for a single class, the dependence on C' vanishes.
find the minimum for each filter. For
or the witness function used in
19

---

Table 3: Comparison of complexities of computing different winess functions. (P) refers to peuning (used in DISCEDIT: SP) and (U) refers unlearning (used in DISCEDIT-U)
Witness Function
Cost (P)
Storage (P)
Cost (U)
Storage (U)
Fixed (a priori)
O(LC) O O(LCn))
O(LC)
O(L)
O(L)
Fisher-type Witness Function
O(LCn O(LCn²)
O
(LL)))
MPM-type Witness Function
O
O
D Variants of the DISCEDIT-SP Algorithm
In this section, we propose a variety of variants of DISCEDIT- CECEEED-SPaagrrttiiiii SPalgorithm. In particular, we highlight how changing the witness function used lower bound the total variation distance can lead to new algorithms. Moreover, we show how TVSPrune [39 can be recovered from DISCEDIT-SP. indeed, is a special case of it.
D.1 Fisher-based Lower Bounds and TVSPrune
We choose ((X) u. xX)). Let
P3.0 EXND, [xxD,Ip(X))
and let
E = ExND. - -1x)))(166(67)) -l/c)
Then, we get
Algorithm 2: DISCEDIT-SP-F
Input: Class conditional distributions De, c E (C), pretrained CNN with parameters
W (W1, Wt). layerwise sparsity budgets B", witness function f
for E [Z] do
syy] ON, Compute w/e Mj Ei Ei for all j.C
Set St
= [s]
Compute ry min, Fish" (D).e: Dj,c) for all j.
j E sorts, ( then
Set S =1
Output: Sparse Masks S return W
If (X) ) is vector of quadratic functions of X, call the algorithm DISCEDIT-SP-FQ.
D.2 Minimax Probability Machine based Algorithms
We define 2de 2 Ej E as previously. We then state the algorithm as follows.
Algorithm 3: DISCEDIT-SP-M
Input: Class conditional distributions De. c E (C), Pretrained CNN with parameters
W- (Wi, Wt), layerwise sparsity budgets B', witness function f
for E [L] = 1s syy ON:
Set St
Compute whe Mr... Ej Σj for all j.C
Compute r min, MPM* (D)... D),e) for all j.
for all j.
j E so Ni) then Set s) 11
Output: Sparse Masks St...., return VV
If (XX) ) is vector of quadratic functions of X, call the algorithm DISCEDIT-SP-MO.
20

---

D.3 EnsemblePrune Taking the best of FisherPrune and MPMPrune
We choose the same witness functions as we did for DISCEDIT-SP-M and SPS,,, and define the moments in the same fashion. We then
Algorithm 4: DISCEDIT-SP-E
Input: Class conditional distributions De, CE [C]. Pretrained CNN with parameters
W = (W1, Wt), layerwise sparsity budgets B', witness function f for I E [L] do
Set S = 1 [N ON.
Compute 2 M3 2z 21, all j,c
Compute
|u (n', 11.aa)
(u (16,. 113,0) )
max
max
u Di + Vulsiy S. 5'
max
2u (E) - 20,,)) )2)
j. if E sort ( then Set a =1
}1=11)
Output: Sparse Masks S...... St
return W
D.4 RobustPrune Accounting for Errors in Moment Measurements
We choose the same witness functions as did for DISCEDIT-SP-M and DISCEDIT-SP-E, and define the moments in the same fashion. We then derive the following algorithm.
Algorithm 5: DiSCEDIT-SP-R
Input: Class conditional distributions De, C [C), pretrained CNN with parameters
W (Wi..., Wt), layerwise sparsity budgets B', witness function f = u "p(x), error
tolerance
for [Z] do
Set St = [+]
s,,,1= ON:
Compute plug-in estimates of 2fen pla En E/, for all j. Compute
1 min
min Mi
(u3. (3i)) T (E)) + En + (ph. + p).))) "M)j u²,c)
s.t. IInj,a 1// s 8/1m
I 1//el1 S 8/
for all j- j E sorts, ((r) ) then Set s) s) =
Output: Sparse Masks return W
St
D.5 Recovering TVSPrune
In TVSPrune, at any layer I, the jth filter is pruned if
Al.
where Ar the minimum Fisher discriminant between pairs of classes. Suppose that we identify important and discriminative filters by measuring the TV distance in pairwise sense. Recall that Corollary 2 gives us a bound that is also monotonic in A.j. If we apply the strategy that we prune all filters with score less than threshold, would prune filter j if
A
< 7
2
A
21

---

for some 7 E (0, 1). We can now find relation between and 1,. First, note that if - - S then A S 4(1 77). Similarly, if we prune expression
A 2+
S then A 27 Equating the two gives us the
3 -
1 27 2
Thus, both TVSPrune and a variant of DISCEDIT-SPP S-P where the TV distance is measured pairwise, and which prunes at threshold are equivalent, as they require pruning the jth filter if
37
A $
4n
27 2
D.6 Using the BatchNorm Random Variables
The BatchNorm random variables for this layer are given by
BN' (x), Y (x) (X), BN (x)
(18)
As stated earlier, our goal is to minimize the TV distance between the distributions the pruned and unpruned features; we the BatchNorm random variables proxy for the features y (X().. Next, the moments of BN' are given by
Ex-D [x) =BN! [at] and Var(BN;(X)) (of)².
(19)
Suppose BN' (X) is drawn from the distribution BN, BN, be the eth class conditional distribu- 3.c
tion, and let D BN. be the distribution of features sampled from the complement of class
E Additional Experiments
In this section, we detail additional experiments conducted the course of this investigation. In particular, provide experimental results highlighting the utility of lower bound on synthetic datasets, and provide expanded results for the Gaussianity tests provided in Section 6.1.
E.1 Measuring the TV Distance between Linearly Inseparable Data
In this section, separated datasets. We choose the "Two Spirals' Dataset, and a dataset consisting of two zero-mean Gaussians with different variances. We choose f(x)
the bounds proposed in Corollary
2 to bound the TV distance between poorly
(((X) (51 f2)/ where ((X), when chosen to be of degree d 2 1, is given by xx() 1 (1 X) +.. (1 x)d. We present our results in Figure 2] We observe that using the lower bound for Fisher outperforms the MPM lower bound on both toy datasets. However, the choice of polynomial witness functions clearly outperforms lower degree choices, particularly in the 'Two Gaussians' case.
F.2 Effective Pruning of Hard-to- -rrune Layers
In this section, we utilize the lower bounds provided in this paper to prune hard-to- prune layers in neural networks. As noted Murti et al. [9 Liebenwein et 34,, some layers, in particular the initial layers in the case of VGG-nets, are difficult to effectively sparsify. In this set of experiments, we aim to show that using the lower bounds proposed in this work, we are able to better identify discriminative filters in hard-to-prune layers, and therefore prune those layers more effectively. Experiment Setup: We select a VGG16 model trained on CIFAR IO. We fix pruning budgets of 40%, 50%, 60%, 70%, 80%. For each model, we then prune three hard-to-prune layers in isolation, and measure the impact on accuracy. We compare the following methods:
TVSPrune: We modify TVSPrune to prune fixed budget, and using the BatchNorm random variables as described Appendix D
DISCEDIT-SP-EO: We apply Algorithm 5 as presented in Appendix D Dusing the features ip(1 X) [1 X, (1 }. Thus, for each c, fie) u (xx(X) (p. (=)/2)).
22

---

Two Spirals Datazet
0.00
1.2
statee bound
T74 spance laver
0.35
0.30
1.00
1.00
Digres
(a) Two Spirats. lower bound: .172
(b) Tho Spirzis. H lower bound: .172
Teo Cauosions with means Dataset
Chass
10-
Cluss
5/752
TV 34400- Lowe Wunr
-15
Pelyoomen Degree
(c) Two Spirals. lower bound: .172
Two Spirals. lower
Figure 2: Comparison of the performance of DISCEDIT-SP with DISCEDIT-SP M with polynomial features the TwoSpicals and Zero- Means Gaussians dalasets.
BAssai Spac Ty Sayes 2
Auseroueely 2ives
Aaceea in battad Lapers
(a) Layers 0-2
(b) Layers
(e) Tayers
Figure Comparison of performance DISCEDIT-SP with TVSPrune and Ll pruning hard-to-prune layers in VGGI6 trained CIFARIO
We use the L1 norms of the filter weights, proposed in 32 Results and Discussion We present our results in Figure The experiments show that DISCEDIT- SP variants using quadratic features (using algorithms outperform both TVSPrune and the L L-nmm--- based pruning strategy. In particular, we see that at 70% sparsity in Layers 1-3, the models obtained by DISCEDIT-SP-ED are 6.6% more accurate than those obtained using TVSPrune.
Li -based Pruning:
E.3 Verifying class-conditional Feature Maps are not Gaussian
In this section, we attempt to validate the assumptions made in Murti et al. [39 about the normality of the class-conditional feature distributions. To do so, we apply the Shapiro-Wilks test [4). a standard test for Normality.
We consider a VGG16 model trained on CIFARIO. Let BN (X) 1 TY For each I.J, we collect 100 samples from each class c E [10]. We then apply the Shapiro- Wilk normality test [49], and we compute p/, values, which are the minimum p-values from the Shapiro- Wilks test computed for the features of the jth filter in layer I conditioned on class We consider that
Experiment Setup:
23

---

filter's s features are unlikely to be Gaussian if p),c < 0.1. We plot the heatmaps of p = min [c] PA.. values for 15 randomly selected filters in Figure to indicate the normality the least Gaussian class-conditional features.
Results: We observe that for most layers, particularly those close to the output, the class-conditional feature distributions are highly unlikely to have been drawn from Gaussian, with P/,e. values for layers 10-12 VGGI6 typically being below le 5. For layers with filters that yield likely- Gaussian features, we observe that for the majority of filters, at least one feature output is likely to be non-Gaussian. We present this data in Figure 4
Laver1 Gramants
Layer Coumterty
Laper Couater
644
(a) Layer
Layer
Layer
Laven Deumaantt
Lepen 2 Gounsion
Leper Gaunalanty
(d) Layer 4
(e) Layer
Layer
Lape Guundar
Lapal Glundiar
(g) Layer 7
(h) Layer
i Layer
Layer 10
(k) Layer
(1) Layer
Figure 4: Shapley Wilkss test p-value heatmaps applied class conditional features generated by filters in each layer, an-axs the class index, and y-axis the filter index. Filters shown the heatmap selected randonly.
E.4 Ablation Experiments for DISCEDIT-SP
In this section, we conduct slate of experiments aimed at demonstrating that the effectiveness of DiSCEDIT-SP is not dependent on particular instantiations of models. Specifically, we apply DISCEDIT-SP to multiple instances of models trained on CIFAR10 CIFAR 100, and Imagenet. We present specific results in the subsequent subsections.

---

E.4.1 CIFARIO Experiments
In this section, run DISCEDIT-SP on 10 different instantiations of different models trained CIFARIO. We observe that DISCEDII-SP performs well on all instances of the models, irrespective of architecture. Our results are tabulated in Table 4
Table Ablations models CIFARIO. consider different and average accuracy after pruning/llae-uuing over Uhem.
Model
Sparsity
Best Acc. drop
Worst Acc. Drop
Mean Acc. Drop
VGG16
VGG16
61.2%
0.19
0.51
0.35 0.03
VGG16
75.6%
1.27
1.38
1.34 0.01
RexNer56
41.2%
+0.03
0.05
0.01 1 0.01.
ResNet56
60.7%
1.21
1.30
1.24 0.02
E.4.2 CIFAR100 Experiments
In this section, we run our experiments on different instances of VGG16, VGG19, and ResNet56 models trained on CIFAR10. We present our results in T Our results show that the effectiveness of DIsCEDrT-SP is unrelated the particular instantiation of the model.
Table S: Ablations for models trained CIFARIOD. We consider different models, average accuracy of pruning algorithm them.
Model
Sparsity
Best Acc. drop
Worst Acc. Drop
Mean Acc. Drop
VGG16
40.8% 60.6%
+00.33 0.12
0.06
0.03 0.06
VGG19
0.19
0.16 + 0.03
+0.13
10001
E.4.3 Imagenet Experiments
In this section, we provide ablation experiments for Imagenet. However, owing to the computational cost of training Imagenet models, we only train additional instantiations of the model. Moreover, we were only able to fine-tune for 30 epochs. However, DISCEDIT-SP broadly works well on all of the instantiations, seen in Table
Ablations
on Imagenet.
models, average accuracy drop.
Best Accuracy
Worst Accuracy Drop
Mean accuracy drop
3.1%
8.3%
6.6%
E,4.4
Pruning without Fine-tuning
In this section, highlight our pruning experiments without fine-tuning models. We fixed layerwise sparsity budgets. We use models trained on CIFARIO and Imagenet datasets. Our experiments, tabulated in Table 7 shows that our method consistently matches or outperforms common baselines.
E.5 Pruning the High Sparsity Regime
In this section, we prune models trained on CIFARIO extensively, with over 80% of parameters pruned in total. Fine-tuning is done in one-shot fashion. We see that DISCEDIT-SP consistently matches our outperforms baselines such as Murti et al. [39 or Sui et Our results are tabulated
in Table g
25

---

Table 7: Pruning results with DISCEDIT SPwithout fine tuning.
Model
Datuset
Sparsity
Acc. Drop lours)
Acc. Drop (39
Acc. Drop [2
Acc. Drop (1.1)
VGG16
VGG16
CIFARIO
41%
4.56
5.21
6.53
16.59
VGG16
CIFARIO
63%
12.79
12.65
32.8
VGG19
CIFARIO
30%
0.98
1.22
6.55
VGG19
CIFARIO
44%
2.56
4.53
13.8
VGG19
CIFARIO
60%
6.16
8.44
23.67
ResNet50
ImageNet
11%
18.37
22.61
21.49
35.52
ImageNet
Table Pruning results with DIsCEbrI-SP CIFAR 10 models high-sparsity regime
Sparsity
Algorithm
Ace. drop (no FT)
Acc. drop (FT)
84.7%
CHIP
83.1%
0,20%
VGG16
TVSPrune
84.7%
DISCES
82.9%
0.1.5%
88.7%
TVSPrune DISCEDIISSP
84.8%
0.17%
VGG19
88.7%
84.7%
0.16%
74.3%
84.0%
ResNe156
74.3%
TVSPrune
83.9%
0.25%
74.3%
DISCEDIT SP
84.2%.
0.28%
E.6
Discriminative Component Discovery for CIFAR10 models
In this section, we provide plots of vf, values for different layers in models trained on CIFARIO. 10. We show plots for different layers for classes, to highlight how discriminative components look.
E.6.1 ResNet56 trained on CIFAR1O
In this section, we produce plots showing discriminative and nondiscriminative components for ResNet56 trained on CIFARIO. Our results are plotted in Figure Note that in layer 0, both class and class have filter as discriminative. Moreover, in subsequent layers, three classes have filters which are somewhat discriminative well.
E.6.2 VGG16 trained on CIFAR10
Similar to the previous subsection, we plot niE values for VGG16 trained CIFAR10 We plot selection of our results in Figure 6 As before, we see that we can clearly identify discriminative and nondiscriminative filters. Note again that class and class share discriminative filters both layer 0 well as in layer 4.
Additional Experimental Details
In this section, we detail additional experiments not mentioned in the main paper, as well as comprehensive description of our experimental setup.
F.1
Pruning Setup
In this section, we discuss our experimental setup.
F1.1
Platform Details
The hardware used for the experiments in this work are detailed below:
1. Server computer with 2 NVIDIA RTX3090Ti GPUs with Intel i9- 19-12700 processors, running Ubuntu 20.04, with Python 3.11 and CUDA Tools 10.2 with PyTorch 2.0.1.
2. Desktop computer with NVIDIA RTX3070Ti GPUs with Intel 17-10700 processor, running Ubuntu 22.04, with Python 3.11 and CUDA Tools 11.7 with PoTorch 1.0.
26

---

12.0
010
0.81
600
0.01
6490
030
660
Cless
Layer
Layer Class
4.10
5-5-
402
88
10
Layer 12,
Layer Class
Layer
615
M
43-
8
(g)) Lager 30, Class
(h) Layer Class
@ Layer 30, Class
8
(i Layer 34, Class
(k) Layer 34, Class 3
( Layer 34, Class 8
Figure 5: rf,, plots for different layers, classes 0, for a ResNet56 trained on CIFARIO, æ axis the filter index, y-axis is rf..-
F.1.2 Models under consideration
We consider the following models.
VGG16/19 trained CIFAR10 and CIEAR100; We use the pre-trained VGGI 1/16/19 models trained on CIFAR10 and CIFAR100, The models achieve accuracies greater than 90% both datasets.
ResNet56 trained on CIFAR10: We consider a ResNer56 model trained on CIFAR10. We not prune layers that are part complex interconnections (such as the final layer each BasicBlock).
ResNet50 trained in Imagenet: We consider a ResNet50 model trained on Imagenet. We do not prune layers that are part of complex interconnections, as was the in ResNet56.
VIT trained CIFAR10 We trained a custom VIT on CIFARIO. The details of the model are given below in Table 9 The accuracies of this models are given Table I0
The accuracies of all models are given in Table II below:
27

---

0.8
000
60080
R
91
8
8
8
Cless
(b) Layer Classs
Layer Classs
0.0
0.4
030
0:00
0:401
Layer
(e) Layer
Layer
6.384
coans
449
8
(5) Layer 8, Class
(h) Layer Class
( Layer Class
8
(i) Layer 12, Class
(k) Layer 12, Class 3
()) Layerr 12, Class 8
Figure 6: rf, plots for different layors, and classes 0, for VGG16 trained on CIFARIO. x axis the filter index, g-axis is r
Parameter
Value
Context Length
65
Embedding dim.
384
Transformer Encoders
7
MLP layers to be pruned
14
Total params. Table 9: Summary model parameters.
6.27m
Dataset CIFARIO
Test Acc.
88.2% 69.5%
110AR100
Table 10: Tost aecuracy for CIFAR datasets.
28

---

Dataset
Model
Test Accuracy
VGGI6
94.16
ResNet56
94.37
CIFARIO
ResNet20
92.2
ViT
88.2
VGG16
74.0
CIFAR100
ResNet56
72.6
VIT
69.5
ImageNet
accurncy for 76.15
ResNet50
models.
Model Provenance
We list the sources the models below.
All CIFARIO and CIFAR100 models were obtained from:
https://github. com/ chenyaofo/pytorch http://ooo//ptoooooofffrrarrra cifar -ifarr-roarmre-ll
ResNer50 trained on Imagenet was obtained from:
attps://drive woogle.ccm/drvvvee.e.eoooeee.eee
which in turn comes from:
The VIT models use code from ottpa///ga//a.a///a//aa/aa/aaarr
https://github. www.
F.13 Dataset Selection
For Pruning Since assume that the training dataset is unavailable to us, utilize the validation set as a proxy for the data-distribution. We detail our dataset splits in Table 12 Note: the subset
Table 12: Breakdown of datasct splits used experiments.
Dataset
Training Set
TV Distance Set
Test Set
CIFARIO
Not used
4000 images from test
6000 images from Test set
CIFAR10
Not used
4000 images from test set
6000 images from Test set
Imagenet
Not used
30000 images from Validation set
20000 images from Val, set
used to compute the TV distances are not reused while measuring the test accuracy.
Class Unlearning We use the training identify discriminative filters, and the test set to measure accuracy. Typically, we get
F.1.4 Hyperparameter Details
In this section, we detail the hyperparameters used when fine-tuning pruned models. We present the hyperparameters for CIFAR10 and Imagenet models only, as we did not fine-tune models that used CIFAR100.
CIFAR10 Fine-tuning
We detail the hyperparameters used our CIFAR10 experiments below.
Batch Size: 128
2. Epochs: 50
3. Learning Rate: ,001
4. Weight Decay: .0005
5. Momentum paramters:
6. Optimizer: SGD
29

---

ImageNet Fine-tuning
We detail the hyperparameters used when fine-tuning Imagenet models.
Batch Size: 128, using gradient accumulation
2. Epochs: 100
3. Learning Rate: 0.08 (initial)
4. Momentum: 0.99
5. Weight Decay:0.0001
6. Optimizer: SGD
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All contributions tally with the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
See section 4 for limitations lower bound, including non-tightness and estimation uncertainty.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
Please see proofs in the appendix. Assumptions are also stated clearly,
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experiments are simple and clearly laid out, experimental details are available in the Appendix, anonymous code submitted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification:
Code submitted, please see link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
casssnooiccyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpipp/.ccppullc//uuuii wwwtc//gguie///eeeeceoocc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See ablation studies in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
33

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: refer to Appendix F for details on compute used.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Code of ethics followed, no interventions with living beings requiring special processing. Only standard datasets were used. No confficts of interest.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See impact statement
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: standard datasets/models used
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Source of models listed in Appendix
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "rO45EP8mUh", "reviewer": "Reviewer_YbA1", "summary": "This paper proposes a method for model editing of convolutional classifier networks. The proposed approach assess the class-discriminative ability of convolutional filters by looking at the distribution of their produced feature maps. Specifically, by comparing the class conditional feature distribution to the marginal feature distribution (which is related to the Bayes error rate of the filter), this work is able to produce a saliency score for each filter that determines its importance for classifying a specific class.  For model pruning, the least discriminative filters are pruned, whereas for class forgetting, the most discriminative filters for the forget class are removed.\n\nThis work involves several technical innovations; primarily,  they approximate computation of the TV distance between the class conditional and marginal feature distributions by proposing a moment-based lower bound, which does not make assumptions about the feature distributions, such as Gaussianity.\n\nNumerical results reported for model pruning and class forgetting using ResNets and VGG nets on ImageNet, CIFAR10, CIFR100. Results demonstrate compelling performance. Interesting additional results in the appendix, e.g., analyzing the distribution of the feature maps.\n\nThe contribution of this work is both theoretical and empirical.", "questions": "* Please fix the minor issues described above.\n* Consider reporting computational performance of proposed method compared to baselines.\n* Motivate the choice of witness functions in numerical results, and provide any ablations where applicable.\n* Report model accuracies in all tables, and not just relative performance drops.\n* Minor; consider adding figures to summarize performance and confidence intervals across all classes.\n* Please consider elaborating and including numerical results indicating the impact of the number of classes on the ability to find class discriminative filters. These can be conducted on ImageNet by training models on subsets of the classes; e.g., 10 classes, 100 classes, 500 classes, 1000 classes. \n* Please consider including experiments with a small vision transformer (e.g., ViT-Tiny) and discuss how the approach might extend beyond convolutional networks.", "limitations": "N.A.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n* This work is novel to the best of my knowledge. In particular, the main innovation in this work is the removal of the Gaussian feature distribution assumption from previous works by expending many pages to providing a moment-based lower bound on the TV distance, which can be estimated form few samples.\n\n**Quality**\n* This work is of high technical quality. Numerical results are compelling, demonstrating significant class forgetting with minimal model editing, and similarly, demonstrating strong performance even with significant pruning.\n\n**Clarity**\n* The paper is well written and relatively straightforward to follow, although some design choices; e.g., choice of witness function in numerical results are not clearly motivated.\n\n**Significance**\n* The problem of model editing is likely of broad interest to the neurips community, and the proposed approach is conceptually simple and does not require access to the loss function.", "weaknesses": "* Numerical results both in the main paper and the appendix do not report the actual accuracies of any of the models. Instead, they only report relative drops in performance on forget and non-forget classes. The link provided for the ImageNet classifier in line 910 in the appendix does not work.\n* Computational complexity of the approach compared to related works is not clear; specifically, what is the cost of computing the saliency scores for each filter in your experiments, and how does it compare to those of related methods.\n* Motivation for the choice of witness functions in numerical results is not well motivated, where do these choices of functions come from?\n* Minor issue in Fisher discriminant eq. (2)... should be $\\Sigma_q$ not $\\Sigma_2$.\n* Minor issue in algorithm description for class unlearning; the filter saliency scores should be class labeled $r^c_{\\ell, j}$ and should be computed using equation (10) not equation (9).\n* Applicability: method is currently restricted to image classification with convolutional networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive appraisal of our work, and especially the detailed feedback and insightful questions. We address the reviewers concerns below.\n\n## Weaknesses\n\n**Numerical results both in the main paper and the appendix do not report the actual accuracies of any of the models. Instead, they only report relative drops in performance on forget and non-forget classes.**\n\nIn this work, following standard practice (e.g. [3], [4]), we presented the accuracy drops. We present a table with the baseline accuracies, as requested by the reviewer in the Questions section. This can be seen as a reworking of Table 1, which will be repeated across tables in the paper.\n\n**The link provided for the ImageNet classifier in line 910 in the appendix does not work.**\n\nThe referee refers to the Gdrive link from which we directly downloaded the VGG16 and ResNet50 models provided in the CHIP (our near baseline) GitHub page that later got updated by the authors of CHIP. We will update the link to the CHIP GitHub repo in the document. \n\n**Computational complexity of the approach compared to related works is not clear; specifically, what is the cost of computing the saliency scores for each filter in your experiments, and how does it compare to those of related methods.**\n\nTypically, complexity of the saliency computation is not provided in related work, such as CHIP [1] or TVSPrune [2]. In our work, we state the complexities of computing different types of witness functions in Appendix C.2, specifically Table 3.  For comparison, we also provide a table of wallclock times of computing saliencies *for all filters in the model* with our approach, as compared to CHIP and TVSPrune.\n\n**Imagenet (ResNet50)**\n\n|Method |Time  |\n|--|--|\n| CHIP[1] | >15 hours |\n| TVSPrune [2] | >15 hours  |\n| Ours | ~13 hours  |\n\n**CIFAR10 (VGG16)**\n\n|Method |Time  |\n|--|--|\n| CHIP [1] |  224 minutes |\n| TVSPrune [2] | 195 minutes|\n| Ours | 24mins |\n\n**Takeaway: Our proposed approach is computationally more inexpensive than common baselines and requires no loss function/backpropagation.**\n\n**Motivation for the choice of witness functions in numerical results is not well motivated, where do these choices of functions come from?**\n\nThe choice of witness function is governed by relations to classical methods (i.e. Fisher discriminants/MPM), speed of saliency computation, and ease of estimation of moments. The witness function used in our pruning experiments (section 6.2) is described in lines 358-359. This enables us to recover the Fisher/MPM bounds stated in Theorems 1 and 2, and corollary 2, which are also easy to compute. Moreover, these witness functions are also in the spirit of connecting classical, discriminant-based classifiers to the TV distance/Bayes error rate, with which we can then derive novel bounds on the excess risk of those classifiers, as seen in section C.1. \nIn section 6.1, we use the witness function stated in line 335, which is similar to the RBF Kernel or the moment generating function. A detailed study on the choice of witness functions is, however, the focus of our ongoing research.\n\n**Typos**\nWe have amended the typos pointed out by the reviewer in the main document\n\n**Applicability: method is currently restricted to image classification with convolutional networks.**\n\nThe results proposed in Theorems 1 and 2 can be applied whenever lower bounds on the total variation distance are required, and not just in the context of model editing. The techniques employed in deriving the key results can be applied to finding witness function-based lower bounds for other divergences as well.\n\nMoreover, when applied to model editing problems such as pruning and classwise unlearning, the proposed methods can be applied whenever we can access conditional data distributions. \n\nOur method applies to other model types as well. As requested by the reviewer, we will shortly present experiments showcasing the use of our methods on ViTs trained on CIFAR10 and CIFAR100. \n\n## Questions:\n\n**Minor; consider adding figures to summarize performance and confidence intervals across all classes.**\n\nWe will do so in the revised manuscript, and thank the reviewer for the suggestion. \n\n**Please consider elaborating and including numerical results indicating the impact of the number of classes on the ability to find class discriminative filters. These can be conducted on ImageNet by training models on subsets of the classes; e.g., 10 classes, 100 classes, 500 classes, 1000 classes.**\n\nWe thank the reviewer for the interesting question. The discriminative filters hypothesis proposed in [2] was supported by several experiments, as well as those in this work (see Figs. 5, 6 of the manuscript). It is expected that discriminative filters are easy to identify if the width of the layers being investigated exceeds the number of classes. We see this for models trained on CIFAR100. The final layers of VGG16 (containing 512 filters) possess, on average, 103 (out of 3072) discriminative filters per class in the final 6 layers, whereas ResNet56 models trained on the same dataset possess on average 21 (in the final layer block). Experiments are ongoing for models trained on Imagenet and its subsets. \n\n**Takeaway: The ability to identify discriminative components depends on whether the number of classes exceeds the width of the network or not**.\n\n### References\n\n[1] Sui et al. *CHIP: CHannel Independence-based Pruning for Compact Neural Networks*\n\n[2] Murti et al. *TVSPrune - Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning*\n\n[3] Shah et al. *Decomposing and Editing Predictions by Modeling Model Computation*\n\n[4] Jia et al. *Model Sparsity Can Simplify Machine Unlearning*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wO94OsLB7a", "reviewer": "Reviewer_H2n6", "summary": "The paper addresses the task of model editing that focuses on modifying critical components within neural networks to improve performance. One of the cornerstone steps is to first identify these components. The authors adopt an approach based on recently proposed discriminative filters hypothesis. Instead of using a Total Variation distance (which is intractable in this case), the authors derive a lower bound on the TV that is subsequently used to discover critical subnetworks responsible for classwise predictions.\nThe authors introduce algorithms for structured pruning and selective class forgetting and experimentally show its performance.", "questions": "NA", "limitations": "The authors has adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written, and the ideas are well motivated.\n- The proposed method looks efficient. The problem that is being addressed is significant, so improvements in this area can potentially benefit the community. \n- The authors provided a thorough analysis and theoretical justifications of the proposed method (including findings in Appendix).", "weaknesses": "- It is quite challenging to evaluate the performance of the propose solution due to the absence of comparison to other methods. While it is possible to see that the method is capable of preserving the overall test accuracy almost unchanged while reducing the quality of predictions on a chosen class by ~80%, it is still challenging to interpret these values. It looks to me that the methods listed in A.4 can be relevant and included into comparison.\n- Related work is moved to Appendix. In my opinion, it would be better to include at least a shortened version into the main part of the paper so that the readers can easier understand the relations of the proposed method to the existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive appraisal of our work, and we address the concerns raised below.\n\n### Weaknesses:\n\n**It is quite challenging to evaluate the performance of the propose solution due to the absence of comparison to other methods. While it is possible to see that the method is capable of preserving the overall test accuracy almost unchanged while reducing the quality of predictions on a chosen class by ~80%, it is still challenging to interpret these values. It looks to me that the methods listed in A.4 can be relevant and included into comparison.**\n\nWe thank the reviewer for the suggestion, and provide a comparison with additional recent baselines in the general response. We refer the reader to the General Response for a summary of our experimental results.\n\n**Takeaway: Our work achieves comparable performance to current state of the art without requiring any fine-tuning or retraining, simply by identifying and editing discriminative filters.** \n\n**Related work is moved to Appendix. In my opinion, it would be better to include at least a shortened version into the main part of the paper so that the readers can easier understand the relations of the proposed method to the existing literature.**\n\nWe thank the reviewer for the suggestion. Should the work be accepted, we will include a summary of the work presented in Appendix A  in the main document.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VBK3YQjhBX", "reviewer": "Reviewer_FULk", "summary": "In this work, the authors propose to tackle two problems at once, class unlearning and structured pruning. To do so, they propose a novel way to compute a lower bound on the total variation distance between distributions of features.\nIn practice, this distance is approximated using the first and second order moments of a transformation of intermediate features generated from samples of specific classes. The aforementioned transformations are specific to the task of class unlearning or structured pruning.\nThe resulting method comes with a strong theoretical background, completed by convincing empirical evidence regarding.", "questions": "I have three questions:\n1. (main point from weaknesses) can the author provide results on ImageNet without fine-tuning to highlight the better selection of filters from their method in the main paper?\n2. It is now standard to include transformers when evaluating pruning techniques. If the authors could provide results on Bert, that would benefit the scope of the paper.\n3. In the article, it is claimed that the method can perform structured pruning with no access to the data. However, from my understanding, the saliency score from equation 9 uses Y(X) which is computed with some data. Could the authors elaborate on this point, please?", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work bridges the gap between, selective class forgetting and efficient inference. In its approach, this work is quite original and generalizes previous methods by removing questionable assumptions.\nThe presentation of the result is overall clear, despite the strong mathematical grounding of the proposed method.\nThe empirical results show that the proposed technique slightly improves over previous similar approaches.", "weaknesses": "I have one minor concern regarding the evaluation of the method:\nIn the main paper, the authors insist on the ability of the proposed method to better discriminate important filters in the case of pruning and selective class forgetting. However, the experiments all include fine-tuning and a quite extensive one on pruning with up to 100 epochs. In my understanding, this empirically validates that the filter selection leads to a selection that can be better fine-tuned, not necessarily that the selection alone better preserves the performance.\nI think that some experiments in that regard were conducted and discussed in the appendices, but they should be moved to the main paper if possible. Also, they should be extended to ImageNet, also, if possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive appraisal of our work, and \n### Questions:\n\n**(main point from weaknesses) can the author provide results on ImageNet without fine-tuning to highlight the better selection of filters from their method in the main paper?**\n\nWe thank the reviewer for the insightful question. We have provided results for pruning ImageNet and CIFAR10 models without fine-tuning in Table 7 of the Appendix.  In that table, we we show that **our structured pruning algorithm, DisCEdit-SP, outperforms baselines to a greater extent in the regime without fine-tuning.** Note that at higher sparsity regimes, the accuracy of Imagenet models falls to less than 1% on all baselines. \n\n\n**It is now standard to include transformers when evaluating pruning techniques. If the authors could provide results on Bert, that would benefit the scope of the paper.**\n\nWe thank the reviewer for the suggestion. We will present results on ViTs trained on CIFAR10 for both unlearning and pruning (without fine-tuning) shortly.\n\n**In the article, it is claimed that the method can perform structured pruning with no access to the data. However, from my understanding, the saliency score from equation 9 uses Y(X) which is computed with some data. Could the authors elaborate on this point, please?**\n\nAs stated in lines 83-84 in Section 1, motivated by recent works highlighting the need to be able to compress models without training data or the loss function, our proposed method requires no access to *original training data (data upon which the model was trained)*; rather, we only require distributional access, either via samples from the original distribution not in the training set, or in the form of *finitely many moments*, which can be used in conjunction with Theorem 2 to identify discriminative components.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r4uwhBsNkc", "reviewer": "Reviewer_crkh", "summary": "This paper applies model editing to address two active areas of research, Structured Pruning and Selective Class Forgetting.\nSpecifically, it adopts a distributional approach to identify important components useful to the model's predictions.\nWith the witness function-based lower bounds on the TV distance, it is able to discover critical subnetworks responsible for classwise predictions, thus achieving Structured Pruning and Selective Class Forgetting.", "questions": "1. The authors should provide a more detailed description of machine unlearning and structured pruning, along with a comprehensive introduction to the current research landscape in these areas.\n\n2. To demonstrate the effectiveness of the proposed DISCEDIT-U, the authors should compare its performance against other works within the field.\n\n3. The authors should incorporate a more detailed introduction of the experimental setup and the results into the main text, rather than relegating them to the appendix without sufficient description.", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper uses lower bounds on the TV distance to approximate the Bayes error rate to quantify the discriminative ability of a filter in a neural network. \n\n2. This paper introduces DISCEDIT-U, which selectively prunes components capable of discriminating each class to facilitate unlearning and DISCEDIT-SP, which prunes non-discriminative filters to achieve structured pruning.\n\n3. The experimental results demonstrate the efficacy of DISCEDIT in both unlearning and structured pruning.", "weaknesses": "1. The code provided by the authors redirects to an empty GitHub repository, lacking reproducibility.\n\n2. This paper focuses on the well-established research areas of machine unlearning and structured pruning. \nHowever, the authors fail to provide an introduction to the field or an overview of the current state of existing research.\n\n3. In the experiments of DISCEDIT-U, the authors do not compare its results with other existing unlearning methods, making it difficult to ascertain whether it outperforms the current baselines.\n\n4. The experimental results of DISCEDIT-SP show no significant improvement over the compared baselines, especially when evaluated against CHIP on ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and useful suggestions. \n\n### Weaknesses:\n\n**The code provided by the authors redirects to an empty GitHub repository, lacking reproducibility.**\n\nWe wish to immediately amend the issue with the empty github and apologize for the mistake. We provide a corrected link below:\n\nhttps://rb.gy/9zaxzz \n\n**This paper focuses on the well-established research areas of machine unlearning and structured pruning. However, the authors fail to provide an introduction to the field or an overview of the current state of existing research.**\n\nWe provide a detailed related works section in Appendix A, lines 538-598.   Moreover, in lines 30-38 and section 3, we introduce the problem of model editing, particularly by using component attribution.\n \n**In the experiments of DISCEDIT-U, the authors do not compare its results with other existing unlearning methods, making it difficult to ascertain whether it outperforms the current baselines.**\n\nWe thank the reviewer for the suggestion, and have compared our results with additional baselines in the General response.\n\n**Takeaway: our proposed method for model unlearning is competitive with current state of the art without requiring additional retraining, simply by identifying and masking discriminative components.**\n\n**The experimental results of DISCEDIT-SP show no significant improvement over the compared baselines, especially when evaluated against CHIP on ImageNet.**\n    \n In the work, we show that with extensive fine-tuning, all models recover nearly the entire accuracy of the original model. However,  in Table 7 of the Appendix, *we present results for pruning without fine-tuning*. Our results clearly outperform near baselines such as CHIP [1] and TVSPrune [2].\n \n **Takeaway: We show that our proposed method outperforms the nearest baselines in this regime (without extensive fine-tuning).**\n\n### Questions:\n\n**The authors should provide a more detailed description of machine unlearning and structured pruning, along with a comprehensive introduction to the current research landscape in these areas.**\n\nThe problems of structured pruning and classwise unlearning are stated in Section 3 in the paper, on lines  145-163. However, we formally state the problems of machine unlearning and structured pruning in the sequel. Should the work be accepted, we will incorporate these problem definitions into Section 3, where we set up the two problems. Moreover, a detailed discussion of related work is presented in Appendix A.\n\n**Machine Unlearning**\nLet $\\mathcal{D}$ be the data distribution, let $\\mathcal{D}\\_{c}$ be the distribution of class $c$ in the dataset, and let $\\mathcal{D}\\_{\\bar{c}}=  \\mathcal{D}\\backslash \\mathcal{D}\\_c$ be the distribution of the remaining classes. Let $f\\_\\theta(X)$ be a neural network with parameters $\\theta$ trained on samples drawn from $\\mathcal{D}$, with loss function $\\mathcal{L}(\\cdot)$. Unlearning class $c$ by editing $\\theta$ can be formalized as finding parameters $\\theta^*$, given some $\\epsilon > 0$, using:\n\n$\\theta^* = \\arg\\max\\_{\\theta'} \\mathbb{E}\\_{X\\sim\\mathcal{D}\\_c}\\mathcal{L}(f\\_{\\theta'}(X))\\quad \\text{s.t. }|\\mathbb{E}\\_{X\\sim\\mathcal{D}\\_{\\bar{c}}}[\\mathcal{L}(f\\_{\\theta'}(X))] - \\mathbb{E}\\_{X\\sim\\mathcal{D}\\_{\\bar{c}}}[\\mathcal{L}(f\\_{\\theta}(X))]|\\leq \\epsilon$\n\nThat is, we edit the parameters to maximize the loss on $\\mathcal{D}\\_c$ while minimizing the effect on $\\mathcal{D}\\_{\\bar{c}}$.\n\nSimilarly, structured pruning with a fixed budget $K$ can be written as a search for sparse parameters $\\theta^*$ \nsuch that \n\n$\\theta^* = \\arg\\min\\_{\\theta'} \\mathbb{E}\\_{X\\sim\\mathcal{D}}[\\mathcal{L}(f\\_{\\theta'}(X)] \\text{ s.t. } ||\\theta'||_0\\leq K$\n\nThus, we aim to find a sparse set of parameters (with at most $K$ nonzero parameters) that minimizes the loss.\n\n**To demonstrate the effectiveness of the proposed DISCEDIT-U, the authors should compare its performance against other works within the field.**\n \nWe thank the reviewer for the suggestion and present a comparison with additional baselines in the General Response. \n\n**Takeaway: Our proposed methods are competitive with the additional baselines without requiring additional retraining/fine-tuning.** \n  \n**The authors should incorporate a more detailed introduction of the experimental setup and the results into the main text, rather than relegating them to the appendix without sufficient description.**\n\nShould this work be accepted, we will add a more detailed description of the experimental setup to the main body of the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the readers for their appreciation of our work. In particular, we thank reviewers for noting:
- The importance of the problem addressed by our work - that is, model editing with a view toward structured pruning and classwise unlearning/
- The efficient and simple nature of our solution to both the problems of structured pruning as well as classwise unlearning by way of model editing, and highlighting the hitherto unknown connection between them by using discriminative components.
- The rigorous derivations of the novel lower bounds on the Total Variation distance that require no assumptions on the distributions being compared (i.e. no Gaussianity assumption). We also take this opportunity to highlight three facts about our results. First, the bounds may be of general interest, and can be used whenever lower bounds on the TV distance are required. Second, the bounds reveal new connections between the TV distance and discriminant based classifiers (such as the minimax probability machine and the Fisher discriminant), which we use to derive novel excess risk bounds for these classifiers in Appendix C.1. Third, the techniques used to derive these bounds can be used to derive lower bounds on other divergences as well.

However, we address a common concerns raised by the reviewers.

**Comparison with Baselines for Model Unlearning**

As suggested by Reviewers, **H2n6**, and **crkh**, we compare our proposed method with recent additional baselines provided in [4]. Following [4],  we state the test accuracies on the forget and remain classes, averaged over all classes in the tables below . In the sequel, GA refers to Gradient Ascent, IU refers to influence unlearning (both as implemented in [4]), and l1-sparse refers to the approach as proposed in [4].  

**CIFAR10 models**

VGG-16 
|Method  |  Forget Class accuracy |Remain Class accuracy|
|--|--|--|
| **Ours** | 9.66% | 82.5% | 
| [4], GA | 22.49% | 88.80% |
| [4], IU | 11.42% | 89.81% |

ResNet-20 

|Method | Forget Class accuracy| Remain Class accuracy|
|--|--|--|
| **Ours** | 6.37% | 83.90% | 
| [4], GA | 11.52% | 85.46% |
| [4], l1-sparse |  1.42% | 90.18%  |

**Takeaway: our model achieves superior forgetting compared to the baselines listed in [4] *notably without any fine-tuning of the model, and without utilizing the loss function in any way*, with minimal difference in remain accuracy.**


### References

[1] Sui et al, 2021. *CHIP: CHannel Independence-based Pruning for Compact Neural Networks*

[2] Murti et al, 2023. *TVSPrune - Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning*

[3] Shah et al, 2024. *Decomposing and Editing Predictions by Modeling Model Computation*

[4] Jia et al, 2023. *Model Sparsity Can Simplify Machine Unlearning*

Meta-review: The reviewers acknowledge the soundness and the technical novelty of the paper. The rebuttal has a positive impact on the final ratings and the reviewers scored 2 accepts, 1 weak accept, and 1 borderline reject. The ACs followed the reviewers' recommendation.
Decision: Accept (poster)