Pruning has a disparate impact on model accuracy
Authors: Cuong Tran
Ferdinando Fioretto
Jung-Eun Kim
Rakshit Naidu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Model pruning
Fairness

Abstract: Network pruning is a widely-used compression technique that is able to significantly scale down overparameterized models with minimal loss of accuracy. This paper shows that pruning may create or exacerbate disparate impacts. The paper sheds light on the factors to cause such disparities, suggesting differences in gradient norms and distance to decision boundary across groups to be responsible for this critical issue. It analyzes these factors in detail, providing both theoretical and empirical support, and proposes a simple, yet effective, solution that mitigates the disparate impacts caused by pruning. 

Pruning has a disparate impact on model accuracy
Cuong Tran Department of Computer Science Syracuse University cutran@syr. edu
Ferdinando Fioretto Department of Computer Science
Syracuse University ffioret edu
Jung-Eun Kim Department of Computer Science North Carolina State University jung kimōncsu. edu
Rakshit Naidu
Department of Computer Science
Carnegie Mellon University Hemaaal@@nddred cmu. edu
Abstract
Network pruning is widely-used compression technique that is able to significantly scale down overparameterized models with minimal loss of accuracy. This paper shows that pruning may create or exacerbate disparate impacts. The paper sheds light the factors cause such disparities, suggesting differences in gradient norms and distance to decision boundary across groups to be responsible for this critical issue. It analyzes these factors in detail, providing both theoretical and empirical support, and proposes simple, yet effective, solution that mitigates the disparate impacts caused by pruning.
Introduction
As deep learning models evolve and become more powerful, they also become larger and more costly store and execute. The trend hinders their deployment in resource-constrained platforms, such as embedded systems edge devices, which require efficient models in time and space. To address this challenge, studies have developed variety of techniques to prune the relatively insignificant or insensitive parameters from neural network while ensuring competitive accuracy 40). When model needs to developed to fit given and certain requirements in size and resource consumption, pruned model which is derived from large, peporously-trainee and (often) over-parameterized model, is regarded as a de-facto standard. That because performs incomparably better than same-size dense model which is trained from scratch, when the same amount of effort and resources are invested.
055 30
In spite of its strengths, pruning has been showed to induce or exacerbate disparate effects in the Intuitively, the removal of model weights affects the process in which the network separates different classes, which can have contrasting consequences for different groups of individuals. This paper further shows that the accuracy the pruned models tends to increase (decrease) more in classes that had already high (low) accuracy in the original model, leading to "the rich get richer" and "the poor get poorer" effect. This Matthew effect illustrated in Figure The figure shows the accuracy of facial recognition task different demographic groups for several pruning rates (indicating the percentage of parameters removed from original models). Notice how the accuracy of the majority group (White) tends to increase while that of the minority groups tends to decrease as the pruning ratio increases.
accuracy of the resulting reduced models 20 19
Following these observations, we shed light on the factors to cause such disparities. The theoretical findings suggest the presence two key factors responsible for why accuracy disparities arise in
"This work was also conducted when Jung-Eun Kim was an assistant professor at Syracuse University.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

No Pruning
Pruning
Pruring 50 %
Pruning 70
Pruning
01
00
Whie Sark Aslar Indianttess
While Black Asia anOthers Groups
Whiee Aeinkh Asantid andther Groups
While Asin inchnDeeers Groups
White ark Adin Indarrctes Groups
Groups
Figure 1: Accuracy of each demographic group in the UTK-Face dataset using Resnet18 [88. at the increasing of the pruning rate.
pruned models: (1) disparity in gradient norms across groups, and (2) disparity in Hessian matrices associated with the loss function computed using a group's 's data. Informally, the former carries information about the groups' local optimality, while the latter relates to model separability. We analyze these factors detail, providing both theoretical and empirical support on a variety of settings, networks, and datasets. By recognizing these factors, we also develop a simple yet effective training technique that largely mitigates the disparate impacts caused by pruning. The method based on alteration of the loss function include components that penalize disparity of the average gradient norms and distance to decision boundary across groups.
These findings are significant: Pruning is key enabler for neural network models in embedded systems with deployments security cameras and sensors for autonomous devices for applications where fairness is an essential need. Without careful consideration of the fairness impact of these techniques, the resulting models can have profound effects on our society and economy.
Related work
Fairness and network pruning have been long studied in isolation. The reader is referred to the related papers and surveys on fairness g on these areas.
24 and pruning [ 30
32 33 40)
for a review
The recent interest in assessing societal values of machine learning models has seen an increase of studies at the intersection of different properties of a learning model and their effects on fairness. For example, Xu al. (39 studies the setting of adversarial robustness and show that adversarial training introduces unfair outcomes in term of accuracy parity [22 Zhu et al. [44 show that semisupervised settings can introduce unfair outcomes the resulting accuracy the learned models. Finally, several authors have also shown that private training can have unintended disparate impacts to the resulting models' outputs
13
36 43) and downstream decisions 22 35
Network compression has also been shown to have a profound impact towards the model fairness. For example, several works observed empirically that network compression may amplify unfairness
in different learning tasks [27] 19 20 (2).. Most of the focus has been on vision tasks and in identifying the set of Pruning Identified Exemplars (PIEs), the samples that are impacted most under the compression scheme and conclude that PIES belongs to low frequency groups (those observed at the tail of the data distribution). Blakeney et al. 66] further investigate how bias could be evaluated and mitigated in pruned neural networks using knowledge distillation while Hosseini et al. [21] observed empirically that knowledge distillation processes may produce unfair student models. The impact of network compression towards fairness has also been assessed in natural language tasks. For example, Du et al. [10] and Xu et al. 33 empirically measure the robustness of compressed large language models, while Ahia et al. (2) look into how compression schemes affects data-limited regimes. Finally, Xu and Hu (8) investigate ways to improve fairness in generative language models by compressing them. We also note that, concurrently to this work, Good et al. [4] studied the relative distortion in recall for various classes. They show that pruning has a Matthews effect on the recall for various classes and propose algorithm to attenuate such effect.
This paper builds on this body of work and their important empirical observations and provides a step towards deeper theoretical understanding of the fairness issues arising as a result of pruning. derives conditions and studies the causes of unfairness in the context pruning as well as introduces mitigating guidelines.

---

2 Problem settings and goals
The paper considers datasets D consisting of datapoints (zi a; y:), with i E [n], drawn i.i.d. from an unknown distribution II. Therein, æ, E X is a feature vector, a; E A with A = [m]] (for some finite m) is demographic group attribute, and y y is class label. For example, consider the case of face recognition task. The training example feature 2; may describe a headshot of an individual, the protected attribute a; may describe the individual's gender or ethnicity, and y; represents the identity of the individual. The goal is to learn predictor fe : - M where θ is a a k-dimensional real-valued vector of parameters that minimizes the empirical risk function:
-= argmin 1(0; D) D)= 1 Σ (fo(z.))))
0
where e: : R. is a non-negative loss function that measures the model quality.
We focus analyzing properties arising when extracting a small model Sa with 0 c Ô of size |0] = k < k. Model fg is constructed by pruning the least important values or filters from vector (i.e., those with smaller values in magnitude) according to a prescribed criterion, such as an E norm 25 31. The paper focuses understanding the fairness impacts (as defined next) arising when pruning general classifiers, such as neural networks.
The fairness analysis focuses on the notion of excessive loss, defined as the difference between the original and the pruned risk functions over some group a A:
Fairness
R(a) = J(O: D.) -((; Da).
(2)
where D2 denotes the subset of the dataset D containing samples (æ), yi) whose group membership a, = a. Intuitively, the excessive loss represents the change loss (and thus, in accuracy) that a given group experiences as result of pruning. Fairness is measured in terms of the maximal excessive loss difference, also referred to as faireess violation:
((D) max IR(a) R(a')),
(3)
a.
defining the largest excessive loss difference across all protected groups. (Pure) fairness is achieved when ((D) = 0, and thus a fair pruning method aims at minimizing the excessive loss difference.
The goal of this paper is to shed light why faimess issues arise (i.e., R(a) > 0) result of pruning, why some groups suffer more than others (i.e., R(a) > R(a')) and what mitigation measures could be taken to minimize unfairness due to pruning.
We use the following notation: variables are denoted by calligraph symbols, vectors or matrices by bold symbols, and sets by uppercase symbols. Finally, || || denotes the Euclidean norm and we use (x) to refer to the model' saft outputs. All proofs are reported in Appendix A
3 Fairness analysis in pruning: Roadmap
To gain insights on how pruning may introduce unfairness, we start with providing a useful upper bound for group's excessive loss. Its goal is isolate key aspects of model pruning that responsible for the observed unfairness. The following discussion assumes the loss function E(-) to be least twice differentiable, which is the case for common ML loss functions, such mean squared error or cross entropy loss.
Theorem 1. The excessive loss of a group a A is upper bounded by
R(a) - | x 0 ø|| ( ||0
( āl| =00 a|r)
(4)
where g¹. = Vo Da) is the vector of gradients associated with the loss function e evaluated ò and computed using group data D,, H v ((;; Da) the Hessian matrix of the loss function f, at the optimal parameters vector ô, computed using the group data D, (henceforth simply referred to as group hessian), and A(E) is the maximum eigenvalue of a matrix E.
With a slight abuse of notation, the results refer to ô as the homonymous vector which is extended with k zeros.
3

---

The bound above follows from second order Taylor expansion of the loss function, Cauchy-Schwarz inequality, and properties of the Rayleigh quotient.
Notice that, in addition to the difference the original and pruned parameters vectors, two key terms appear in Equation (9) (1) The norms of the gradients g and (2) the maximum eigenvalue of the Hessian matrix H for a group a. Informally, the former is associated with the groups' local optimality while the latter relates to the ability of the model separate the groups data. As we will show next these components represent the main sources of unfairness due to model pruning.
The following is important corollary of Theorem I It shows that the larger the pruning, the larger will be the excessive loss for a given group.
Corollary 1. Let k and K' be the size of parameter vectors 0 and 0', respectively, resulting from pruning model fa- where k < K (i.e., the former model prunes more weight than the latter one). Then, for any group E A,
R(a, 0) 2 Ría, 0'),
(5)
where Rla, w) is the excessive loss upper bound computed using pruned model parameters (Eg.
(
The corollary above indicates that the excess risk for a group increases as the pruning regime increase. Building on this result, the paper illustrates next why unfairness can become more significant as the pruning regime increases.
The next sections analyze the effect of gradient norms and the Hessian unfairness in the pruned models. The theoretical claims are supported and complemented by analytical results. These results the UTKFaace dataset [41 for vision task whose goal is to classify ethnicity. The experiments use a ResNet-18 architecture and the pruning counterparts remove the P% parameters with the smallest absolute values for various P. All reported metrics are normalized and an average of 10 repetitions. While the theoretical analysis focuses on the notion of disparate impacts under the lens of excessive loss, the empirical results report differences accuracy of the resulting models. The empirical results thus reflect the setting commonly adopted when measuring accuracy parity [2).
We report glimpse of the empirical results, with the purpose of supporting the theoretical claims, and extended experiments, as well as additional deseriptions of the datasets and settings, are reported in Appendix C
Why disparity in groups' gradients causes unfairness?
This section analyzes the effect of gradients norms on the unfairness observed in the pruned models. In more detail, it shows that unbalanced datasets result in model with large differences in gradient norms between groups (Proposition []. connects gradients norms for a group with the resulting model errors in such a group (Proposition 2). and connects these concepts with the excessive loss (Theorem [ to show that unfairness in model pruning is largely controlled by the difference in gradient norms among groups.
Gradient norms and group sizes. The section first shows that imbalanced datasets lead model to have imbalanced gradient norms across groups. The following result assumes that the training converges to a local minima.
Group size
0.4
l19!!11
0.3
0.2
Proposition Consider a dataset with two groups, a and b with ID,l \Dol. Then || [|o/6ll
0.0 White Black Asian Indian Others Groups
That is, groups with more data samples will result in smaller gradients norms than groups with fewer data samples and vice-versa. Figure 2 illlustrates Proposition [] The plot shows the relation between groups sizes [Dal and their associated gradient norms llg'6l on the UTK dataset and settings described above. Notice the strong trend between decreasing group sizes and increasing gradient norms for such groups. These theoretical considerations can be used to explain why underrepresented groups are often subject to larger performance impacts after network pruning [20]. These groups tend to exhibit large gradient norms at convergence, relative to other groups, thus, by Theorem they are also subject to larger excessive losses due to pruning.
Figure 2: Group size gradient norms.

---

Gradient norms and accuracy. Next, the section shows strong connection between the gradient norms of group and its associated accuracy. The following assumes the models adopt a cross entropy loss (or mean squared enror for regression tasks, as shown Appendix Aj.
Proposition 2. For given group A, gradient norms can be upper bounded
1 o (a.y)ED
Σ
Eror
The above relates gradient norms with an error measure of the classifier to a target label multiplied by the gradient of the predictions. For example, in classification task with cross entropy loss, {(fo(z),y) = Exy fo(z)yy, where fā( represents the z-th element of the output associated with the soft-max layer of model fe, and y one-hot encoding of the true label y, with y representing its z-th element, then,
lla.ll = 1I D..
1/Dall Σ (a.yeeD
x
Vofa
Vipu
Σ
(fe(x) - X (
0.8
(2RY)EDD
+ا Indian Others
s 1/1,,1
Σ (n,y))D,
(If(()) yll X V (
0.4
Accuracy
l|sšil
0.2
A similar observation holds for mean square error loss, as illustrated in Appendix [ The observation above sheds light on the correlation between the prediction еггоr of group and its model gradients. This relation emphasized in Figure which illustrates that the gradient norm for given group increases as its prediction accuracy decreases.
0.0
White
Black
Asian
Groups
Figure 3: Accuracy vs. gradient norms.
Proposition 2 allows us to link the gradient norms with the group accuracy the resulting model, which, together with the result above will be useful to reason about the impact of gradient norms on the disparities the group excessive losses.
The role of gradient norms in pruning. Having highlighted the connection between gradients norms of group with the accuracy of the pruned model on such group, this section provides theoretical intuitions on the role of gradient norms in the disparate group losses during pruning.
From Theorem 1 notice that the excessive loss is controlled by term lلg'll x 1l0- è Il. As already noted in Corollary I the term ||0- Ô regulates the impact of pruning on the excessive loss, as the difference between the pruned and non-pruned parameters vectors directly depends on the pruning rate. For a fixed pruning rate, however, notice that groups with different gradient norms will have a disparate effect the resulting term. In particular, groups with very small gradients norms (those generally associated with highly accurate predictions) will be less sensitive to the effects of the pruning rate. Conversely, groups with large gradient norms will be affected by the pruning rate to greater extent, with larger pruning rates, lypically reflecting in larger excessive losses.
These observations of the factors of disparity, accuracy, and group size, can also be appreciated empirically in Figures 4a and 4bb The plots report accuracy (a) and gradient norms (b) on the UTKFace datasets for a variety of pruning rates. Consider group White (containing 42% of the total samples) and Others (containing 7% of the total samples). The unpruned model has high accuracy on the former group and small gradient norms. The accuracy of this group is insensitive to various pruning rates and even increases at large pruning regimes. In contrast, group Others has much lower accuracy and larger gradient norms in the unpruned model. As the pruning rate increase, their accuracies drastically drop. As a result, in high pruning regimes, this minority group exhibits poor accuracy and very high gradient norms.
Notice that the empirical results apply to much more complex settings than those which can be analyzed formally, thus they complement the theoretical observations.
5

---

No Pruning Pruning 10%
2000
0.8
Godd nams
25
1500
Pruting 20%
Pruning
0.6
Pruning 40%
1000
Pruning 50%
Pruning
0.2
10-
500
Pruning 90%
Pruning 70%
Pruning 90%
Bc
Black
Asian
indian
Others
White
Back
Aslen
Indian Others
White
Asian Indian Others
Groups
Groups
Groups
(a) Accuracy
(b) Gradient Norm: lIa''l
(c) Group Hessian: A(H))
Figure 4: Accuracy, gradient norm, and group Hessian max eigenvalues of each ethnicity group, before and after increasing pruning ratios for UTK-Face dataset. The percentage of data samples across groups White, Black, Asian, Indian, and Others is 0.42.0.19.0.15,, 0.15, 0.07, respectively.
5 Why disparity in groups' Hessians causes unfairness?
Having examined the properties of the groups gradients and their relation to unfairness in pruning, this section turns on analyzing how the Hessian associated with the loss function for a group is linked to the unfairness observed during pruning. In more detail, it connects the groups' Hessian to the distance to the decision boundary for the samples in that group and their resulting model errors (Theorem 2), it illustrates strong positive correlation between groups' Hessian and gradient norms, and links these concepts with the excessive loss (Theorem[] [ to show that unfairness in model pruning controlled by the difference in maximum eigenvalues of the Hessians among groups.
Group Hessians and accuracy. The section first shows that groups presenting large Hessian values may suffer larger disparate impacts due to pruning, when compared with groups that have smaller Hessians. It does by connecting the maximum eigenvalues of the groups Hessians with their distance to decision boundary and the group accuracy. The following result sheds light on these observations. restricts its attention models trained under binary cross entropy losses, for clarity of explanation, although an extension to multi-class case is directly attainable.
Theorem 2. Let fo be a binary classifier trained using binary cross entropy loss. For any group a E A, the maximum eigenvalue of the group Hessian XHH) can be upper bounded by:
2(H)) S
Σ (w.))ED
(1-((0) [|Vofgg(>)]|
(6)
D
Closeness decision bowndory
Ervor
The proof relies on derivations of the Hessian associated with model loss function and Weyl inequality. In other words, Theorem Serreem@@iihh highhlight direct connection between the maximum eigenvalue of the group Hessian and (1) the closeness to the decision boundary of the group samples, and (2) the accuracy of the group. The distance to the decision boundary derived from (9. Intuitively this term is maximized when the classifier is highly uncertain about the prediction: fx - 0.5, and minimized when is highly certain fz(x)) 0 or 1, showed in the following proposition.
Proposition Consider a binury classifier fo(x). For a given sample a E D, the term fz(x)(1 fz()) is maximized when fz( = 0.5 and minimized when fz((r) e (0, 1).
Observe that group consisting of samples that are far from the decision boundary will have smaller Hessians and, thus, be less subject to drop in accuracy due to model pruning. These results can be appreciated in Figure 5 Notice the inverse relationship between maximum eigenvalues of the groups' Hessians and their average distance the decision boundary. The same relation also holds for accuracy: the higher the Hessians maximum eigenvalues, the smaller the accuracy. This is intuitive as samples which are close to the decision boundary will more prone to errors due to small changes the model due to pruning, when compared with samples lying far from the decision boundary.
X Dec. Boundary
0.3
XIM))
Decision boundary Accuracy
Accurso
0.1
0.0
White Black Asian Indian Others
Figure 5: Group Hessians, distance to decision boundary, and accuracy.

---

Correlation between group Hessians and gradient norms. This section observes a positive correlation between maximum eigenvalues of the Hessian of group and their gradient norms. This relation can be appreciated in Figure While mainly empirical, this observation is important as it illustrates that both the Hessian 2 (HH) and the gradient llg'll terms appearing in the upper bound of the excessive loss R(a) reported in Theorem I are in agreement. This relation was observed in all our experiments and settings. Such observation allows us to infer that it is the combined effect of gradient norms and group Hessians that is responsible for the excessive loss of group and, in turn, for the exacerbation of unfairness in the pruned models.
The role of the group Hessian in pruning. Having highlighted the connection between Hessian for a group with the resulting accuracy of the model such group, this section provides theoretical intuitions on the role of the Hcssians in the disparate group losses during pruning.
A04O)
0.4
[|9
0.3
0.2
In Theorem I notice that the excessive loss is controlled by term |H x ||0- 0 IP2. As also noted in the previous section, the term 1 è || regulates the impact of pruning on the excessive loss as the difference between the pruned and non-pruned parameter vectors directly depends on the pruning rate. Similar the obser- vation for gradient norms, with a fixed pruning rate, groups with different Hessians will have a disparate effect on resulting term. particular, groups with small Hessians eigenvalues (those generally distant from the decision boundary and highly accurate) will be less sensitive to the effects of the pruning rate. Conversely, groups with large Hessians eigenvalues will be affected by the pruning rate to greater extent, typically resulting in larger excessive losses. These observations can further be appreciated empirically in Figures 4a (for accuracy) and (for maximum group Hessian eigenvalues) on the UTKFace datasets for a variety of pruning rates.
0.1
White
Black Asian Indian Others Groups
Figure 6: Group Hessians and gra- dient norms.
6 Mitigation solution and evaluation
The previous sections highlighted the presence of two key factors playing a role in the observed model accuracy disparities due to pruning: the difference in gradient norms, and the difference in Hessians losses across groups. This section first shows how to leverage these findings to provide a simple, yet effective solution reduce the disparate impacts of pruning. Then, the section illustrates the benefits of this mitigating solution on a variety of tasks, datasets, and network architectures.
6.1 Mitigation solution
To achieve fairmess, the aforementioned findings suggest to equalize the disparity associated with gradient norms lلg'اl and Hessians 2 ((H)) across different groups a E A. For this goal, we adopt a constrained empirical risk minimization approach:
minimize D) such that: لل//لl = Ilg'll, XHH) x(H² Va A.
where g (oJ D) and H vg J(: D) refer to the gradients and Hessian associated with loss function E, respectively, and computed using the whole dataset D. The approach (7) is common strategy adopted in fair learning tasks, and the paper uses the Lagrangian Dual method of Fioretto al. [2 which exploits Lagrangian duality extend the loss function with trainable and weighted regularization terms that encapsulate constraints violations (see Appendix C for additional details).
A shortcoming of this approach is, however, that requires computing the gradient norms and Hessian matrices of the group losses in each and every training iteration, rendering the process computationally unviable, especially for deep, overparametrized networks.
To overcome this computational burden, we will use two observations made earlier in the paper. First, recall the strong relation between gradient norms for a group and their associated losses. This aspect was noted in Proposition That is, when the losses across the groups are similar, the gradient norms across such groups will also tend to be similar. Next, Theorem 2 noted positive correlation between model errors (and thus loss values) for a group and its associated Hessian eigenvalues. Thus, when the losses across the groups are similar, the group Hessians will also tend to be similar. This intuition
7

---

Overall Accuracy
0.55
Accuracy
0.46
0.78
Via
0..4
Fair Vol
0.6
0.50
0.42
0.74
MPtgatten
0.45
Miigatinn far Kroning
Fruning
S4 Rorting
FAB Pnrling
0.72
0.40
Tar b-t)
d/Pruming
FaB 20th
hurnng
for Fruning TurBol"
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6 Pruning Rate
0.8
Pruning Rate
Pruning Rate
Pruning Rate
Figure 8: Accuracy and Fairness violations attained by all models on ResNet50, UTK-Face dataset with ethnicity (5 classes) as group attribute (and labels) [left] and age (9 classes) [right].
also complemented by the strong correlation between group Hessians and gradient norms reported in Section Based on the above observations, we propose simpler version of the constrained minimizer defined as
minimize J(0; D) such that: J(): J(0; D) Va E A.
that substitutes the gradient norms and max eigenvalues of group Hessians equality constraints with proxy terms capturing the group J(;; Da) and population J(0; D) losses.
The impact of such proxy terms in the fairness- constrained problem above can be appreciated, empirically, in Figure 7 The plots, that use the protected
0.5
0.4
UTK-Face dataset, with Ethnicity group, show an original unfair model (top) and fair counterpart obtained through problem (8) (bottom). Both top and bottom figures use an unpruned model. The top sub-figure shows the performance of an original unpruned model trained by minimizing the empirical risk func- tion while the bottom one shows the effect of solving Problem (8. i.e., constrains the empir- ical risk function with the various group loss terms. Notice how enforcing balance the group losses also helps reducing and balancing the gradient norms and group's s average distance to the decision boundary. As a consequence, the resulting model fairness is dramatically enhanced
64-
39
0.2
Acoracy
112211
Deds bouncaty
4..5
12-
0.1 Slacc Amar Frchm
0.1
Figure 7: Effects of fairness constraints in balanc- ing not only group accuracy (left) but also gradient norms (middle) and group average distance the decision boundary (right).
(bottom-left subplot).
6.2 Assessment of the mitigation solution
Datasets, models, and settings. This section analyzes the results obtained using the proposed mitigation solution with ResNet50 and VGG19 the UTKFace dataset [41], CIFAR-I0 [23), and SVHN (26) for various protected attributes. The experiments compare the following four models:
No Mitigation: it refers to the standard pruning approach which uses no faimess mitigation strategy.
Fair Bf Pruming: it applies the fairesss mitigation process (Problem (8) exclusively to the original
large network, thus before pruning.
Fair Aft Pruning: it applies the mitigation exclusively to the pruned network, thus after pruning.
Fair Both: it applies the mitigation both to the original large network and to the pruned network.
The experiments report the overall accuracy of resulting models as well as their fairness violations, defined here as the difference between the maximal and minimal group accuracy. The reported metrics are the average of 10 repetitions. Additional details on datasets, architectures. and adopted, well additional and extended results are reported in Appendix C
Effeets on accuracy. The section first focuses on analyzing the effects of accuracy drop due to applying the proposed mitigation solution for fair pruning. Figure [ compares the four models on the UTK- dataset using a ResNet50 architecture. The left subplots use ethnicity as protected group and class label, with y 5, while the right subplots age protected group and class label,
8

---

0.10
Overall Accurazy
0,760
0.32
Acouracy
0.926
Fairtr trunin
Fairkk Pouuung
0.755
Falr Vil
0.924
0.00
Fair Born
0.922
Fair
0.30
0.04
Fniriig Pi
FAY R0
Nuriig
0.920
Prunnn Far Buth
0.02
0.745
Far 30th
Fol Buth
0.918
0.4
0.6
0.8
0.4
0.6 Pruning Rate
0.8
0.4
0.6 Pruning Rate
0.8
0.00
0.4
0.6
0.8
Pruning Rate
Pruning Rate
Figure 9: Accuracy and Fairness violations attained by all models on VGG-19, CIFAR-10 dataset (left) and SVHN (right) with 10 class labels also used as group attribute.
Dataset
version
Class-wise accuracy
Overall accuracy
UTK-age bins
full
0.856, 128,8, 0.145, 0.319, 0.331, 0.342, 0,181, 0.334, 0.512
0.395
relaxed
0.810, 0,096, 0,141, 0.284, 0.385, 0.324, 0.227, 0.257, 0.533
0.390
full
0.830, 0.876
0.857
UTK-gender
relaxed
0.868, 0.845
0.852
full
0.864, 0,911, 0.869, 0.819, 0.887, 0.784, 0,840, 0.877, 0,805, 0.856
0.857
SVHN
relaxed
0.824, 0,910, 0.775, 0.726, 0.827, 0.752, 0.747, 0.789, 0.713, 0.755
0.795
full
0.998, 0,996, 0.993, 0,998, 0.994, 0.991, 0,991, 0.993, 0,992, 0.985
0.993
MNIST
relaxed
0.994, 0,988, 0.989, 0.986, 0.987, 0.979, 0,981, 0.988, 0.969, 0.994
0.986
Table I: Full (Equation
vs relaxed (Equation 8) versions of the proposed mitigation solutions.
with |3]| Notice that, as expected, all compared models present some accuracy deterioration as the pruning rate increases. However, notably, the deterioration of the models that apply the fair mitigation steps are comparable (or even improved) those of the "No mitigation" model, which applies standard pruning.
similar trend can be in Figure that reports results CIFAR (left) and SVHN (right). Both use the ten class labels as protected attributes. These results clearly illustrate the ability of the mitigating solution to preserve highly accurate models.
comparison of the "full" (Equation 7 and "relaxed" (Equation g) versions the proposed mitigation solutions is provided in Table I We note that while the "full" version leads to fairer results, the reduction in the various groups accuracy is often insubstantial. We also note that the running time of the "full" version is largely (over order magnitude) longer than the relaxed counterpart. This is due to the calculation of gradient norms and the Hessian terms associated with each group.
Effects fairness. The section next illustrates the ability of the proposed solution to achieve fair pruned models. Table 2 iillustrates the results for the UTKFaces dataset with ethnicity as class labels and age protected attributes for a CNN with two convolutional layers and three lincar layers and prune amounts: 30%, 50%, 70%, and 90%. Notice how Fair Aft pruning and Fair both achieve relatively lesser faireess violations compared to the No mitigation and the Fair bf Pruning methods in most cases.
Next, the second and fourth subplots presented in Figures g iillusstttt the faimeess violations obtained by the four models analyzed on different datasets and settings. We make the following observations: First, all the plots exhibit consistent trend in that the mitigation solution produces models which improve the fairness of the bascline, "No mitigation" model. Observe that, as already illustrated in Figure 7 the fair models tend to equalize the gradient norms and group Hessians components (and thus the distance to the decision boundary across groups). Thus, the resulting pruned models also attain better fairness, when compared their standard counterparts.
Next, notice that "Fair Aft Pruning" often achieves better fairness violations than "Fair Bf Pruning". especially at high pruning regimes. This is because the former has the advantage to apply the mitigation solution directly to the pruned model to ensure that the resulting model has low differences in gradient norms and group Hessians. The presentation also illustrates the application of the mitigation strategies both before and after pruning (Fair Both) which shows once again the significance of applying the mitigation solution over the pruned network.

---

Methods
Overall
accuracy
Fairness violations
30%
50%
70% 90 %
30%
50%
70% 90%
No mitigation
0.546 0.545 0.529 0.559
0.179 0.186 0.152 0.134
Fair bf Pruning
0.539 0.557 0.529 0.540
0.189 0.190 0.174 0.238
Fair Aft Pruning 0.538 0.532 0.497 0,472
0.172 0.161 0.163 0.05
Fair both 0.525 0.541 0,508 0.484
0,170 0.144 0.156 0.073
Table 2: Accuracy and fairness violations for the UTKFaces dataset with ethnicity as class labels and age as protected attributes and prune amounts of 30%, 50%, 70%, and 90%.
Finally, it is notable that "Fair Aft Pruning" achieves good reductions in fairness violation. Indeed, pre-traime pre-trained large (non-prume con-pruned) fair models may not be available and the ability to retrain these large models prior to pruning may be hindered by their size and complexity.
7 Discussion and limitations
This section discusses three key messages found in this study. First, we notice that pruning affecting model separability and distance to the decision boundary is related to concepts also explored in robust machine learning [5 28). Not surprisingly, some recent literature in network pruning has empirically observed that pruning may have negative impact on adversarial robustness [16]. These observations raise questions about the connection between pruning, robustness, and fairness, which we believe is an important direction to further investigate.
Next, although the solution proposed in Problem ( allows it to be adopted in large models, the size of modern ML models (together with the amount of hyperparameters searches) may hinder retraining such original massive models from incorporating fairness constraints. Notably, however, the proposed mitigation solution can be used as a post-processing step to be applied during the pruning operation directly. The previous section shows that the proposed method delivers desirable performance in terms of both accuracy and fairness.
Finally, notice that the results amalyzed in this paper pertain to losses that are twice differentiable. Lifting such an assumption will be an interesting and challenging future research avenue.
Ethical considerations. The analyses and solutions reported in this paper should not be intended as an endorsement for using the developed techniques to aid facial recognition systems. We hope this work creates further awareness of the unfairness caused by pruning mechanisms in ML systems in the context of models that could be deployed in energy-efficient devices, such as smart cameras or access control systems, etc.
8 Conclusion
This work observed that pruning, while effective in compressing large models with minimal loss of accuracy, can result in substantial disparate accuracy impacts. The paper examined the factors causing such disparities both thcoretically and empirically showing that: (1) disparity in gradient norms across groups and (2) disparity in Hessian matrices associated with the loss functions computed using a groups' data are two key factors responsible for such disparities to arise. By recognizing these factors, the paper also developed simple yet effective retraining technique that largely mitigates the disparate impacts caused by pruning.
As reduced versions of large, overparameterized models become increasingly adopted in embedded systems to facilitate autonomous decisions, we believe that this work makes an important step toward understanding and miligating the sources of disparate impacts observed in compressed learning models and hope it will spark awareness in this important area.
Acknowledgments
This research is partly funded by NSF grants SaTC-1945541, SaTC-2133169, and CAREER-21 2114706. Fioretto also supported by a Google Research Scholar Award and an Amazon Research Award. The views and conclusions of this work are those of the authors only.
10

---

References
[1] N. Aghli and E. Ribeiro. Combining weight pruning and knowledge distillation for cnn compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3191-3198, 2021.
[2] O. Ahia, J. Kreutzer, and S. Hooker. The low-resource double bind: An empirical study of pruning for low-resource machine translation. In EMNLP, 2021.
[3] E. Bagdasaryan, O. Poursaeed, and V. Shmatikov. Differential privacy has disparate impact on model accuracy. In Advances in Neural Information Processing Systems, pages 15479-15488, 2019.
[4] S. Barocas, M. Hardt, and A. Narayanan. Fairness in machine learning. Nips tutorial, 1:2, 2017.
[5] C. Baykal, L. Liebenwein, I. Gilitschenski, D. Feldman, and D. Rus. Sipping neural networks: Sensitivity provable pruning of neural networks. arXiv preprint 12..5:291. 2019.
[6] C. Blakeney, N. Huish, Y. Yan, and Z. Zong. Simon says: Evaluating and mitigating bias in pruned neural networks with knowledge distillation. ArXiv, abs/2106.07849, 2021.
[7] D. Blalock, J. J. G. Ortiz, J. Frankle, and Guttag, What is the state of neural network pruning? arXiv preprint arXiv:2003.03000 2020.
[8] S. Caton and C. Haas. Fairness in machine learning: A survey. arXiv preprint arXiv:2010.04053,
2020.
[9] Cohen, Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smooth-
ing. In International Conference on Machine Leurning, pages 1310-1320. PMLR, 2019.
[10] M. Du, S. Mukherjee, Y. Cheng, M. Shokouhi, X. Hu, and A. H. Awadallah. What do compressed large language models forget? robustness challenges in model compression. ArXiv, abs/21 10.08419 2021.
[11] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Faimeess through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
[12] F. Fioretto, P. V. Hentenryck, T. W. Mak, C. Tran, F. Baldo, and M. Lombardi. Lagrangian duality for constrained deep learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 118-135. Springer, 2020.
[13] Fioretto, C. Tran, P. Hentenryck, and K. Zhu. Differential privacy and fairesss in decisions and learning tasks: A survey. CoRR, abs/2202.08187, 2022.
org/ /bs/
2202. 08187]
[14] A. Good, J. Lin, X. Yu, H. Sieg, M. Ferguson, Zhe, J. Wieczorek, and T. Serra. Recall distortion in neural network pruning and the undecayed pruning algorithm. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2022.
[15] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2014. URL //tp:v////xivvee// org/ abs/ 141//141..... 6572
[16] Y. Guo, C. Zhang, C. Zhang, and Y. Chen. Sparse duns with improved adversarial robustness. In Proceedings of the International Conference on Neural Information Processing Systems (NeurlPS), page 240-249 2018.
[17] M. Hardt, E. Price, E. Price, and N. Srebro. Equality of opportunity in super- vised learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran As- sociates, Inc., 2016. URL https::/pprooeddng...errr// wttpppppe//1////1..... 1426233773333eeeccb22224999
pd.f.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[19] S. Hooker, A. C. Courville, G. Clark, Dauphin, and A. Frome. What do compressed deep neural networks forget. arXiv: Learning, 2020.

---

[20] S. Hooker, N. Moorosi, G. Clark, S. Bengio, and E. L. Denton. Characterising bias in compressed models, ArXiv, abs/2010.03058, 2020.
[21] S. Hosseini, M. A. Shabani, M. M. Jahanara, and B. Salamatian. Learning fair from unfair teachers.
[22] V. Joseph, S. A. Siddiqui, A. Bhaskara, G. Gopalakrishnan, S. Muralidharan, M. Garland,
S. Ahmed, and A. R. Dengel. Going beyond classification accuracy metrics in model compres- sion. 2020.
[23] A. Krizhevsky, V. Nair, and G. Hinton. Cifar-10 (canadian institute for advanced research). URL httt:://////w cs. toronto. wdww-erii/deeeeeeeeee
.html
[24] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan. A survey bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
[25] M. C. Mozer and P. Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Norgan-Kaufmann http:///A00.///./././///////1///
1988. URL https: / //poceeengs Paper .pdf]
neurips. /aarrppss.cppppe//////eeeee 1ttpper/1998///11e//////////////
[26] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[27] M. Paganini. Prune responsibly. arXiv preprint SXKiv:2009.0993333 2020.
[28] N. Papernot, P. McDaniel, and I. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv: Arxiv- 07277 2016.
[29] D. Pujol, R. McKenna, S. Kuppam, M. Hay, A. Machanavajihala, and G. Miklau. Fair decision making using privacy-protected data. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 189-199, 2020.
[30] A. Renda, J. Frankle, and M. Carbin. Comparing rewinding and fine-tuning in neural network pruning. International Conference on Learning Representations, 2020. URL https
//openreview net htt///orrmmidd S1gS S1gSjoNKVBB ONKvB
[31] J.T. S. Han, J. Pool and W. J. Dally, Learning both weights and connections for efficient neural networks. In NIPS, 2015. URL htt.ht h/tpxs///aaxvv../ wwwrxiiv.orrrree ab06 02626v3
[32] V. Schwag. S. Wang. P. Mittal, and S. Jana. Towards compact and robust deep neural networks. preprint arXiv: www.:09066011110 2019.
[33] W. Toussaint, A. Mathur, F. Kawsar, and A. Y. Ding. Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows, 2022. URL https://arxiv. .rg/ /bw/ab//0000 07677
[34] C. Tran, M. Dinh, and F. Fioretto. Differentially private empirical risk minimization under the fairness lens. In Advances in Neural Information Processing Systems, 2021.
[35] C. Tran, F. Fioretto, P. V. Hentenryck, and Z. Yao. Decision making with differential privacy under a fairness lens. In Z.. Zhou, editor, International Joint Conference on Artificial Intelligence (IJCAI), pages 560-566, 2021.
[36] A. Uniyal, R. Naidu, S. Kotti, S. Singh, P. J. Kenfack, F. Mireshghallah, and A. Trask. Dp-sgd
vs pate: Which has less disparate impact on model accuracy? ArXiv, abs/2106. 12576, 2021.
[37] C. Xu, W. Zhou, T. Ge. K. Xu. J. McAuley, and F. Wei. Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10653-10659, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.
18653/v1/2021. htt/2/221..mm/pmmma... 832. URL
orgpaaannooooooooo//22200220 emnlp-main
832
[38] G. Xu and Q Hu. Can model compression improve nip faimess. ArXiv, abs/2201 2022.
[39] H. Xu, X. Liu, Y. Li, A. Jain, and J. Tang. To be robust or to be fair: Towards fairness in adversarial training, 2021.
[40] T. Zhang, S. Ye, K. Zhang, J. Tang, W. Wen, M. Fardad, and Y. Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 184-199, 2018.
12

---

[41] Z. Zhang, Y. Song. and H. Qi. Age progression/eegresion by conditional adversarial autoencoder.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
[42] H. Zhao and G. Gordon. Inherent tradeoff's in learning fair representations. Advances in neural information processing systems, 12:15575-15685 2019.
[43] K. Zhu, P. Van Hentenryck, and F. Fioretto. Bias and variance of post-processing in differential privacy. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11177-11184 2021.
[44] Z. Zhu, T. Luo, and Y. Liu, The rich get richer: Disparate impact of semi-supervised learning. arXiv preprini arXiv:21 2021.
13


Reviews:
[{"review_id": "q7eAVodPgOx", "reviewer": "Reviewer_Zs7H", "summary": "The work studies effect of pruning neural networks on their losses for specific groups and the differences between them, termed as unfairness. It shows that gradient norm and Hessian of the loss function are important quantities controlling the upper bound on group-specific losses. As a way to mitigate unfairness from pruning, the effect of adding constraints to the learning problem that penalise the two quantities is explored. Some of the claims are validated through experiments on a face images used to train an ethnicity detection model.", "questions": "Major points to address in the response\n\n1. There is an unacknowledged jump in reasoning from Corollary 1 to the main claim of the work that pruning increases unfairness. Corollary 1 shows that the upper bound on excessive loss for *a given* sensitive group may increase due to pruning. But it does not guarantee that *relative* excessive loss may increase. Also, the results are on the upper bound of the excessive loss and not the actual value of the excessive loss. The difference in actual loss across groups need not be smaller than the difference between upper bound. The theoretical results may motivate investigating gradient norms and the Hessian in experiments but the motivation is not as clear as presented. Thus the claims for studying disparate excessive loss in the rest of the results are overstated. The claims should be stated more carefully.\n\n2. Please discuss the work by Hooker et al. 2020 which seems to have the same conclusion. Please discuss relationship between the findings.\nHooker et al. 2020, Characterising Bias in Compressed Models. https://arxiv.org/abs/2010.03058\n\nAnother related work is Ahia et al. 2021 which posits that pruning affects infrequent sentences more. This relates to Proposition 1’s claim that the proportion of data points from the group is a potential source of disparate impact of pruning.\nAhia et al. 2021, The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation. https://aclanthology.org/2021.findings-emnlp.282/\n\n3. Validating the relaxed version of the mitigation method eq. (8) is a bit unsatisfactory. Partly because it is not directly related to the analysis. Consider solving the optimization problem in eq. (7) instead of the relaxed version on a smaller dataset and model. This will help in validating the constraints motivated by the theoretical analysis instead of simplifying the problem. Perhaps only the gradient norm penalty, which is relatively quicker to estimate than max eigenvalue of the Hessian, can be tested in experiments.\n\nSecondly, the relaxed version is an existing fairness method which is not discussed. It is also considered in the domain generalization problem where one approach is to equalize losses across domains that are groups in this case. In this sense, the contribution can be rephrased as taking an existing method and showing its relevance for fairness of pruning.\nSee Donini et al. 2018 https://papers.nips.cc/paper/2018/hash/83cdcec08fbf90370fcf53bdd56604ff-Abstract.html, and Williamson and Menon 2019 https://proceedings.mlr.press/v97/williamson19a.html for the fairness method, and Section 2.4 of Krueger et al. 2021 https://proceedings.mlr.press/v139/krueger21a.html for the domain generalization method.\n\n4. Finally, please address the ethical concern of using the UTKFace dataset as the only exemplar in the work.\n\n\nMinor points or suggestions (no response is being expected)\n\nHandling one failure mode (disparity due to pruning, here) in isolation risks unintended consequences on other failure modes. Please consider adding a note that the question of how the proposed solution impacts fairness on other attributes, robustness, and interpretability of the pruned network.\n\nGiven the discrepancy between the theoretical results and claims, the empirical results become essential to the significance of the work. Consider repeating the analyses in Figures 2 to 6 on more models and datasets as considered in Figure 9.\n\nIn the line 88, fairness wrt pruning is taken to be zero excessive loss \\xi(D) =0. While terming this as fair pruning is ok, it can be commented that the pruned network can still be unfair in terms of differences in losses between groups.\n\nClarify the accuracy metric used in Figure 3.\n\nMention the discrepancy between loss metrics (like squared loss) used in the results and accuracy metric (0-1 loss) used to evaluate them in the experiments.\n\nProposition 3 can be a remark as it is a known fact.\n\nAdd error bars specifically in Figure 8 and 9 where different methods are compared. Ideally error bars should be reported in other figures also to convey robustness of the findings.\n\nI think a more natural way to visualise Figure 1 would be a multiple line plot of accuracy vs pruning percentage with separate lines for each group.\n\nReport the R^2 value for the correlation between the two variables shown in Figure 2, 3 and 6 which is easier to interpret.\n\nIt makes more sense to normalize the gradient norm plot in Figure 8 since two different models are compared.\n\nInclude a reference to Weyl’s inequality e.g. ones in https://terrytao.wordpress.com/tag/weyl-inequalities/.\nWeily’s -> Weyl’s in Appendix.\n\nConsider using the first person in sentences as it is less ambiguous, The paper uses -> we use.", "limitations": "Some of the limitations are presented. Please discuss limitations including relevance of theoretical results to differences in excessive loss, reliance on assumptions like being able to find the local minima. Also, mention the effect of other pruning methods, and uncertainty around effect on other properties of the pruned model than fairness. Prominently write the assumptions in one place and reference them in the results like in Proposition 1 which does not state the assumptions.\n\nPlease consider clarifying why ethnicity detection task from face images was used as an example to illustrate effects of pruning. Since it is a controversial application of machine learning with damaging consequences (with or without fairness concerns), it becomes imperative to motivate its inclusion in the paper. In my personal opinion (disagreements are welcome), the inclusion of facial recognition application should not appear as an endorsement of the technology.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Provides clear experimental evidence that backs the claim of relation between excessive losses from pruning and properties of the network such as gradient norm and the max eigenvalue of the Hessian.\n- Concepts and results are presented clearly.\n- Studies a practically relevant problem.\n\nWeaknesses\n- The main claim on unfairness from pruning is not well-motivated by the theoretical results. The discussion centres on unfairness that is difference in excessive losses across groups while the result analyzes excessive losses for each group separately.\n- Missed related work which makes a similar observation that pruning may cause unfairness.\n- Experimental results are not presented as clearly as possible. Error bars can be reported and correlation coefficient can be reported in plots where the claim is that the plotted quantities are correlated.\n\nThe evidence presented in this work that pruning causes unfairness is an important one because of the prevalence of pruning in practice. Also, the observations on gradient norm and max eigenvalue of the Hessian are interesting. However, the presentation of the results seems to overstate the significance of the theoretical results. I think that these can be addressed through minor writing edits.\n\n---\n\nAfter the response\n\nThe response adequately clears my concerns on the proposed relaxation for the solution and related work. The response to the over-interpretation of the theoretical result in the paper is adequate. The promised change to the text after Corollary 1 is good. I would strongly encourage the authors to propagate this change to the rest of the paper e.g. the text in the heading of Sections 4 and 5 can be changed to may cause. \n\nThe work presents concrete evidence for the relation between two training properties during pruning and unfairness which is an important problem due to prevalence of pruning. Thus I am increasing the score to 7, Accept. The motivation from theory needs improvement which needs to be adequately acknowledged in a revision. The presentation should be improved also including error bars in figures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}, {"review_id": "gh6enpj1rfA", "reviewer": "Reviewer_B2Lo", "summary": "This paper studies the relationship between pruning and group disparities in model accuracy. Using the UTKFace dataset and ResNet and VGG models, the authors show that increased levels of pruning show an increase in disparities in accuracy between groups. That is, groups that had lower accuracy than average had their accuracies get lower, while groups with higher than average accuracy have their accuracies improve. The authors relate these changes theoretically to the gradient norm and Hessian and propose mitigation techniques with these insights. ", "questions": "In equation 3, why is the maximum chosen for fairness violation? Was the average, standard deviation, etc. also considered? Does max have desirable theoretical properties that the others don’t? Using the maximum implies a worst case fairness violation, but it could be interesting to understand the behavior of, say, the average fairness violation as well. This may be worth discussing in the text. \n\nRather than having the two bars per group in figure 2, would it be possible to made a 2D plot of group size vs gradient norm instead? For this figure, it’s not crucial to know which groups are which, but instead to see the correlation between group size and gradient norm more directly. It can be inferred from the bars, but a scatter plot might convey the results more clearly. \n\nGiven that a relationship between gradient norms and loss is shown, what is the relationship between fairness violation and accuracy? Is there an inherent tradeoff? In interpreting the results of figure 9, I found myself wondering whether having a lower accuracy automatically meant having a higher fairness violation. It was hard to tell from the plots themselves, but I think this is worth some discussion.\n", "limitations": "The primary limitation of the work is that it only considers pruning in the context of a single dataset and task (with multiple models). So, some of the empirical results may not generalize to, say, a BERT model performing sentiment classification. That being said, the theoretical results are not task dependent. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper’s main strengths are its originality, clarity, and quality. The effect of pruning on group disparity metrics has not been studied in the past, and this paper does a very thorough theoretical and empirical treatment of the effect. In particular, the insights about the relationship of the gradient norm and Hessian eigenvalues to fairness violation were well done. The paper is systematic in its approach and exposition, with an easy-to-follow motivation.\n\nThe main weakness of the paper is that the mitigation results presented are difficult to interpret. While the results showing the relationship between pruning and fairness violation are quite clear, the results shown in figure 9 are difficult to read and understand. For example, consider the fairness violation plot in the bottom left of Figure 9. Here there is no smooth behavior relative to the pruning fraction, despite the fact that the theoretical results seem to indicate that there should be. It looks as though there may be some statistical uncertainty in these experiment results. \n\nEven without these mitigation results, however, the theoretical treatment and empirical results on pruning are interesting and could be a paper on their own. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mo-4YxVFlhX", "reviewer": "Reviewer_3Nrs", "summary": "This paper has studied two factors causing disparity of network pruning, that is differences of gradient norm and Hessian matrix cross groups. It further introduced a simple training strategy to mitigate this disparity. With extensive experiments on UTK-Face, CIFAR-10, and SVH datasets,  this strategy has shown to mitigate the disparity well and keep comparable pruning performance to those without migration.", "questions": "1. It is unclear to me how to do the pruning with mitigation. Does the mitigation after pruning need retraining the model? \n2. In Figure 7, although the fairness among groups is improved,  the pruning accuracy of all groups except “Others” equally decreases with that. The unfair model (top) seems to be an unpruned model. Is the fair model (bottom) the one before pruning or after pruning? If it is after pruning, please clarify the pruning rate. \n3. If I understand correctly, all the experiments have the class labels as the protected groups. How does this mitigation strategy work when they are different? For example, the model on the UTK-Face dataset uses ethnicity as class labels and uses age as protected groups. \n4. In Proposition 2 and Theorem 2, $||f_{\\theta}(x) -y||$ is for the error instead of the accuracy, $(f_\\theta(x)(1-f_\\theta(x)))$ and the distance to the decision boundary seems negative proportional. The larger  $(f_\\theta(x)(1-f_\\theta(x)))$, the smaller the distance to the decision boundary is. \n5. In line 94, what is the soft output? Is it the output after softmax or before it? ", "limitations": "The proposed mitigation solution uses the loss disparity cross groups as the fairness proxy. Although much simpler, it is not so novel that it can be proposed without knowing the gradient norms and Hessian matrices as the cause of the disparity. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is well written and the idea is clear and easy to follow. \n2. It is novel to explain the fairness with gradient norms and Hessian matrices across groups theoretically.\n3. The proposed method is simple yet efficient to mitigate the disparity across groups. \nPlease look at the Questions for the weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept