SWAP: Sparse Entropic Wasserstein Regression for Robust Network Pruning
Authors: Lei You
Hei Victor Cheng
Venue: ICLR 2024 poster (2024)
Keywords: Network Pruning
Model Compression
Optimal Transport
Wasserstein Distance
Deep Learning
Primary area: general machine learning (i.e., none of the above)

Abstract: This study addresses the challenge of inaccurate gradients in computing the empirical Fisher Information Matrix during network pruning. We introduce SWAP, a formulation of Entropic Wasserstein regression (EWR) for network pruning, capitalizing on the geometric properties of the optimal transport problem. The “swap” of the commonly used linear regression with the EWR in optimization is analytically demonstrated to offer noise mitigation effects by incorporating neighborhood interpolation across data points with only marginal additional computational cost. The unique strength of SWAP is its intrinsic ability to balance noise reduction and covariance information preservation effectively. Extensive experiments performed on various networks and datasets show comparable performance of SWAP with state-of-the-art (SoTA) network pruning algorithms. Our proposed method outperforms the SoTA when the network size or the target sparsity is large, the gain is even larger with the existence of noisy gradients, possibly from noisy data, analog memory, or adversarial attacks. Notably, our proposed method achieves a gain of 6% improvement in accuracy and 8% improvement in testing loss for MobileNetV1 with less than one-fourth of the network parameters remaining.

Published as a conference paper at ICLR 2024
SWAP: SPARSE ENTROPIC WASSERSTEIN REGRESSION FOR ROBUST NETWORK PRUNING
Lei You*
Hei Victor Cheng
Department of Engineering Technology Technical University of Denmark Ballerup, DK-2750, Denmark leiyoldtu.dk.
Department of Electrical and Computer Engineering
Aarhus Universrity
Aarhus, DK-8200, Denmark hvcêece au.dk
ABSTRACT
This study addresses the challenge of inaccurate gradients in computing the em- pirical Fisher Information Matrix during neural network pruning. We introduce SWAP, a formulation of Entropic Wasserstein regression (EWR) for pruning, cap- italizing the geometric properties of the optimal transport problem. The "swap" the commonly used linear regression with the EWR in optimization is analyt- ically demonstrated to offer noise mitigation effects by incorporating neighbor- hood interpolation across data points with only marginal additional computational cost. The unique strength of SWAP is its intrinsic ability to balance noise reduction and covariance information preservation effectively. Extensive experiments per- formed various networks and datasets show comparable performance of SWAP with state-of-the outperforms the SoTA when the network size the target sparsity is large, the gain is even larger with the existence of noisy gradients, possibly from noisy data, analog memory, or adversarial attacks. Notably, our proposed method achieves a gain of 6% improvement in accuracy and 8% improvement in testing loss for MobileNetVI I with less than one-fourth of the network parameters remaining.
(SoTA) network pruning algorithms. Our proposed method
INTRODUCTION
The advent of deep learning has revolutionized various domains of artificial intelligence, with neu- ral networks showing remarkable performance across an array of applications. Nonetheless, the increase in model complexity has led escalating computational demands and substantial memory requirements. This poses significant challenges for deploying these models in resource-constraie environments such as mobile or internet of things (IoT) devices. Therefore, the concept of neural network pruning emerges as a critical solution. It aims to optimize the network by removing less important parameters, which reduces computational overhead while maintaining the performance of the original model.
In the realm of state-of the-art deep learning, the models often exhibit substantial size and complex- ity, with up trillions of parameters, exemplified by models such GPT-4. The immense compu- tational demand, energy inefficiency, and the challenges with model interpretability associated with these models highlight the need for innovative and efficient optimization techniques. These tech- niques should ideally minimize the model size while improving their robustness and interpretability. Considering the limitations of previous work, especially those arising from the influence of noisy data and noisy gradients, the paper proposes a promising pathway for robust pruning.
Below, inspect the network pruning problem from optimization perspective, with a concise introduction of the most relevant existing works. Then a sketch of our approach is given.
Related Work on Pruning Optimization. Denote by w R trained model and L(w) the loss function given arbitrary model w. The loss function can be locally approximated around w with
& *Correspondence to both Leiyo@dtu. dk and hvcšece dk. Lei You is supported by Thomas B. Thriges Fond 5041-2402. Hei Victor Cheng is supported by the Aarhus Universitets Forskningsfond under Project AUFF 39001.

---

Published as a conference paper at ICLR 2024
Taylor Expansion shown in (1).
1 [(w) = ((w) + VL(W) - 2 - >>] w) + O(l -w1²) (I)
Consider a neural network with a loss function [(w)
Σ L:(w), where (((w) R& the The goal of network pruning is to find set of w
loss incurred at data point i (i 1, 1,...,N). N). such that there are & (k < p) elements of w being zero while keeping the newly obtained model w's performance as good possible to the original one W. Mathematically, we want to find some W E Rp that satisfies both [(w) & [(W) and || wwll & k. with < p.
This line of research can be dated back to (LeCun et al., 1989), where the approximation in equation (1) is adopted. Under the assumption that gradient V V[(w) = when the network trained, the net- work weights
pruned one-by-one in decreasing order based on the value of (w- w) WHH((W (w- w). In their approach, the H is approximated as a diagonal matrix; this is later extended in (Hassibi & Stork, 1992) to include the whole Hessian matrix, and the authors also proposed using the Fisher information matrix (FIM) as an approximation to the Hessian. Later, (Singh & Alistarh, 2020) proposed reduce the computation complexity by using block diagonal Hessian, and FIM is ap- proximated using small subset of the training data.
These approaches all use equation (1) to prune the network a one-by-one manner, namely the weight with the least importance is set to zero according to the different approximations of equation (1). this way, the potential interactions of pruning multiple weights are ignored. To explore this, the network pruning problem is formulated as mixed integer quadratic programming (MIQP) in (Yu et al., 2022). Namely, an objective function
f(w) = (w- w) H(w w) + A \/| - w| (A=0)
(2)
is minimized, and Hessian is approximated as s V2L(W), subject to the sparsity constraint ||wllu k, where À is regularization parameter. Although this approach shows significant im- provements, it suffers from scalability issues as a full Hessian matrix is required.
Sparse Linear Regression (LR) Formulation. To reduce the computational complexity, the Hes- sian matrix can be approximated by the empirical FIM, using n samples as in (Chen et al., 2022; Benbaki et al., 2023). Denote G [VR....., Ve,, RR*x,, where V4, (L((W). For simplic- ity, V4, is used in this document to represent the derivative of the data point i's loss at w consis- tently in this paper unless specified otherwise. The Hessian is approximated through the expression
H (1/n) Σi= Ve,Ve (1/n) G, which is the so-called FIM. Denote xq Vl, and y = Vl, w, (2) is formulated to the sparse LR problem shown in (3) below.
+n min Q(w) Σ [[x,(w) vill + n2 ||w w/f s.t. w/lo & k
This formulation has computational advantage, as empirical FIM needs not to be computed explic- itly. It is shown that the formulation scales large neural network pruning (Chen et al., 2022).
Motivation of Combating Against Noise. In practice, it is not always easy to obtain the correct gradients for pruning large neural networks. There can be noise contained in the data samples, and the gradients can also be corrupted due to various reasons, e.g., distributed or federated learning (Turan et al., 2022), or adversarial attacks such as data poisoning (Steinhardt et al., 2017).
As pointed out by (Mahsereci et al., 2017; Siems et al., 2021), conditioning on the underlying true gradient V((w)= = O, there are mini-batch gradients which are not informative anymore can be fully explained by sample noise and the vanishing gradients. These gradients would not contribute to the covariance information of the empirical FIM but serve as outliers Hessian approximation.
In the scenarios of federated learning (FL), gradients computed by different clients are skewed and consequently, local models move away from globally optimal models (Huang et al., 2022), imposing challenges for constructing informative FIM. Besides, noise can be added to the gradient for privacy concerns in communications (Li et al., 2020). Additionally, the clients usually have inevitable noisy samples and labels, making models suffer from significant performance drop (Tuor et al., 2021). Additionally, over- werrtthe-iit air communications itself suffer from unavoidable noises (Ang et al., 2020; Yang et al., 2020). These lead to concerns for network pruning with noisy gradients. Finally, analog
2

---

Published as a conference paper at ICLR 2024
memory recently gained attention for deep learning model deployment (Garg et al., 2022). When neural network parameters and data are stored in these analog devices, they are susceptible to device- related noise, affecting the performance of network compression (Isik et al., 2023).
Approach Sketch. We revisit the MIQP network pruning optimization from a perspective of en- tropic Wasserstein regression (EWR). which leverages Wasserstein distance model the dissimilar- ity between two distributions. In our context, it measures the dissimilarity of distributions relevant to model parameters and gradient magnitudes before and after pruning. Namely, V is a p dimensional distribution, capturing geometric properties of the loss at W before pruning. Both w and w perform projections for V to a I-D distribution respectively as Ve w and Ve w. Computing the distance between ve w and Ve w falls into the framework of sliced probability divergence (Nadjahi et al., 2020). Under this framework, pruning optimization essentially fine-tunes w and selectively reserves its elements such that the divergence is minimized subject to the sparsity constraint.
Our approach's 's effectiveness in combating noisy gradients established both analytically and nu- merically. We demonstrate that pruning through the Wasserstein regression implicitly enacts gradi- ent averaging using Neighborhood Interpolation. This entails a nuanced balance between capturing gradient covariance and diminishing gradient noise. Notably, the sparse LR formulation is merely specific instance of ours. Yet, our proposed algorithm doesn't demand a markedly higher computa- tional expense. This modest additional effort bestows upon us enhanced robustness.
2 PROBLEM SETUP AND FORMULATION
We first introduce the optimal transport (OT) problem in Kantorovich formulation with entropic regularization, which measures the distance between two distributions, defined in (4) below. The Wasserstein regression formulation as a generalization of the LR formulation then proposed.
The Kantorovich Problem. Denote P2 the set of probability measures with finite second moments. Let μu, v E P2 and let [T2,, v) denote the set of probability measures in P2 with marginal distribu- tions equal y and v. The Wasserssteii distance is defined as
dr
(2(mv) v) inf
|| vll dr(x, v) + log
dr.
(4)
nell(u.)) Jgaxa
Jgex dd
This is also referred to as the entropic OT problem, where the first term is the transportation cost between the two measures and the second term is the entropic regularization with multiplier e.
Sparse EWR Formulation. The pruning problem formulation is defined in (5) below.
min w
Q(w) = (2(W). v) + A ||w- - w| |
(5a)
S.t. ||llo & k
(5b)
The term (x( v) is a Wasserstein distance between the two one-dimensional distributions x and y (or sliced Wasserstein distance for Ve with two one-dimensional projections). The opti- mization is to alter w such that the distance between the two distributions is minimized.
Let and y follow the empirical distributions {xr)] and {w:)=11 Denote by μ4 and is the mass of the data points v; and yi, respectively. We use II to refer to a matrix representing the transporta- tion probability between and y, and II the set of all such matrices, i.c. II [II] Σ Tyy y4 Vj and Σj=1 mij 14 Vi), where μ4 and Dy are marginal distributions. Then (5) reads:
n
min Q(w)= inf w Mei
Σ Σ
[w)(w) vill Tiz +
log
T 14U,i
Tij
+ > l|w
w| (6a)
s.t.
||wll & k
(6b)
LR as a Special Case. Let E = 0. Once we set II to be a diagonal matrix with constant value 1/n, i.e. (iag(1/n), the mass transportation happens only between data point pairs (xt, ys) for i = 1,..., n. Therefore we have
Qn-diag(1/a)) =
[zx(w)) w) - vill? 2 + > ||w
=
1
3

---

Published as a conference paper at ICLR 2024
Le. the formulation (6) in this case degrades to the LR formulation in (3).
3 THEORETICAL ASPECTS
This section reveals some good theoretical properties of the Sparse EWR formulation for network pruming. We start with Proposition 1 below (proof in Appendix A.1) that states geometry prop- erty of OT with squared Euclidean distance cost. Additionally, we demonstrate the Neighborhood Interpolation mechanism that happens implicitly in solving the EWR. Moreover, we show that such mechanism strikes balance in capturing gradient covariance and reducing gradient noise, with brief discussion on the advantage using entropic regularization terms of sample complexity.
Proposition 1 (Convex Hull Distance Equality). Consider set S and its convex hull Conv( S) in Euclidean space, and an arbitrary point x in the space. For any probability measure ù on S, we can find point yf in Conv(S) as y' yyyddllyy Sydv(y) such that ||x v/1 measure on Conv(S) (S).
y12 di(y), where v is
Neighborhood Interpolation. In formulation (5), let Wa, be the first term of W for an arbitrary given æ, i.e., Wa = S [|=((w) - y/1 dr(+)=x)v), where <-|[x> (z) is a conditional measure given x. Now, divide the Euclidean space Ed by subspaces SÝ. Sg.... Sy for y. For any conditional measure (+)xx)))) defined any S (i .,...,, n), there exists measure v(y) defined on Conv(S,) (S,) such that the weighted distance from S to x equals the distance from x to a point y' in Conv(S.)) Hence
[ J US
W₂
> yll dT(-1=)(3))
1
v/l s.t. v. Comm(sy) y du(v), UE V.(x), i i= I,...n.
where (,(x) is the set of measures v that make the equality holds and V.(z) * by Proposition 1. Similarly, we define Wy for any given y and subspaces ST, S ........ S
W
U.S.H I ull (.l()l)))))
ST
| u s.t. x Conr(Sp)
du(x), μ Ui(y), i i= 1....n.
where j is measure defined on Conv(ST) and U.(y) Ø.
We demonstrate the concept of "Neighborhood Interpolation" through an empiricul distribution example. Define S subset of y such that for every element y, E Rz, > Without loxs of gener- ality, can denote S= {yn, y2, ys. ys. ys. The area shaded in gray, denoted as Conv(S), represents the convex hull of S. We computes a weighted summation of distances between and the points yr.... yo- The weights s......., Nz.s are decided by OT. significant n... typi- cally implies that y. is in proximity to x, indicating neighborhood relation. By Proposition I, this weighted distance is analogous to the distance between and where is derived from Comv(S).
1/5
I
0/a
y
5/
02
94
Revisit the EWR formulation. The integral of either Wz or W,, respectively on or y gives the first term of W One can then reformulate (5) as (8) below.
1 min Q(w) Q(w)= inf (Wa(()) dd(()) [W
l/w w/l2
wi| wlok
The objective function calculates the Euclidean distance between a point r and n distinct points. These n points originate from convex hulls, each shaped by different subspaces within y. Similarly, the function measures the distance between each point y and m unique points derived from convex hulls, each formed by distinct subspaces within ...
Interpretation:
4

---

Published as a conference paper at ICLR 2024
We claim that the EWR formulation is more resilient to noisy gradient than its counterpart, the LR formulation given by (3). To understand this claim better, let us reimagine the problem using z; and Yi as substitutes for Sj
empirical distributions, as indicated by (6). In this context, and S.- Moreover, the integration in both W and Wy is replaced with summations, offering a more insightful version of our initial EWR formulation, shown as (9).
Q(w) = inf @m(()) + ΣΣ log ==1==1
Ti 1442
min w:|lw
Tij
(9)
IT
The notation Qu (w) defined in (10) denotes the part of the objective function given fixed II:
Qn(w)= = Σ Σ [[x(w) Uill Tej + > l|w w/1
(10a)
Σ ||:;; will +
[<((w) - will" +1 ||w w//
(10b)
= 2
KI
K
In Qn(w), for each index i, points xf and vo are chosen from the convex hulls formed by points in æ and y. as per the guidelines of Proposition 1. Now, contrasting this with the LR model in (3), the objective ((w) aims for regression directly over the data points whereas every point from one empirical set is matched for Euclidean distance computation to a point derived from convex combination of the other.
The infimum in (9) seeks the OT plan, II, that aligns the empirical distributions æ and closely. In practical terms, for each data point Fı, only a subset of {y,]__ will transport substantial mass, rather than the entire set. This behavior of IT effectively defines "neighborhoods" for each data point 22 within the empirical distribution of y- Here, "neighhorhood" refers to a group of data points in y that proximate to specific z; in the Euclidean sense.
Neighborhood Size Control. A critical aspect of this formulation the entropic regularization term, which is used to modulate the size of these neighborhoods. Specifically, increasing the value of E amplifies the impact of the entropy term. This change broadens the neighborhoods, drawing more data points into the fold of the associated convex hulls. An illustrative extreme case is when E = 0. Here, the OT does one-to-one matching, implying that each data point y; primarily forms the convex hull independently. On the contrary, when oc, all data points are equally weighted by II and hence involved in forming the convex hull as a neighborhood.
Capturing Covariance With Gradient Noise Reduction. For arbitrary the EWR for- mulation essentially strikes a balance between gradient noise reduction and covariance captur- (w) follows similarly. Note that s/ 1 Vej w, where (+) are convex combination coefficients by Proposition 1.
ing. We show the analysis for K( in (10), and k=()(w)
(0)
2 3 yj
Denote VeT Σn VeJ. and G' - (Vl, Ve, The term k(I) expands as follows.
K (1) = Σ (Ve. w VL, w) T -
VeVeT w VVVV w VEVE w W VVVVE w)
(11)
Examining (()) from (11), we see that it effectively replaces half of Ve, with VL,, a version obtained through weighted gradient averaging. Now let's compare the covariance between Ve, and VY. Assume that vl, (1 & & n) are i.i.d. with the same covariance matrix Σ, then G' with equal less noise than G. To show this, denote the covariance matrix of each V by
2 PppE.
Σ Cov
Ve,
5

---

Published as a conference paper at ICLR 2024
The total variance of each gradient in G' (i.e., the trace of Σ/) is then
trace(E)) trace Σ 1 [1)))2))
Σ [ [ trace(2)) & trace(].).
10 < 1, which is a consequence of the The last inequality follows from the fact that E Cauchy-Schwarz inequality given that the coefficients //j form a convex combination.
Originally, the covariance information of all data points is embedded in V&VE, for 1.2..... ......., An alternative representation Ve,VE, which prioritizes noise reduction, but sacrifices some co- variance information. Both Ve,ve, and VLVE highlight trade-off. Notably, both the original covariance V., and its reduedd counterpart V.,V are retained in (11).
Difference From Averaging Prior to Optimization: Next, we show that such gradient av- eraging differs from the averaging operation conducted prior to optimization. Let G' [VE,, Ve,,..., V such that G' represents the row-wise convex combination of G. Approxi- mating the Hessian of the MIQP (2), two scenarios emerge: using G that not performing averaging (case 1) and G' that performs averaging before optimization (case 2).
Case 1 is the original LR formulation (3). Denote by K below its term corresponding to K (1),
K = (w- " N G - -W)
n Σ V.VEL w VVVV w VEVE + w VVVVV. w)
(12)
Case 2 uses the less-noisy row-wise convex combination matrix G' instead G. Yet, the original covariance Ve. is lost: Denote by K' the corresponding term, and we have
K' Σ w VEVE, w - VEVE VVVV VVEVE w) (13)
Inspecting the expressions, it can be observed that (0O)) 11 (also k(²) strikes a balance between K and K". There are two notable extreme cases for II:
1. IT diag(1/n). This corresponds to the LR formulation, as detailed in Section 2. A smaller value & steers the optimization in this direction.
2. II (1/n²)1 1
This arises when oc, meaning the entropy term holds sway in the OT optimization. Here, mutual information minimized to ensure an even contribution from data points in the convex combination. Both o and y/ are the arithmetic means of their respective sets, and all VL, are equivalent to the averaged gradient over the points. Importantly, the original covariance remains intact even in this edge case.
As n grows indefinitely, the empirical OT formulation from (6) approaches its continuous coun- terpart given by (5). Intuitively, a large dataset of high-quality training samples makes the empir- ical fisher close approximation to the true fisher. In such situations, E is set to zero. Brenier's theorem (Peyré, 2019) then suggests that the OT plan turns into a monotone map for costs repre- sented by squared Fuclidean distances. This means IT tends towards diag 1/n). Consequently, the Wasserstein distance formulation reduces to the Euclidean distance formulation, delivering optimal performance with ample data.
An advantage of employing the EWR formulation is its inherent capability of gradient averaging. This approach negates the need to manually determine the convex combination coefficients or re- sort to density estimation to pinpoint the nearest gradient neighbors for averaging. Importantly, this seamless trade-off has an advantage over using Euclidean distance with gradient averaging per- formed prior to optimization. The reason is that the original covariance information will inevitably be lost in the formulation (13), irrespective of the chosen averaging method.
Sample Compexity of W (H, is narrowed to (11/vm) from O(1ntt) by the entropic regular- ization term. Please see Appendix A.2 for details.

---

Published as a conference paper at ICLR 2024
4 ALGORITHM DESIGN
Algorithmic Framework. The algorithm addresses the network pruning problem defined in (5). Drawing inspiration from (Chen et al., 2022), the algorithm incrementally adjusts the sparsity of the weights vector w by using a descending sequence of non-zero elements each sparsity level, the weights and the transportation plan II (can be obtained with efficient algorithms; see Appendix A.3) are refined iteratively.
During
Algorithm Sparse Entropic WAsserstein Regression Pruning (SWAP)
Input: Number of pruning stages T pre-pruning weights w, target sparsity k, regularization pa- rameter A, E, batches Bo, Bi ..., Br, optimization step size 7 > 0.
Output: Post-pruning weights w, satisfying || & k
Set ko, .e....,,y ky descending sequence, with Ko < p and kr
2: w(1))
3: fort O, To
4:
Compute G [Vl]( (w). ..... [W)] with batch B,
5: x. Gw(t), Gw
6:
Compute the pairwise Euclidean distance matrix C between x and
7:
Compute OT planning (I (see Appendix A.3)
8:
VQ + G (II (Gl((G(()).. - Gw)) + A(w(t) - w)
w +)= - w(t) VVQ
9:
10:
(++ - Select from (++ he components having largest absolute values; Others zero 11: W W ((+1)
end for
13: w w- (T+1)
Weights Optimization. The weights optimized using the stochastic gradient descent (SGD) paired with the iterative hard thresholding (IHT) algorithm. We use VQ to represent the derivative of ((w) for brevity, with its comprehensive derivation in Appendix A.4. The expression is
VQ= = G
Gw)) + A(w - wW
(14)
Following the weight updates driven by SGD (as seen in line 9 of Algorithm 1), the IHT method applied. Here, the ki components of w with the largest magnitudes are retained, while the remaining are set to zero, ensuring adherence to the sparsity criteria.
A vital component of the optimization process is the choice of the stepsize (referenced in line 9). Although a straightforward approach might be to set r = +. (where L denotes the Lipschitz constant of Q), better performance proposed in (Chen et al., 2022, Algorithm 2). For the quadratic function Q, the Lipschitz constant given by nA + |G where ||-1lopp indicates the foremost singular value.
be achieved when the stepsize is optimized using the methodology
Line 10 in Algorithm 1 employs the IHT method that is commonly used in sparse learning, which together with line 9, forms a projected gradient descent algorithm. It finds sparse representation of the updated gradient in line 9. Intuitively, IHT keeps the dominant model weights and essentially preserves the most impactful aspects of the to-be-trimmed model. Although there exist alternative strategies for refining the IHT solution-including active set updates, coordinate descent, and the Woodbury formula's back-solving- discussion on these falls outside the scope of this paper. For in-depth exploration, especially with respect to the specialized case described in (7), one can consult (Bhatia et al., 2015; Chen Mazumder, 2020; Hazimeh & Mazumder, 2020; Benbaki al., 2023).
NUMERICAL RESULTS
Our method compared with several existing SoTA methods including MP (magnitude pruning (Mozer & Smolensky. 1989)), WF (WoodFisher (Singh & Alistarh. 2020)), CBS (Combinatorial Brain Surgeon (Yu et al., 2022)), and LR (i.e. the sparse LR formulation adopted by (Chen et al., 2022)). We refer our proposed method EWR (i.e. sparse entropic Wasserstein regression).
7

---

Published as a conference paper at ICLR 2024
Table 1: Model Pruning Accuracy Benchmarking. Five runs are taken for LR and EWR, with the mean and 95% confidence interval (in the brackets) reported. The data of MP, WF, and CBS are copied from (Yu et al., 2022). The superscript "+o" indicates that 20% of data is with noise. Bold texts imply the best performance, with 0.1 percentage as tolerance in difference. The sparsity column is the target sparsity.
Network
Sparsity
MP
WF
CBS
LR (EWRIT-T:))
EWR (proposed)
0.5
93.93
94.02
93.96
95.26 (+0.03)
95.24 (+0.03)
0.6
93.78
93.82
93.96
95.13 (+0.02)
95.13 (+0.01)
0.7
93.62
93.77
93.98
94.93 (+0.03)
95.05 (+0.04)
MLPNet
0.8
92.89
93.57
93.90
94.82 (+0.04)
94.84 (+0.03)
on MNIST
0,9
90.30
91.69
93.14
94.32 (+0.05)
94.30 (+0.05)
(93.97%)
0.95
83.64
85.54
88.92
92.82 (+0.06)
92.86 (+0.05)
0.95+a
90.11 (+0.08)
90.50 (+0.07)
0.98
32.25
38.26
55.45
84.43 (+0.10)
85.71 (+0.09)
0.98 +
82.12 (+0.11)
83.69 (+0.10)
0.5
88.44
90.23
90.58
92.06 (+0.044
92.04 (+0.03)
0.6
85.24
87.96
88.88
91.98 (+.00
91.98 (+0009)
0.7
78.79
81.05
81.84
91.09 (+0.10)
91.89 (+0.10)
ResNet20
0.8
54.01
62.63
51.28
89.00 (+0.12)
90.15 (+0.09)
on CIFARIO
0.9
11.79
11.49
13.68
87.63 (+0.11)
88.82 (+0.10)
(91.36%)
0.95
80.25 (+0.17)
81.33 (+0.15)
0.95
77.37 (+0.18)
79.05 (+0.16)
0.98
68.15 (+0.27)
69.21 (+0.24)
0.98
65.04 (+0.277
68.01 (+0.25)
0.95
83.75 (+0.14)
84.96 (+0.15)
ResNet50
0.95+a
82.34 (+00116)
84.92 (+0.17)
CIFAR10
(92.78%)
0.98
81.04 (+0.14)
82.85 (+0.20)
0.98+t
80.11 (+0.23)
82.94 (+0.22)
0.3
71.60
71.88
71.87
71.14 (+0.08)
71.87 (+0.05)
0.4
69.16
71.15
71.45
71.12 (+0.10)
71.44 (+0.07)
0.5
62.61
68.91
70.21
70.12 (+0.13)
71.12 (+0.18)
MobileNetV on ImageNet
0.6
41.94
60.90
66.37
70.05 (+0.22)
70.92 (+0.18)
0.7
6.78
29.36
55.11
68.15 (+0.17)
69.26 (+0.13)
(71.95%)
0.8
0.11
0.24
16.38
65.72 (+0.119
66.82 (+0.114
0.8
60.29 (+0.18)
63.62 (#0.15)
0.9
47.65 (+0.15)
49.43 (+0.13)
0.9
44.55 (+0.16)
47.98 (+0.16)
Note that LR is special instance EWR, with II diag(1/n). All the methods are benchmarked on pre-trained neural networks: MLPNet (30K parameters) trained MNIST (LeCun al., 1998), ResNet20 (200K parameters) and ResNet50 (25M parameters) (He et al., 2016) trained on CIFARIO (Krizhevsky al.. 2009), and MobileNetV1 (Howard et al., 2017) (4.2M parameters) trained on ImageNet (Deng et al., 2009). The experiment setup for reproducibility' is detailed in AppendixA.5. We deliver more experiments results Appendix A.6-A.9.
Model Accuracy Performance Benchmarking Table compares different networks across var- ious sparsity levels, utilizing different methods. MLPNet's performance on MNIST is consistent across different sparsity levels, with both LR and the proposed EWR method showing superior per- formance. The advantages EWR over the others are reflected by the three more challenging tasks ResNet20 and ResNer50 on CIFAR 10 and MobileNerV1 noisy gradients. In summary, the proposed EWR method consistently outperforms or matches other methods. The LR method performs well at lower sparsity levels but is surpassed otherwise.
ImageNet, especially in the presence of
'The code is available on ? https: / / /iggituuu thub. cwt.bo/ommolleeeeeeoo e 202 wtt.oppi-wwsserrteeeeeeeeee wasserstein-Prrnt
8

---

Published as a conference paper at ICLR 2024
Table 2: Comparison of testing loss values (no fine-tuning) for ResNet20. The result is averaged over 25 runs. The 90% confidence interval is reported. The target sparsity is set to be 0.95.
10% Noisy Data
Sparsity
Noise
Noise 20
LR
EWR
Diff
LR
EWR
Diff
0.95
2.83 (+0.02)
2.75 (+0.02)
2.87%
2.86 (+0.02)
2.74 (+0.01)
4.35%
0.84
1.58 (+0.01)
1,54 (=0.01)
2.73%
1.62 (+0.03)
1.54 (+0.02)
5.15%
0.74
0.66 (+0.00)
0.64 (=0.00)
3.03%
0.66 (+0.00)
0.65 (+0.00)
1.32%
0.63
0.35 (+0.00)
0.35 (+0.00)
0.00%
0.35 (+0.00)
0.35 (+0.00)
0.85%
25% Noisy Data
Sparsity
Noise o
Noise 2a
EWR
Diff
LR
EWR
Diff
0.95
2.87 (+0.02)
2.77 (#0.01)
3.50%
2.89 (+0.02)
2.79 (+0.02)
3.82%
0.84
1.72 (+0.02)
1.65 (+0.02)
4.07%
1.76 (+0.02)
1.69 (+0.022)
4.05%
0.74
0.67 (+0.01)
0.67 (+0.000
0.49%
0.68 (+0.00)
0.67 (+0.000)
1.55%
0.63
0.36 (+.0)
0.36 (=.00)
0.00%
0.35 (+00000)
0.35 (+0.00)
0.00%
Table 3: Comparison of testing loss values (no fine-tuning) for MobileNetV1. The result is averaged from 10 runs. The 90% confidence interval is reported. The target sparsity is set be 0.75.
10% Noisy Data
Sparsity
Noise =
Noise = 2a
LR
EWR
Diff
LR
EWR
Diff
0.75
4.54 (+0006)
4.34 (#0.06)
4.41%
4.62
(+0.07)
4.40 (+0.06)
5.02%
0.63
2.53 (+0.04)
2.46 (#0.03)
2.61%
2.56 (+0.04)
2.48 (+0.04)
3.23%
0.53
1.64 (+0.02)
1.56 (=0.02)
5.16%
1.66 (+0.03)
1.56 (#0.00)
6.01%
0.42
1.30 (+0.00)
1.30 (+00.00)
0.00%
1.30 (+0.00)
1.30 (+0.000
0.00%
25% Noisy Data
Sparsity
Noise
Noise 20
LR
EWR
Diff
LR
EWR
Diff
0.75
4.93 (+0.06)
4.53 (=0.06)
8.13%
5.02 (+0.06)
4.66 (+0.055
7.92%
0.63
2.54 (+00.44)
2.44 (+-0.03)
4.01%
2.57 (+0.004)
2.45 (+-0.03)
5.00%
0.53
1.66 (+00.2)
1.57 (+00002)
5.16%
1.66 (+0002)
1.55 (++0.02)
6.63%
0.42
1.30 (+0.00)
1.30 (+0.00)
0.30%
1.31 (+0.00)
1.31 (#0.00)
0.00%
Robustness with Noisy Gradient. From Section 3, EWR differs from LR in terms of gradient noise reduction achieved by solving the OT problem obtain group of non-trivial data pair weighting coefficients. Hence, LR that has the transportation plan II fixed to diag(1 1/n) naturally serves as baseline for evaluating the effectiveness of such optimization in terms of robustness against noise. In two noisy scenarios, 10%, and 25%, we evaluate loss at noise levels of a and 2a across varying sparsity. Tables 2 and contrast the loss difference between LR and EWR. EWR consistently outperforms LR in both ResNet20 and MobileNetVI. most notably in noisy conditions and at higher sparsity. The peak performance difference 133% favoring EWR on MobileNetV1 at 0.75 sparsity with 25% noise. Hence, EWR outperforms LR.
CONCLUSIONS AND FUTURE IMPACT
The paper offers a novel formulation based on EWR, which strikes a balance between covariance in- formation preservation and noise reduction. The work suggested promising avenues for applications in large-scale model compression, though it may require further empirical validation and exploration of practical implementations.

---

Published as a conference paper at ICLR 2024
REFERENCES
Fan Ang. Li Chen, Nan Zhao, Yunfei Chen, Weidong Wang, and F Richard Yu. Robust federated learning with noisy communication. IEEE Transactions on Communications, 6866):3452-3444 2020.
Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Fast as chita: Neural network pruning with combinatorial optimization. arXiv preprint arXiv:2302. 2023.
Kush Bhatia, Prateck Jain, and Purushottam Kar. Robust regression via hard thresholding. Advances in neural information processing systems, 28, 2015.
Wenyu Chen and Rahul Mazumder. Multivariate convex regression at scale.
arXiv preprint
arXiv:2005. 11588 2020.
Wenyu Chen, Riade Benbaki, Xiang Meng, and Rahul Mazumder. Network pruning at scale: A discrete optimization approach. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Sahaj Garg, Joe Lou, Anirudh Jain, Zhimu Guo, Bhavin J Shastri, and Mitchell Nahmias. Dynamic precision analog computing for neural networks. IEEE Journal of Selected Topics in Quantum Electronics, 29(2: Optical Computing): 1-11 2022.
Aude Genevay, Lénaic Chizat, Francis Bach, Marco Cuturi, and Gabriel Peyré. Sample complex- ity of sinkhorn divergences. In The 22nd international conference on artificial intelligence and statistics, pp.
PMLR, 2019.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problem. Advances in neural information processing systems, 19, 2006.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Hussein Hazimeh and Rahul Mazumder. Fast best subset selection: Coordinate descent and local combinatorial optimization algorithms. Operations Research, 68(5):1517-1537, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04670044611 2017.
Hong Huang, Lan Zhang, Chaoyue Sun. Ruogu Fang, Xiaoyong Yuan, and Dapeng Wu. Fedtiny: Pruned federated learning towards specialized tiny models. arXiv preprint arXiv:2212.01977, 2022.
Berivan Isik, Kristy Choi, Xin Zheng, Tsachy Weissman, Stefano Ermon, H-S Philip Wong, and Armin Alaghi. Neural network compression for noisy storage devices. ACM Transactions on Embedded Computing Systems, 22(3):1 2023.
Hicham Janati, Boris Muzellec, Gabriel Peyré, and Marco Cuturi. Entropic optimal transport be- tween unbalanced gaussian measures has a closed form. Advances in neural information process- ing systems, 13:10648-14479 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
10

---

Published as a conference paper at ICLR 2024
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2322, 1998.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50-60 60, 2020.
Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig, Early stopping without validation set. arXiv preprint S.KIv:17033095800 2017.
Michael C Mozer and Paul Smolensky. Using relevance to reduce network size automatically. Con- nection Science, 1(1):3-16, 1989.
Kimia Nadjahi, Alain Durmus, Lénaic Chizat, Soheil Kolouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. Advances in Neural Information Processing Systems, 33:20802-20812. 2020.
Kimia Nadjahi, Alain Durmus, Pierre Jacob, Roland Badeau, and Umut Simsekli. Fast approx- imation of the slised-wasserstein distance using concentration of random projections. Advances in Neural Information Processing Systems, 14:12411-12424, 2021.
Gabriel Peyré. Course notes on computational optimal transport. 2019.
Galen Reeves. Conditional central limit theorems for gaussian projections. In 2017 IEEE Interna- tional Symposium on Information Theory (ISIT), pp. 3045-3049, IEEE, 2017.
Julien Niklas Siems, Aaron Klein, Cedric Archambeau, and Maren Mahsereci. Dynamic pruning of neural network via gradient signal-to-noise ratio. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neurul Information Processing Systems, 13:18098-18109 2020.
Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
Vladimir Nikolaevich Sudakov. Typical distributions of linear functionals finite-dimensional spaces of higher dimension. In Doklady Akademii Nauk, volume 243, pp. 1402-1405. Russian
Academy of Sciences, 1978.
Tiffany Tuor, Shiqiang Wang, Bong Jun Ko, Changchang Liu, and Kin K Leung. Overcoming noisy and irrelevant data in federated learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 5020-502) IEEE, 2021.
Berkay Turan, César Uribe, Hoi-To Wai, and Mahnoosh Alizadeh. Robust distributed optimization with randomly corrupted gradients. IEEE Transactions Signal Processing, 70:3484-3498, 2022.
Kai Yang, Tao Jiang, Yuanming Shi, and Zhi Ding. Federated learning via over-the-air computation. IEEE transactions on wireless communications, 19(3):2022-25) 2020.
Xin Yu, Thiago Serra, Srikumar Ramalingam, and Shandian Zhe. The combinatorial brain surgeon: Pruning weights that cancel one another in neural networks. International Conference on Machine Learning, pp. 25668 25633. PMLR. 2022.

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PROOF OF PROPOSITION 1
(Convex Hull Distance Equality) Consider set S and its convex hull Conv(S) in Euclidean space, and an arbitrary point x: in the space. For any probability measure on S, we can find point y' in Conv(S) as y (ydv(y) such that || - y'122 SS y do(y), where is measure on (onv(S).
Proof. We define function f(v) || y'll where y' ydl((). This function takes the em- pirical measure / as input and computes the squared Euclidean distance between x and y'. Similarly, we define a function g(v) = S || y/l dv(y). which computes the weighted average of squared Euclidean distances between x and the points in the set S according to the probability measure v.
Without loss of generality, let's assume that S is contained within its convex hull (onv(S). Then, the right-hand side of the equation to be proved in the theorem takes the minimum and maximum values, respectively, at points within Conv(S) (),, i.e.,
Imin inf ||2 dv()
and
Jmax sup S | du()).
The function f(v) takes its maximum value at 2, where 2 is the farthest point inside Comv(S) from x, i.e.,
fmax = sup || - y'1² | =1
Similarly, the minimum value is obtained at y/ = 2', where 2' is the closest point inside Conv(S) to x. The minimum value can reach zero x is inside Comv(S). Formally,
mmi
inf {1|x 2| || Conv(S))
To establish that fmax e Imax, we consider the maximum values of the two functions. The function f(v) takes its maximum value at z, which is the farthest point inside Conv(S) from x. This means that fmax is the squared Euclidean distance between x and 2, i.e., fmax = |z - =|2. On the other hand, the function g(v) computes the weighted average of squared Euclidean distances between and the points in S. The maximum value of g(v), denoted as fmax, corresponds to the squared Euclidean distance between x and the farthest point in S. Since Conv(S) contains S, follows that 2 is farther from r' than any point in S. Therefore, fmax 2 Imax- Similarly, since Conv(S) contains S and z' is the closest point in Conv( (S) x. it follows that fmin & Jmin
We apply the intermediate value theorem to finish the proof. For any measure v. let h(p)) f(u) g(v). Once can find two measures μ4 and Pa such that h(p1) & and h(pz) > 0, hence the zero point of h exists, followed by that there is corresponding value of such that f(u) g(v).
Hence, the conclusion.
A.2 SAMPLE COMPLEXITY
The efficiency of estimator is often measured by its ability to deliver accurate estimates with fewer samples, trait referred to as "good sample complexity'. The entropic Wasserstein distance formulation is posited to enhance noise reduction by optimizing this sample complexity. When 0, the sample complexity of W2(,,,) stands 0 ((1/nt). Specifically,
lim W P.) W (n, w = O(1/nt)
as per (Nadjahi et al., 2020, Corollary 2). As approaches infinity, Wi( 1/) leverages the beneficial characteristics of the maximum mean discrepancies (MMD) (Gretton et al., 2006). which narrows the sample complexity to ((1//n) according to (Genevay et al., 2019, Theorem 3). The coefficient acts regulator, adjusting the sample complexity within this specified range.
12

---

Published as a conference paper at ICLR 2024
A.3 OT PLAN OPTIMIZATION
For an arbitrary w(t), the transportation plan IT(O) is obtained by solving (x(((()), y). The optimization of the transportation plan II in line 7 of Algorithm I is based the Sinkhorn- -nopp algorithm, shown in Algorithm 2 below.
Algorithm 2 Sinkhorn-Knog Algorithm for Regularized OT
Input: C (n Fuclidean distance matrix), Jr, (probability mass of x and y), (regularization parameter), e (tolerance for stopping criterion)
Output: II
1: Initialize K = exp( C/s), u= 1,/n, v= 1
2: repeat
3: u K/Kv
4:
p/KTu 5: until sup {||a- (vygu ||b diag(v)K
"ullz)] <
6:
return II diag(u) K diag(v)
The Sinkhorn- Knopp algorithm an iterative method used to solve regularized OT problems, aiming to find a transportation plan that minimizes total cost while adhering to specific source and target probability distributions. of the negative cost matrix divided by regularization parameter, and two vectors are initialized as uniform distributions. These vectors are then iteratively updated using rules derived from the Kullback-Leible divergence, sceking to align the row and column sums of the resulting matrix with the given source and target distributions. The process continues until the maximum difference between the actual and desired row and column sums falls below specified tolerance, ensuring the solution is feasible.
the algorithm, an initial matrix K is formed using the exponential
Note that and which the dimension is the number of parameters of the neural network, a.k.a. p. Prior research has demonstrated that when certain moderate criteria met, the distribution of lower-dimensional versions of nigh-dimensional random variables tends to closely follow Gaussian (or normal) distri- bution (Sudakov, 1978; Reeves, 2017: Nadjahi al., 2021). In the research by (Janati al., 2020), it is highlighted that given x and y obeying the Gaussian distribution. W2(x,, xr y) can be reduced to concise closed-form solution. As illustrated in Algorithm 3, the OT plan can be directly computed relying solely the statistics of the empirical distributions and y.
one-dimensional projections of high-dimensional random variables VA, of
Algorithm Closed Form Algorithm for Regularized OT
Input: & v 5. a mean(x), b mean(y), of Output:
var(x), of var(y); do (40.0fan+ (1))
~ N ( oa 2
o²
1 [aadwoo
ob
Specifically, the OT plan, represented as II, is governed by the mean and variance of both and y, tandem with the regularization parameter e. To derive the n matrix II, (x,, y1). where i spans from to are employed to extract samples from the distribution produced in Algorithm 3.
As depicted in Figure 1, a comparative evaluation of the two algorithms in terms of attaining the OT objective, W is presented. The entropic regularization term's 's value has been omitted considering its non-impact on the pruning optimization of w, given a constant II. Notably, the disparity between the two algorithms in their objective optimization magnifies as increases. Typically, in real-world applications, the value of & oscillates between 0 and 10 (and we use = 1 most frequently in our experiments). The variance in the performance of the two algorithms concerning OT planning remains trivial, echoing our practical observations during the algorithmic implementation in this study.
13

---

Published as a conference paper at ICLR 2024
Kinkhom-Knopp vs Closed-Foo Form Solution
1,150
SinphornKnopp
(Without Regularization Term)
Closed-Fore
OT Objective W
1,100
10.24%
1,050
1.80%
1,000
0
20
40
60
80
100
Figure I: Comparison between the Sinkhorn- Ponkhorn-KKopp (i.c. Algorithm 2) and the closed-form solution (i.e. Algorithm 3). The plot is made based the data of ResNet20 trained on Cifar10. The relative difference is computed by (red blue) blue.
A.4 DERIVATIVE OF Q(w)
Let's start by revisiting the function Q(w):
Q(w)
Σ Σ [l:(ww) - will méj
x 1| w1
(15)
Given:
I.(w): = ve, w
y Vej
Differentiating Q(w) with respect to w:
VQ(w) - VeT w - VeJ w/ | Tée + w/// w/?
2>(w w) +
Σ Σ
Veg w Vej
(16)
For the gradient of the inner term, consider:
a
n
Ve w
vej
w
Tij
2n Ve, w- VeTwiVe.
(17)
Expressing in Matrix Form. Given matrices:
Vl Vl
G
Vee
π Matrix with elements mij
14

---

Published as a conference paper at ICLR 2024
Consider the double summation:
mez VE w Ve].wvee
For each i, the term Ve, w projects vector w onto V4,. To compute this for every i simultancously. it is Gw. This results in an n x 1 column vector. Similarly, for each j. it is Gw. This also produces an n x 1 column vector.
The difference between these two projections for each i and j is Gw- Gw. This results in an x column vector. To incorporate the Tij weights, we have
II (Gw - Gw)
This operation gives an n 1 column vector, where each element a summation over j for the term Tay (V,, w Ve, w).
To finalize the summation, we multiply it by G yielding
III Gw - Gw))
(18)
Combining (16), (17), and (18), we get:
VQ(w)= 2\(w w) 2G 2 Gw)) 2| (II(GIWWWW.... [[[))--W)) X(w w)]
(19)
A.5 EXPERIMENT SETUP
The models MLPNet, ResNer20, and MobileNetV1 underwent a pre-training phase of 100 epochs utilizing 4 NVIDIA Tesla V100 32 GB GPUs connected with NVlink. The training times were approximately hour for MLPNet, 3 hours for ResNer20, and day for MobileNetV1. Pre-pruning accuracy levels for these models are detailed under their respective names in Table For the pruning process, either utilized 2 NVIDIA Tesla V100 32 GB GPUs with NVlink or single Tesla A PCIE (available in 40 or 80 GB configurations). It's worth emphasizing the time- inte-intessive nature of training and pruning the MobileNetVi on ImageNet; thus, harnessing multiple GPUs for concurrent training is highly recommended.
In Table set the pruning stage of LR and EWR be 15 for MLPNet and ResNet20 and 10 for MobileNetV1. 1. The sparsity k1, .2....Kr in Algorithm 1 is arranged following an exponential gradual pruning schedule
(Ru - kr) (1-T)
4 = kr
T
with the initial sparsity ky set zero. The fisher sample size setup follows (Chen al., 2022, Table 2), shown as Table 4 of this paper below.
Table Comparison of Sample and Batch Sizes for Different Models
Model
MLPNet
ResNet20/50
MobileNet
Sample
Batch
Sample
Batch
Sample
Batch
WF & CBS
1000
1000
400
2400
LR & EWR
1000
1000
1000
16
In Table 2, Table 3, Table 6, Table 7, and additional results provided in the Appendix, sparsity is set using a linear gradual pruning strategy, progressing from 0 to 0.75 or 0.95 across ten distinct stages for MLPNet and ResNet, and from 0 to 0.75 across eight distinct stages for MobileNetV 1. The values are computed with linear incremental steps, from zero to the target sparsity. Notably, all recorded loss values captured immediately post-pruning, devoid of any subsequent fine-tuning.
15

---

Published as a conference paper at ICLR 2024
This approach ensures that the loss values exclusively reflect the impact of the pruning algorithms, without being clouded by external factors. Contrasting with Table I, where each row represents a full pruning cycle, the loss values here are recorded across the ten incremental pruning stages. The empirical fisher is computed based on 100 samples with a batch size 1.
In calibrating noise for data in neural networks, start with well-trained network. First, calculate the standard deviation a of the network's derivative. Then, we add Gaussian noise with zero mean the data. After adding the noise, the standard deviation of the network's derivative changes to a new value, a', which is always greater than sigma. The goal is to adjust the standard (referred to noise level being σ)
deviation of the Gaussian noise that o' becomes o' or o' = a + 2α (referred to as noise level being 2σ).
Throughout the paper, we À in the optimization problem (6) 0.01. The regularization multiplier is set to unless specified otherwise. The noise level is set to be the standard deviation of the original gradients.
A.6 ABLATION STUDY
Table 5: Comparison of loss values in terms of for ResNet20. The results are obtained from 25 runs, with 10% Noisy data and noise level a. The target sparsity is 0.95.
Sparsity
LR Loss
EWR Loss
E 0
E
10
E = 00
0.95
2.89
2.77
2.78
2.78
2.79
2.97
0.84
1.76
1.69
1.61
1.68
1.71
2.62
0.74
0.68
0.68
0.67
0.66
0.66
0.81
0.63
0.36
0.36
0.36
0.36
0.36
0.37
0.53
0.31
0.31
0.31
0.31
0.31
0.31
0.42
0.31
0.31
0.31
0.31
0.31
0.31
0.32
0.30
0.30
0.30
0.30
0.30
0.30
0.21
0.30
0.30
0.30
0.30
0.30
0.30
0.11
0.31
0.31
0.31
0.31
0.31
0.31
Loss Reduction
0.15
(LR -EWR)
0.1
0.05
0
=
E = 2
10
1 Sparsity = 0.74
Sparsity = 0.84
Sparsity 0.95
Figure 2: Loss reduction with different The result averaged over 25 runs for ResNet20, with 10% Noisy data and noise level a. The error bar shows 90% confidence interval. The target sparsity 0.95.
The ablation study centered ResNet20 gives insight into the influence of the entropic regulariza- tion multiplier, E, on pruning. The aim is to understand how different values of this parameter affect the loss L(w) during the pruning process.
Observations from Loss Values. From Table 5, we can glean the following points. For higher sparsity levels (0.95, 0.84, 0.74), EWR Loss consistently outperforms the LR Loss, except for
16

---

Published as a conference paper at ICLR 2024
CD. The optimum performance of the EWR Loss, across varying sparsities, tends to occur when is set at values between and 2. As E tends towards infinity, the EWR loss exceeds the LR loss. This phenomenon aligns with the neighborhood interpolation elaborated upon in Section 3. Specifically, incorporating an excessive number of distant data points into the interpolation detrimentally impacts performance. For lower sparsity levels (from 0.63 downwards), the differences between LR Loss and EWR Loss across different E values are minuscule.
Loss Reduction Insights. Referencing Figure 2, it's evident that the loss reduction (difference between LR and EWR) is more pronounced at higher sparsity levels. For a sparsity of 0.84, = demonstrates the most significant loss reduction. The trend of EWR loss for different values is consistent across varying sparsity levels. The impact E is clearly visible at higher sparsity levels. For mid high sparsity levels, lower values of e (specifically and 2) seem to achieve the best balance in terms of loss.
EWR Loss for Different Values
3
0
E = 1
E = 2
E = 10
EWR Loss
2
0.8
0.6
0.4
0.2
Sparsity
Figure 3: EWR Loss in function of the sparsity. The result is obtained over 25 runs ResNet20, with 10% Noisy data and noise level a. The target sparsity is 0.95.
Figure 3 illustrates EWR Loss for various s values against sparsity, there is a clear trend of de- creasing loss sparsity decreases, consistent across all values. Particularly s 00, there's pronounced increase in loss at higher sparsity, suggesting that extreme entropic regularization might hinder optimal pruning. However, at low sparsity levels, the loss consistent across all empha- sizing the minimal impact of E in limited pruning scenarios. This underscores the significance of choosing an appropriate e, balaneing between regularization and pruning efficiency.
The ablation study provides insights into the role of the parameter in pruning. Its influence di- minishes at low sparsity levels but becomes significant extremely high sparsity. For ResNet20, an optimal range for appears to be between and 2, ensuring effective pruning. Generally, while exact choice e doesn't drastically alter pruning outcome, exceedingly high values, such as 1 x 10a, might lead to less than ideal pruning decisions in very sparse networks.
A.7 ANALYSIS OF PRUNING STAGE VERSUS PERFORMANCE
Analysis Loss. Figure 4 depicts the relationship between the sparsity and the difference in loss (LR EWWR under the influence of varying noise levels.
The left plot, representing data with a 10% noise level, depicts a prominent decrease in the difference of loss sparsity is reduced for both noise levels 2o and σ. Notably, in the 2x noise scenario, there is sharp decline in loss difference when sparsity transitions from 0.95 to 0.53. Beyond this threshold, the loss difference stabilizes and remains near For the noise level a, the decrease
17

---

Published as a conference paper at ICLR 2024
10% Noisy Data
25% Noisy Data
0.14
Noise 20
Noise 20
Noise
4.35%
0.12
Noise
3.82%
Difference in Loss (LR EERR
0.12
0.1
3,509
0.1
5.15%
0.08
4.05%
0.08
1.32%
0.06
0.06
4.07%
2,875
0.04
0.04
2.73%
0.02
0.02
1.55%
0.85$
0.49%
3.03%
0
0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
1
Sparsity
Sparsity
Figure 4: Difference in loss between LR and EWR for ResNet20. The data is from Table 2. The relative loss improvement of EWR over LR is reported. The target sparsity 0.95.
10% Noisy Data
25% Noisy Data
0.25
Noise 2a
Noise 20
0.4
8.13%
Noise a
Noise
Difference in Loss (LR -EWR)
5.02%
7.92%
0.2
4.41%
0.3
0.15
0.2
0.1
6.01 %
5.16%
3.23%
5.00%
2.61%
0.1
6.63%
4.01%
0.05
5.16%
0
0
0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
Sparsity
Sparsity
Figure 5: Difference in loss between LR and EWR for MobileNetV1. The data is from Table 3. The relative loss improvement of EWR over LR reported. The target sparsity 0.75.
appears gradual. It is of interest to observe that the loss difference diminishes more swiftly for 2o compared to a. The error bars offer insights into data variability, showcasing broader intervals at elevated sparsity levels, which suggests greater unpredictability at these levels, particularly in the 2a setting.
The right plot represents data with 25% noise. While the trends in loss difference share similarities with the 10% noisy data, the exact values differ slightly. In this 25% noise setting, the decline in loss difference between the noise levels is similar. The error bars, indicating confidence intervals,
18

---

Published as a conference paper at ICLR 2024
Pruning Stage Accuracy (%)
3
Accuracy Difference (%)
2
0
2
4
8
Pruning Stage
Pruning Stage vs. Top 5 Accuracy (%)
Top 5 Accuracy Difference (%)
1
3
5
7
Pruning Stage
Figure 6: Accuracy difference vs. pruning stage for ResNet20. The difference defined be (EWR LR) / LR. The data is obtained over 5-25 runs over combinations of different setups: noise level and 20, noisy gradient proportion 10% and 25%, & = 1,2. The target sparsity is 0.95.
highlight the increased variability larger sparsity levels. This variability is most noticeable for the 2a setting at the top sparsity levels.
Figure depicts the difference in loss between LR and EWR algorithms applied the MobileNetVI. 1. In both two cases 10% and 25% noisy data, sparsity increases, the loss difference diminishes, par- ticularly when sparsity approximately 0.42 less. The 10% noisy data reveals that the difference in loss for noise level 2o is marginally higher than that of a for most sparsity levels. In contrast, the 25% noisy data sometimes exhibits a reversal in this trend, especially at the highest sparsity level of 0.74.
Confidence intervals provided at select data points underscore the reliability of the data, with the 25% noisy data showing tighter intervals compared to the 10% scenario. This infers higher sistency in the measurements or a minimized effect of outliers in the 25% noisy data. These plots accentuate the interplay between sparsity, noise, and the performance difference between the two al-
19

---

Published as a conference paper at ICLR 2024
Pruning Stage vs. Accuracy (%)
15
Accuracy Difference (%)
10
5
0
2
3
4
6
Pruning Stage
Pruning Stage vs. Top 5 Accuracy (%)
Top 5 Accuracy Difference (%)
8
4
2
2
3
5
7
Pruning Stage
Figure 7: Accuracy difference vs. pruning stage for MobileNetV1. The difference defined to be (EWR LR) / LR. The data is obtained over 5-10 runs combinations of different setups: noise level a and 2a, noisy gradient proportion 10% and 25%, E 1, 2. The target sparsity is 0.95.
gorithms, emphasizing the significance of noise levels in algorithmic performance evaluations across various sparsity conditions. The underlying rationale is that as the noise intensity is too high, the performance of both LR and EWR tends to deteriorate. Consequently, their performances converge, resulting in diminished differential between the two.
Analysis of Accuracy. The box plots in Figure together with Table show how the ResNer20 model performs at different pruning stages in terms of top-1 accuracy (also referred to accuracy or the overall accuracy) and top-5 accuracy difference. Top-5 accuracy means any our model's top 5 highest probability answers match with the expected answer. The difference is computed for EWR, using LR as the baseline. When we look closely, we can see patterns that help us understand how much pruning affects the model.
In the earlier pruming stages, both the overall accuracy and the top accuracy differences for EWR minimal, suggesting that EWR remains closely aligned with the baseline LR in terms of per-
20

---

Published as a conference paper at ICLR 2024
Table 6: Comparison of testing Top-1 accuracy (%) (without the fine-tuning steps imposed) values for LR and EWR for ResNet20. The result is averaged over 25 runs. The 90% confidence interval reported. The target sparsity be 0.95.
10% Noisy Data
Sparsity
Noise
Noise = 2a
LR
EWR
Diff
LR
EWR
Diff
0.95
11,65 (+.)211)
11.70 (=0.15)
0.47%
11.53 (#0.14)
11.72 (#0.11)
1.70%
0.84
57.01 (+0.10)
58.17 (=0.10)
2.03%
56.03 (+0.09)
57.50 (#0.09)
2.63%
0.74
80.53 (+0.05)
81.12 (=0.07)
0.73%
80.23 (+0.05)
81.06 (+0.06)
1.03%
0.63
89.85 (+0.06)
89.90 (+0.06)
0.00%
88.95 (+0.06)
88.96 (+0.05)
0.85%
25% Noisy Data
Sparsity
Noise =
Noise = 2a
LR
EWR
Diff
LR
EWR
Diff
0.95
11.63 (+0.17)
11.98 (+0.115)
3.03%
11.68 (+0.18)
12.04 (+0.16)
3.04%
0.84
56.90 (+0.09)
58.90 (+0.09)
3.52%
57.90
(+0.07)
59.76 (+0.066)
3.22%
0.74
80.37 (+005
80.76 (=0.10)
0.49%
81.50
(=0.08)
82.27 (=0.05)
0.95%
0.63
89.62 (+.05
89.62 0+.0)
0.00%
90.56 (+0.06)
90.56 (+0.05)
0.00%
Table 7: Comparison of testing Top-1 accuracy (%) (without the fine-tuning step imposed) values for LR and EWR for MobileNetV1. The result is averaged from 10 runs. The 90% confidence interval reported. The target sparsity to be 0.75.
10% Noisy Data
Sparsity
Noise =
Noise = 20
LR
EWR
Diff
LR
EWR
Diff
0.75
17.10 (+)))
17.79 (=0.15)
4.02%
17.05
(+0.21)
17.84 (#0.18)
4.62%
0.63
43.92 (+005
44.92 (+0.05)
2.27%
43.27
(+0.10)
44.53 (+0.10)
2.92%
0.53
61.06 0+044
61.84 0=00)
1.28%
60.51
(+0.05)
63.90 (+0.5
2.30%
0.42
68.09 (+.5)
68.09 (=000)
0.00%
67.24 (+0.05)
67.24 (=0.05)
0.00%
25% Noisy Data
Sparsity
Noise =
Noise = 2a
LR
EWR
Diff
LR
EWR
Diff
0.75
13.97 (+0.14)
14.88 (+00226)
6.52%
13.78 (+0.21)
14.78 (+0.25)
7.29%
0.63
43.45 (+0.10)
44.78 (#0.09)
3.05%
43.21 (+0.10)
45.13 (=0.11)
4.44%
0.53
60.59 (+.05
61.26 (#..04)
1.10%
60.07 (+0005)
61.56 (=0.05)
2.48%
0.42
67.33 (+0.04)
67.46 (+0005)
0.20%
66.92 (+00066
66.92 (+0005
0.00%
formance. This minimal deviation can be viewed an advantage, as implies that even with simplifications brought by pruning, EWR retains its effectiveness compared to LR.
However, as pruning intensifies, the patterns begin reveal more about EWR's relative strengths. Although the accuracy difference increases, this increase in the context of the baseline suggests that EWR might be better at handling intense pruning than LR. Notably, in the top 5 accuracy, the discrepancies remain relatively low compared to the overall accuracy up until the more aggressive pruning stages. This suggests that while the model's primary prediction confidence may decrease, the true class is still frequently among its top 5 predictions. In essence, EWR seems retain a broader spectrum of potential correct classifications even when it's uncertain about the primary prediction.
Figure together with Table 7 shows the performance difference between EWR and LR using Mo- bileNetV1 across various pruning stages. In the first plot showcasing the overall accuracy, there's an evident upward trend in the median accuracy difference as pruning intensifies. Initially, the dif- ference is marginal, which suggests that EWR's performance closely mirrors L.R during the early
21

---

Published as a conference paper at ICLR 2024
Scalability Test ResNet50
LR
300
EWR
Execution Time (seconds)
200
100
0.5
1.5
2.5
Parameters in Pruning
-107
Figure 8: Scalability of Algorithm 1. This plot shows the comparison between LR and EWR of the single-round execution time of Lines 4-11. The test is performed on an NVIDIA Tesla V100 32GB GPU. The fisher sample size is set to 1000. Layers of the networks gradually added for pruning (hence the number of model parameters p increases). The OT planning II is solved using the Sinkorn-Kang Knopp method shown in Algorithm 2.
pruning stages. However, as we progress to the 5th and 6th stages, the accuracy difference widens considerably, indicating that EWR might be outpacing LR. The 7th stage particularly striking, with median accuracy difference surpassing 6%, pointing towards potential superiority of EWR extreme pruning scenarios.
The second plot focuses on the top 5 accuracy differences, presenting more varied pattern. The early stages indicate a tight performance race between EWR and LR. By the 2nd stage, there's 's a minor dip, hinting at EWR's possible underperformance. This scenario changes by the 3rd stage as EWR regains momentum. The later stages, especially the 5th and 6th, denote a significant rise in the median difference in EWR's favor. Much like the accuracy chart, the 7th stage is distinct, with EWR showcasing a considerable advantage in the top 5 accuracy over LR.
A.8 ALGORITIM SCALABILITY
In this section, we analyze the scalability of Algorithm 1 with respect the number of model parameters involved in pruning. The result is shown in Figure 8. It can be observed that the execution time scales linearly with the number of pruning parameters. The extra cost of solving the OT is marginal. Theoretically, could derive this linear scalability by inspecting Line 9, which is the most consucinn step. The required operations can be decomposed as sequential operations: matrix-vector multiplications Gw and Gw in ((p)), the vector subtraction Gw Gw in O(n), a matrix-matrix multiplication II(Gw Gw) in O(n²), matrix transposition and multiplication with G in (np), and the vector subtraction and scalar multiplication A(w
in Op)). Thus, the overall complexity is (mp)), with significantly larger than n practically. Given fixed fisher sample size n, the loop of Algorithm scales linearly with the number of pruning parameters p.
A.9 OPTIMAL TRANSPORT VISUALIZATION
In this section, showease the optimized OT plan denoted as II. This was derived from the pruning applied to ResNet20. Figure 9 displays two data sets: 1) {x,},_} This is Gw where w is the pruned model. 2) (v:) 1 This represents Gw where is the original unpruned model.
22

---

Published as a conference paper at ICLR 2024
In Figure 10, the matrix II is shown through vibrant heatmaps that adjust with varying values. For small E, the majority the data remains near a diagonal. As increases, there's a broader data distribution, notably for central data points.
KDE
60
KDE
0.2
Frequency
40
Density
0.1
20
60
40
20
0
20
40
60
40
20
0
40
Figure 9: The histogram and kde plot of the empirical distribution {a.)7_) and {ys 1-1
(a)
(b) 0.01
(
(d) E
Figure 10: Heatmap of the optimized OT plan II across different c. Darker color indicate larger value II.
23


Reviews:
[{"review_id": "20q40TxpSF", "reviewer": "Reviewer_5Zk8", "summary": "Robust Network Pruning With Sparse Entropic Wassertein Regression\n\nIn this paper, the authors propose a method to prune neural networks. In particular, in the Sparse Linear Regression Formulation of network pruning, the authors replace the first $l_0$ regression term with Wasserstein regression. Theoretical justifications and empirical experiments show that the proposed pruning strategy is effective and robust against gradient/data noise.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and the problem is well-motivated.\n- The proposed method has desirable properties and shows improved performance over previous methods, especially at larger sparsity.", "weaknesses": "- “The noise level σ is set to be the standard deviation of the original gradients”. Why is this the noise level for both gradients and data? I would like to see a more detailed explanation how how the noise is added to data and gradients.\n- Can the authors also provide an accuracy table for Table 2 and Table 3?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RtYQfYzV1q", "reviewer": "Reviewer_e6Ww", "summary": "The authors propose a technique for pruning (sparsification) of neural networks that relies on robust estimation of the empirical Fisher Information Matrix as a surrogate for the Hessian of the training loss.  Earlier work has relied on a decomposition of the FIM to motivate a sparse LR formulation of an MIQP framework. \n\nIn contrast, in this work the authors propose a framework to address instances of contaminated gradients. In this situation, one must leverage robust estimators of the FIM, or risk a significant drop in empirical performance. By studying the original MIQP problem from the perspective of entropic Wasserstein regression, the authors propose a variation of the sparse LR formulation which amounts to substituting the 2-Wasserstein distance with entropic regularization for the quadratic regression loss. Notably, without entropic regularization, the formulation is equivalent to that of the sparse LR framework. \n\nTheoretically, the authors demonstrate that pruning via entropic Wasserstein regression exactly corresponds to gradient averaging using Neighborhood Interpolation, with the entropic regularization term governing the size of the neighborhood. Algorithmically, the method is simple and computationally efficient. Finding solutions to the problem is done via coupling sinkhorn iterations with SGD. Numerically, the performance of the method exceeds that of previous work and is competitive with the state of the art. \n\nThe method is simple and implies an elegant interpretation, as explored by the authors. Numerically, improvements over existing methods are observed- particularly when the training gradients are corrupted by noise. However, there is a significant number of grammatical mistakes and instances of poor phrasing. I do recommend this paper for acceptance, but suggest that the authors devote more time to proofreading the manuscript.", "questions": "I may have missed it, but it is not obvious to me what kind of assumption is made regarding the noise. Additionally, it is unclear what kind of noise is introduced in the experiments. What kinds of noise can this method be a good choice for? What is an appropriate choice of the regularization weight for different kinds / magnitudes of noise?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The following are the primary strengths of this paper:\n\n- The authors propose a straightforward (but novel) modification to the sparse LR framework for neural network pruning. The modification amounts to an additional regularization term grounded by an interpretation using the principles of optimal transport. The optimization problem remains efficiently solvable. \n\n- The authors motivate their method via an analysis of the robustness  properties exhibited by solutions to their optimization problem. Namely, by proving that pruning using their technique implicitly corresponds to gradient averaging via a certain neighborhood interpolation and naturally trades off between a measure of robustness and the quality of the covariance estimator.\n\n- Additional discussions on sample complexity, ablations on the sparsity and regularization parameter, and alternative methods for computation of the EWR solution are comprehensive and provided in the appendix.\n\n- The method proposed by the authors improves results over existing methods, particularly when the training gradients are corrupted by noise. \n\n- Code is provided by the authors as a github link, which is appreciated.", "weaknesses": "As a reviewer, I highlight that I am unfamiliar with the current state-of-the-art pruning techniques. I defer to other reviewers regarding the thoroughness of the comparative experiments.However, the method seems grounded. The structure of the manuscript is OK. The writing and clarity of this paper could be significantly  improved. In particular, many phrases and statements are unclear, beginning with the abstract:\n\n_This study unveils a cutting-edge technique for neural network pruning that judiciously addresses noisy gradients during the computation of the empirical Fisher Information Matrix (FIM)._\n\nAdditionally, some choices could be better motivated. E.g. the pruning step (step 10 of alg 1) as a projection onto the l-0 norm ball. However, as (reasonably) referenced by the authors, analysis of the optimization problem lies outside the scope of the paper. \n\nThroughout the main text, there are many typos and grammatical errors. Although the draft is readable in its current form, I would suggest the authors carefully review the manuscript and improve the writing- e.g. the following are some examples:\n- Now let’s comparing the covariance between…\n- Intuitively, a large dataset of high-quality training samples diminishes concerns over gradient noise, making the empirical fisher a close approximation to the true fisher.\n- Importantly, this seamless trade-off eludes the combination of the Euclidean distance with gradient averaging.\n- Remark that LR is a special case of EWR…", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALRApVy471", "reviewer": "Reviewer_WJBJ", "summary": "This paper reformulate the network pruning problem into an Wasserstein distance regularized sparse linear regression problem. The author shows that the ordinary sparse linear regression is just a special case when only diagnal entries exists in the transportation matrix. The authors also show that it can be viewed as neighborhood size control, which trade off between covariance capturing and gradient noise reduction. Numerical results show improvement on MLPNet, ResNet20 and MobileNetV1 architectures, over existing magnitude prunning or CVS approaches.", "questions": "Why using 0.84, 0.74, 0.75, 0.63 ... values in Table 2 and 3? This is very uncommon and even in Table 1, the results are following traditional sparsity levels.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The reformulation is novel as far as I know. The author successfully connect the reformulation to existing sparse regression set up, which makes a good story here.\n2. The analysis on neighborhood control is also insightful.", "weaknesses": "1. The experiments are weak, without test on state-of-the-art architectures like transformers, or larger models like ResNet50, making it suspicious that the proposed approach does not work well on larger model sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i2jBreCPP8", "reviewer": "Reviewer_u9iD", "summary": "This paper proposed a new network pruning method based on Wasserstein distance. Under the convex hull distance equality, the problem is reformulated using Neighborhood Interpolation. Under this formulation, the authors analyzed that compared to the traditional LR formulation, this method learned a data-adaptive gradient averaging weights to smooth noise in gradient estimation. An iterative optimization method is provided. Experiments on several backbones and datasets demonstrated the utility.", "questions": "Just to be curious, as the main contribution lies in better estimating the gradient under the existence of noise, is it possible to extend the proposed method to other applications beyond network pruning?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The noise corruption in gradient estimation considered in this paper is an interesting and important problem. \n2. The idea of leveraging Wasserstein distance, especially the reformulation through neighborhood interpolation in combating noise is interesting. \n3. The authors have conducted supportive experiments to validate the advantages. Particularly, I like Figure 9 which illustrates the distribution of learned $\\Pi_t$ as the noise level varies. \n4. The paper is well-organized and written.", "weaknesses": "1. Is it possible to conduct a convergence analysis for algorithm 1? If it is difficult, does the main challenge lie in the simultaneous optimization of $\\Pi_t$?\n2. It would be more interesting to compare with more baselines, in addition to the LR method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPC6bV2ID6", "reviewer": "Reviewer_yr3z", "summary": "Computation of the empirical Fisher Information Matrix (FIM) is an important part of neural network pruning and subjected to \nnoisy gradients. As a solution, this paper proposes an entropic Wasserstein regression (EWR) formulation to address the issue above. The method EWR is demonstrated to able to implicitly enacts gradient averaging using Neighborhood Interpolation, resulting in a balance in capturing gradient covariance and reducing gradient noise.  The paper demonstrates the proposed methods in combatting noisy gradients through theoretical analysis (section 3), and empirical evidence (section 4, 5). The empirical evidence is showcased using various models and vision datasets.", "questions": "1. Suggestion to change some of the cited references to be included in parentheses for better presentation. Eg in page 3 Computing the distance between .... (Nadjahi et al., 2020).\n2. In the introduction, GPT-4 is used an example of model with substantial size and complexity, Is it possible to show case such a pruning in a language model?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provide clear motivation about combating against noise that may comes from different aspects. \n2. Provide theoretical analysis for the proposed method and empirical evidence.", "weaknesses": "1. Is it possible to show an analysis in term of computational expense incurred, as the scale of the model increases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)