Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions
Authors: Murad Tukan
Loay Mualem
Alaa Maalouf
Venue: NeurIPS 2022 Accept (2022)
Keywords: Coresets
Convex Geometry
Neural Network Pruning

Abstract: Pruning is one of the predominant approaches for compressing deep neural networks (DNNs). Lately, coresets (provable data summarizations) were leveraged for pruning DNNs, adding the advantage of theoretical guarantees on the trade-off between the compression rate and the approximation error. However, coresets in this domain were either data dependant or generated under restrictive assumptions on both the model's weights and inputs. In real-world scenarios, such assumptions are rarely satisfied, limiting the applicability of coresets. To this end, we suggest a novel and robust framework for computing such coresets under mild assumptions on the model's weights and without any assumption on the training data. The idea is to compute the importance of each neuron in each layer with respect to the output of the following layer. This is achieved by an elegant combination of L\"{o}wner ellipsoid and Caratheodory theorem.
Our method is simultaneously data-independent, applicable to various networks and datasets (due to the simplified assumptions), and theoretically supported. Experimental results show that our method outperforms existing coreset based neural pruning approaches across a wide range of networks and datasets. For example, our method achieved a $62\%$ compression rate on ResNet50 on ImageNet with $1.09\%$ drop in accuracy.

Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions
Murad Tukan* muradtuk@gmail.com com
Loay Mualem* t 1oaymua@gmail.cm com com
Alaa Maalouf f
Alaamaalouf www.faii00aaa.....m com
Abstract
Pruning one of the predominant approaches for compressing deep neural net- works (DNNs). Lately, coresets (provable data summarizations) were leveraged for pruning DNNs, adding the advantage of theoretical guarantees on the trade-off between the compression rate and the approximation error. However, coresets in this domain were either data-dependent or generated under restrictive assumptions on both the model's weights and inputs. In real-world scenarios, such assumptions are rarely satisfied, limiting the applicability of coresets. this end, we suggest a novel and robust framework for computing such coresets under mild assumptions on the model's s weights and without any assumption on the training data. The idea is to compute the importance of each neuron in each layer with respect to the output the following layer. This is achieved by a combination Löwner ellipsoid and Caratheodory theorem. Our method is simultaneously intependdeenn,nt, applicable to various networks and datasets (due to the simplified assumptions), and theoreti- cally supported. Experimental results show that method outperforms existing coreset based neural pruning approaches across wide range networks and datasets. For example, our method achieved 62% compression rate on ResNet50 on ImageNet with 1.09% drop in accuracy.
1 Introduction and Backround
Deep neural networks (DNNs) achieved state-of-the-ar (SOTA) performance large variety of tasks, e.g., in computer vision [33] 50 and natural language processing (NLP; [87 20]). However, DNNs usually contain millions
billions of parameters in order achieve SOTA performances resulting in large storage requirements and long inference time. This is obstructive when, e.g., dealing with limited hardware real-time systems such autonomous cars and text/speech translation. To this end, a large body of research is dedicated to reducing the size and inference costs of DNNs.
Pruning. A dominant approach widely used for reducing the size of DNNs is to utilize pruning algorithm to remove redundant parameters from the original, over over-parameterized network. In general, pruning can be categorized into two main types: (i) Unstructured pruning [1 [ reduces the number of non-zero parameters by inducing sparsity into weight parameters, which can achieve high compression rates but requires specialized software and/or hardware in order to achieve faster inference times. (ii) Structured pruning [35 60) (77 modifies the structure of the underlying weight tensors, by removing filters/neurons from each layer, usually resulting in smaller compression rates while directly achieving faster inference times with no specialized software; see section
"These authors equally contributed to this paper. I Department of Computer Science, University of Haifa
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1.1 Coresets for Pruning
potentially due to the empirical observation that existing filter pruning approaches consistently yield impressive results. However, most pruning methods are based on heuristics, lacking theoretical guarantees on the trade-off between the compression rate and the approximation error. This was the motive for introducing coresets [82 600 to the world of pruning.
Notably, many recent papers focused on various types of filter pruning 88 79]
Coresets. In machine learning, we are (usually) given an input set P C Ie of n points, its corresponding weights function w P IR, feasible set of queries X, and a loss function Φ: P x X + [0, oo). The tuple (P, X, Φ) is called query space, and it defines the optimization problem at hand. For a given problem that is defined by its query space (P, w, X, Φ), and an error parameter (0, 1), e-coreset is is a small weighted subset of the input points that approximates the loss of the input set for every feasible query x, up to a provable bound of E.
Since coresets approximate the cost of every query, traditional (possibly inefficient) algorithms/solvers can be applied on coresets to obtain an approximation of the optimal solution on the full data, using less time and memory; see Section B. 3.2 in the appendix for more details.
Layer input neurons and output naurons, representing of points in it (d=5)
Each output neuron 1s 88 (d-5).
output some neuron in the layer #(( + 1)
(P
x e R
:5
Asproeimating the
the ourput a()
C
My R
Wi x)
Figure 1: Illustration of our neuron coreset construction on toy example.
Pruning via coresets. Recently, some inspiring innovative frameworks 82 60 5 leveraged the idea of coresets for pruning DNNs. Any layer l can be represented as a set P (P), Pa} of n >> points in Rd, where d is the number of imput neurons, and n in the number of output neurons, i..., each point p P, represents specific neuron using its d weights (parameters). When the layer receives an input vector x E Rd, it outputs the vector (ø(pf x). ... <(TTx) x)). where Φ R - TR is activation function which defines non-linear mapping. Focusing layer that follows f, defined by its corresponding vector of weights w = (W1, ... wn), we set in the context of coresets, w(p:) := w; for every i E {I,.... n) this is just a mapping from p; to we to simplify the writing and reading. Note that the output of this neuron is Epep w()) (p" (p"x)) v). Assuming that we are given an e-coreset (C, u) for the query space (P, w, 12, Φ) where C C P. and + R. we have that (C, u) approximates the output of this specific neuron 1) for every query using less parameters; see Figure I To formalize the stated above, we now define coresets in the context of activation functions.
single neuron v in the
Definition 1.1 (Coreset for activation functions). Let E 10, and let (P, w. Rdd .)) be a query space. Then the pair (C, u), is an e-coreset for (P, w. Rd (.)) (f) C C P, (ii) u: C - 10, oo), and (iii) for every x E X, Eneo u()) PCP w(p))(p"z) <
Since C' is a subset of P. we can remove (assign zero to) all weights from w that corresponds to points not chosen to be in C' from P. and replace the weights of the chosen points in C' with the new weights vector u; see Figure in 82 for a visual illustration. To prune neurons, we refer the reader to Section 2.5 as it is simple extension. Prior work showed that such approaches successfully result in high compression rates across a wide range of networks and datasets, and even achieves SOTA performance on verity of them.
The main (strong) advantage the coreset approach over others was the provided provable theoretical guarantees the tradeoff between the compression rate and the approximation error, which supports

---

worse case scenarios. In addition. coresets play an important role in improving the generalization properties the trained networks [5 78)
Sensitivity sampling for constructing pruning coresets. To compute such coresets, both 660
utilised the known sensitivity sampling framework p [52] In short the sensitivity of point p P in some query space (P, X, Φ) corresponds to the importance of this point with respect to the query w(p))ffp.z)
space at hand, and it is defined as s(p) supzex Eger w(1)0(q,2) where the denominator is not
equal to zero. Once we bound these sensitivities, we can sample points (neurons) from P according sensitivity bounds, and re-weight the sampled points to obtain coreset. The size of the sample proportional to the sum of these bounds. See Section B.1 and Theorem B.2 for more details in the Appendix.
1.2 Our contribution
Prior coreset methods for pruning DNNs either (i) imposed restrictive assumptions both on the model's weights and inputs [82), i.e., the input set P representing the neurons, and the query set X which represents the inputs of the layer, are enclosed in ball in IR of radius rı and r2, respectively, (ii) the methods data-dependent, i.e., use mini-batch of the input set measure the influence
of each parameter on the loss function S 660
To this end, in this work, we take corescts step further into the realm of pruning by introducing a unified framework with provable guarantees for pruning DNNs (weights and ncurons/filters while minimally affecting the generalization error. Our main improvement is that simultaneously (i) data-independent, (ii) requires a single assumption on the model's 's weights, and (iii) provably guarantees multiplicative factor approximation, which is favourable upon additive approximations; see Theorem 2.6] The approach is based on the widely used theory of coresets allowing
framework is
suggest provable guarantee on the tradeoff between the approximation and compression rate for each layer.
We conducted experimental results which established new SOTA benchmarks for structured pruning via coresets across a wide range of networks and datasets. We share all of our resulted models [I4].
Method
In general, the coreset (for pruning) technique hinges upon the insight that any linear layer such as convolutions, can be casted as a matrix multiplication [82]. Hence, we focus in what follows on fully connected (FC) layers, while the details holds for any linear layer. Furthermore, for simplicity, assume in what follows that the weights of P are all equal to and thus our query space is denote by
<(p.2) (P, (R, o), and the sensitivity of point E P is simply s(p) = suprex Eger ø(g.z) Note that our proofs are easily extended to the general case where we are given a weight function w P as discussed in Section 2.5
2.1 Preliminaries
Notations. For a positive integer n. use [n] denote the set {1, ... n). For Rd and symmet- ric positive definite matrix G E Rdddd define E (G, =
Rd (xr c) G (z c) S 1 to be the ellipsoid defined by and G. For an ellipsoid F. (G, c). each endpoint of semi principal axis is called a vertex of E (G, c). We define rank (P) for any set P C I to be the dimension of the affine subspace that P lies on. For a set C Rd the convex hull of is denoted by Conv (P). Finally, vectors are treated column vectors.
2.2 Novelty Löwner ellipsoid meets Carathéodory
Our method hinges upon a combination two known tools from convex geometry. The novelty of our approach exploits the following observation. Most activation functions & are continuous non-decreasing functions, which indicate that for every query x and a set of points P. the maximal contribution to Epep o(p" 2) with respect to such activation function is associated to point the convex hull of P. By finding geometrical body B of bounded number of vertices, that is (i)
3

---

(i)
(iii)
(iv)
Figure 2: Our novelty in a nut-shell: The very first steps of our technique rely on bounding the ReLu activation function by the fr-regression loss function, e-g.. for ReLU (p" z).. where p (1,0) 0) in this example (shown in (i), we first bound it by the fi-regression loss function (shown in (ii)) using Definition 2.3 Following this step, a set of points can be projected into a low dimensional subspace of dimension rank (P) using any dimensionality reduction algorithm as presented in (iii), resulting in the set P.. (iv) The convex hull (blue dashed lines) /' (red points) is enclosed by its Löwner ellipsoid (depicted in green). (v) Finally, for each vertex (magenta star) of the enclosed ellipsoid (black ellipsoid), its Carathédory set is found (red points connected by cyan dashed lines).
enclosed in Conv (P) and (ii) with some dilation factor (expanding) enclose Conv (P), we will be able represent each point on the boundary of the convex hull of / as a convex combination of two points P1, P2, one each (p) ) on and the second (p2) on its dilated form, which is formalized as the set fa(x c) + E B, Va E [0, 1]], where here denotes the center of B. For such task, Löwner ellipsoid is leveraged.
Theorem 2.1 (John-Löwner ellipsoid[42).. Let L S. IR be a set of points such that the convex hull of L has nonempty interior. Then, there exists ellipsoid E (G,c) (also known the MVEE), where G E Rdxdd is positive definite matrix and c E R. of minimal volume such that (G, - e) + c C. Conv (I.) S E (G, If L symmetric around the center e, then the dilation factor can be reduced to
Va
Algorithm 1: Lx0 -CORESET (P, (m.
Algorithm 2: CORESET (P,
Input :P C Rd of n points with rank
input :A set PC Rd of n points, and a sample size
Output A subset S C P that satisfies Lemma(2.5 (Y, affine subspace that P lies on, i.. _ {=YYYT + 2 E Rd)
output: A weighted set (C,u)
Q P, i C 0
2 while |Q| 2 2rank (Q) do S = lox CG--COESSETCC (Q)
2 P = [p' := pr
Let map: P' + P a function that maps
for every p E S do
every p/ P" to its corresponding point
s(p) Qr)
PEP
end
S = 0; k 0
Q= Q -
F. (G, == MVEE (Conv (P'))
8 end
V == the vertices of ellipsoid
for every p Q do
r (E, c) c
s(p) = 2rank(Q)
10
for every u V do
11 end
K =
I Epep&(p (p)
KUCARATHEODORY-SETT (u, PP) // See Algorithm 4 in the supplementary material.
13 C an i.i.d sample of m points from P,
where each E P is sampled with probability *(p)
end
10 = {map (4)la E K)
u(p) m-e(p) for every p E C
return S
15 return (C,u)
Afterwards, P1 and Pa should be represented by points from P. Each point on B (specifically, P1) can be represented via a convex combination of + points from P. The same holds for points on the dilated form of B (e.g., P2) but via a conical combination (linear combination where the weights are non-negative and the sum of weights not necessarily 1). This problem solved by invoking Carathéodory theorem.

---

Theorem 2.2 (11O) 951). For any A C. Rd and E Conv(A), there exists m S d + points PI..... Pm E A (denoted by Carathéodory set of p) such that p E Conv ((PI,..,,Pml
Finally, it is known that some functions, including the ReLU function, do not admit an e-coreset of size o(n) 82 8T. Thus, we use a generalized form of what is known as the complexity measure of a set of points, which was first introduced in [81] and later leveraged in [76). This measure is used to determine the complexity of a given set P with respect to ReLU, and the coreset size theoretically.
Definition 2.3 (Regression Complexity Measure), Let C Rd x (1), the regression complexity (ue 1 [=x "|
measure of P is defined as y (P) supze Ege Eye(per],T.sop "| where the denominator is 2
and the last entry of every p E P 1, reserved for the bias/intercept term.
2.3 Our Pruning Scheme
In what follows, we present our data summarization technique for ReLU on the dot product function.
Then in Section 2.5] we discuss that our results can be easily extended to a wide family of activation functions including the Sigmoid function, as recently shown in (76). First we present Algorithm which serves as a stepping stone towards bounding the sensitivities.
Overview of Algorithm | The algorithm receives as input a set P C Rdd whose rank is r E [d] and deterministically finds a subset S C P which satisfies that for every j E [d], X E RRdx and u E Rd. maxper [[pp=)xx]],,
max maxpes|[o-ooxx [ < ,,15 To do so, first, find the affine hyperplane that P lies on, followed by computing the low dimensional representation of P. denoted by P"; see Lines 2 Note that if PP) d, then we can either keep P as it is (i.e., P' : P). or use dimensionality reduction tricks as detailed in Section 2.5 To compute the output S, we first bound the convex hull of P" by its Löwner ellipsoid E c) in Line 5 followed by computing the dilated ellipsoid of E c). namely, (E - c) + Let V be the set of vertices of such ellipsoid: see Line 6 Now, for each point v V. we represent it as a convex combination of r + 1 points from P' via Theorem 2.2 and store the union of such sets (each of size most + 1) of points into K done in Lines 7 For each point in K, we map it back to Rd to satisfy Lemma 2.5 To sum up Algorithm we observe that the vertices can be used via canonical combinations with their dilated form to describe every point on the convex hull of the input data (in our end, it would the network's 's weights). Hence the Carathéodory set of these vertices from the input points lying on the convex hull can be further used also represent points lying on the convex hull. This is the core idea which enable us in forming our for coreset for any Ep regression problem where p (0, oc).
We now discuss Algorithm 2 which is responsible for constructing an E-coreset with respect to activation functions. Its input set P C Rdd and sample size m 1.
Overview of Algorithm 2. First set Q == P. At each iteration i, the algorithm obtains subset S. C Q as output to a call to f COREESE(()) as stated in Line 3 of Algorithm [ such that for
every x E Rdd holds that maxpep p
maxpes, 1p" S 1.5 with r being the rank of Q. The sensitivity of each
point in S: is bounded from above by 2d as stated in Lines 4 6 The idea behind these bounds lies in our proof of Theoreh [2.6] The set S; removed from Q, and this procedure is repeated with respect to Q until the size of Q is small enough. The obtained sensitivities are the ones needed for computing the pruning coresets. Finally, we utilize the sensitivity sampling framework of [9] to obtain the desired coreset; see Lines 13
2.4 Analysis
In this section, we prove the correctness of our algorithms. The following lemma shows that for each point p that is inside some convex hull S, its l distance to any affine subspace is always bounded from above by e distance from the same affine subspace, of some other point g E S.
Lemma 2.4. Let d,e, m 2 be integers. Let p E IR and A S Rdd be a set of m points with P E Conv(A) so that there exists a A > [0, 1] such that Ene A a(g) = 1 and Ene a(g) .
Then for every Y E ddxxt and E TR. 1|(p - e) S max [1(q - v) Yl12-
The following states the provable guarantees of Algorithm
5

---

Lemma 2.5 (l -ooreset for & rregressioo)) Let P S Rd be a set of points, and r be the rank of P.
Let j E [d 1] and let S be the output of a call to lac CCORESETT))).. Then (i) [S] E O (r²), and
(ii) for every X E [dxx and E Rd, maxxe>llla--ae x
maxges (q o) 21.2r1.).
The following theorem states our main result.
Theorem 2.6 (ReLU e-coreset). Lel P S RR, E, E (0,1), r rank (P), 1 (P) be as in Def- inition 2.3 and let m E EO M(P)rs logn (log (u (P) rlg n)) + log (t)) Let (C, w) be the
5² output of a call to CORESET( P. m): see Algorithm 2
Then, with probability at least 1 - 8, (C, w) is
an E-coreset for (P. Rd, ReI.U (-)); see Definition L.1
Time analysis. Letting r be the rank of P. the time complexity of Algorithm 1 can be dissected to two main parts: (i) Computing the Löwner ellipsoid in Onr logn) time using the method
proposed in [98] and (ii) computing the Carathédory set in O(nr + r* n) time via [70]. Since V can contain up Orr²) points, the overall time for Algorithm D is (nr log n + nr + r log n). As for Algorithm| 2 it takes O(n2,² log + nr n). Indeed, as explained in Section 2.5
dimensionality reduction algorithm may applied to improve the run time (reducing the rd factor). Furthermore, the run time of our algorithm can be improved, using the merge-and-reduce tree from the literature of coresets to reduce the 2 terms to n log n, i.c., the running time can be reduced to On log" (n)r²² + nr*). ). For a data- aate-independent provable method, this running time is reasonable.
Our advantages over previous results. Our coreset supports different activation functions without the need to change the sensitivity that much. Specifically, it will only be multiplied by some scalar, unlike previous coresets where different losses impose drastically different sensitivitiesslevrrage scores and algorithms. This since our coreset unlike other coresets is in its essence a framework of coresets for different to losses, as it can be used as is for different E, losses and yet still attain e-approximation. In addition, when the rank of the input points small, then our method outperforms previous methods. If the input data is of full rank, previous methods obtain faster coresets construction.
On the boundness of the regression complexity measure. First of all, there exists an example where the complexity measure is unbounded, e.g., consider a set of points distributed evenly on unit ball. In this case, you can always find point where a hyperplane separating it from the rest of the points can be found such that the one half-space of this hyperplane contains only this point while the other half-space contains the rest of the points. This leads to infinite complexity measure. Such an example is also mentioned in [82), when assessing the hardness of generating multipicative-aproximmtioo coresets for ReLU functions.
Theoretically, the complexity measure is influenced by how free can the bias term be (the last entry of x); see Definition 2.3 This term is the only thing that ensure that one point can be separated from the rest in the sense of finding a separating hyperplane, leading to an infinite complexity measure. Bounding this term, leads to bounded complexity measure from a theoretical point of view.
In the context of model pruning, from the perspective of the complexity measure, the model's 's weights are the input denoted by a matrix P E [g=((d+1) while the query is now gd x {1}. Thus the complexity measure is now u (P) suppERdxp) Eve(pepioT |9"=
(pEel:/TT 1v With this in mind,
observe that the complexity measure is now an instance of the complexity measure used in (76). The complexity measure now relies entirely on the structure of the model's weights, where the goal is to find the largest ratio between the sum of the absolute of the values inside the rectified neurons prior to applying the rectification, and the sum values of non-rectified neurons. To bound this measure, we can use variant of the algorithm described in the proof of Theorem 3 in 81
2.5 Extensions
Our suggested scheme can be extended to support many other variants of the pruming problem.
Various activation functions. Our result can be extended to family of activation functions called "Nice hinge functions"; see Definition D.I. Let (P, X, σ) be query space, where ø is a "Nice hinge functions"". To bound the sensitivity of a point p, we first bound the nominator of s(p) by proving that Vo E X p(p" 2) < lp" zz For bounding the denominator from below, recently (66 proved
that Vz E X Epep ReLU (p" z) E Epep o(p"2) 2); see full detail in Section D.3.

---

Weighted Input. In the context of deep learning, the output of each neuron is multiplied with scalar which brings the necessity of having the ability to deal with weighted set of points. Algorithm 2 can be extended easily to the case as generously detailed in Section D.1 in the Appendix.
Dimensionality reduction. All coreset-based pruning methods rely heavily on the dimensionality of the model's s layers, as well as our method. To ensure sufficient pruning ratio, we apply either PCA, TSNE, MDS, or the JL transform on the weights of each layer prior generating its coreset.
From weight to neuron pruning. Most coreset-based pruning methods, e.g., [ 12], first provide a scheme for (provable) weight pruning, which is then used stepping stone towards pruning neurons as follows. [5] first suggested coreset-based neuron pruning via the use of a generated controled set of queries to evaluate the importance of weights. Any neuron that has a maximal activation value lower or equal to zero, will be pruned from the network as its impact on the rest of the neurons is minimal. On the other hand (22 altered the definition of sensitivity such that it takes into account the sensitivity of a neuron in a layer & with respect to all the neurons in the layer e + 1, which basically means that the sensitivity of each neuron is taken be the maximal sensitivity over every weight function (neuron in the next layer) defined by the layer. Hence, we follow the same logic for such method; see Section D.2 in the supplementary material.
From neuron to filter pruning. Convolutions can be expressed as matrix multiplications, which enables our method to prune filters from any model as done by [60] [2 61]
3 Experimental Results
In this section, we study various widely used network architectures and benchmark data-sets. Fol- lowing [82], to test the robustness of our methods on each of the neuron and filter pruning tasks independently, two sets of experiments are conducted. The first focuses on pruning neurons (Sec- tion 3.Ip whereas the second focuses on pruning filters (Section 3.2) both via our coreset method.
The setting. In all experiments we report the Pruning ratio - the percentage of the parameters that were removed from the original mode. Here, PR stands for pruning ratio, FR stands for soating-point reduction ratio and Err the percentage of misclassified test instances of our method compared to coreset-based pruning methods and more. Baseline Err is the error of the original uncompressed network, while Pruned Err is the classification error of the compressed model. In our experiment compress and fine-tune the network once, iterative pruning was applied, thus, the compared methods also satisfy this setting. Each experiment was conducted 5 times, in the tables, we report for our method the best error achieved and we highlight in parentheses next it the average error and standard deviation across the 5 trails. In all of our experiments, the models are fine-tuned till convergence (after pruning). Implementation details are given in Section E in the Appendix.
Software/Hardware. Our algorithms were implemented in Python 3.6 [008] using Numpy (83) and Pytorch (84). Tests were performed on NVIDIA DGX A100 servers with 8 NVIDIA A100 GPUs each, fast InfiniBand interconnect and supporting infrastructure.
Baselines. Our results are compared to (i) PFP 160 (ii) FT 588 (iii) SoftNet 34 (iv) ThiNet 668
and (v) PvC [82], (vi) Soft Pruning [34, (vii) CCP [85]
(viii) FPGM [36). (ix) ThiNet-70, (x)
ThiNet-50 688 (xi) Pruning via Coresets (PvC) [82] (xii) Pruning from Scratch (PIS) [I1, and
(xiii) Rethinking the value network pruning (Rethink) 65
3.1 Neuron Pruning
We test our method on VGGI6 [92] using CIFARIO 49], and LeNe1300-100 using MNIST [55]
Diseussion. Table present the results of LeNet300-100 and VGG16. Observe that in both architec- tures, our method outperformed the competing methods under the same compression scenarios. For example, we pruned roughly 90% of the parameters of the LeNet-300- the accuracy of the original model. We witness a similar phenomena on the VGG16 16 model, where we pruned roughly 90% of the parameters of the dense layers resulting accuracy improvement. This confirms the insights in [5] that coresets help in improving the generalization properties of DNNs.
model while improving
7

---

Table 1: Pruning of LeNet-300-100 architecture on the MNIST dataset and of VGG-16 on the CIFAR-10 dataset. Here, we report the compression with respect to the fully connected layers only.
Model
Method PFP FT
Baseline Err. (%)
Pruned Err. (%)
PR (%)
1.59
2.00
84.32
1.59
1.94
81.68
LeNet-300- 100
SoftNet
1.59
2.00
81.69
ThiNet
1.59
12.17
75.01
PvC
2.16
2.03
90
Our method (90)
2.07
1.98 (2.02 + 0.04)
90
Our method (92.6)
2.07
2.64 (2.74 0.1)
92.6
Our method (94.6)
2.07
3.51 (3.58 + 0.07)
94.6
PvC
8.95
8.16
75
VGG-16
Our method
5.9
5.9 (6.2 + 0.3)
90
3.2 Filter pruning
We compressed the convolutional layers of (i) ResNet50 33 on ILSVRC-2012
18]
(ii)
ResNet56 33 (iii) VGG19 92 on CIFAR10 and (iv) VGG16 92 on CIFARIO.
Table Filter pruning results on different neural networks with respect the CIFARIO dataset.
Model
Method
Bascline ErT. (%)
Pruncd Err. (%)
PR (%)
FR (%)
PIS
6.4
6.29
52
NA
Rethink
6.5
6.22
80
VGG-19
Structured Pruning
6.33
6.20
88
NA
PvC
6.33
6.02
88
NA
Our method (88)
6.33
5.85
(6.03 = 0.18)
88
NA
Our method (91.28)
6.33
6.23
(6.35
0.12)
91.28
ThiNet
7.11
9.21
63.95
64.02
FT
7.11
8.22
80.09
80.14
VGG-16
SoftNet
7.11
7.92
63.95
63.91
PFP (94)
7.11
7.61
94.32
85.03
PFP (87)
7.11
7.17
87.06
70.32
PFP (80)
7.11
7.06
80.02
59.21
Our method (95.32)
7.11
7.31
(7.55 +: 0.24)
95.32
85.09
Our method (87)
7.11
(6.76
87
68.2
Our method (79.53)
7.11
6.3 (6.38 0.08)
79.53
59 14
ThiNet
7.05
8.433
49.23
49.74
Channel Pruning
7.2
8.2
N/A
50
AMC
7.2
8.1
64.78
ResNet56
CCP
6.5
6.42
57
52.6
PvC
6.21
7.0
55
N/A
Our method
6.61
7.26 (7.56 1. 0.3)
63.95
50
Table 3: Filter pruning ResNet50 ImageNet (ILSVRC-2012).
Method
Baseline Err. (%)
Pruned Eir. (%)
PR (%)
FR (%)
PFP
23.87
24.79
44.04
30.05
Soft Pruning 23.85
25.39
49.35
41.80
CCP
23.85
24.5
56
48.8
FPGM
23.85
25.17
53.5
ThiNet-70
27.72
26.97
33.72
36.78
ThiNet-50
27.72
28.0
51.5
55.82
PvC
23.78
25.11
N/A
Our method
23.78
24.87
(25.07 0.2)
62
61.5
Our method (18.01)
23.78
24.1
(24.2 = 0.1)
18.01
10.82

---

Filter pruning of DNNs on CIFARIO and ImageNet (ILSVRC-2012), For Cifar10, we used PyTorch implementations of VGG19 and VGG16. We compressed both models using our approach by different compression rates as shown in Table 2 with a comparison to other methods. For ImageNet, we compressed the baseline model of ResNet50 X roughly by 62% in terms of number of parameters. Table 3 provides comparison between our method and other baselines.
Discussion. As can be seen in Tables
and our method either outperforms the competing methods or achieves comparable results. As for the coreset methods, our algorithm achieves better result than all of them this setting, e.g., we compressed 62% of ResNet50 trained on ImageNet while incurring 1.09% drop in accuracy, improving the recent coreset result of PvC [82] for the same compression ratio, while PFP [60] compressed 44.04% achieve comparable results.
Related work
DNNs can be compressed before training 997
110 57], during training [188 115 51], or after
training [93]. Furthermore, such procedures may also be repeated iteratively [88]. As previously noted pruning can be categorized into structured and unstructured pruning.
Unstructured pruning. Weight pruning (56) techniques aim reduce the number of weights in a layer while approximately preserving its output. Approaches of this type include the works I (2], where the desired sparsity embedded as constraint or via regularizer into the training pipeline, and those of [31
of [54] 21 39
30) where weights with absolute values below a threshold use mini-batch of data points to approximate the influence of
88
removed. The approaches of (5 6
each parameter on the loss function. Other data-informed techniques include (28) 63 80, 77 [16]. A thorough overview of recent pruning approaches is given by [27 8. However, unlike our approach. weight-based pruning approaches generate sparse models instead of smaller ones thus requiring
specialized hardware and sparse linear algebra libraries in order speed up inference.
Structured pruning, Pruning entire neurons and filters directly shrinks the network leading to smaller storage requirements and improved inference-ting performance any hardware (59] 67.. Lately, these approaches were investigated in many papers (64, [59 11 36 22 46 114 113 Usually, filters
are pruned by assigning an importance score each neuron/filter, either solely weight-based [37 35
or data-informed 116,, 66), and removing those with a score below a threshold. The procedure can be embedded into iterative pruning scheme (88) that requires potentially expensive retrain cycles.
Tensor decomposition. Some of the work in DNN compression entails decomposing the layer 4]
into multiple smaller ones, c.g., via low-rank tensor decomposition [99
74
48
96 40 105
117 53 61. Other approaches to tensor decomposition include weight sharing, random projections,
and feature hashing [112 3 91 12 B 107). However, such techniques usually require expensive approximation algorithms or use heuristics since tensor decomposition is generally NP-hard.
Coresets. In the recent years, coresets got increasing attention, and where leveraged to compress the input datasets many machine learning algorithmss, improving there performance, e.g., regression [72
38 81
47
103, decision trees 44 matrix approximation [26 70
25 89 [73], data discretization
75
clustering 24
29
66
4 45
106], Eg --rreression [16 17 94], SVM [32 101 99 100 [02]. deep
learning models [69 5 60 and even for path planning in the field of robotics [104.. For extensive
surveys on coresets, we refer the reader to [2 86 43 7
5 Conclusions and Future Work
In this paper, we provided a coreset-based pruning technique that hinges upon combination of tools from convex geometry, while achieving SOTA results with respect to coresct- structured pruning approaches on variety of networks. Our main improvement is that our coreset is (training) data-independent and assumes a single assumption on the models weights.
Future work includes (i) suggesting a coreset based budget allocation framework, to determine the (optimal) per layer prune ratio while achieving an overall desired compression rate, (ii) extending our coreset technique to other layers such as attention layers, and (iii) bridging the gap between coreset based pruning approaches and censor-decompositioo methods, as both techniques are theoretically supported by bounding the approximation error given specific compression rate, we can leverage these

---

bounds to formulate the compression problem as an optimization problem which iterates between the two approaches search for the local minimum.
References
[1] Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg. Net-trim: Convex pruning of deep neural networks with performance guarantee. In Advances in Neural Information Processing Systems, pages 3180-3189, 2017.
[2] Jose M Alvarez and Mathieu Salzmann. Compression-awee caaare training of deep networks. In Advances in Neural Information Processing Systems, pages 356-867, 2017.
[3] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In International Conference on Machine Learning, pages 254-263, 2018.
[4] Olivier Bachem, Mario Lucic, and Silvio Lattanzi. One-shot coresets: The case of k-clustering. In International conference on artificial intelligence and statistics, pages 784-792. PMLR, 2018.
[5] Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski. Dan Feldman, and Daniela Rus. Data- dependent coresets for compressing neural networks with applications generalization bounds. In International Conference on Learning Representations, 2019.
[6] Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Sipping neural networks: consitivity-informmn provable pruning of neural networks. arXiv preprint arXiv:1910.05422,222 2019.
Gantavya Bhatt and Jeff Bilmes. Tighter m-DPP Coreset Sample Complexity Bounds. In ICML 202/ Workshop: SubSerMI:: Subset Selection in Machine Learning: From Theory to Practice, Virtual, July 2021.
[8] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state neural network pruning? In Proceedings of Machine Learning and Systems 2020, pages 129-146, 2020.
[9] Vladimir Braverman, Dan Feldman, and Harry Lang. New frameworks for offline and streaming coreset constructions. arXiv preprint arXiv:1612.000 2016.
[10] Constantin Carathéodory. Über den variabilitätsbereichh der koeffizienten von potenzreihen, die gegebene werte nicht annehmen. Mathematische Annalen, 64(1):95-115, 1907.
[1I] Jianda Chen, Shangyu Chen, and Sinno Jialin Pan. Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
[12] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pages 2285-2294, 2015.
[13] Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen. Com- pressing convolutional neural networks. CoRR, hbb//1506.04449) 2015.
[14] Code. All resulted pruned models presented in this paper, 2022, The authors commit to publish upon acceptance of this paper or reviewer request.
[15] Michael Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM). 68(1):1-33 2021.
[16] Michael B Cohen and Richard Peng. Lp row sampling by lewis weights. In Proceedings of the sevett annual ACM symposium Theory of computing, pages 183-192, 2015.
10

---

[17] Anirban Dasgupta, Petros Drineas, Boulos Harb, Ravi Kumar, and Michael W Mahoney. Sam- pling algorithms and coresets for 'ell p regression. SIAM Journal on Computing, 38(5):0060- 2078, 2009.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[19] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural information processing systems, pages 1269-1277 1277, 2014.
[20] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[21] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer- wise optimal brain surgeon. In Advances in Neural Information Processing Systems, pages 4860 2017.
[22] Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: more complicated network with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5840-5848, 2017.
[23] Dan Feldman. Core-sets: An updated survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, http:://anivv.rr/a///00100
10(1):e1335, 2020.
[24] Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. In Advances in neural information processing systems, pages 2142-21525 2011.
[25] Dan Feldman, Morteza Monemizadeh, Christian Sohler, and David P Woodruff. Coresets and sketches for high dimensional subspace approximation problems. In Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms, pages 630-649. Society for Industrial and Applied Mathematics, 2010.
[26] Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant- size coresets for k-means, pca and projective clustering. In Proceedings of the Eveety-Fourt Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1434-1453 1453-445. SIAM, 2013.
[27] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 2019.
[28] Noah Gamboa, Kais Kudrolli, Anand Dhoot, and Ardavan Pedram. Campfire: Compress- ible, regularization-fres structured sparse training for hardware accelerators. arXiv preprint arXiv:2001.0333 002530 2020.
[29] Lei Gu. A coreset-based semi-supverised clustering using one-class support vector machines. In Control Engineering and Communication Technology (ICCECT), 2012 International Con- ference on, pages 52-55. IEEE, 2012.
[30] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances In Neural Information Processing Systems, pages 1379-1387, 2016.
[31] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149,, 2015.
[32] Sariel Har-Peled, Dan Roth, and Dav Zimak. Maximum margin coresets for active and noise tolerant learning. In IJCAI, pages 836-84444 2007.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 770-778, 2016.
11

---

[34] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks.
[35] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2234-2240, AAAI Press, 2018
[36] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4340-4349, 2019.
[37] Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[38] Jonathan Huggins, Trevor Camphell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances in Neural Information Processing Systems, 29:4080-400 4088, 2016.
[39] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint 1602v
2016.
[40] Yani Ioannou, Dunean Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Crimin- isi. Training cnns with low-rank filters for efficient image classification. arXiv preprint arXiv:1511.06744 2015.
[41] Max Jaderberg. Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
[42] Fritz John. Extremum problems with inequalities as subsidiary conditions. In Traces and emergence of nonlinear programming, pages 197-215, Springer, 2014.
[43] Ibrahim Jubran, Alaa Maalouf, and Dan Feldman. Introduction to coresets: Accurate coresets. arXiv preprint arXiv:1910008707, 2019.
[44] Ibrahim Jubran, Ernesto Evgeniy Sanches Shayda, Ilan Newman, and Dan Feldman, Coresets for decision trees of signals. Advances in Neural Information Processing Systems, 34, 2021.
[45] Ibrahim Jubran, Murad Tukan, Alaa Maalouf, and Dan Feldman. Sets clustering. In Interna- tional Conference Machine Learning, pages 5005. PMLR, 2020.
[46] Minsoo Kang and Bohyung Han. Operation- aware soft channel pruning using differentiable PMLR, 2020.
masks. In International Conference Machine Learning, pages
[47] Zohar Karnin and Edo Liberty. Discrepancy, coresets, and sketches in machine learning. In Conference on Learning Theory, pages 1975-1993. PMLR, 2019.
[48] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530 2015.
[49] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 105, 2012.
[51] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. arXiv preprint 2r11v:2002.03231, 2020.
12

---

[52] Michael Langberg and Leonard J Schulman. Universal E e-approximators for integrals. In Proceedings of the twenty-firs. annual ACM-SIAM symposium on Discrete Algorithms, pages 698-607.200 SIAM, 2010.
[53] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In ICLR (Poster), 2015.
[54] Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, pages 2554- 2564. IEEE, 2016.
[55] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278-233) 2324,, 1998.
[56] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598 0888655,11 1990.
[57] Namhoon Lee, Thalaiyasingam Ajamthan, and Philip Torr. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference Learning Representations, 2019.
[58] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 08710, 2016
[59] Yawei Li, Shuhang Gu, Lue Van Gool, and Radu Timofte. Learning filter basis for convolu- tional neural network compression. In Proceedings of the IEEE International Conference Computer Vision, pages 5623-5632, 2019.
[60] Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. International Conference on Learning Representations, 2020.
[61] Lucas Liehenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. Compressing neural networks: Towards determining the optimal layer-wise decomposition. Advances in Neural Information Processing Systems, 34:5328-5344, 2021.
[62] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Advances in Neural Information Processing Systems, pages 2178-2188, 2017.
[63] Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
[64] Zechun Liu, Haoyuan Mu. Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang- Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 3296-3305, 2019.
[65] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
[66] Mario Lucic, Olivier Bachem, and Andreas Krause. Strong coresets for hard and soft bregman clustering with applications to exponential family mixtures. Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 1-9, Cadiz, Spain, 09-11 May 2016. PMLR
[67] Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. arXiv preprins arXiv: 1005: 1805.08941, 2018.
[68] Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. Proceedings of the IEEE international conference on computer vision, pages 5058-5066 2017.
13

---

[69] Alaa Maalouf, Gilad Eini, Ben Mussay. Dan Feldman, and Margarita Osadchy. A unified approach to coreset learning. arXiv preprint arXiv:2111.03044, 2021.
[70] Alaa Maalouf, Ibrahim Jubran, and Dan Feldman. Fast and accurate least-mean-Be - solvers. In Proceedings the 33rd International Conference on Neural Information P Processing Systems, pages 2307-8318, 2019.
[71] Alaa Maalouf, Ibrahim Jubran, and Dan Feldman. Introduction to coresets: Approximated mean. arXiv preprint
2021.
[72] Alaa Maalouf, Ibrahim Jubran, and Danny Feldman. Fast and accurate least-mean squcaes solvers for high dimensional data. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
[73] Alaa Maalouf, Ibrahim Jubran, Murad Tukan, and Dan Feldman. Coresets for the average case error for finite query sets. Sensors, 21 (191)6666,, 2021.
[74] Alaa Maalouf, Harry Lang, Daniela Rus, and Dan Feldman. Deep learning meets projective clustering. In International Conference on Learning Representations, 2020.
[75] Alaa Maalouf, Murad Tukan, Eric Price, Daniel M Kane, and Dan Feldman. Coresets for data discretization and sine wave fitting. International Conference on Artificial Intelligence and Statistics, pages 10622-10639. PMLR, 2022.
[76] Tung Mai, Anup B Rao, and Cameron Musco. Coresets for classification simplified and strengthened. arXiv preprint arXiv: 106. 2021.
[77] Zelda Mariet and Suvrit Sra. Diversity networks: Neural network compression using determi- nantal point processes. arXiv preprint
2015.
[78] Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
[79] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1 11264-11 11272, 2019.
[80] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convo- lutional neural networks for resource efficient inference. arXiv preprint arXiv: 1611 1.06440, 2016.
[81] Alexander Munteanu, Chris Schwiegelshohn, Christian Sohler, and David P Woodruff. On coresets for logistic regression. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6562-6571, 2018
[82] Ben Mussay, Dan Feldman, Samson Zhou, Vladimir Braverman, and Margarita Osadchy. Data-independent structured pruning of neural networks via coresets. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[83] Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
[84] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 0088802668077,,, 2019.
[85] Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In International Conference on Machine Learning, pages 5113-5122. PMLR, 2019.
[86] Jeff M Phillips. Coresets and sketches. arXiv preprint arXiv:1601.0061 2016.
[87] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9 2019.
14

---

[88] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing finettning and rewinding in neural network pruning. International Conference on Learning Representations, 2020.
[89] Tamas Sarlos. Improved approximation algorithms for large matrices via random projections. In 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS(06), pages 143-152. IEEE, 2006.
[90] Melanie Schmidt, Chris Schwiegelshohn, and Christian Sohler. Fair coresets and stream- ing algorithms for fair k-means. In International Workshop on Approximation and Online Algorithms, pages 232-251. Springer, 2019.
[91] Qinfeng Shi, James Petterson, Gideon Dror, John Langford, Alex Smola, and SVN Vish- wanathan, Hash kernels for structured data. Journal of Machine Learning Research, 10NNov):211--2637 2009.
[92] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1553 1556, 2014.
[93] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. arXiv preprint arXiv:2004. 14340, 2020.
[94] Christian Sohler and David P Woodruff. Subspace embeddings for the 11 no-m with appli- cations. In Proceedings of the forty-third annual ACM symposium Theory of computing. pages 755-764, 2011.
[95] Ernst Steinitz. Bedingt konvergente reihen und konvexe systeme. Journal für die reine und angewandte Mathematik (Crelles Journal), (013(143::128-111 1913.
[96] Cheng Tai, Tong Xiao, Zhang, Xiaogang Wang. et al. Convolutional neural networks with low-rank regularization. arXiv preprini arXiv: 15/1 1.06067, 2015.
[97] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neu- ral networks without any data by iteratively conserving synaptic flow. arXiv preprins arXiv:2006.0544 2020.
[98] Michael Todd and E Alper Yildırm. khachiyan's algorithm for the computation of minimum-volume enclosing ellipsoids. Discrete Applied Mathematics, 155(13):1731-1744 2007.
[99] Ivor W Tsang, James T Kwok, and Pak-Ming Cheung. Core vector machines: Fast svm training very large data sets. Journal of Machine Learning Research, 63333)3333-39, 2005.
[100] Ivor W Tsang, James Tin- Yau Kwok, and Pak-Ming Cheung. Very large svm training using core vector machines. AISTATS, 2005.
[101] IW-H Tsang, JT-Y Kwok, and Jacek M Zurada. Generalized core vector machines. IEEE Transactions on Neural Networks, 17(5): 126:)1122-1140, 2006.
[102] Murad Tukan, Cenk Baykal, Dan Feldman, and Daniela Rus. On coresets for support vector machines. Theoretical Computer Science, 2021.
[103] Murad Tukan, Alaa Maalouf, and Dan Feldman. Coresets for near-convex functions. Advances in Neural Information Processing Systems, 33:997-1009, 2020.
[104] Murad Tukan, Alaa Maalouf, Dan Feldman, and Roi Poranne. Obstacle aware sampling for path planning. arXiv preprint arXiv:2203.04075. 2022.
[105] Murad Tukan, Alaa Maalouf, Matan Weksler, and Dan Feldman. No fine-tuning, no cry: Robust svd for compressing deep networks. Sensors, 21(16)55599, 2021.
[106] Murad Tukan, Xuan Wu, Samson Zhou, Vladimir Braverman, and Dan Feldman. New coresets for projective clustering and applications. In International Conference on Artificial Intelligence and Statistics, pages 5391-5415. PMLR, 2022.
15

---

[107] Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprint arXiv 04008, 2017.
[108] Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009.
[109] Kasturi Varadarajan and Xin Xiao. A near-linear algorithm for projective clustering inte- ger points. In Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms, pages 1329-1342. SIAM, 2012.
[110] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference Learning Representations, 2020.
[111] Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, and Xiaolin Hu. Pruning from scratch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1223-122288 2020.
[112] Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg, Fea- ture hashing for large scale multitask learning. In Proceedings of the 26th annual international conference on machine learning, pages 1113-1120, 2009.
[113] Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller-norm-less intesss-ifommaivee assumption in channel pruming of convolution layers. In International Conference Learning Representations, 2018
[114] Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. International Conference on Machine Learning, pages 10820-10830. PMLR, 2020.
[115] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv:1812.0892. 2018.
[116] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pages 9194-9203. 2018
[117] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 7370-7379, 2017.
[118] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878. 2017.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 2
(b) Did you include complete proofs of all theoretical results? [Yes] See Sections 2.4 and C
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
16

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasig new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "Yzf1iJsQ1vk", "reviewer": "Reviewer_Uwaw", "summary": "This paper proposes a method for pruning neural network weights at each layer. This improves on previous algorithms for activation function coresets by attaining tight sensitivity bounds with no data assumptions and mild assumptions on the network weights. The coreset construction relies on combining two key ideas from convex geometry: specifically finding the Caratheodory set of each vertex of a John-Löwner ellipsoid. Experiments show that the proposed approach outperforms several recent baselines, pruning more weights while achieving lower classification error.", "questions": "- Why different baselines used for different tables/models/experiments?\n- Is Theorem 2.8 directly comparable with previous work?\n- Do you have any intuition/empirical results on what dimensionality reduction method works best in this setting?\n- Can this method be applied to attention-based networks such as transformers?\n", "limitations": "This paper describes limitations as opportunities for future work, but does not address any potential negative social impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Originality/Significance/Quality\n- Builds on previous work on network pruning in a novel way via tools from convex geometry\n- Cites related work where appropriate\n- Elegant idea that with intermediate results that may be of independent interest\n- Experimental results look impressive, although different baselines are used in different experiments\n\n## Clarity\n- The paper is generally a pleasure to read, but I have a couple of suggestions:\n    - The weights assumption (Regression Complexity Measure) is a key contribution of the paper, and the presentation would be improved it if was mentioned earlier, perhaps with an informal statement and/or background motivation in Section 1\n    - The John ellipsoid is defined with a dilation of $d$ (ambient dimension), but is used in Lemma 2.7 with a dilation of $r$ (rank).\n- Typos\n\t- Line 140: \"with respect ReLU\" should be \"with respect to ReLU\"\n\t- Line 148: \"at [74]\" should be \"in [74]\"\n\t- References [5-7] are identical\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P0SOXo9UNB", "reviewer": "Reviewer_B4JF", "summary": "This paper aims to advance coreset-based neural network pruning by relaxing the assumptions made in the previous works (especially Mussay et al.). This work claims to provide data-independent sample complexity bounds for layer pruning for multiplicative coresets including for ReLU-like activations which were considered to be non-approximable with fewer than O(n) samples. Novelty aspects come by providing a different upper bound for sensitivity, to be used for importance sampling. This upper bound on sensitivity uses certain conditions satisfied in Algorithm 1 which uses the Carathéodory theorem applied to the vertices of maximal volume ellipsoid contained within the convex hull of weights in a layer. Having mentioned these theoretical results, the paper empirically shows the efficacy of their algorithm on different architectures (LeNet, VGG, and ResNet) on different datasets (MNIST, CIFAR-10, and ImageNet-1K). ", "questions": "As mentioned in the main review.", "limitations": "As mentioned in the main review.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The theory seems to be reasonably sound and novel (usage of Löwner-John ellipsoid with Carathéodory theorem). \n- The Paper is decently written albeit I’ve some questions (described later)\n-  Empirical experimentation involves a variety of networks and datasets and baselines. \n\nWeaknesses:\n\nWeaknesses of this work are mostly my questions and some suggestions which call for small experimentation for coreset guarantee. I describe them as follows:\n\n- Since the work follows the same notations as in Mussay et al. can they also include a neural network diagram so that it would be easier to follow what is mentioned in line 77? \n- Line 699 claim 1: why does the ellipsoid have at most 2r vertices? Shouldn't that be exactly 2r? \n- What is special about the vertices of the ellipsoid that is used to get the Carathéodory set?\n- The upper bound on sensitivity (and hence on the normalizing constant t) is in terms of regression complexity measure $\\mu(P)$. Is there any bound on this term, or this can get arbitrarily large (somewhere as large as the o(n)?).\n- I am unable to comprehend why following this procedure would lead to the possibility of multiplicative coresets. The proof given by Mussay et al. (Appendix A.1) describes a possibility of the case that shows that for ReLU like activation functions sample complexity is O(n). Why should this procedure not fall into the case mentioned by Mussay et al.?\n- In Definition 2.5, it may be better to use $x \\in \\mathbb{R}^{d} \\times {1}$ and $P \\subseteq  \\mathbb{R}^{d+1} $. \n- Algorithm 2:  r is not defined. \n- Algorithm 2: To run the while loop runs iff $|Q| \\geq 2*rank(Q)^2$. Where did this condition come from? In neural network pruning setting where |Q| at the beginning would be the #neurons in the next layer and rank(Q) can be at max #neurons in the next layer, how often does this while loop ever get executed? \n- On the empirical side: similar to Mussay et al, can authors include a curve showing true multiplicative error in some simulations, comparing it with baselines, showing the efficacy of their method? \n- Can authors include pruning results without re-training as well like Mussay et al.? It is important to see how performance looks when not re-trained.  \n- All the results need to have the standard deviation attached for the testing error.  \n- Table 1: Why is the PR different for the methods but PvC or  Algo2? \n\n\nAdd the following to the related work:\n\n- Gantavya Bhatt and Jeff Bilmes. Tighter m-DPP Coreset Sample Complexity Bounds. In ICML 2021 Workshop: SubSetML: Subset Selection in Machine Learning: From Theory to Practice, Virtual, July 2021. This paper aims to improve bounds for DPP-based coreset and show possible applications to neural networks pruning (similar to mussay et al.) \n\n- Diversity Networks: Neural Network Compression Using Determinantal Point Processes. Zelda Mariet, Suvrit Sra\n\n\nI'm willing to raise my rating upon satisfactory response. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JCtNbABymuB", "reviewer": "Reviewer_8WWu", "summary": "This paper presents a new method for network pruning based on coresets which relies on inner and outer approximations of the convex hull of the weights of each layer of the neural network, with the inner approximation being obtained from a subset of the weights of the neurons and the outer approximation being a dilation of the former. This form of determining which weights to prune is almost completely independent from the dataset, which is one of the main advantages claimed by the authors.", "questions": "1) In Section 1.1, line 42, what do you mean by finding a query x*? My understanding is that you are actually looking for a subset of all queries, not for a single one.\n\n2) In Section 2, line 83, what do you mean by w(p_i) := w_i? I understand that every neuron of the top layer has a weight for the neuron at the bottom layer, but it is not a function of the value of its weights.\n\n3) What is the reason for assuming all the weights of P as equal to 1 in the main paper?\n\n4) In Section 2.1, between lines 114 and 116, what do you mean by \"Once we bound these sensitivities\" and \"The size of the sample is proportional to the sum of these bounds\"?\n\n5) In Theorem 2.3, line 127, what do you mean by \"points with nonempty interior\"? Points do not have interior.\n\n6) In Section 2.2, line 133, can you explain what do you mean by the dilated form of B consisting of a conical combination? Wouldn't it make more sense to say that this is a weighted sum in which the sum of the weights now exceed 1?\n\n7) In Section 3, why is every network pruned with a different subset of approaches? What happened with the missing ones in each of those cases?\n\n8) In Table 1, what is the meaning of \"Baseline Err.\" and \"Pruned Err.\"? Is the first the complement of the accuracy and the second the drop due to pruning? Why is the first one almost the same but not always across all methods tested?\n\n9) In Table 2, what is the meaning of \"FR\"?\n\n10) In Table 3, is \"Orig. Err.\" the same as \"Baseline Err.\" from before? If so, please make it uniform.\n\n11) Why is only one pruning rate reported for each model and approach?\n\n12) How do you think coreset-based pruning relates to what happens in the case exact compression methods (e.g., https://arxiv.org/abs/2102.07804 & https://arxiv.org/abs/2107.07467)?", "limitations": "No; Section 5 talks about future work, which is not the same thing as limitations of the current work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents some interesting ideas, but it is not easy to follow. I have several comments to the authors about how to improve the writing below. Among other things, some concepts are used before definition, some explanations are repeated, and often it is only possible to understand what is going on when a new and more accessible definition is presented. However, what really concerns me is the limited experimental evaluation: the last sentence of the abstract sounds an alarm of hand-picking. Indeed, the results in Section 3 are very limited: only one pruning rate is used for each method tested, which is far from ideal. I would not expect a new method to always beat all other methods in order to accept a paper, but I do expect to see a thorough assessment of where and why a new method performs better. Finally, I believe that the authors understand quite well what they are doing and are very excited about it, but they also need to provide more context for the readers by going over their design choices in more detail and with more context and explanations to non-experts.\n\n### Comments on the writing\n\nSection 1:\n- In the Abstract, line 5, you write data dependant. In the Introduction, line 61, you write data-dependent. Please choose one convention for spelling and hyphenation.\n- The whole paragraph on coresets is very difficult to read and I could only understand most of it (except for my question about x*) after reading the paragraph \"How to prune using coresets?\" in Section 2, in which the useful parts are actually repeated. I would encourage the authors to rewrite this and perhaps merge both paragraphs at a convenient point.\n- Line 46: \"corestes\" -> coresets\n- Line 69: \"guarantees\" -> guarantee\n\nSection 2:\n- Line 88: \"See\" -> see\n- By the time that [n] is defined in line 92, it has already been used in line 83.\n- Definition 2.2 repeats a lot of terms that were already introduced in the second paragraph of Section 2. Again, I would encourage the authors to think of rewriting the paper with less repetition, more clarity, and each term introduced at the most appropriate time.\n- Line 113-114: the excerpt \"and it corresponds to the importance of this point with respect to the query space at hand\" seems to refer to s(p) rather than its denominator alone, but it is not clear from context.\n- Line 118: \"Our method hinges upon an elegant combination of\": Academic papers should not contain self-praise; please remove \"elegant\". Likewise for \"an elegant combination\" in line 326.\n- Line 137: One of the many examples of the use of \"e.g.\" looks a bit unusual: are you saying that ReLU, among many others, does not admit such an approximation? Please also revisit other uses in the paper (line 148) and compare them with the form in which this abbreviation is commonly used.\n- Line 139: Replace \"at\" with \"in\" (two incidences; see also lines 148, 156, 194 twice, 195, 208, 286)\n- Caption of Figure 1: Replace \"novelty\" with \"contribution\"? In the fourth line, replace \"A\" with \"a\"\n- Line 161: Add \"an\" after \"constructing\"\n- Line 221: \"Deep\" -> deep\n\nSection 3:\n- Line 254: Remove quotes from \"Numpy\"\n- Line 293: \"obtain a\" -> obtain an; remove comma between \"coreset\" and \"when\"\n\nSection 4:\n- Line 322: \"Decision Trees\" -> decision trees\n\nSection 5:\n- Line 328: \"on variety\" -> on *a* variety\n\nReferences:\n\n- Remove [5]: duplicate of [6] with wrong publication year", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-zYb6ig2bL8", "reviewer": "Reviewer_4TWt", "summary": "**Summary of results**\nThe main scope is compression of neural networks with coreset: How to remove neurons or filters after training such that we achieve almost the same performance. The paper proposes an $\\ell_{\\infty}$ coreset for $\\ell_1$ regression (in Lemma 2.7). This coreset is exploited for pruning a single-layer neural network with ReLU activations with all-one output weights (details in Definition 2.2). Compared to existing results in the literature,  the proposed theoretical guarantee is data-independent without explicit assumptions on the norm of inputs. Experimentally comparisions have been conducted on standard neural networks to compare the proposed coreset-based pruning with baselines. \n\n**Comments on relevance and importance of the scope** \nPruning neurons not only enhance the storage and computational complexity of neural networks but also may enhance the generalizability of the nets. Thus, it is a core topic in neural computing. \n\n\n\n", "questions": "1. There are coreset designed for l1 regression with similar techniques:(i) Lowner ellipsoid and (ii) Caratheodory set (see, for example, the survey on Sampling Algorithms and Coresets for $\\ell_p$ Regression). I wondered what the novelty of the proposed algorithm compared to these exciting methods for l1 regression. \n2. Can authors validate boundedness of regression complexity measure for simple cases? \n3. Is it possible to characterize a worse case for the regression complexity measure?\n", "limitations": "This project does not have negative social impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Advantages**\n1. Compared to existing results, the coreset size is data-independent up to the regression complexity measure.   \n2. The proposed algorithm handles low-rank weight matrices that are essential in pruning neural networks. \n3. Nice literature review and introduction \n4. Experimental comparison with various existing methods \n5. In my view, the proposed $\\ell_{\\infty}$ coreset for $\\ell_1$ regression in Lemma 2.7 is more interesting and solid than the main theorem and coreset design for neural networks. \n6. Details of the coreset design with complexity analysis of algorithm \n\n**Main Concern** \n1. The proposed bound strongly depends on the regression complexity measure, which is not proven to be **bounded**. Notably, we can design sets for which this complexity measure scales with the size of the set. For such a set, the established coreset-size is greater than the number of neurons, hence the theoretical bound is vacuous. \n2. To compare the baselines, I recommend checking comparisons for networks with the same accuracy levels and comparison rates (in tables 1-3). Indeed, the compression has to be conducted on the same input networks (so with the same level of accuracy). Comparing compression algorithms on different networks may not lead to a fair comparison. \n\n\n**Minor Comments**\n Adding details about the computational complexity of methods to find Lowner ellipsoid would improve clarity.  \n\n**Post Rebuttal** \nThe response address my major concerns so I updated my score. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept