Neural expressiveness for beyond importance model compression

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:04:40

Abstract: Neural Network Pruning has been established as driving force in the exploration of memory and energy efficient solutions with high throughput both during training and at test time. In this paper, we introduce a novel criterion for model compression, named "Expressiveness". Unlike existing pruning methods that rely on the inherent "Importance" of neurons' and filters' weights, "Expressiveness" emphasizes a neuron's or group of neurons ability to redistribute informational resources effectively, based on the overlap of activations. This characteristic is strongly correlated to a network's initialization state, establishing criterion autonomy from the learning state ($\textit{stateless}$) and thus setting a new fundamental basis for the expansion of compression strategies in regards to the "When to Prune" question. We show that expressiveness is effectively approximated with arbitrary data or limited dataset's representative samples, making ground for the exploration of $\textit{Data-Agnostic strategies}$. Our work also facilitates a "hybrid" formulation of expressiveness and importance-based pruning strategies, illustrating their complementary benefits and delivering up to 10$\times$ extra gains w.r.t. weight-based approaches in parameter compression ratios, with an average of 1\% in performance degradation. We also show that employing expressiveness (independently) for pruning leads to an improvement over top-performing and foundational methods in terms of compression efficiency. Finally, on YOLOv8, we achieve a 46.1\% MACs reduction by removing 55.4\% of the parameters, with an increase of 3\% in the mean Absolute Precision ($mAP_{50-95}$) for object detection on COCO dataset.

Main paper: 
 introduction: 
 To address the computational constraints of existing models, Model Compression [7] has emerged as a prominent solution in exploring models that achieve comparable performance, but with reduced computational complexity [52] . Within this scope, Floating Point Operations (FLOPs) are used to estimate a model's computational complexity, by measuring the arithmetic operations required for a forward pass, while parameters (params) are associated with a model's size in terms of memory space [48] and their reduction can be a precursor towards more energy efficient solutions [5] . Although FLOPs and params often correlate, their relationship isn't strictly linear. For instance, VGG16 [43] has 17× more parameters than ResNet-56 [17] but only 3× more FLOPs, largely due to VGG16's extensive use of fully connected layers. At first sight, this can be attributed to the differences in network topologies. From a deeper perspective, the intricacies of various operations at handling computational workloads, such as residual structures [17, 55] , depthwise separable convolutions [19] , inverted residual modules [18] , channel shuffle operations [59] and shift operations [53] , coupled with their interplay, may significantly affect the relationship between FLOPs and params in a neural network. In a nutshell, besides the use of more computationally efficient operations as above-Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. mentioned, Model Compression aims to maintain model performance while optimizing the two aforementioned metrics via tensor decomposition, data quantization, and network sparsification [7] .In this paper we emphasize on the sparsification strategy of pruning [49] , which we use as a basis framework to introduce "Expressiveness" as a new criterion for compressing neural networks.Existing pruning methods focus on removing redundant network elements -be they weights, neurons, or structures of neurons -in ways that minimally affect the overall performance of a network, based on the criterion of "Importance", e.g. [38, 58, 20, 30] . Importance-based methods address questions like "How much does the removal of a network's element cost in terms of performance degradation?" and "How much information does a network element contain?" in various ways. More specifically, they are motivated by the information inherent in network elements, such as the magnitude of weights [15, 28] , similarity of weights or weight matrices [29, 60] ; and their sensitivity to the network's loss function, such as the magnitude of gradients [38] and more [49, 3] . Such dependencies on weights' distributions constitute the aforementioned pruning methods to be "data-aware" since they intrinsically rely on the input data and the information state of the model, making the importance estimation of the network's elements challenging and often costly due to factors like i) the stochasticity from training with minibatches, ii) the presence of plateau areas in the optimization space, and iii) the complexity introduced by nonlinearities [38] . Liu et al. [36] have also discussed limitations in the perception of importance within trained models, i.e. the authors criticize the ability of network's elements importance to generalize to pruned derivatives, while also questioning the necessity of training large-scale models prior pruning.Inspired by the concepts of "Information Plasticity" [2] and the "Lottery Ticket Hypothesis" (LTH) [12] , we aim to address the limitations of previous importance-based methods through elaborating the "Expressiveness" criterion in model compression. In contrast to "Importance", we focus on understanding the capability of network elements to redistribute informational resources to subsequent network elements. We define "Expressiveness" as -"A neuron's or group's of neurons potential (when a network is not fully trained) or ability (when it is trained) to extract features that maximally separate different samples". As derived by [2] , the early training phase of a model is crucial in shaping its expressiveness, with the formation of critical paths -strong connections that determine the "workload distribution"-being particularly significant during these initial stages. It's essential to note that the network's initialization state influences the formation of those paths, which interestingly enables "Expressiveness" to be a fit criterion for compression during all time instances of a networks' convergence [12] , setting a baseline for answering the question of "When to prune?" [42] . Our proposed pruning metric centers on measuring the overlap of activations between datapoints of the feature space. In that way, expressiveness is based on effectively evaluating the inherent ability of the network's neurons to differentiate sub-spaces within the feature space. We experimentally show that utilizing either small sets of arbitrary data points from the feature space or stratified sampling [34] from each class yields consistent estimations of expressiveness. Finally, we propose and implement a new "hybrid" pruning optimization strategy that cooperatively searches, exploits and characterizes the complementary benefits between "Importance" and "Expressiveness" for model compression.In summary, this work offers the following four-fold contribution: (i) we propose Expressiveness, a novel criterion based on the overlap of activations for model compression; (ii) we provide an in-depth theoretical analysis of both the fundamental principles and the technical intricacies of the proposed criterion; (iii) we validate the hypothesis that Expressiveness can be approximated with little to none input data, opening the road for data-agnostic pruning strategies; and (iv) through extensive experimentation we offer a thorough comparison w.r.t to both foundational and state-ofthe-art methods demonstrating the efficiency and effectiveness of the proposed technique in model compression, while also examining the feasibility and effectiveness of a "hybrid" expressivenessimportance pruning strategy. Specifically, we validate "Expressiveness" on the CIFAR-10 [24] and ImageNet [40] datasets using a variety of models with different design characteristics [44, 17, 45, 21, 19] . We demonstrate the superiority of our novel criterion over existing solutions, including many top performing structural pruning methods [31, 61, 58, 32, 23, 46, 11] , and show significant params reduction while maintaining comparable performance. We experimentally explore and analyze the complementary nature of expressiveness and importance, showing that summary numeric evaluation provides up to 10× additional parameter compression ratio gains, with an average of 1% loss decrease w.r.t group ℓ1norm [28] . Finally, we experiment on the current state-of-the-art computer vision model (YOLOv8 [9, 22] ), showcasing notable compression rates of 53.9% together with performance gains of 3% on the COCO dataset [33] , and highlighting the ability of more expressive neurons to better recover lost information from the pruning operation. 
related work: 
 Weight (Non-Structural) Importance. Han et al. [15, 14] and Guo et al. [13] approached the importance of weights based on their magnitude, removing connections below given thresholds.However, earlier works [25, 16] emphasized on the Hessian of the loss and have questioned whether magnitude is a reliable indicator of weight's importance, as small weights can be necessary for low error. In this direction, several studies [4, 47, 41, 8] have proposed strategies of iterative magnitude pruning, in the form of "adaptive weight importance", where weights are ranked based on their sensitivity to the loss. From a different perspective, Yang et al. [56] address the limitations of individual weight's saliency that fail to account for their collective influence and provide a formulation of weight's importance based on the error minimization of the output feature maps. Expanding on this concept, Xu et al. [54] propose a layer-adaptive pruning scheme that encapsulates the intra-relation of weights between layers, focusing on minimizing the output distortion of the network. Amongst other factors and limitations (as also discussed in 1), weight importance is very expensive to measure, mainly because of the increased complexity induced by the mutual influences of the weights among interconnected neurons. This, coupled with the requirement for specialized hardware to manage the irregular sparsity patterns resulting from weight pruning [57] , has shifted research focus towards structural pruning [28] , where neurons or entire filters are removed.Neuron and Filter (Structural) Importance. Many where driven by the success of Iterative Shrinkage and Thresholding Algorithms (ISTA) [6] in non-structural sparse pruning and proposed filter-level adaptations [28, 29, 32, 26] , based on the relaxation (ℓ1 and ℓ2) of ℓ0 norm minimization.However, the loss of universality of such magnitude-based methods remains a limitation in the approximation of importance even in the structural scope. Yu et al. [58] further elaborate on the idea of error propagation ignorance, where the analysis is limited to the statistical properties of a single [28, 29] or two consecutive layers [37] . The authors suggest that the importance of neurons is better approximated from the minimization of the reconstruction error in the final response layer from which it is propagated to previous layers. In contrast to this view, Zhuang et al. [61] emphasize on the discriminative power of a filter as a more effective measure of importance and highlight that this aspect is not effectively assessed by the minimization of the reconstruction error. In a manner that reflects the progression of weight importance, Molchanov et al. [38] define "adaptive filter importance" as the squared change in loss and apply first and second-order Taylor expansions to accelerate importance's computations. Predominantly, the data-awareness imposed by most pruning strategies is added to their already high-complexity -i.e. mostly non-convex, NP-Hard problems that require combinatorial searches. This renders the estimation of importance both computationally expensive and labor-intensive, similarly to non-structural approaches. Notably, Lin et al. [30] propose a less data-dependent solution based on the observation that the average rank of multiple feature maps generated by a single filter remains constant. HRank [30] , alongside several other feature-guided filter pruning approaches, are valuable indicators towards data independence. Such works form a principle that pruning elements are better evaluated in the activation phase, where the importance of information and the richness of characteristics for both input data and filters are better reflected. In this work, we expand on this belief and we through extensive experimental analysis, we demonstrate that neither the information state nor the input data is required for the discriminative characterization of an element. i , where l∈L is indicative of the neuron's layer in a network with L = {l 0 , ..., l l , ..., l |L| } layers and i of its position in the given layer l = {a 0 , ..., a i , ..., a |l| }. Another important element are the learning parameters of the network. Otherwise the weights represent the strength of connections between neurons in adjacent layers and are denoted by w (l) ij , where i and j index the neurons in the current and previous layers. In that manner, neuron's can be perceived as switches that allow or block information from propagating through-out a network. The activation (or not) of a neuron a (l) i depends on the output value of its activation function σ(•), where there are many popular options for the definition of σ, e.g., sigmoid, tanh, and ReLU functions. Specifically, a neuron's output is defined as follows,a (l) i = σ j w (l) ij a (l-1) j + b (l) i (1)where b (l) i denotes the bias term. From eq. 1, we observe that a neuron's activation is affected by the activation of the previous layers, hence affecting in the same way the consecutive layers. This interdependence between activations a (l) , for a given layer l defines a recurrent form that can be generalized as follows, a (l) = σ W (l) f a (l-2) , . . . , a (1) + b (l) .(EQUATIONOn the other hand, weights are a more static representation of information as they modulate how much influence one neuron's activation has on another's, compared to activations that control the flow of information in a network. This differentiation has motivated us to define two axes of study in the categorisation of pruning criteria, one based on the weights ("importance") and one based on the activation phase ("expressiveness").Generalization of concepts in a structural level. The aforementioned principles extend to the structural representations of weights and activations, the most common being Convolutional Neural Networks (CNNs). For a CNN model with a set of K convolutional layers, where C l is the l -th convolutional layer. We denote filters (weight maps) and feature maps (activation maps) as F l k and C l k respectively, where k the is index within a layer. Given filter with dimensions m × n, eq. 1 is adapted as follows,EQUATIONwhere (i, j) and (x, y) are the coordinates of weights and output activations within the filter and the output activation map respectively. Similarly, a convolution layer l can be analyticaly expressed as follows,EQUATIONwith X being the input to the first layer of the network, and where symbol * denotes convolution operation and denotes the concatenation operation. Within this context1 , eq. 2 is generalized as follows,EQUATIONConceptualization of information propagation. Consider a task with X = {x i } |D| i=1 denoting dataset samples, where |D| is the size of the dataset. Given the information state (weight state) of a CNN model with K convolutional layers at a given time t i , X is mapped through the network asf (X, W ti ), where W ti = {F 1 ti , . . . , F l ti , . . . , F |K| ti } and F l ti = {F (l,1) ti , . . . , F (l,k) ti , . . . , F (l,K (l) ) ti },with K (l) being the amount of weight maps (filters) in a given layer l. This process can be further analyzed as follows,EQUATIONwhere F l represents the mapping operation of convolutional layer l.Based on eq. 2 and eq. 5, the equivalent of the previous based on the activations of the layers can be expressed as,EQUATIONHere, C (l) represents the activation map of the l-th layer, where C (l) = F l (C (l-1) ; F l ti ) aligns with the structure defined in eq. 4. In this formulation, C (1) is the activation map of the first layer, computed using the input X and the first layer's filters F 1 ti . Subsequent layers' activation maps C (l) are derived from the previous layer's output C (l-1) and their respective filters F l ti . Assuming a classification task, the final layer C (|K|) is considered the classification layer, effectively summarizing the hierarchical feature extraction and transformation process across all convolutional layers.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: While our work does not implicitly provide a Discussion section, we have incorporated any discussions on the limitations and the intricacies of the provided solution at its section separately.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
mathematical foundation of neural expressiveness.: 
 We observe that the training parameters of the model, in this case W ti2 , are responsible for transforming the original input feature space X into a sequence of intermediate feature spaces{C (1) , . . . , C (|K|-1) }, progressing towards the final prediction formulated by the prediction layer C (|K|) .Based on this intrinsic characteristic of neural networks and inspired by the goal of optimizing feature discrimination, akin to the entropy reduction strategy in decision trees [51] , we assess network elements ability, in this scenario filters, to extract features, i.e., activation patterns, that maximally separate different input samples x i . In other words, we score the expressiveness of the filters within W ti , based on the discriminative quality of the intermediate feature spaces they generate, where the feature space generated by a filter F l k , is denoted as C l k .Neural Expressiveness foundational concept. When assessing the expressiveness of an element within W ti based on its generated feature spaces, e.g., N EXP (F l ti ; C l ), we cooperatively evaluate all of its preceding elements, as derived from eq. 5. This can be formulated as,EQUATIONwhich can be further extended to incorporate the inter-dependencies between the examined element and its predecessors, in accordance with eq. 7, as detailed below:EQUATIONThe aforementioned eqs. 8 and 9 provide the foundational concepts for utilizing the evaluation of the activation phase, in an endeavor to encourage the development of more universal solutions by addressing the limitations of universality inherent in the assessment of the weight state alone (as also discussed in sections 1 and 2).Formulation of Neural Expressiveness (NEXP) Score. Diving deeper into the Neural Expressiveness (NEXP) scoring process, we follow eq. 9 previously and assume a mini-batchX ′ = {x ′ i } N i=1 ,with N being the number of samples in it. Mapping the batch through the network, based on eqs. 6 and 7, generates a set of sequences of feature spaces (activation maps), denoted asS = {s 1 , . . . , s i , . . . , s N }, where s i = {x ′ i , . . . , C l i , . . . , C |K| i } is the sequence of the activation patterns generated from sample x ′ i ∈ X ′ and |s i | = |K| + 1 is its cardinality, including the feature space of sample x ′ i .To evaluate a specific filter k in layer l, denoted as F l k , we utilize the retrieved activation patterns from that filter, denoted as {s l i,k } N i=1 , where s l i,k = C l i,k is the activation pattern retrieved from filter k in layer l.To score the Neural Expressiveness of F l k , we first construct a N × N matrix that expresses all possible combinations of the activation patterns derived from the different input samples. This table can be visualised as follows, where s l (i,j),k denotes the dissimilarity of activations patterns between the i-th and the j-th sample of the batch. In other words, the matrix in eq. 10 represents all the possible combinations of NEXP calculations, where each element s l (i,j),k derives from f (s l i,k , s l j,k ), with f being any dissimilarity function. Without loss of generality, for the rest of the study, we use the Hamming distance as the operator implementing dissimilarity function. Activations are first binarized (values greater than 0 become 1, and the rest become 0), i.e. enabling to evaluate the degree of overlap between the binary activation patterns using f .EQUATION)We note that the matrix's diagonal, where i equals j, along with the elements below the diagonal, where i is greater than j, do not contribute additional value to quantifying the discriminative ability of an element. The diagonal elements represent comparisons of the same sample's activation patterns, rendering them redundant. Meanwhile, the lower triangular elements are considered duplicates since s l (i,j),k is equal to s l (j,i),k , thereby not adding any new information. Drawing from these two observations, we define the Neural Expressiveness score (NEXP) as follows,EQUATIONThe more similar the activation patterns derived from an element are, the less expressive it is declared to be. In eq. 11, we also normalize the score w.r.t the total amount of combinations( N (N -1)2), thereby deriving the average expressiveness score. This average score is then utilized to characterize the discriminative capability/capacity of the examined network element. In this study, we used the mean operation, however, we note that alternate statistical measures, e.g., minimum, maximum, median, etc., could feasibly be applied in the computation of the overall score.NEXP evaluates the inherent property of network elements to maximally distinguish between input samples. We extend this line of thought and assess its sensitivity to input data X and mini-batch size N , in order to delineate the dependence between NEXP and the input data. To achieve that, we perform a sensitivity analysis of NEXP to the mini-batch data X, using two input sampling strategies to assemble a batch with 60 samples, namely random sampling (denoted as 'random') and class-representative sampling via k-means (denoted as 'k-means'). We define the true NEXP score (denoted as 'non-approx') for each filter as the value obtained by comparing all activation patterns across the entire training dataset (more info in A.1). Fig. 1 presents a detailed comparative illustration of the results that highlight the similarities in NEXP estimations across various trained networks, including VGGNet [44] , ResNet [17] , MobileNet [19] and DenseNet [21] on CIFAR-10 dataset.Columns represent the aforementioned sampling strategies, while colors indicate expressiveness levels, with higher values signifying greater expressiveness. In each sub-figure, the x-axis indicates Algorithm 1 NEXP Pruning AlgorithmDefine: N EXPmap = {{N EXP (F l k )} |C l | k=1 } |K| l=1Require: A mini-batch X, a neural network N (Wt i ), a theoretical speed-up target, denoted τ , and the allowed amount of pruning steps, denoted steps max . Ensure: NEXP can be effectively estimated using random and limited data samples. Detailed results of this analysis, are presented in Appendix A. The comparative analysis reveals that a mini-batch of 60 samples (0.4% of D in this case) effectively approximates the NEXP scores calculated from the entire dataset, yielding consistent similarity scores above 99% across most similarity metrics (Table . 3).FLOPs(N ) FLOPs(N pruned ) ≥ τ 1: Initialize N EXPmap ← f (X; Wt iAlg. 1 describes the proposed NEXP-based pruning process, and it has been implemented as extension in the DepGraph pruning framework [11] . A target theoretical speed-up is specified, referred to as the Compression FLOPs Ratio (↓) and denoted by τ . This ratio is calculated using the formula original FLOPs compressed FLOPs . To achieve this target ratio, the network may undergo pruning in one or several steps, dictated by the intricacies of the pruning criterion and adjusted according to the quantity of elements removed at each step. For example, NEXP benefits from additional steps, since a filter's score is reliant on its preceding elements (Section 3.2), and a more gradual update on the scores allows for improved pruning precision. A more in-depth analysis of Alg. 1 along with more details on the implementation options are presented in Appendix B. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
comparison w.r.t. state-of-art model compression strategies: 
 Image Classification on CIFAR-10 and Imagenet-1k. We compare against a plethora of foundational and top-performing approaches, ranging from filter magnitude-based [28, 32, 29] and loss sensitivity-based [58] methods to feature-guided strategies [23, 30] and search algorithms [35, 31] .Outcomes and Discussion. Our findings for various target FLOPs pruning ratios are presented in Tab. 1 (and Tab.6-9 in Appendix D.2) for CIFAR-10, and in Tab. 2 for ImageNet. It is essential to acknowledge the subjectivity in reported performance metrics (accuracy), influenced by the finetuning process post-pruning, e.g. the authors in DCP [61] fine-tune for 400 epochs, in contrast to ours 100. We observe that our approach yields consistent improvements in params reduction compared to other methods for given FLOPs ratios, which notably scale significantly for regimes of higher target FLOPs compression ratios τ . For example, on ResNet-56 we show +0.92× average params reduction gains in the 2×-2.20× FLOPs reduction regime, with -0.38%, +0.05% and -0.37% percentage difference in loss respectively to ABC [31] , SCP [23] and HRank [30] , while on ResNet-110 we show +1.21× average params reduction gains in the 2.87×-3.27× FLOPs reduction regime, with -0.67% and +0.26% percentage difference in loss respectively to ABC [31] and HRank [30] . Similar observations are evident across all tables, where in certain regimes we also show notable performance gains, up to +1.5%, especially for VGGNet, which is more prone to params reductions due to its plain structure.Object Detection with YOLOv8. We evaluate expressiveness against four importance based methods, i.e layer-adaptive magnitude-based pruning (LAMP) [26] , network slimming (SLIM) [35] ,Wang's et al. proposed method (DepGraph) [11] and random pruning that serves as a generic pruning baseline [3] . The experiments were conducted on the YOLOv8m model version [22] , utilizing the DepGraph pruning framework [11] with an iterative pruning schedule of 16 steps, where after each pruning step the model was fine-tuned for 10 epochs using the coco128 dataset.Outcomes and Discussion. We report the comparative pruning progress of expressiveness versus the baseline methods, i.e. the remaining percentage of the original model in terms of MACs and params after each pruning step, named MACs Size Percentage (MSP) and Parameters Size Percentage (PSP) respectively, and highlight the mAP val 50-95 both after pruning (pruned mAP) and fine-tuning (recovered mAP). We observe that expressiveness outperforms the rest of the reported methods across the whole pruning spectrum, as shown in Fig. 2 (more in Appendix D.2), preserving the initial performance of the model for percentage sizes that reach up to 40% (2.5 ↓) of that of the original model, with less than 0.5% of recovered performance degradation. Our method even achieves a 3% increase in recovered mAP for 46.1% MSP (2.17 ↓), in comparison to the baselines that showcase weak recovery capabilities after the 60% (1.67 ↓) mark in both MSP and PSP. This can be attributed to the intrinsic property of expressiveness to maintain network elements that are more robust to information redistribution, in contrast to "important" labeled structures by other methods. In our experimental scenario, that characteristic is further amplified by the iterative pruning format and the higher amount of fine-tuning epochs at each step, in comparison to conventional pruning schedules that fine-tune for 1 epoch after each iteration or perform a unified fine-tuning session after the last pruning iteration. Interestingly, our criterion also demonstrates significant resistance to performance loss after pruning, achieving 18% increased average performance in terms of pruned mAP compared to the importance-based methods. We have empirically observed that expressiveness benefits from increased cardinality in pruning granularity settings, e.g amount of intermediate steps to achieve a given compression ratio. This stems from expressiveness interactive nature of all elements, as also explained in Sec. 3, where smaller pruning steps combined with iterative fine-tuning, enhance pruning precision and allow for "smoother" redistribution of information in a network, thus contributing to the increased resistance to performance deficits after each pruning step.In this section, we assess the potential efficiency of "hybrid" pruning strategies exploiting the cooperation between importance and expressiveness. We explore the solution space of "hybrid" compression, using a linear combination of importance and neural expressiveness criteria. We guide exploration through the scoring function: W imp • IMP + W nexp • NEXP and conduct experiments with) • W imp + α • W nexp = 1.We use group L1-norm [28] as the importance criterion (IMP) and assess all permissible combinations across a linear scale, denoted as τ , representing the target FLOPs compression ratios that we utilized for pruning, on ResNet-56 for CIFAR-10. The outcomes are visualized in Figure 3 , which maps our predetermined τ values on the x-axis against the various parameter compression ratios achieved by each combination. Regarding performance, we report the averaged percentage differences in top-1 accuracy between the baseline importance method (L1) and each hybrid format: -0.21% for hb-0.2, -0.96% for hb-0.4, -1.55% for hb-0.6, -1.07% for hb-0.8, and -2.18% for NEXP.Observations. A consistent pattern is observed across the values of α, where larger values yield higher params compression ratios. Notably, hybrid derivatives allow us to explore sub-spaces with higher parameter compression ratios by sacrificing slight performance accuracy. We also observe that the solution vectors corresponding to IMP and EXP act as extremal points in the solution space of hybrid combinations, thus suggesting a degree of partial orthogonality between the two criteria.Furthermore, the findings reveal a polynomial relationship between parameter compression ratios and FLOPs reduction, with compression ratios increasing polynomially to linear increments in FLOPs reduction, and thus enabling more efficient explorations.The nature of NEXP allows to be applied in a weight agnostic manner, i.e. on untrained networks.An extended version of the section's 3.3 analysis, which also includes untrained models (Appendix A), reveals that N EXP map 's obtained at initialization and after network convergence share some expressiveness pattern similarities, particularly in the initial layers. Our numeric evaluation shows a notable correlation between the initialization and converged states for DenseNet-40 and VGG-19, with cosine similarities of 84.10% and 86.82%, respectively. It also indicates greater consistency in neural expressiveness measurements for the first layers of all networks, which could be considered important for the formation of critical paths [2] . Motivated by these observations, we also assess the efficacy of expressiveness as criterion for Pruning at Initialization against various SOTA approaches [27, 50, 46] (Appendix D.1). Our method consistently outperforms (in terms of top-1 acc) all other algorithms, particularly in regimes of lower compression, up to 10 2 (↓) with an average increase of 1.21% over SynFlow, while maintaining competitiveness at higher compression levels, above 10 2 (↓)with an average percentage difference of 4.82%, 3.72% and -2.74%, compared to [50] , [27] and [46] .In summary, under the assumption that the selection of hyperparameters remains congruent with the initialization [12] , consistent map measurements between initial and final states can effectively evaluate NEXP's ability to identify winning tickets. However, a robust evaluation should also consider the initial state quality and the training process, while addressing the "When to prune" question [42] . 
conclusions: 
 In this work, we have introduced "Neural Expressiveness" as a new criterion for model compression.In our NEXP steps, we will explore optimal solutions for the "When" and "How" to prune questions.A Duality of Independence: Data (X) and Information State (W t i )Fig. 4 presents a detailed comparative illustration that highlights the similarities in NEXP estimations across various networks, including VGGNet [44] , ResNet [17] , MobileNet [19] and DenseNet [21] on CIFAR-10 dataset. Specifically, for each network architecture, we showcase expressiveness A.1 True NEXP value (non-approx).We define the true NEXP score for each filter as the value obtained by comparing all activation patterns across the entire training dataset D. In that way, the ability of each element to extract maximal features is evaluated for every data-point in the input feature space of a task at hand. In this study however, due to GPU memory constraints (limited to 12GB of GDDR6 SDRAM), we employed 25% of the total training set, ensuring class distribution is preserved, to determine these exact NEXP scores, denoted as non-approx.A.2 Data Agnostic.To evaluate NEXP's sensitivity to input data, we conduct a similarity analysis for each row in as also highlighted by Frankle et al. [12] . In that manner, under the assumption that the selection of hyperparameters remains congruent with the initialization, "Expressiveness" can be considered a fit criterion for Pruning at Initialization (PaI). In summary, the consistency of map measurements between initial and final states may serve as a solid metric for evaluating NEXP's ability to identify winning tickets. Nevertheless, a more robust process of its evaluation should also take into account the quality of the initial state as well as the subsequent training process. To maintain consistency across the y-axis, we have interpolated each layer's feature maps (pixel-wise) to match the layer with the most feature maps. Columns denote different sampling strategies and different colors denote different expressiveness values (the higher the value, the more expressive the feature map). To approximate the expressiveness score of each element, denoted as "non-approx", we used 25% of all dataset's samples (not 100% due to memory limitations) maintaining the label's distribution. As can be seen, the rank of each feature map (column of the sub-figure) is almost unchanged (the same color), regardless of the image batches. Hence, even a small number of images can effectively estimate the average rank of each feature map in different architectures. and GraSP [50] , as well as one state-of-the-art iterative pruning strategy, named SynFlow [46] . For our approach, we implement one-shot pruning, utilizing a batch of 64 arbitrary data points for the estimation of expressiveness.Experimental Setup. We adopt the experimental framework of Tanaka et al. [46] , who assess algorithm performance across an exponential scale (10 r ) of parameters compression ratios r ∈ {0.00, 0.25, 0.50, 0.75, . . . }. Their proposed settings also enable for the evaluation of an algorithm's resilience to "layer collapses", typically observed at higher compression levels. Results. We prune VGG-16 on CIFAR-10 and compare against the findings of [46] . We remain consistent with our adversaries and train the model for 160 epochs, using a batch size of 128 and an initial learning rate of 0.1, which is reduced by a factor of 10 after 60 and 120 epochs. The results are illustrated on Fig. 5 . Observations. Our method consistently outperforms all other algorithms, particularly in regimes of lower compression, up to 10 2 (↓) with an average increase of 1.21% over SynFlow, while maintaining competitiveness at higher compression levels, above 10 2 (↓) with an average percentage difference of 4.82%, 3.72% and -2.74%, compared to GraSP, SNIP and SynFlow respectively. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies all the training and test details necessary to understand the results. Each experiment is accompanied by a discussion of its experimental details and a reference to its experimental settings (Section 4). Additionally, an overview of the experiment settings can be found in Appendix C, and an in-depth analysis of the pruning procedure, including its implementation choices, is described in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: While we do not explicitly address the statistical significance of each experiment (in a quantitative manner), we do discuss in great detail any assumptions or statistical implications of our experiments (in a qualitative manner).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: While we do not provide the exact times of executions and memory requirements for each experiment, we do provide an in-depth analysis of all the parameters and experimental specifications, along side with the overview of the configurations that were used for this work Appendix C and Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: While our work does not directly discuss societal impacts, we do reference the eco-friendly implications of efficient models in the Introduction section 1. Additionally, we highlight the potential indirect societal benefits that can arise from optimizing models, such as in the case of YOLOv8 D.2. . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work emphasizes on efficiently compressing Neural Networks and does not target any specific use-case scenario, rather it addresses the greater challenge of Vision as whole.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We present further experiments and comparisons with state-of-the-art methods, including HRANK [30] , GAL [32] , ABC [31] and DCP [61] , specifically for GoogLeNet and MobileNet-v2 networks. For MobileNet-v2, our method attains an increased compression ratio of 0.94× in parameters and 0.75× in FLOPs (↓), with a minimal decrease of only -0.09% in performance compared to DCP. In the GoogLeNet case, we demonstrate a notable enhancement in parameters compression within the 1.60× to 2.20× FLOPs compression range, surpassing GAL and HRANK with margins of 1.8× and 1.52× respectively, with an average improvement of 7.5% in performance degradation. Motivation. YOLOv8 [22] is the current state-of-the-art for Object Detection and Image Segmentation, and has already been widely adopted by many for a variety of real-time applications, e.g.Traffic Safety [1] , Medical Imaging [39] , Rip Currents Detection [10] , and more. Such applications could majorly benefit from model compression optimizations, achieving higher throughput ratios that translate to increased resolution (FPS), and enabling deployment on hardware with strict resource constraints. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We do not include pooling and batch normalization layers in the formulations; however, the equations can be expanded to incorporate them as intermediate steps based on each architecture.Bias terms are excluded for simplicity. 


Decision: The reviewers are not fully convinced by the paper. Although the authors could sort out some issues in the rebuttal, the reviewers ratings are 2 x 4 and 2 x 5, making this paper a clear borderline case. After carefully reading the paper and the discussions with the reviewers, I came to the conclusion that the paper is below the acceptance threshold. Although the paper has some merits and is certainly of interested to the NeurIPS community, the strength of the contribution is limited as mentioned by reviewer ha2w.
Therefore, the paper can not be accepted at NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper aims to handle network pruning problem. Specifically, it proposes to use a new importance measurement, called expressiveness, to decide the pruning process. It jointly considers the model state to leverage on the proposed measurement. In addition, it can also combined with typical importance based pruning methods to improve the model efficiency.
strength_weakness: strength_weakness: 1. Adding more discussion in the conclusion part helps to improve the paper readability.
2. Since this paper proposes a new pruning metric, it is better to show more visualization and network behavior analysis to illustrate the intuition.
3. The compared baseline models are relatively old, adding more recent publications helps to support this paper.
questions: questions: Please refer to the weaknesses section above.
limitations: limitations: None
paper_summary: paper_summary: This paper propose a new structured pruning approach NEXP. It works by computing the dissimilarity score of the feature activations across samples and removing those filters with smaller variances. Experimental results on several models and datasets demonstrate the effectiveness of the proposed approaches.
strength_weakness: strength_weakness: 1. The conclusion section is too short and fails to characterize the main contribution of this paper. What does it mean as to “when” and “how” to prune? The authors should elaborate on these further.   

2. I think that this is not a scalable approach for computing the pruning metrics. The pruning metric proposed in the paper requires computing a N by N matrix for each filter in the network, where N is the number of samples. This could grow quite computationally infeasible for large networks and batch sizes. The authors also fails to discuss this aspect on the pruning efficiency in the paper.  

3. In terms of experiments, I am not sure why the authors compare each methods under different compression ratio? If the pruned models in each method have different parameters, it can be hard to compare the accuracy numbers.  

4. I feel a lot of the content in section 3 is not necessary and they can could go into a separate preliminary section. Section 3.1 and early parts of Section 3.2 takes up a lot of space and in the meantime do not provides us with the motivation and insights for the later introduced methods.
questions: questions: 1. The authors mostly focus on convolutional neural networks. However, Vision Transformers are becoming the de facto architecture for many computer vision tasks. Does the proposed approach apply to Vision Transformers as well?   

2. Figure 3 seems to suggest that the proposed pruning metric is uniformly superior to importance based weight metric. Why is this the case? It seems unintuitive to me that a pruning metric that discard the weight information could works better than magnitude based approaches.  

3. Have the authors evaluate the practical runtime speedup of pruned models? This would make the results more comprehensive.  

4. What does Figure 1 depicts exactly? What does each figure represent, e.g., the row and column?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper works on weight pruning for CNNs. It proposes an evaluation metric, i.e., "expressiveness", to evaluate whether a neuron/groups of neurons should be pruned or not. The metric focuses on the neurons' ability to redistribute informational resources. As the evaluation of expressiveness requires data samples, the paper includes studies on arbitrary data or limited dataset’s representative samples. The experiments are conducted for image classification tasks on ResNet architectures, and the object detection task on YOLOv8m.
strength_weakness: strength_weakness: 1. Limited practicality of the approach. The method is mainly focused on removing redundant filters from CNNs. Though CNN is one category of DNNs, recent works have shifted to more advanced model architectures such as transformers and Mamba, which are mainly composed of FC layers instead of CNNs. The practicality of the approach is highly restricted to SOTA model architectures for image classification tasks.   

2. Limited performance improvements. This is a major concern. The performance gain of the proposed method is not obvious compared with baselines. For instance. on CIFAR-10 VGG-16, SCP and reduce the parameters 15.28$\times$ with a 93.85\% accuracy while the proposed method can only reduce the parameters 5.62$\times$ with a slightly better accuracy 93.87\%. HRank also provides better performance than the proposed method with higher accuracy 93.96\% (0.09\% higher than NEXP) and higher reductions of FLOPs (4.26$\times$ (HRank) v.s. 4.01$\times$ (NEXP) ). On DenseNet-40, Hrank also shows better performance across all metrics. Hrank v.s. NEXP: Accuracy 95.05\% v.s. 94.64\%, parameter reduction 3.31$\times$ v.s.3.12$\times$, FLOPs reduction 3.38$\times$ v.s. 2.51$\times$.
questions: questions: 1. Can this method be extended to model architectures beyond CNNs? How is the performance compared with other methods?

2. What is the acceleration performance of the proposed method?

3. For the data selection, in extreme cases, such as if the data samples all come from the same classification class, what will the performance be like?
limitations: limitations: Please refer to weaknesses and questions. The major concern is that the method does not show better performance than baselines, and is also limited in model architectures.
paper_summary: paper_summary: In this paper, the author proposes "Expressiveness," a metric that measures the dissimilarity of feature maps produced by different filters. Subsequently, the author introduces NEXP, a technique to prune filters based on their expressiveness. The proposed method is tested on tasks such as image classification and object detection.
strength_weakness: strength_weakness: 1. The notation is very confusing. For example, in the section 'Generalization of concepts at a structural level,' $\ell$ is the index of a certain layer, but the upper-case K is the total number of layers, and the lower-case k is the index of a filter/channel in a certain layer.
2. The concept of expressiveness is not new in the context of pruning [1] and neural architecture search [2,3,4].
3. The performance improvement by NEXP is not prominent. For example, in Tables 1 and 2, NEXP often shows a higher compression ratio but lower accuracy. This makes it unclear if NEXP offers a significant advantage over other methods.

***
**Minor Mistakes:**

Line 163: "where k the is" should be corrected.

[1] Tanaka, Hidenori, et al. "Pruning neural networks without any data by iteratively conserving synaptic flow." Advances in Neural Information Processing Systems 33 (2020): 6377-6389.

[2] Lin, Ming, et al. "Zen-NAS: A zero-shot NAS for high-performance image recognition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Wang, Haoxiang, et al. "Global convergence of MAML and theory-inspired neural architecture search for few-shot learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[4] Chen, Wuyang, Xinyu Gong, and Zhangyang Wang. "Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective." arXiv preprint arXiv:2102.11535 (2021).
questions: questions: In section D.1, how is NEXP applied to PaI methods? If I understand correctly, SNIP, GraSP, and SynFlow are all weight pruning methods, whereas NEXP is structured/filter pruning method.
limitations: limitations: The authors have discussed limitation in the manuscript.

Comments: title: Addressing Reviewer ha2w's Concerns Regarding W1 and W2
Addressing Concerns Regarding Performance Improvements (part b)
Addressing Concerns Regarding Performance Improvements (part a)
Reply to Reviewer 9m28
Rebuttal
Thank you for your rebuttal.
comment: We would first like to thank you for taking the time and effort to address our initial rebuttal (especially at this time of year).  Your insightful comments and questions are greatly appreciated and have enabled us to further clarify the motivations of our work and improve the presentation quality of the paper.

To further address your concerns regarding the applicability (W1) and performance improvements (W2) of our proposed method, we provide more detailed discussions below.

**W1 (extended).** Indeed, the paper does not provide experimental evidence on how our proposed approach will perform in transformer blocks. While we have incorporated our proposed pruning approach as an extension of the latest DepGraph version (and thus its pruning compatibility aligns with that of the DepGraph framework) and have provided theoretical 'guarantees' via the generalization of the foundational concepts and intuitions of Neural Expressiveness—i.e., estimating the contribution of a neuron or group of neurons based on their ability to extract features that maximally separate sub-spaces within the feature space using the overlap of activations—we believe that, in order to perform a robust analysis and provide concrete experimental evidence, the intricacies of the various operations handling computational workloads within different neural network architectures should be explored and discussed separately. For that reason, the format of this paper prioritizes providing a solid basis on both the theoretical and algorithmic properties of NEXP, while hinting at potential future directions. To motivate further research and analysis of NEXP in task-specific and model-specific optimizations, we kindly encourage readers to experiment with different settings of NEXP throughout the paper, and we provide specific guidelines (e.g., lines 219-222, 234-236).

**W2 (extended).** For an in-depth discussion on the performance improvements achieved by NEXP, as well as its effectiveness beyond the predictive performance (quality) of the pruned models, we kindly refer you to the responses titled *'Addressing Concerns Regarding Performance Improvements (part a)'* and *'Addressing Concerns Regarding Performance Improvements (part b)'* in reviewer pBA2's comments section.

W2.a. Addressing the Example on ABC: The ABCPruner can be categorized as part of the subset of Evolutionary-Based Search Algorithms within the domain of Model Compression [3]. In contrast to most pruning methods, which focus on removing redundant network elements—whether weights, neurons, or structures of weights and/or neurons— Neural Architecture Search (NAS) involves searching through a vast space of possible architectures to find the optimal one, often requiring the training of many candidate models from scratch. Specifically, NAS-Evolutionary-Based pruning approaches adopt evolutionary algorithms to explore and identify optimal sparse subnetworks; in this context, ABCPruner employs the artificial bee colony algorithm to efficiently discover the optimal pruned structure. While NAS-based algorithms often do not constitute a fair comparison to pruning methods due to their more intensive and costly explorations (when compared to the more lightweight and straightforward intuition of elimination in pruning approaches), we have included ABC as a more generic compression baseline to further emphasize NEXP's consistent ability to achieve higher compression ratios beyond those of native pruning methods.

We are more than willing to address any further questions or concerns you might have to ensure that our submission meets the highest quality standards.

[3]. He, Yang, and Lingao Xiao. "Structured pruning for deep convolutional neural networks: A survey." IEEE transactions on pattern analysis and machine intelligence (2023).
3. Thank you for acknowledging NEXP's superiority in consistently achieving higher compression ratios. While it is true that in some cases, as seen in Tables 1 and 2, the predictive quality of models generated by NEXP is not always superior to other methods, it is equally important to note the numerous instances where NEXP demonstrates notable performance improvements (e.g., Table 1, right: lines 1-3, 8-9; Table 1, left: lines 10-11; Table 2: lines 6-8). In this context, we believe that a fair comparison, which includes cases where our approach does not outperform alternative methods, offers a valuable basis for a more thorough discussion and for identifying ways to address potential limitations. In this context, as also discussed in point 1, we address the potential limitations in predictive performance by providing a detailed discussion on the advantages of using an iterative pruning format (as demonstrated in object detection, where NEXP shows significant performance improvements) compared to the one-shot pruning format employed for image classification in Tables 1 and 2. A more in-depth analysis of the pruning settings is also available in Appendix B ('Pruning Process: An In-Depth Analysis').

Overall, we concur with the notion that the pruned architecture, rather than the inherited 'important' weights, is more crucial to the efficiency of the final model, as discussed in Q2 of reviewer 9m28 and outlined in [4]. However, under different experimental settings and evaluation criteria (e.g., Tables 1 and 2), 'important' weights may prove to be a superior metric for achieving higher predictive performance, especially when predictive accuracy is a more critical constraint than parameter reduction. To address this potential trade-off between important and expressive methods along the efficiency-quality curve, we have included an assessment of the hybrid compression space (Subsection 4.2). This assessment highlights the partial orthogonality of the two compression criteria (i.e., importance of weights and expressiveness of neurons) and provides both an intuitive understanding and a solid algorithmic basis for hybrid optimizations, and thus enables a more comprehensive exploration of the trade-off between the quality of the pruned architecture and the inherited 'important' weights (*in our opinion, an open and intriguing challenge in the domain of model compression*). To the best of our knowledge, this is the first work to propose hybrid solutions for model compression.

We are more than willing to address any further questions or concerns you might have to ensure that our submission meets the highest quality standards.

[4]. Liu, Zhuang, et al. "Rethinking the value of network pruning." arXiv preprint arXiv:1810.05270 (2018).
Thank you for taking the time and effort to address our initial rebuttal (especially at this time of year). Your insightful comments and suggestions are greatly appreciated and have enabled us to further clarify the motivations of our work and improve the presentation quality of the paper. 

To further address your concerns regarding performance improvement by NEXP (W3), we would like to reiterate some of the key points from the paper, with emphasis on the experimental section, for clarity.

The experimental setup and structure of this work are motivated by our understanding that the pruning criteria for each compression use case scenario may vary significantly, e.g., availability of computational resources, cost of deployment, specifications of the target hardware device, availability of data, target quality and efficiency of the pruned model, and many more. Below, we present some specific (1 and 2) and general (3, 4, and 5) examples to highlight the intricacies of defining the (combinatorial) pruning constraints during the compression process and to demonstrate the effectiveness of NEXP beyond the predictive performance (quality) of the pruned models.

1. DCP employs 400 training epochs for fine-tuning post-pruning to recover predictive performance, compared to our method, which uses only 100 epochs (as outlined in lines 276-277). This may result in a significant increase in operational costs for cloud-based hardware infrastructures. Specifically, given that the pruned models generated by both NEXP and DCP have similar compression sizes and inference speeds, DCP will require $4\\times$ more funds to generate the pruned model compared to NEXP. 

2. HRANK utilises 500 input samples to estimate the average rank, while Network trimming [3] requires over ten thousand input
images to estimate the sparsity of feature maps, compared to ours random selection of 64 for the estimation of NEXP. This results to a $\\approx8\\times$ reduction in dependency on input data compared to HRANK and several orders of magnitude less than [3].

3. NEXP requires only forward passes, unlike gradient-based pruning methods that also require backward passes, which are also intrinsically linked to the availability and quality of data. In contrast, NEXP is independent of both the quality and quantity of samples, as demonstrated in Appendix A and discussed in Section 3.3 (*"Dependency to Input Data"*) of the paper.

4. NEXP maintains consistent performance without a significant drop in computational efficiency as models and tasks grow in complexity. In contrast, weight-based pruning methods incur increased computational overhead as model sizes increase, as discussed in W2 of reviewer 9m28, primarily due to their dependence on the dimensions and cardinality of layer weight matrices. Unlike these methods, NEXP's efficiency is not correlated with model size, demonstrating strong scalability as model complexity increases.

5. NEXP yields consistent estimations across the evolutionary (learning) stages of a neural network, making it a suitable criterion for pruning both untrained and trained networks, as demonstrated in Subsection 4.3 and discussed in Appendix A. To the best of our knowledge, our proposed approach is the first criterion designed for both Pruning at Initialization (PaI) and Pruning after Training (PaT). This stands in contrast to other fundamental principles of pruning metrics in the literature, which are specifically designed to address only one of these challenges and do not extend across the full spectrum of the convergence process.

Next, we also present 3 specific examples from the experimental section to highlight the performance improvement achieved by NEXP:

1. Object Detection with YOLOv8: Our work demonstrates the superiority of NEXP for the more complex task of object detection, yielding significant performance improvements across the entire pruning spectrum compared to all adversary methods (as shown in Figs 2 and 6). Specifically, in lines 302-315, we discuss the superiority of our method using an iterative pruning format to further highlight the intrinsic property of expressiveness in maintaining network elements that are more robust to information redistribution during re-training, in contrast to the 'important' labeled structures identified by other methods. The experimental sections are arranged in this order to address the potential limitations of employing one-shot pruning for NEXP, as demonstrated in the image classification experiments in the previous subsection.

2. NEXP at Initialization: NEXP consistently outperforms all other approaches in top-1 accuracy, particularly in parameters compression regimes with up to $100\\times$ smaller networks.

3. Tables 1 and 2 for image classification: we kindly ask you to refer to part b of this response.

[3]. Hu, Hengyuan, et al. "Network trimming: A data-driven neuron pruning approach towards efficient deep architectures." arXiv preprint arXiv:1607.03250 (2016).
Thank you for taking the time and effort to address our initial rebuttal (especially at this time of year). Your insightful comments and suggestions have given us the opportunity to further clarify the intuition and effectiveness of our proposed approach, while also improving the quality of the paper. A discussion on the **Computational Scalability** of NEXP, as well as a comparison (both theoretical and experimental) against weight-based methods, has been included in the updated version of the paper.

We greatly appreciate your increased score.
I thank the authors' for providing the response to my questions. After reading the rebuttal, I still have concerns towards: 1) The application scenarios towards other model architectures, especially to more latest building blocks such as transformer blocks with more fully connected layers rather than convolutional layers. There is no evidence how the method will perform to these models. 2) The performance improvements. Though the reported accuracy is the base accuracy of the dense model, the performance improvements comparing to baselines are not obvious. For instance, in table 1, ABC only degrades the accuracy by 0.03\% with 2.18$\times$ parameters reduction and 2.18$\times$ FLOPs reduction. However, NEXP degrades the accuracy by 0.41\%, with 2.11$\times$ FLOPs reduction and 2.87$\times$ parameters reduction. 

Thus, I would keep my current rating and thanks the authors' efforts.
I would like to thank the authors for the rebuttal. My concerns are mostly addressed. Thus I increase my score. I would suggest that the authors include the speed evaluation of the pruning method in the updated version.
Thank you for your detailed rebuttal. After reviewing the authors' response and reviews from other reviewers, my primary concern W3 remains unchanged. The experimental evaluation presented in the manuscript demonstrates that while NEXP can be applied to many scenarios, it is often not the best-performing method when compared to the baselines. Or in some cases, the comparisons do not show which method is superior.

Put simply, NEXP may be a good approach, however, the evidence provided does not convincingly show that it offers a significant improvement over existing baselines. Therefore, I do not find sufficient justification to revise my initial rating. As a result, I will be maintaining my original rating.

Thank you again for your efforts.