Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models (2024)

Abstract: Neural network pruning has become increasingly crucial due to the complexity of these models and their widespread use in various fields. Existing pruning algorithms often suffer from limitations such as architecture specificity, excessive complexity and reliance on demanding calculations, rendering them impractical for real-world applications. This paper introduces KEN: a straightforward, universal and unstructured pruning algorithm based on Kernel Density Estimation (KDE). KEN1 aims to construct optimized transformers by selectively preserving the most significant parameters while restoring others to their pre-training state. This strategy preserves model performance while enabling storage of only the optimized subnetwork, leading to substantial memory savings. Extensive evaluations across seven different LLMs demonstrate that KEN achieves equal or better performance than their original unpruned versions, with a minimum parameter reduction of 25%. Furthermore, in-depth comparisons with established pruning and PEFT algorithms confirm KEN effectiveness. We further introduce KENviz , an explainable tool that visualizes the optimized model composition achieved by KEN from different points of view.

heading: 1 Introduction
text: Large Language Models (LLMs) have become the best and simplest solution for achieving state-ofthe-art results in many natural language processing (NLP) applications. However, the increasing use of neural networks (NNs) and transformers (Vaswani et al., 2017) has resulted in a rise in computational cost due to the complexity of arithmetic calculations, larger matrices and the addition of more layers. Consequently, the weight and structure of these models become more complex, requiring high demands in computation and memory.
1Code available at https://github.com/ itsmattei/KEN
One of the best approaches to address the overwhelming size of LLMs is to reduce their resources through pruning algorithms. These algorithms can eliminate parameters or entire components in a NN, making it lighter without compromising its original performance. Pruning algorithms emerged in parallel with the earliest use of NNs (Mozer and Smolensky, 1989; Janowsky, 1989; LeCun et al., 1989), but they have gained significant importance in the last decade due to the widespread use of these networks in various fields. There are many pruning algorithms in literature (Blalock et al., 2020), each with a unique approach or adapted old algorithms for these new architectures (Benbaki et al., 2023). However, the complexity of neural networks can pose a challenge when creating pruning algorithms, as these may require new complex theorems to make the models lightweight (Dong et al., 2017; Malach et al., 2020). Additionally, existing pruning algorithms often exhibit shortcomings in their completeness (Blalock et al., 2020) and fail to consider a critical aspect: the efficient storage of the pruned result. Some algorithms compress models at runtime but lack mechanisms to preserve the reduced NN for future use. Consequently, most algorithms prioritize the speed of reduction and execution, neglecting this critical final stage essential in resource-limited environments (Yang et al., 2017; Sze et al., 2017).
11361
This paper presents KEN (Kernel density Estimator for Neural network compression): a universal, simple, magnitude-based transformer pruning algorithm that leverages Kernel Density Estimation (KDE) for parameter pruning. Unlike other pruning methods that rely on minimizing loss functions or exhaustive parameter search, KEN - inspired by the winning ticket pruning hypothesis (Frankle and Carbin, 2018) - identifies and retains the most influential parameters using KDEs while resetting the others to their original pre-trained values. This innovative approach streamlines the optimization process by leveraging the natural distribution of model parameters, eliminating any architecture-specific considerations. KEN effectively reduces the size of transformer models by a minimum of 25% without compromising performance. The pruned models consist only of a subnetwork of trained parameters, which can be seamlessly downloaded and injected into its pre-trained version as needed. This feature enables dynamic model reconfiguration and saves significant memory space that would otherwise be required to store the fully trained model. Comparative evaluations demonstrate KEN exceptional capabilities, surpassing existing transformer pruning and PEFT algorithms. Finally, we introduce KENviz: an explainable tool that graphically depicts the optimized model from various perspectives. KENviz highlights the KEN-selected parameters, their layerwise differences and neighbor counts for each matrix that made up the analyzed model. Using KEN, we employed a non-parametric method widely used in statistics, to create an efficient and intuitive pruning algorithm. Our approach achieved excellent results in terms of efficiency and performance, making it a practical alternative to other more complex pruning algorithms.

heading: 2 Background
text: Compression algorithms can be summarized in three areas of research: weight pruning (Han et al., 2015; Zhu and Gupta, 2017), quantization (Gong et al., 2014; Zhu et al., 2016) and knowledge distillation (Ba and Caruana, 2014; Kim and Rush, 2016). These techniques aim to make models lighter, but each of them takes a different approach. Weight pruning removes model parameters according to the chosen algorithm and strategy, while quantization reduces the number of bits necessary to represent each parameter. Knowledge distilla-
tion, instead, tries to minimize the learned large knowledge of a model into a smaller one without affecting its validation.
Focusing on pruning algorithms, there are different approaches depending on the strategy and algorithm adopted. Pruning algorithms can be classified as either structured or unstructured, based on the approach applied and magnitude-based or impactbased, according to the algorithm used. Structured pruning (Huang et al., 2018; Wang et al., 2019; Gordon et al., 2020) removes weights in groups, such as entire neurons, filters or layers, while unstructured pruning (Han et al., 2015; Frankle and Carbin, 2018; Lagunas et al., 2021; Benbaki et al., 2023) does not consider any relationship between parameters and selects weights to prune based on their impact or magnitude. Magnitude-based algorithms (Hanson and Pratt, 1988; Mozer and Smolensky, 1989; Gordon et al., 2020) analyze the absolute value of each parameter to determine its importance. In contrast, impact-based algorithms (LeCun et al., 1989; Hassibi and Stork, 1992; Singh and Alistarh, 2020) work on the loss function and its variation caused by removing a parameter. The winning ticket hypothesis (Frankle and Carbin, 2018), is a recent advancement in pruning techniques. A winning ticket is a subnetwork within a trained model that - when trained in isolation - can achieve performance comparable to the original model even after significant pruning. To identify the winning ticket, a pruning criterion is applied to zero-mask weights and the remaining network is retrained. This process can be repeated multiple times or in a one-shot manner.

heading: 3 Related Work
text: In this section, we present three algorithms that are relevant benchmarks for our proposed algorithm, KEN. These algorithms have some similarities with it: the first two, called FLOP and BMP, are pruning algorithms designed to reduce the size of transformer models by employing algebraic or geometric techniques. The third, LoRA is the SoTA parameter-efficient algorithm for LLMs.
Factorized Low-rank Pruning (FLOP: Wang et al., 2019) is a magnitude-based pruning algorithm that employs matrix factorization to reduce the size of matrices in transformer models. This approach involves decomposing each matrix into rank components, which are then multiplied together to form the original matrix. For attention layers,
FLOP decomposes each matrix into smaller rank-1 components based on the magnitudes of the matrix entries. Instead, for embedding layers, FLOP adaptively prunes dimensions based on word clusters. This means that FLOP only prunes dimensions that are not frequently used (Joulin et al., 2017; Bastings et al., 2019), which helps to reduce the model size without sacrificing performance.
Block Movement Pruning (BMP: Lagunas et al., 2021) introduces an extension to the movement pruning technique used in transformers (Sanh et al., 2020). This approach reduces the size of each matrix in a transformer model by dividing it into fixed-sized blocks. Regularization is then applied, and the NN is trained through distillation to match the performance of a teacher model. Our focus is on two pruning methods: Hybrid and HybridNT. The key difference between these two approaches is that HybridNT does not involve the use of a teacher model during training (No Teacher).
Low-Rank Adaptation of Large Language Models (LoRA: Hu et al., 2021) is a novel fine-tuning method that leverages low-rank decomposition to reduce the parameter size of LLMs while preserving their performance. This approach involves decomposing LLMs weight matrices into low-rank components, which are then fine-tuned along with
the original weights. LoRA enables efficient parameter adaptation to specific tasks without compromising the model generalization capabilities.

heading: 4 KEN pruning algorithm
text: KEN (Kernel density Estimator for Neural network compression) pruning algorithm is designed to identify and extract the most essential subnetwork from transformer models following the main idea of the winning ticket hypothesis (Frankle and Carbin, 2018). Our algorithm effectively prunes the network by employing Kernel Density Estimators (KDEs), retaining only the essential parameters and resetting the rest to their pre-trained values. The optimized subnetwork can be stored independently and seamlessly integrated into its pre-trained configuration for downstream applications.
KEN through KDEs, generalizes the point distribution of each transformer matrix, capturing the smoothed version of the original fine-tuned model. To prevent the complete deconstruction of the initial matrix composition, KEN applies KDEs to individual rows. The KDE calculation requires a k value, which defines the number of points employed in the distribution calculation which directly influences the number of retained fine-tuned parameters (Fig. 1). Thus, a lower k value indicates a closer resemblance to the pre-trained model while
a higher k value reflects a closer alignment with its fine-tuned version.
KEN algorithm operates in three primary steps:
Step 1: Parameter Extraction and KDE Calculation Given a pre-trained matrix W 0 of a layer l:
W 0 = {w01,1, ..., w0n,m} | W 0 ∈ Rn×m
and its corresponding fine-tuned counterpart W t:
W t = {wt1,1, ..., wtn,m} | W t ∈ Rn×m
for each row rti of the fine-tuned matrix W t:
rti = {wti,1, ..., wti,m} ∀i ∈ [1, n]
KEN calculates the KDE distribution of the row rti using a bandwidth parameter h determined following Scott’s rule of thumb (Scott, 2015).
h = 1.06 · σ̂ · n− 15
where σ̂ is the standard deviation of rti .
Step 2: Parameter Retention and Pre-trained Value Reset The k points that best fit the rti row distribution are identified using the KDE likelihood, while the others are reset to their pre-trained values. This process results in an optimized row r̂i:
r̂i = {ŵi,1, ..., ŵi,m} ∀i ∈ [1, n] computed using the following binary function:
f(ŵi,j) =
{ wti,j if w t i,j ∈ KDE likelihood
w0i,j otherwise (1)
Step 3: Matrix Replacement and Optimized Fine-tuned Model After applying the previous step on each row, the optimized matrix Ŵ :
Ŵ = {ŵ1,1, ..., ŵn,m} | Ŵ ∈ Rn×m
will replace the original fine-tuned matrix W t within the model. KEN operates iteratively, replacing the W t matrix with Ŵ during each iteration. Therefore, after the t − th iteration, the model will have t− optimized matrices, effectively replacing the fine-tuned matrices without creating any additional versions of the model. This versatility allows KEN to prune the entire model or specific layer ranges.
Algorithm 1: Generate the optimized Ŵ matrix using KEN
Data: W 0 = {w01,1, ..., w0n,m}, W t = {wt1,1, ..., wtn,m}, k Result: Ŵ Ŵ [n,m]← 0 for i = 1 to n do
best_points← KDE(rti , k) for j = 1 to m do
r̂ti ← [] if rti [j] in best_points then
r̂ti [j]← rti [j] else
r̂ti [j]← r0i [j] end
end Ŵ [i]← r̂ti
end return Ŵ
Algorithm 1 provides a more formal explanation of the three steps described for generating the optimized matrix Ŵ . Additionally, the graphical representation in Fig. 2 offers a clear and comprehensive visualization of all KEN steps, while Fig. 3 displays different Ŵ matrices obtained using various k values.

heading: 5 Experiments
text: To validate our algorithm, we conducted a series of extensive case studies. Sec. 5.1 describes the experimental setup, including the models employed and the k values tested. Additionally, Sec. 5.2 focused on investigating the feasibility of saving and loading compressed data.

heading: 5.1 Experimental set-up
text: To evaluate KEN pruning algorithm performance across different architectures and datasets, we conducted a thorough series of experiments using seven distinct transformer models. To maintain consistent evaluation conditions, we uniformly divided each dataset into training, validation and test sets. These divisions remained consistent throughout our experiments and across models. All datasets were imported from Huggingface2. To achieve optimal performance, we fine-tuned each model before applying KEN algorithm per each dataset, adjusting
2https://huggingface.co/datasets
the number of epochs until the fine-tuned model achieved the best F1-weighted score. Despite what the literature suggests, we used the F1 measure instead of classical accuracy as a comparison metric - if not explicitly used by the comparison benchmarks - because it delivers more reliable predictions, particularly on strongly unbalanced datasets.
To fully assess KEN capabilities, we gradually increased the k value required by the algorithm, starting from a low k value and incrementally increasing it until its fine-tuned version was reached. This incremental approach allowed us to identify the critical threshold value whereby the compressed model obtained results similar to its fine-tuned version or when the compression value k leads to a catastrophic decline of performances, as reported in Apx. A.
To provide a comprehensive analysis of KEN, we selected different transformer models with unique architecture, attention mechanisms, training approaches or different versions of the same model. Tab.1 compares the architectures of the models examined, emphasizing the number of layers and the number of parameters of each.

heading: 5.2 Model compression
text: Transformer models, like many neural networks, often have large file sizes. A fine-tuned transformer can range from 500 MB to 2GB or more. However,
the KEN algorithm reduces this size by selecting and retaining a subset of k parameters while restoring the rest to their pre-trained values. This process creates a more concentrated model that only includes the essential k values for each matrix, resulting in significant weight reduction. To quantify the effectiveness of KEN, we save the compressed model generated during this phase and compare it to its original, unpruned version. To ensure a fair comparison, we use the same technique to save both the compressed and original fine-tuned models. However, KEN requires a support file, such as a dictionary, to load the k parameters saved into their appropriate positions during the loading process. This is because during loading, the k fine-tuning values must be loaded into a pre-trained model and the support file provides the necessary mapping to ensure proper placement. Sec. 6.2 provides a comprehensive overview of the compression results obtained during this analysis.

heading: 6 Results and Discussion
text: In this section, we present the results obtained for each KEN main goals. Sec. 6.1 discusses the effectiveness of KEN-pruned models compared to their unpruned counterparts, pruning benchmarks and the state-of-the-art PEFT algorithm. Sec. 6.2 focuses on the process of saving and loading the subnetwork extracted by KEN, comparing the reduced file sizes achieved by it with those of the original models. Finally, Sec. 6.3 shows KENviz, illustrating its applications.

heading: 6.1 Experiment results
text: To evaluate the efficacy of KEN, we conducted a series of experiments across different classification and sentiment analysis datasets. For each dataset, we implemented KEN multiple times, employing
varied k values and calculating the mean and standard deviation of the resulting F1-weighted scores. The complete dataset list can be found in Apx. B. As evidenced in Tab. 2, KEN successfully compressed all analyzed models without sacrificing their original, unpruned performance. We observed a remarkable reduction in overall model parameter count, ranging from a minimum of 25% to a substantial≈ 70% for certain models. Intriguingly, the models with both the highest and lowest parameter counts exhibited the most significant parameter reduction. Additionally, for each model under examination, we observed no substantial difference in performance as the percentage of reset parameters increased, maintaining a remarkable resemblance to the unpruned model performance. This observation underscores KEN exceptional generalization capability, balancing performance and compression even at middle-high compression rates.
We compared KEN to other pruning algorithms specifically designed for transformer models, including FLOP, Hybrid and HybridNT as described in Sec. 3. It is essential to note that Lagunas et al. (2021) models (Hybrid and HybridNT) only prune the attention layers and not the entire model. To facilitate a comprehensive and standardized comparison of all algorithms, we recalibrated the size of their models based on our holistic perspective, ignoring any partial considerations. We combined the results obtained in their publication with those
obtained from KEN and FLOP in Tab. 3. KEN outperformed all other compared models with a significant performance gap while utilizing fewer parameters in every instance. In addition to these findings, we conducted a thorough analysis of FLOP, which is the most complete pruning algorithm studied and, like KEN, decomposes original matrices to derive pruned ones. We conducted additional experiments on all models where FLOP could be applied, using the datasets listed in Tab. 2. We compared the results obtained from FLOP with those of KEN, which employed fewer parameters than FLOP. As shown in Tab. 4, FLOP outperforms KEN in only one instance. For all other models and datasets analyzed, KEN consistently outperforms FLOP.
Although KEN belongs to the winning ticket pruning algorithms family, it shares similarities with Parameter Efficient Fine-tuning (PEFT) algorithms. This is because both approaches aim to identify a subset of optimal parameters within the fine-tuned model. We thoroughly evaluated KEN and compared it to LoRA, which is currently the state-of-the-art PEFT algorithm. We applied LoRA and KEN to the same layers of each model and then trained the LoRA-based models using five times more training epochs than their KEN-based counterparts. Additionally, we gradually increased the number of rank decomposition matrices for each model from 16 to 768, which is the average size of the matrices in the tested models. In each LoRA-based experiment, only the LoRA-specific parameters were designated as either trainable or not. Our results, presented in Fig. 4, demonstrate that KEN consistently outperforms LoRA in terms of F1-measure while utilizing fewer trained parameters. However, when LoRA parameters are not the only ones trained, KEN and LoRA generally produce similar results. Nevertheless, LoRA consistently requires a larger parameter count than KEN. These compelling results provide strong evidence supporting our hypothesis that strategically selecting a subset of parameters and resetting the remainder offers a promising alternative to conventional pruning techniques.

heading: 6.2 Compression values
text: One of KEN main goals, is to significantly reduce the overall size of transformer models, including their file sizes. To achieve this, KEN leverages a
subnetwork comprising only k-trained parameters, allowing it to be saved and then injected into its pretrained counterpart. This process requires a support file, like a dictionary, that specifies the precise location of each saved parameter within the pre-trained model. To guarantee an unbiased comparison between the original and compressed model sizes, the compressed one is saved using identical techniques and format as the original model. For each transformer analyzed, two compressed versions are generated using both high and low k values.
As shown in Tab. 5, both versions of the compressed models demonstrate substantial memory savings, directly proportional to the number of saved parameters. Specifically, transformers saved using a high k value, thus closely mirroring the structure of the unpruned model, conserving ≥≈100 MB per each. This value increases further as the number of trained parameters saved
diminishes. The support dictionary for parameter injection, stored using the Lempel-Ziv-Markov chain data compression algorithm, has a negligible impact on the final model weight, which remains significantly smaller than the original. Additionally, the time required to load the injected parameters into the pre-trained model scales linearly with the transformer architecture and the compression employed.
6.3 KENviz KENviz is a visualization tool that provides a clear understanding of matrices composition after the application of KEN pruning step. It offers various views to explore the pruned model, including:
1. Single Matrix View: This view offers a clear understanding of the parameters retained by KEN, leaving the pruned elements blank (Fig. 3, Fig. 5a).
2. Neighbor Count View: It visualizes the number of non-zero neighbors (horizontally and vertically) for each point in a given matrix. This provides additional information about the remaining parameters and potential parameter clusters that might emerge (Fig. 5b).
3. Layer-wise View: This view shows the impact of KEN on the entire model architecture. It iteratively applies the Single matrix view to the same matrix on all layers it appears. This allows a layer-by-layer comparison, revealing how the pruning of the same matrix changes in different parts of the network.
The examples in Fig. 5 and Apx. C both indicate that the number of non-zero neighbors for each point remains consistently high even in cases with high reset parameters. This suggests that the chosen parameters not only represent the most effective elements but also display a well-proportioned distribution within each matrix.

heading: 7 Conclusions
text: This paper introduces KEN: a novel nonarchitecture-specific pruning algorithm that leverages KDE to construct an abstraction of the parameter distribution and selectively retain a finite subset of them while resetting the others to their pre-trained states. Our extensive evaluations on seven different transformer models demonstrate that KEN consistently achieves remarkable compression rates, reducing unnecessary parameters by a minimum of 25% up to ≈ 70% on some models,
without compromising model performance. Moreover, by leveraging the KEN core idea, is possible to store only the active subnetwork, leading to substantial memory savings.
We also present KENviz: the KEN visualizer that provides insights into the algorithm operation. KENviz reveals that KEN uniformly selects parameters across matrices, preventing parameter clusters. With KEN we demonstrate how a simple, non-parametric algorithm commonly used in statistics, can be effectively adopted for model pruning achieving excellent results in both compression and performance.

heading: 8 Limitations
text: One of the major limitations of KEN is its computational efficiency, especially when analyzing large models. Although KEN excels at generating rich distributions with high k values, it faces a trade-off in computational efficiency, particularly for large models. Processing time scales linearly with the model size, number of layers, and chosen k. This primarily impacts the parameter selection stage, not compressed model storage or loading.
Our focus on the sequence classification task ensured consistent results, but unpublished experiments suggest KEN effectiveness extends to other tasks. Future work will explore this broader applicability and investigate potential optimizations for large-scale deployments.

heading: A How to prove the importance of selected parameters
text: To assess the effectiveness of KEN core idea, which involves selecting parameters based on their distribution using Kernel Density Estimation (KDE), we conducted parallel experiments. In these experiments, we compared KEN against random parameter pruning. Random parameters were either retained or reset to pre-trained values. This enabled us to determine if the selection method used in KEN resulted in a better subnetwork compared to random selection
Formally, for each matrix in a generic model, the optimized matrix Ŵ contained k randomly selected fine-tuned parameters. Our goal is to determine whether the parameters introduced into a generic transformer model by KEN constituted an optimal subnetwork or if equivalent results could be achieved by randomly selecting the same number of parameters. To address this question, we performed an experiment using the AG-NEWS dataset, comparing the performance differences between extracting Ŵ matrices using KEN and using k random values for each matrix row.
The results, illustrated in Fig. 6, consistently show that KEN outperforms its random counterpart. KEN achieves a lower error rate and maintains higher performance at reasonable compression levels. However, for all models tested, a compression threshold exists beyond which performance inevitably declines. The KEN algorithm effectively compresses models while minimizing this decline, preserving high performance. Conversely, random parameter selection reaches this threshold earlier, resulting in a larger performance drop and higher er-
ror rate. Notably, the best achievable performance with random selection is always lower than or equal to the average achieved with KEN.
Furthermore, when using KEN, the error rate remains minimal within the threshold. This suggests that the subnetwork derived from KEN is not random; rather, it consistently selects the most effective portion of the original network.

heading: B Additional results
text: This appendix presents additional results obtained using KEN that are not included in Tab. 2.
Tab. 7 provides a comprehensive overview of all datasets analyzed in the paper. In contrast to Tab. 2, Tab. 6 focuses on the specific results included in this appendix. Here, we highlight cases where KEN achieves F1-weighted scores that match or surpass the original unpruned model performance, focusing on the highest percentage of reset parameters for each model on each dataset. This emphasizes KEN remarkable compression capabilities, allowing it to achieve comparable or even improved performance while significantly reducing the model parameter count.
C KENviz examples
KENviz generates visual representations of the model pruning after the KEN application. Here, we focus on key matrices from layers 0 and 12 of a BERT model trained on the glue-sst2 dataset (details in Sec. 6.1). For each layer, we present both a single matrix view and a neighbor count view, as described in Sec. 6.3.
BERT was chosen for this experiment due to its exceptional performance across a range of k values during testing (Tab. 2 and Tab. 6). To comprehensively explore how parameter selection patterns evolve, we employed three different k values, representing varying degrees of parameter selection. This allowed us to observe how parameter choices shift as the amount of parameter resetting increases.
Fig. 7 and Fig. 8 consistently reveal a uniform distribution of parameters within each matrix row across all configurations and layers. This implies an absence of well-defined clusters of selected parameters. Furthermore, the number of neighbors for each parameter remains consistent regardless of the chosen k value.

heading: Future Work
text: Future work will explore this broader applicability and investigate potential optimizations for large-scale deployments