One-for-All Pruning: A Universal Model for Customized Compression of Large Language Models
Authors: Rongguang Ye, Ming Tang
Accepted at: acl25

One-for-All Pruning: A Universal Model for Customized Compression of Large Language Models

Rongguang Ye, Ming Tang*
Department of Computer Science and Engineering
and the Research Institute of Trustworthy Autonomous Systems
Southern University of Science and Technology, Shenzhen, China
yerg2023@mail.sustech.edu.cn, tangm3@sustech.edu.cn

Abstract

Existing pruning methods for large language models (LLMs) focus on achieving high compression rates while maintaining model performance. Although these methods have demonstrated satisfactory performance in handling a single user’s compression request, their processing time increases linearly with the number of requests, making them inefficient for real-world scenarios with multiple simultaneous requests. To address this limitation, we propose a Universal Model for Customized Compression (UniCuCo) for LLMs, which introduces a StratNet that learns to map arbitrary requests to their optimal pruning strategy. The challenge in training StratNet lies in the high computational cost of evaluating pruning strategies and the non-differentiable nature of the pruning process, which hinders gradient backpropagation for StratNet updates. To overcome these challenges, we leverage a Gaussian process to approximate the evaluation process. Since the gradient of the Gaussian process is computable, we can use it to approximate the gradient of the non-differentiable pruning process, thereby enabling StratNet updates. Experimental results show that UniCuCo is 28 times faster than baselines in processing 64 requests, while maintaining comparable accuracy to baselines.

1 Introduction

Large language model (LLM) compression (Ma et al., 2023; Zhu et al., 2024; Yu et al., 2024) aims to reduce the size and computational demands of pre-trained models while preserving their performance. Among commonly used techniques, pruning (Frantar and Alistarh, 2023; Yin et al., 2023) reduces the size and complexity of pre-trained models by removing less critical weights or layers while retaining their core functionality. By employing these techniques, LLMs can be deployed efficiently in resource-constrained environments, such as edge devices (Tseng et al., 2024).

In practical applications, users have diverse compression requests (defined by their goals of model size reduction while preserving performance) due to the varying capabilities of their devices. Numerous LLM pruning methods have been developed to address specific compression requests (Yin et al., 2023; Kim et al., 2024). These methods can be categorized into optimization-based approaches (Sieberling et al., 2024) and score-based approaches (Men et al., 2024; Frantar and Alistarh, 2023). Optimization-based approaches frame LLM pruning as an optimization problem, utilizing heuristic algorithms (Yu and Gen, 2010) (e.g., evolutionary algorithms) to preserve the performance of the pruned LLM under a specific compression request. In Fig. 1, optimization-based approaches are highly effective in maintaining performance through iterative refinement of pruning strategies during the search process. However, their runtime efficiency in handling multiple compression requests is significantly limited, as each request necessitates an independent heuristic search, leading

![A comparison of various approaches in terms of effectiveness and efficiency when providing pruning strategies for compression requests.](page_728_355_670_377.png)

Figure 1: A comparison of various approaches in terms of effectiveness and efficiency when providing pruning strategies for compression requests.
to substantial time overhead. In contrast, score-based approaches calculate importance or sensitivity scores for each LLM layer, which are then used to determine the layers to prune to meet compression requests. By reusing the computed scores, the efficiency of handling multiple compression requests is enhanced. However, score-based approaches exhibit limited effectiveness in preserving performance, since the pruning strategies generated from importance scores fail to satisfy the monotonicity property (Sieberling et al., 2024). Motivated by these findings, we pose the following research question: How can we effectively and efficiently handle multiple compression requests?

To address this problem, we propose Request-Conditional Pruning (UniCuCo) for handling multiple compression requests simultaneously. Specifically, we introduce a StratNet that maps an arbitrary compression request to its corresponding optimal pruning strategy, enabling the handling of diverse requests. It is applicable for StratNet to a wide range of pruning approaches, such as depth pruning and non-uniform pruning. The challenges in training the StratNet are twofold: First, it is difficult to balance the reduction in model size with the preservation of performance when optimizing StratNet to optimally match the user’s request. Second, evaluating pruning strategies is computationally expensive. Third, applying pruning strategies (such as binary masks) to the LLM is a non-differentiable operation, which disrupts the backpropagation process and prevents StratNet from being updated using gradient-based methods. To address these challenges, we introduce the weighted Tchebycheff function in the optimization of StratNet, enabling it to effectively derive a pruning strategy that optimally aligns with the given request. Furthermore, we introduce a Gaussian process estimator to evaluate pruning strategies, significantly reducing evaluation time. Since the gradient of the Gaussian process is computable, we leverage it to restore the parts of StratNet that are disrupted in the backpropagation process. Notably, we propose an alternating update scheme where the Gaussian process and StratNet are updated in an interleaved manner. The main contributions of this paper are as follows:

• We introduce the problem of multiple request pruning for LLMs, which requires algorithms to efficiently and effectively generate pruning strategies tailored to diverse requests.

• We propose UniCuCo, a framework that maps arbitrary compression requests to tailored pruning strategies. UniCuCo includes a Gaussian process estimator, which significantly reduces evaluation time of pruning strategies and subtly solves the non-differentiable issue in UniCuCo. Additionally, we introduce an alternating scheme for updating the Gaussian process and StratNet.

• Experimental results show that UniCuCo processes 64 compression requests on the Mistral-7B model with a speed at least 28 times faster than optimization-based approaches, while maintaining comparable accuracy. Meanwhile, UniCuCo achieves an average accuracy improvement of 3% across five benchmark datasets in a non-uniform pruning scenario with 70% sparsity when compared with score-based approaches on the Mistral-7B model.

2 Related Works

2.1 Depth Pruning

Depth pruning treats each transformer block as a unit and removes entire blocks for pruning. The most common approaches are score-based methods, which compute block importance scores and remove those with lower scores based on a compression request. For example, Weight Subcloning (Samragh et al., 2023) is a simple yet effective technique that transfers pre-trained model knowledge to smaller variants by evaluating block importance using the ratio of \( \ell_2 \) norms between output embeddings with and without residual connections. Shortened LLaMA (Kim et al., 2024) measures block contribution by removing each block from a pre-trained model and assessing its impact on perplexity. ShortGPT (Men et al., 2024) determines importance through cosine similarity between block inputs and outputs, where lower similarity indicates higher importance. Gromov et al. (Gromov et al., 2024) groups consecutive blocks and evaluates their importance using cosine similarity. However, according to EvoPress (Sieberling et al., 2024), score-based approaches in depth pruning are not monotonic. That is, a pruned LLM with a higher cumulative importance score does not necessarily lead to higher effectiveness in preserving performance. To address this limitation, several optimization-based approaches have been proposed for depth pruning. For example, Sheared-LLaMA (Xia et al.) introduces a mask learning phase to identify prunable components across blocks.
2.2 Non-Uniform Pruning

Non-uniform pruning is a more fine-grained pruning scenario, where each transformer block is assigned a sparsity value between 0 and 1, rather than being simply set to 0 or 1. Score-based approaches in non-uniform pruning include Wanda (Sun et al.) and SparseGPT (Frantar and Alistarh, 2023). Wandb evaluates weight importance by assessing their impact on the calibration dataset. Specifically, it computes the dot product between the absolute value of the parameter matrix and the \( \ell_2 \) norm of the input to calculate the weight importance score. SparseGPT computes the Hessian matrix for the weights within each block and generates the corresponding mask matrix based on the compression request. For optimization-based approaches, He et al. (He et al., 2018) and Ashok et al. (Ashok et al., 2018) employed reinforcement learning to guide the LLM compression process. However, these approaches are hindered by high computational complexity, leading to significant time overhead when processing a single compression request. To mitigate this issue, the recent OWL method (Yin et al., 2023) improves compression efficiency by pruning LLMs using layer-wise sparsity ratios proportional to their activation outlier ratios. EvoPress (Sieberling et al., 2024) formulates compression requests as constraints and employs heuristic search to determine the sparsity of each transformer block. Once the sparsity for each block is determined by EvoPress, SparseGPT (Frantar and Alistarh, 2023) is applied to perform the sparsification of the LLM. Despite these advancements, such methods still incur significant time costs, which are proportional to the number of compression requests. Our work aims to efficiently handle multiple requests while preserving the effectiveness of the pruned LLM.

3 Methodology

In this section, we introduce UniCuCo for customized compression of LLMs. Fig. 2 illustrates the flowchart of UniCuCo. First, the core idea is to introduce a StratNet that learns to map arbitrary requests to corresponding pruning strategies (Section 3.1). However, updating the StratNet requires evaluating a large number of pruning strategies on a calibration dataset, which is a time-consuming process. Thus, we propose the use of a Gaussian process to estimate the evaluation process, thereby reducing the computational overhead of evaluating pruning strategies (Section 3.2). Finally, we introduce methods for updating StratNet and the Gaussian process (Section 3.3).

3.1 UniCuCo Framework

We consider a cloud server that provides pruning strategies \( x \in \mathbb{R}^d \) for diverse compression requests, where \( d \) is the number of LLM blocks. Each element \( x_i \in [0, 1] \) represents the sparsity ratio of the \( i \)-th block. If \( x_i \) is binary, it corresponds to depth pruning, whereas if \( x_i \) takes a value in the continuous range \([0, 1]\), it corresponds to non-uniform pruning. Subsequently, we define the compression request and incorporate it into the pruning optimization task. Then, we introduce StratNet and discuss its optimization method.

3.1.1 Request Formulation

Each compression request is associated with model size reduction and performance preservation. To learn the pruning strategy for any given request (i.e., the Pareto front corresponding to all requests), we propose representing compression requests using \( \lambda \in \mathbb{R}_+^2 \), where \( \lambda_1 + \lambda_2 = 1 \). Each \( \lambda_i \) is used to balance the trade-off between model size reduction and performance preservation. The set of all requests over these objectives defines the request space \( \Lambda = \{ \lambda \in \mathbb{R}_+^2 \mid \sum_{i=1}^2 \lambda_i = 1 \} \).

Given pruning strategy \( x \), we quantify the model size reduction objective as:

\[
\min_x f_1(x) = 1 - \frac{\sum_{i=1}^d x_i}{d},
\]

where \( \frac{\sum_{i=1}^d x_i}{d} \) represents the average sparsity of
the pruned LLM. A smaller value of \( f_1(x) \) leads to a smaller model size, as it reflects a higher sparsity.

Following (Sieberling et al., 2024), we adopt the KL-divergence between the outputs of the pruned and unpruned LLM to characterize performance preservation:

\[
\min_x f_2(x) = \mathcal{D}_{\mathrm{KL}}(P_{\mathcal{M}_x} \| P_{\mathcal{M}}),
\]

where \( P_{\mathcal{M}} \) denotes the output distribution of the unpruned LLM, while \( P_{\mathcal{M}_x} \) is the output distribution of the pruned LLM determined by the pruning strategy \( x \). \( f_2 \) quantifies the discrepancy in model outputs on the calibration dataset \( \mathcal{D} \), with smaller values indicating better performance preservation.

3.1.2 StratNet
We propose a StratNet \( \phi_\theta \), parameterized by \( \theta \), that maps the request \( \lambda \) to the corresponding pruning strategy \( x \), as follows:

\[
x = \phi_\theta(\lambda).
\]

We consider optimizing StratNet with respect to two objectives, \( f_1 \) and \( f_2 \). Thus, the optimization of StratNet is formulated as a bi-objective optimization problem: \( \min_\theta [f_1(x), f_2(x)] \). A straightforward approach to solving this problem is to compute a weighted sum of \( f_1 \) and \( f_2 \):

\[
\min_\theta g_{ws}(x \mid \lambda) = \min_\theta \sum_{i=1}^2 \lambda_i f_i(x).
\]

As shown in Fig. 3, \( g_{ws}(\cdot) \) is applicable only to convex Pareto fronts, and not to concave ones. To overcome this limitation, we propose the following weighted Tchebycheff function (Miettinen, 1999):

\[
\min_\theta g_{tch}(x \mid \lambda) = \min_\theta \max_{i \in [2]} \{ \lambda_i (f_i(x) - z_i^*) \},
\]

where \( z_i^* \) is ideal value for objective \( f_i \) (i.e., the lower bound of \( f_i \)). The function \( g_{tch}(\cdot) \) can be used to identify the optimal pruned strategy in different kinds of Pareto front.

To enable the StratNet to learn pruning strategies for all possible requests, we optimize StratNet over the entire request space \( \Lambda \) as follows:

\[
\min_\theta \mathbb{E}_{\lambda \sim \Lambda} [g_{tch}(x = \phi_\theta(\lambda) \mid \lambda)] .
\]

For each sampled request \( \lambda \) in Eq. (6), \( f_1(x = \phi_\theta(\lambda)) \) in \( g_{tch} \) can be directly computed by Eq. (1). In contrast, \( f_2(x = \phi_\theta(\lambda)) \) cannot be directly computed solely from \( x \). Its computation requires generating the compressed model based on \( x \) and performing inference on a calibration dataset to derive its value. This introduces two key challenges.
(I) **Significant computational overhead**. For each sampled compression request \( \lambda \), evaluating \( f_2 \) with \( g_{tch}(\cdot) \) is time-consuming, as it requires the pruned LLM to perform inferences on the calibration dataset.
(II) **The feasibility of updating the StratNet**. The computation of the gradient \( \nabla_\theta g_{tch} \) is necessary for updating the StratNet. However, computing \( \nabla_\theta g_{tch} \) is challenging because \( \nabla_\theta f_2 \) in \( \nabla_\theta g_{tch} \) involves the following chain rule:

\[
\nabla_\theta f_2(x) = \nabla_\theta \phi_\theta(\lambda) \cdot \nabla_x \mathcal{M}_x \cdot \nabla_{\mathcal{M}_x} f_2(x),
\]

where \( \nabla_x \mathcal{M}_x \) cannot be computed, as the mapping from pruning strategy \( x \) to a pruned model \( \mathcal{M}_x \) is a non-differentiable operation. It disrupts the computational chain for Eq. (7).

3.2 Gaussian Process for Efficient Estimation
We introduce a Gaussian process denoted by \( \mathcal{G} \), to compute \( f_2(x) \) when solving problem (6). The key idea is to use \( \mathcal{G} \) as an estimator to approximate \( f_2(x) \) without requiring expensive inference on the calibration dataset for new pruning strategies. This Gaussian process can address challenges (I)-(II). We begin by presenting the Gaussian process and integrating it into our framework.

The Gaussian process \( \mathcal{G} \) is defined by a mean function \( \mu(\cdot) \) and a covariance function \( k(\cdot, \cdot) \):

\[
\hat{f}_2 \sim \mathcal{G}(\mu(x), k(x, x)).
\]

Initially, we construct a set of \( C \) observed samples \( \{x^{(c)}, f_2(x^{(c)})\}_{c=1}^C \). These samples are used to train the Gaussian process, i.e., to estimate the mean function and covariance function.

Once \( \mathcal{G} \) is trained, it can be used for *inference* on new pruning strategies. When a new pruning strategy \( x^{new} = \phi_\theta(\lambda^{new}) \) is generated by StratNet

![The optimal pruning strategy x obtained using (a) the weighted sum function and (b) the weighted Tchebycheff function under a concave Pareto front.](page_1012_180_388_246.png)

Figure 3: The optimal pruning strategy \( x \) obtained using (a) the weighted sum function and (b) the weighted Tchebycheff function under a concave Pareto front.
\( \phi_\theta \) in problem (6), \( \mathcal{G} \) computes the posterior mean \( \hat{\mu}(\mathbf{x}^{new}) \) and variance \( \hat{\sigma}^2(\mathbf{x}^{new}) \) for \( \mathbf{x}^{new} \). The posterior mean \( \hat{\mu}(\mathbf{x}^{new}) \) serves as an estimate of \( f_2(\mathbf{x}^{new}) \), while the variance \( \hat{\sigma}^2(\mathbf{x}^{new}) \) quantifies the uncertainty of this estimate. To balance exploration (trying uncertain strategies) and exploitation (focusing on good predicted performance), we incorporate uncertainty into the estimation of \( f_2 \) for \( \mathbf{x}^{new} \) by employing criteria such as the Lower Confidence Bound (LCB) or Upper Confidence Bound (UCB). Specifically, when applying LCB, the estimate of \( f_2 \) for \( \mathbf{x}^{new} \) is given by
\[
\hat{f}_2(\mathbf{x}^{new}) = \hat{\mu}(\mathbf{x}^{new}) + \kappa \hat{\sigma}(\mathbf{x}^{new}),
\]
where \( \kappa \geq 0 \) is a constant used to balance posterior mean and uncertainty. Using Gaussian process to estimate \( f_2 \) significantly reduces the computation time from tens of seconds on the calibration dataset to just tens of milliseconds, achieving a thousandfold improvement in computational efficiency and addressing challenge (I).

Meanwhile, the Gaussian process can effectively tackles challenge (II). The existence of Gaussian process ensures that \( \nabla_{\theta} f_2(\mathbf{x}) \) can be estimated:
\[
\nabla_{\theta} \hat{f}_2(\mathbf{x}) \approx \nabla_{\theta} \phi_\theta(\mathbf{\lambda}) \cdot \nabla_{\mathbf{x}} \mathcal{G},
\]
where \( \nabla_{\mathbf{x}} \mathcal{G} \) estimates \( \nabla_{\mathbf{x}} f_2(\mathbf{x}) = \nabla_{\mathbf{x}} \mathcal{M}_x \cdot \nabla_{\mathcal{M}_x} f_2(\mathbf{x}) \). Then, when StratNet is updated, its gradient \( \nabla_{\theta} g_{tch} \) can be estimated by
\[
\nabla_{\theta} g_{tch}(\mathbf{x} \mid \mathbf{\lambda}) =
\begin{cases}
\lambda_1 \nabla_{\theta} f_1(\mathbf{x}) & \text{if } \lambda_1 f_1(\mathbf{x}) \geq \lambda_2 \hat{f}_2(\mathbf{x}), \\
\lambda_2 \nabla_{\theta} \hat{f}_2(\mathbf{x}) & \text{if } \lambda_1 f_1(\mathbf{x}) < \lambda_2 \hat{f}_2(\mathbf{x}).
\end{cases}
\]
In the above equation, when \( \lambda_1 f_1(\mathbf{x}) = \lambda_2 \hat{f}_2(\mathbf{x}) \), a subgradient is employed, given by \( \lambda_1 \nabla_{\theta} f_1(\mathbf{x}) \).

3.3 Updating Gaussian Process and StratNet
The accuracy of the Gaussian process in estimating \( f_2 \) is crucial for optimizing StratNet. We present StratNet’s update method and a dynamic update approach for the Gaussian process. The Gaussian process is updated once per epoch while StratNet undergoes \( I \) updates per epoch. Their updates are divided into steps (A) and (B).

(A) Initializing Gaussian process and optimizing StratNet. We randomly initialize a set of \( N \) pruning strategies \( X^0 = \{ \mathbf{x}^j \}_{j=1}^N \). Next, we prune the LLM based on \( X^0 \) and compute the corresponding \( f_1 \) and \( f_2 \) values for each pruned LLM. Here, \( f_2 \) is evaluated on a calibration dataset, while \( f_1 \) measures the model size reduction. Together, these values form the set \( F^0 = \{ (f_1(\mathbf{x}^j), f_2(\mathbf{x}^j)) \}_{j=1}^N \). Finally, we train the Gaussian process \( \mathcal{G}^0 \) on the pairs \( \{ (\mathbf{x}^j, f_2(\mathbf{x}^j)) \}_{j=1}^N \) by maximizing the marginal likelihood (Rasmussen, 2003).

To optimize StratNet at the first epoch, we apply Monte Carlo sampling to estimate the expectation of requests in Eq. (11) and then use gradient descent with \( I \) steps for optimization:

\[
\theta \leftarrow \theta - \eta \sum_{k=1}^K \nabla_{\theta} g_{tch}(\mathbf{x} = \phi_\theta(\mathbf{\lambda}^k) \mid \mathbf{\lambda}^k),
\]
where \( K \) is the number of sampled requests and \( \eta \) is the learning rate.

(B) Incremental Gaussian process update and continuous StratNet update. During the \( t \)-th epoch, stage (B) selects new samples to expand the training dataset \( \{ \mathbf{X}^{t-1}, \mathbf{F}^{t-1} \} \) in epoch \( t-1 \) of the Gaussian process, thereby enhancing its prediction accuracy. Then, the Gaussian process updates on the increased training dataset, while StratNet is updated according to Eq. (11).

To increase the training dataset, we first generate a strategy candidate pool \( X_p^t \) based on StratNet. To do this, we sample a set of \( C \) requests \( \{ \mathbf{\lambda}^c \}_{c=1}^C \) from the request space \( \Lambda \). The StratNet then maps these vectors \( \{ \mathbf{\lambda}^c \}_{c=1}^C \) into the candidate pool, i.e., \( X_p^t = \{ \mathbf{x}^c = \phi_\theta(\mathbf{\lambda}^c) \}_{c=1}^C \). Next, we compute the \( f_1 \) of \( X_p^t \), and use the Gaussian process to predict the \( f_2 \) of \( X_p^t \), resulting in the objective values \( \hat{\mathbf{F}}_p^t = (f_1(X_p^t), \hat{f}_2(X_p^t)) \).

To select a subset from candidate set \( \{ X_p^t, \hat{\mathbf{F}}_p^t \} \) that provides the maximum benefit to Gaussian process training, we use hypervolume (HV) (Guerreiro et al., 2021) to assess the quality of the objective set \( \mathbf{F} \). HV is calculated by the area enclosed between each point in \( \mathbf{F} \) and a predefined reference point \( r \):

\[
\mathcal{H}_r(\mathbf{F}) = \{ a \in \mathbb{R}^2 \mid \exists \mathbf{f} \in \mathbf{F}, \mathbf{f} \leq a \leq r \},
\]
where a larger value of \( \mathcal{H}_r(\mathbf{F}) \) reflects a higher quality of the set \( \mathbf{F} \). The key in improving the Gaussian process lies in measuring the improvement brought by adding the selected subset to the \( (t-1) \)-th epoch training dataset \( \{ \mathbf{X}^{t-1}, \mathbf{F}^{t-1} \} \). Thus, we identify a subset \( \{ \mathbf{X}_s^t, \hat{\mathbf{F}}_s^t \} \) from \( \{ X_p^t, \hat{\mathbf{F}}_p^t \} \) with the largest hypervolume improvement (\( \mathcal{HI} \)) for \( \mathcal{H}_r(\mathbf{F}^{t-1}) \), as follows:

\[
\mathcal{HI}(\hat{\mathbf{F}}_s^t) = \mathcal{H}_r(\mathbf{F}^{t-1} \cup \hat{\mathbf{F}}_s^t) - \mathcal{H}_r(\mathbf{F}^{t-1}).
\]

Based on Eq. (13), \( \mathbf{X}_s^t = \arg \max_{\mathbf{X}_s^t} \mathcal{HI}(\hat{\mathbf{F}}_s^t) \). To add the selected \( \mathbf{X}_s^t \) to the \( (t-1) \)-th epoch
<table>
  <tr>
    <th rowspan="2">Sparsity</th>
    <th rowspan="2">Method</th>
    <th colspan="5">Mistral-7B</th>
    <th colspan="5">Llama-3-8B</th>
  </tr>
  <tr>
    <th>Wiki2 (J)</th>
    <th>C4 (J)</th>
    <th>FW (J)</th>
    <th>Avg. (J)</th>
    <th>Latency (s)</th>
    <th>Wiki2 (J)</th>
    <th>C4 (J)</th>
    <th>FW (J)</th>
    <th>Avg. (J)</th>
    <th>Latency (s)</th>
  </tr>
  <tr>
    <td>0%</td>
    <td>Dense</td>
    <td>4.82</td>
    <td>7.72</td>
    <td>6.41</td>
    <td>6.32</td>
    <td>–</td>
    <td>5.54</td>
    <td>8.80</td>
    <td>7.62</td>
    <td>7.32</td>
    <td>–</td>
  </tr>
  <tr>
    <td rowspan="5">12.5%</td>
    <td>Cosine (Window)</td>
    <td>7.19</td>
    <td>10.18</td>
    <td>8.39</td>
    <td>8.59</td>
    <td>8s</td>
    <td>13.21</td>
    <td>19.56</td>
    <td>14.27</td>
    <td>15.68</td>
    <td>9s</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>5.74</b></td>
    <td><b>9.07</b></td>
    <td><b>7.43</b></td>
    <td><b>7.41</b></td>
    <td>26.7m</td>
    <td>7.68</td>
    <td>12.33</td>
    <td>10.20</td>
    <td>10.07</td>
    <td>15.4m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>7.19</td>
    <td>10.18</td>
    <td>8.39</td>
    <td>8.59</td>
    <td>&lt;1s</td>
    <td>13.21</td>
    <td>19.56</td>
    <td>14.27</td>
    <td>15.68</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>7.19</td>
    <td>10.18</td>
    <td>8.39</td>
    <td>8.59</td>
    <td>&lt;1s</td>
    <td>13.21</td>
    <td>19.56</td>
    <td>14.27</td>
    <td>15.68</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>5.93</td>
    <td>9.39</td>
    <td>7.67</td>
    <td>7.66</td>
    <td>&lt;1s</td>
    <td><b>7.42</b></td>
    <td><b>12.03</b></td>
    <td><b>9.80</b></td>
    <td><b>9.75</b></td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="5">25%</td>
    <td>Cosine (Window)</td>
    <td>34.94</td>
    <td>33.7</td>
    <td>15.08</td>
    <td>27.91</td>
    <td>15s</td>
    <td>5527.47</td>
    <td>11588.16</td>
    <td>2388.11</td>
    <td>6501.25</td>
    <td>14s</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>10.35</b></td>
    <td><b>13.44</b></td>
    <td><b>10.63</b></td>
    <td><b>11.47</b></td>
    <td>26.2m</td>
    <td><b>14.77</b></td>
    <td>21.30</td>
    <td>16.74</td>
    <td>17.60</td>
    <td>19.6m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>43.26</td>
    <td>40.16</td>
    <td>29.29</td>
    <td>37.57</td>
    <td>&lt;1s</td>
    <td>5527.47</td>
    <td>11588.16</td>
    <td>2388.11</td>
    <td>6501.25</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>43.26</td>
    <td>40.16</td>
    <td>29.29</td>
    <td>37.57</td>
    <td>&lt;1s</td>
    <td>5527.47</td>
    <td>11588.16</td>
    <td>2388.11</td>
    <td>6501.25</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>13.73</td>
    <td>17.2</td>
    <td>13.53</td>
    <td>14.82</td>
    <td>&lt;1s</td>
    <td>15.05</td>
    <td><b>20.78</b></td>
    <td><b>16.18</b></td>
    <td><b>17.34</b></td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="5">37.5%</td>
    <td>Cosine (Window)</td>
    <td>1038.98</td>
    <td>2362</td>
    <td>1013.9</td>
    <td>1471.62</td>
    <td>15s</td>
    <td>64402.73</td>
    <td>13833.98</td>
    <td>3908.76</td>
    <td>27381.82</td>
    <td>17s</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>31.91</b></td>
    <td><b>30.86</b></td>
    <td><b>22.47</b></td>
    <td><b>28.41</b></td>
    <td>25.7m</td>
    <td><b>66.21</b></td>
    <td><b>80.48</b></td>
    <td><b>53.82</b></td>
    <td><b>66.84</b></td>
    <td>15.4m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>2899.74</td>
    <td>2327.1</td>
    <td>1023.7</td>
    <td>2083.50</td>
    <td>&lt;1s</td>
    <td>64402.73</td>
    <td>13833.98</td>
    <td>3908.76</td>
    <td>27381.82</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>2899.74</td>
    <td>2327.1</td>
    <td>1023.7</td>
    <td>2083.50</td>
    <td>&lt;1s</td>
    <td>64402.73</td>
    <td>13833.98</td>
    <td>3908.76</td>
    <td>27381.82</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>42.48</td>
    <td>36.8</td>
    <td>25.34</td>
    <td>34.87</td>
    <td>&lt;1s</td>
    <td>76.89</td>
    <td>98.54</td>
    <td>55.86</td>
    <td>77.10</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="5">50%</td>
    <td>Cosine (Window)</td>
    <td>3410.85</td>
    <td>1950.6</td>
    <td>1695.4</td>
    <td>2352.29</td>
    <td>10s</td>
    <td>2054.46</td>
    <td>1116.51</td>
    <td>692.89</td>
    <td>1287.95</td>
    <td>12s</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td>4148.65</td>
    <td>2943.6</td>
    <td>2937.8</td>
    <td>3343.32</td>
    <td>26.3m</td>
    <td><b>496.86</b></td>
    <td><b>396.78</b></td>
    <td><b>261.37</b></td>
    <td><b>385.00</b></td>
    <td>13.0m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>2423.38</td>
    <td>2135.4</td>
    <td>1104.9</td>
    <td>1887.89</td>
    <td>&lt;1s</td>
    <td>1664.06</td>
    <td>1739.99</td>
    <td>1622.69</td>
    <td>1675.58</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>2423.38</td>
    <td>2135.4</td>
    <td>1104.9</td>
    <td>1887.89</td>
    <td>&lt;1s</td>
    <td>1664.06</td>
    <td>1739.99</td>
    <td>1622.69</td>
    <td>1675.58</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td><b>235.08</b></td>
    <td><b>148.85</b></td>
    <td><b>120.33</b></td>
    <td><b>168.09</b></td>
    <td>&lt;1s</td>
    <td>983.97</td>
    <td>632.06</td>
    <td>447.78</td>
    <td>687.77</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="5">62.5%</td>
    <td>Cosine (Window)</td>
    <td>8663.29</td>
    <td>7568.5</td>
    <td>8644.3</td>
    <td>8292.01</td>
    <td>8s</td>
    <td>6552.93</td>
    <td>2756.67</td>
    <td>2839.64</td>
    <td>4049.75</td>
    <td>9s</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td>3629.51</td>
    <td>3091.1</td>
    <td>2597.9</td>
    <td>3088.33</td>
    <td>28.1m</td>
    <td><b>4711.99</b></td>
    <td><b>4041.00</b></td>
    <td>4036.07</td>
    <td>4263.02</td>
    <td>15.2m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>12539.6</td>
    <td>10536</td>
    <td>4755.3</td>
    <td>9276.92</td>
    <td>&lt;1s</td>
    <td>56522.08</td>
    <td>23863.46</td>
    <td>12350.08</td>
    <td>30911.87</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>12539.6</td>
    <td>10536</td>
    <td>4755.3</td>
    <td>9276.92</td>
    <td>&lt;1s</td>
    <td>56522.08</td>
    <td>23863.46</td>
    <td>12350.08</td>
    <td>30911.87</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td><b>1846.28</b></td>
    <td><b>1170.4</b></td>
    <td><b>971.79</b></td>
    <td><b>1329.49</b></td>
    <td>&lt;1s</td>
    <td>8405.46</td>
    <td><b>2173.26</b></td>
    <td><b>1845.45</b></td>
    <td><b>4181.39</b></td>
    <td>&lt;1s</td>
  </tr>
</table>

Table 1: Depth pruning results of various methods across five sparsity levels, evaluated by perplexity (PPL) and averaged PPL. Latency refers to the time required to handle a single compression request. The best results are highlighted in bold, while the second-best results are underlined.

training dataset \( \{ X^{t-1}, F^{t-1} \} \) of the Gaussian process, we need to evaluate \( X_s^t \) on the calibration dataset, as \( f_2 \) of \( X_s^t \) is still estimated by the Gaussian process. To do this, we obtain the pruned LLMs based on \( X_s^t \), and compute their corresponding \( f_1 \) and \( f_2 \) values, forming \( F_s^t \). The training dataset at t-epoch is represented as \( \{ X^t, F^t \} = \{ X^{t-1} \cup X_s^t, F^{t-1} \cup F_s^t \} \). The Gaussian process is then updated on \( \{ X^t, F^t \} \).

Afterwards, StratNet performs I steps of Eq. (11) in t-epoch. The pseudocode of UniCuCo is given in Algorithm 1 of Appendix.

4 Experiments

In this section, we validate the effectiveness and efficiency of our proposed ReCoP against state-of-the-art baselines in both depth pruning and non-uniform pruning scenarios. We further analyze the impact of different scalarization functions (i.e. Eqs. (4), (5) and others) in Appendix B.3.

4.1 Experimental Setups

Baselines. For the depth pruning scenario, where the pruning strategy for each layer is represented by binary values (0 and 1), we compare our approach with several baselines. These include the optimization-based method EvoPress (Sieberling et al., 2024), as well as score-based methods such as ShortGPT (Men et al., 2024), Weight Subcloning (Samragh et al., 2023), and Sliding Window Cosine Similarity (referred to as Cosine (Window)) (Gromov et al., 2024). Evaluation. All competitive methods use Fineweb-Edu (FW) (Penedo et al., 2024) as the calibration data. We evaluate perplexity on the WikiText-2 (Wiki2) (Merity et al., 2016) and C4 (Raffel et al., 2020) datasets to measure the performance of pruned LLMs. Additionally, we assess accuracy on zero-shot tasks across a range of datasets, including WinoGrande (Sakaguchi et al., 2021), PiQA (Tata and Patel, 2003), HellaSwag (Zellers et al., 2019), and both ARC-easy and ARC-challenge (Clark et al., 2018), using the LM Eval Harness (Gao et al., 2021).

4.2 Depth Pruning Results

From the depth pruning results in Table 1, two key conclusions can be drawn: (I) Our proposed UniCuCo outperforms the three score-based methods by delivering pruning strategies in less than one second, similar to the speed of the two fastest score-based methods. Additionally, UniCuCo significantly improves average perplexity, especially as model sparsity increases. (II) UniCuCo achieves competitive results compared
<table>
  <tr>
    <th>Sparsity</th>
    <th>Method</th>
    <th>Wiki2 (s)</th>
    <th>C4 (s)</th>
    <th>ArcC (%)</th>
    <th>ArcE (%)</th>
    <th>HS (%)</th>
    <th>PiQA (%)</th>
    <th>WG (%)</th>
    <th>Avg. (%)</th>
    <th>Latency (s)</th>
  </tr>
  <tr>
    <td rowspan="4">0%</td>
    <td>Dense</td>
    <td>4.82</td>
    <td>7.72</td>
    <td>48.90</td>
    <td>79.60</td>
    <td>60.9</td>
    <td>80.30</td>
    <td>73.90</td>
    <td>68.72</td>
    <td>–</td>
  </tr>
  <tr>
    <td>OWL</td>
    <td>5.69</td>
    <td>8.94</td>
    <td>43.90</td>
    <td><b>76.90</b></td>
    <td>55.4</td>
    <td>78.50</td>
    <td>70.30</td>
    <td>65.00</td>
    <td>40m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>5.48</b></td>
    <td><b>8.69</b></td>
    <td><b>44.88</b></td>
    <td>76.85</td>
    <td><b>56.46</b></td>
    <td><b>79.16</b></td>
    <td><b>71.35</b></td>
    <td><b>65.74</b></td>
    <td>122m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>5.68</td>
    <td>8.93</td>
    <td>43.70</td>
    <td>76.70</td>
    <td>55.70</td>
    <td>78.40</td>
    <td>71.00</td>
    <td>65.10</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>5.65</td>
    <td>8.95</td>
    <td>44.11</td>
    <td>76.73</td>
    <td>55.66</td>
    <td>78.40</td>
    <td>71.27</td>
    <td>65.23</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="4">50%</td>
    <td>OWL</td>
    <td>7.50</td>
    <td>11.34</td>
    <td>38.50</td>
    <td>71.90</td>
    <td>46.90</td>
    <td>75.10</td>
    <td>70.20</td>
    <td>60.52</td>
    <td>40m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>7.12</b></td>
    <td><b>10.91</b></td>
    <td>38.05</td>
    <td><b>72.56</b></td>
    <td><b>49.91</b></td>
    <td><b>76.01</b></td>
    <td>68.98</td>
    <td><b>61.10</b></td>
    <td>121m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>7.78</td>
    <td>11.86</td>
    <td>38.00</td>
    <td>72.40</td>
    <td>49.40</td>
    <td>75.00</td>
    <td>69.30</td>
    <td>60.82</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>7.44</td>
    <td>11.46</td>
    <td><b>39.33</b></td>
    <td><b>72.90</b></td>
    <td><b>49.91</b></td>
    <td><b>75.79</b></td>
    <td><b>69.93</b></td>
    <td><b>61.57</b></td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="4">60%</td>
    <td>OWL</td>
    <td>17.22</td>
    <td>21.66</td>
    <td>27.90</td>
    <td>62.60</td>
    <td>38.60</td>
    <td>67.00</td>
    <td>63.50</td>
    <td>51.92</td>
    <td>40m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>9.73</b></td>
    <td><b>14.63</b></td>
    <td><b>33.45</b></td>
    <td><b>67.13</b></td>
    <td><b>43.91</b></td>
    <td><b>72.63</b></td>
    <td><b>65.27</b></td>
    <td><b>56.48</b></td>
    <td>120m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>23.08</td>
    <td>30.03</td>
    <td>27.10</td>
    <td>60.90</td>
    <td>36.10</td>
    <td>65.90</td>
    <td>59.40</td>
    <td>49.88</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>15.88</td>
    <td>22.08</td>
    <td>29.10</td>
    <td>64.27</td>
    <td>39.05</td>
    <td>69.04</td>
    <td>62.90</td>
    <td>52.87</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="4">70%</td>
    <td>OWL</td>
    <td>17.22</td>
    <td>21.66</td>
    <td>27.90</td>
    <td>62.60</td>
    <td>38.60</td>
    <td>67.00</td>
    <td>63.50</td>
    <td>51.92</td>
    <td>40m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>9.73</b></td>
    <td><b>14.63</b></td>
    <td><b>33.45</b></td>
    <td><b>67.13</b></td>
    <td><b>43.91</b></td>
    <td><b>72.63</b></td>
    <td><b>65.27</b></td>
    <td><b>56.48</b></td>
    <td>120m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>23.08</td>
    <td>30.03</td>
    <td>27.10</td>
    <td>60.90</td>
    <td>36.10</td>
    <td>65.90</td>
    <td>59.40</td>
    <td>49.88</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>15.88</td>
    <td>22.08</td>
    <td>29.10</td>
    <td>64.27</td>
    <td>39.05</td>
    <td>69.04</td>
    <td>62.90</td>
    <td>52.87</td>
    <td>&lt;1s</td>
  </tr>
</table>

Table 2: Non-uniform pruning results on the Mistral-7B, evaluated at three sparsity levels, with perplexity for Wiki2 and C4 datasets, and the average zero-shot accuracy (Avg.) across the ArcC, ArcE, HS, PiQA, and WG datasets.

![Line plots comparing total time for generating pruning strategies between UniCuCo and EvoPress as the number of requests increases.](page_186_613_482_246.png)

Figure 4: The comparison of total time for generating pruning strategies between UniCuCo and EvoPress as the number of requests increases.

to optimization-based EvoPress, while maintaining significantly shorter latency. Although EvoPress achieves the best perplexity in six out of ten cases across different sparsities and models, it requires approximately 13 to 26 minutes to compute the pruning strategy for each request. In contrast, our proposed UniCuCo takes less than one second in handing each request, while achieving the best results in four out of ten cases. This demonstrates that UniCuCo not only offers fast inference but also remains highly competitive in effectiveness.

We further compare the total time overhead (including both model training time and latency) of UniCuCo and EvoPress in Fig. 4. The results show that as the number of users increases, UniCuCo outperforms EvoPress in total time overhead. Specifically, on Mistral-7B, UniCuCo achieves 56 times more efficiently than EvoPress when scaling to 64 requests. This is because, once the StratNet in UniCuCo is trained, it can generate pruning strategies for any request, whereas EvoPress requires re-searching for each request.

4.3 Non-Uniform Pruning Results

In Table 2, due to the finer granularity of pruning in Non-Uniform Pruning, the distinction between algorithms is less noticeable at low sparsity, while it becomes more significant at 70% sparsity. In addition, Table 2 shows that our proposed UniCuCo not only achieves lower latency but also outperforms Uniform by 0.1%, 0.7%, and 3% in average accuracy across three sparsities. Furthermore, while EvoPress achieves the best average accuracy in two out of three sparsities, it comes at a prohibitively high time cost for handing single request. Although OWL is more time-efficient than EvoPress, its ac-

![Radar charts showing the impact of Gaussian process updates on depth pruning, evaluated using adjusted perplexity.](page_1012_613_482_246.png)

Figure 5: The impact of Gaussian process updates on depth pruning, evaluated using adjusted perplexity (higher values indicate better effectiveness).
curacy is lower by 0.74%, 0.68%, and 4.56% at the three sparsity levels, respectively. In contrast, our proposed UniCuCo achieves a favorable balance, being approximately 2400 times faster than OWL and 7000 times faster than EvoPress per request in terms of efficiency, while outperforming OWL by an average of 0.6% in accuracy. We provide additional results for non-uniform pruning on Llama-3-8B and Llama-2-13B in Appendix B.2.

4.4 Effects of Gaussian Process Updates

Fig. 5 presents the pruning effectiveness based on normalized perplexity, comparing results with and without Gaussian Process updates. Results show that when the Gaussian Process is not dynamically updated, pruning effectiveness significantly declines compared to when updates are applied. Notably, on Llama-2-13, the absence of Gaussian Process updates leads to effectiveness that is lower than that of the score-based method, ShortGPT. This outcome is intuitive, as StratNet’s performance relies on the Gaussian Process’s estimation of \( f_2 \). Insufficient training samples hinder this estimation, thereby degrading the quality of the pruning strategies generated by the StratNet.

Figure 6: The impact of uncertainty on non-uniform pruning across three large models, evaluated by averaged perplexity and averaged zero-shot accuracy.

Figure 7: Effect of \( \kappa \) on Mistral-7B across two datasets.

4.5 Effects of Uncertainty Estimates

Recall that in Eq. (9), the prediction of pruning strategies is guided the uncertainty provided by the Gaussian process. Fig. 6 compares UniCuCo with LCB, UCB, and no uncertainty (None) against Uniform and EvoPress. An interesting observation is that ignoring uncertainty generally leads to worse effectiveness, but it still outperforms Uniform. In contrast, UniCuCo incorporating uncertainty, such as LCB and UCB, delivers superior effectiveness compared to UniCuCo (None). This is intuitively reasonable, as LCB and UCB balance exploiting well-understood regions with exploring areas of higher uncertainty, thereby improving overall effectiveness. Additionally, we show the results on the impact of the uncertainty weight \( \kappa \) in Fig. 7. The results indicate that variations in \( \kappa \) within the range of 0.1 to 1 have a marginal effect. However, when \( \kappa = 2 \), the pruning perplexity deteriorates as sparsity increases. This is because a larger \( \kappa \) causes the Gaussian process to overly emphasize uncertainty, neglecting the predictive mean.

5 Conclusion

In this paper, we proposed UniCuCo, an efficient method for handling multiple compression requests while preserving effectiveness. UniCuCo contains a StratNet that learns to map any given request to an optimal compression strategy. To overcome the challenges of high computational cost and gradient incomputability in updating the StratNet, we employ a Gaussian process to approximate updating process, thereby enabling effective learning of the StratNet. Experimental findings indicate that UniCuCo is at least 28 times more efficient than optimization-based methods for processing 64 compression requests. Additionally, it achieves 3% higher averaged accuracy in a non-uniform pruning scenario with 70% sparsity when compared with score-based methods.
Limitations

Our work currently applies UniCuCo to LLMs with 54 to 80 transformer blocks, ranging from 7B to 13B parameters. The effectiveness of UniCuCo relies on the training of the Gaussian process. As the size of LLMs increases, with hundreds of transformer blocks, the fitting space for Gaussian process training expands, and the pruning strategy dimension in the request space also increases. In this context, the effectiveness of UniCuCo requires further analysis and validation. Additionally, while our work addresses multiple compression requests for a single LLM, a more complex and realistic scenario involves handling multiple compression requests for multiple LLMs. These aspects will be explored in future research.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62202214 and Guangdong Basic and Applied Basic Research Foundation under Grant 2023A1515012819.

References

Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, and Kris M Kitani. 2018. N2n learning: Network to network compression via policy gradient reinforcement learning. In International Conference on Learning Representations.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. 2021. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10:8–9.

Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. 2024. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887.

Andreia P Guerreiro, Carlos M Fonseca, and Luís Paquete. 2021. The hypervolume indicator: Computational problems and algorithms. ACM Computing Surveys (CSUR), 54(6):1–42.

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. 2018. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 784–800.

Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. 2024. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.

Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.

Kaisa Miettinen. 1999. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media.

Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Carl Edward Rasmussen. 2003. Gaussian processes in machine learning. In Summer school on machine learning, pages 63–71. Springer.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yéjin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. 2023. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv preprint arXiv:2312.09299.

Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. 2024. Evopress: Towards optimal dynamic model compression via evolutionary search. arXiv preprint arXiv:2410.14649.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations.

Sandeep Tata and Jignesh M Patel. 2003. Piqa: An algebra for querying protein data sets. In 15th International Conference on Scientific and Statistical Database Management, 2003., pages 141–150. IEEE.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396.

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations.

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. 2023. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175.

Xinjie Yu and Mitsuo Gen. 2010. Introduction to evolutionary algorithms. Springer Science & Business Media.

Zhongzhi Yu, Zheng Wang, Yuhan Li, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reddy Bommu, Yang Zhao, and Yingyan Lin. 2024. Edge-llm: Enabling efficient large language model adaptation on edge devices via unified compression and adaptive layer voting. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pages 1–6.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.

Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2024. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556–1577.
A Experimental Details

Baselines. We provide detailed descriptions of the four baseline methods used for comparison in depth pruning as follows:

• ShortGPT (Men et al., 2024): Blocks are scored based on the average cosine similarity between their input and output embeddings, including the residual stream.

• Weight Subcloning (Samragh et al., 2023): Blocks are scored using the ratio \( \frac{||\mathcal{M}(\mathcal{E})||}{||\mathcal{M}(\mathcal{E})+\mathcal{E}||} \), where \( \mathcal{E} \) is the input embedding and \( \mathcal{M}(\mathcal{E}) \) is the output of block, excluding the residual stream.

• Sliding Window Cosine Similarity (Gromov et al., 2024): Sets of consecutive blocks are scored based on the cosine similarity between the embeddings before and after the blocks, including the residual stream.

• EvoPress (Sieberling et al., 2024): Determines whether to discard blocks under given compression constraints using an evolutionary algorithm.

For non-uniform pruning, all baselines and our proposed UniCuCo adopt SparseGPT (Frantar and Al-istarth, 2023) as a fast and efficient one-shot layer pruning framework. SparseGPT generates sparsified blocks with varying sparsity levels across layers. The following baselines focus on searching for the optimal sparsity level for each layer:

• Uniform: Directly set a uniform sparsity level for all layers and extract the corresponding sparse model generated by SparseGPT.

• OWL (Yin et al., 2023): OWL uses Layer Outlier Distribution (LOD) metric as a measure of layer saliency, and computes a sparsity profile that is weighted by LOD.

For OWL, we used the same hyperparameter grid as the original work and took the configuration yielding the best perplexity for each model. Notably, EvoPress can also be applied to non-uniform pruning.

Hyperparameters. Following the setup in (Sieberling et al., 2024), the Calibration tokens and Evaluation tokens for the benchmark datasets are set to 524288. The maximum number of tokens that Mistral 7B and Llama-3-8B can process at once is 8192, while Llama-2-13B can process up to 4096 tokens. StratNet employs a multi-layer perceptron (MLP) architecture for training. In the process of increasing the training set in Gaussian model training based on hypervolume contribution, the candidate pool size is set to \( C = 2240 \). In each epoch, 10 new samples are selected and incorporated into the training set. To ensure a fair comparison, we set both EvoPress and UniCuCo to run for 50 epochs (corresponding to \( T = 50 \) in line 3 of the algorithm 1). For the StratNet in UniCuCo, the number of iterations in each epoch is set to 1000 (corresponding to \( I = 1000 \) in line 6 of the algorithm 1) with a learning rate of 1e-3. The Matérn 5/2 kernel is used in the Gaussian process.

Hardware Details. All experiments are conducted on a server running Ubuntu 22.04.5 LTS, equipped with an Intel® Xeon® Platinum 8383C CPU @ 2.70GHz and two NVIDIA L40 GPUs (46GB RAM each). All the implementations are performed using the PyTorch framework.

Algorithm 1 UniCuCo algorithm

<table>
  <tr>
    <th colspan="2">Algorithm 1 UniCuCo algorithm</th>
  </tr>
  <tr>
    <td>1:</td>
    <td><b>Input:</b> StratNet \( \phi_\theta \)</td>
  </tr>
  <tr>
    <td>2:</td>
    <td>// Initialize the parameters \( \theta \) and the training dataset \( \{X^0, F^0\} \) for the Gaussian process \( \mathcal{G} \)</td>
  </tr>
  <tr>
    <td>3:</td>
    <td><b>for</b> \( t = 0 \) <b>to</b> \( T \) <b>do</b></td>
  </tr>
  <tr>
    <td>4:</td>
    <td>Training \( \mathcal{G} \) on \( \{X^t, F^t\} \);</td>
  </tr>
  <tr>
    <td>5:</td>
    <td>Sampled requests \( \{\lambda^k\}_{k=1}^K \sim \Lambda \);</td>
  </tr>
  <tr>
    <td>6:</td>
    <td><b>for</b> \( i = 0 \) <b>to</b> \( I \) <b>do</b></td>
  </tr>
  <tr>
    <td>7:</td>
    <td>// StratNet updating</td>
  </tr>
  <tr>
    <td>8:</td>
    <td>Update \( \phi_\theta \) by Eq. (11);</td>
  </tr>
  <tr>
    <td>9:</td>
    <td><b>end for</b></td>
  </tr>
  <tr>
    <td>10:</td>
    <td>// Gaussian process updating</td>
  </tr>
  <tr>
    <td>11:</td>
    <td>Generating a pruning strategy pool \( X_p^t = \{x^c = \phi_\theta(\lambda^c)\}_{c=1}^C \);</td>
  </tr>
  <tr>
    <td>12:</td>
    <td>Select a subset \( \{X_s^t, F_s^t\} \) based on Eq. (13);</td>
  </tr>
  <tr>
    <td>13:</td>
    <td>\( \{X^t, F^t\} \leftarrow \{X^{t-1} \cup X_s^t, F^{t-1} \cup F_s^t\} \);</td>
  </tr>
  <tr>
    <td>14:</td>
    <td><b>end for</b></td>
  </tr>
  <tr>
    <td>15:</td>
    <td><b>Return:</b> \( \phi_\theta \)</td>
  </tr>
</table>

B Additional Experiments

B.1 Results of UniCuCo on Depth Pruning for LlaMA-2-13B

Table 3 shows that our proposed UniCuCo responds to compression requests in less than one second and outperforms both ShortGPT and Cosine (Window) methods in terms of perplexity, which also respond within one second. Moreover, as sparsity increases, UniCuCo not only achieves competitive
<table>
  <tr>
    <th>Sparsity</th>
    <th>Method</th>
    <th>Wiki2 (↓)</th>
    <th>C4 (↓)</th>
    <th>FW (↓)</th>
    <th>Avg. (↓)</th>
    <th>Latency (↓)</th>
  </tr>
  <tr>
    <td rowspan="5">12.5%</td>
    <td>Dense</td>
    <td>4.57</td>
    <td>6.45</td>
    <td>5.84</td>
    <td>5.62</td>
    <td>–</td>
  </tr>
  <tr>
    <td>Cosine (Window)</td>
    <td><b>5.67</b></td>
    <td>8.09</td>
    <td>6.97</td>
    <td><b>6.91</b></td>
    <td><u>16s</u></td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>5.42</b></td>
    <td><b>7.65</b></td>
    <td><b>6.62</b></td>
    <td><b>6.56</b></td>
    <td>29m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>5.88</td>
    <td>8.36</td>
    <td>7.19</td>
    <td>7.14</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>5.97</td>
    <td>8.46</td>
    <td>7.26</td>
    <td>7.23</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>5.94</td>
    <td>8.07</td>
    <td>6.94</td>
    <td>6.98</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td rowspan="5">25%</td>
    <td>Cosine (Window)</td>
    <td><b>8.99</b></td>
    <td><b>12.57</b></td>
    <td>10.34</td>
    <td><b>10.63</b></td>
    <td><u>24s</u></td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>7.20</b></td>
    <td><b>9.93</b></td>
    <td><b>8.32</b></td>
    <td><b>8.48</b></td>
    <td>28.6m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>17.91</td>
    <td>19.89</td>
    <td>15.73</td>
    <td>17.84</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>17.91</td>
    <td>19.89</td>
    <td>15.73</td>
    <td>17.84</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>10.39</td>
    <td>13.40</td>
    <td>10.14</td>
    <td>11.31</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td rowspan="5">37.5%</td>
    <td>Cosine (Window)</td>
    <td>95.98</td>
    <td>72.35</td>
    <td>50.46</td>
    <td>72.93</td>
    <td><u>24s</u></td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>13.26</b></td>
    <td><b>16.69</b></td>
    <td><b>13.00</b></td>
    <td><b>14.32</b></td>
    <td>26.8m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>52.28</td>
    <td>46.61</td>
    <td>35.24</td>
    <td>44.71</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>52.28</td>
    <td>46.61</td>
    <td>35.24</td>
    <td>44.71</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>20.49</td>
    <td>23.66</td>
    <td>17.74</td>
    <td>20.63</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td rowspan="5">50%</td>
    <td>Cosine (Window)</td>
    <td>1124.49</td>
    <td>649.33</td>
    <td>316.09</td>
    <td>696.64</td>
    <td><u>17s</u></td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>52.06</b></td>
    <td><b>35.75</b></td>
    <td><b>28.48</b></td>
    <td><b>38.76</b></td>
    <td>28.5m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>187.99</td>
    <td>165.59</td>
    <td>132.47</td>
    <td>162.02</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>187.99</td>
    <td>165.59</td>
    <td>132.47</td>
    <td>162.02</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>170.34</td>
    <td>97.43</td>
    <td>76.79</td>
    <td>114.85</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td rowspan="5">62.5%</td>
    <td>Cosine (Window)</td>
    <td>160231.22</td>
    <td>5219.97</td>
    <td>4944.90</td>
    <td>56798.70</td>
    <td><u>13s</u></td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td>4437.52</td>
    <td>3945.83</td>
    <td>3930.84</td>
    <td>4104.73</td>
    <td>23.5m</td>
  </tr>
  <tr>
    <td>ShortGPT</td>
    <td>1204.75</td>
    <td>864.48</td>
    <td>622.41</td>
    <td>897.23</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>Weight Subcloning</td>
    <td>1204.75</td>
    <td>864.48</td>
    <td>622.41</td>
    <td>897.23</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td><b>588.07</b></td>
    <td><b>451.34</b></td>
    <td><b>383.87</b></td>
    <td><b>474.43</b></td>
    <td><b>&lt;1s</b></td>
  </tr>
</table>

Table 3: Depth pruning results of Llama-2-13B across five sparsity levels, evaluated by validation perplexity (PPL). Avg. represents the averaged perplexity of three datasets.

<table>
  <tr>
    <th>Sparsity</th>
    <th>Method</th>
    <th>Wiki2 (↓)</th>
    <th>C4 (↓)</th>
    <th>ArcC (↑)</th>
    <th>ArcE (↑)</th>
    <th>HS (↑)</th>
    <th>PiQA (↑)</th>
    <th>WG (↑)</th>
    <th>Avg. (↑)</th>
    <th>Latency (↓)</th>
  </tr>
  <tr>
    <td rowspan="4">0%</td>
    <td>Dense</td>
    <td>5.54</td>
    <td>7.10</td>
    <td>50.40</td>
    <td>80.10</td>
    <td>60.20</td>
    <td>79.70</td>
    <td>72.60</td>
    <td>68.60</td>
    <td>–</td>
  </tr>
  <tr>
    <td>OWL</td>
    <td>8.13</td>
    <td>13.12</td>
    <td>43.80</td>
    <td>75.80</td>
    <td>54.00</td>
    <td>75.70</td>
    <td><b>72.20</b></td>
    <td>64.30</td>
    <td>30m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>7.64</b></td>
    <td><b>12.53</b></td>
    <td><b>43.94</b></td>
    <td><b>76.18</b></td>
    <td><b>54.92</b></td>
    <td><b>76.17</b></td>
    <td><b>72.14</b></td>
    <td><b>64.67</b></td>
    <td>139m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>8.05</td>
    <td>13.07</td>
    <td>43.60</td>
    <td>75.70</td>
    <td>54.20</td>
    <td>76.10</td>
    <td>71.70</td>
    <td>64.26</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>7.97</td>
    <td>12.96</td>
    <td>43.90</td>
    <td>75.84</td>
    <td>54.30</td>
    <td><b>76.28</b></td>
    <td>71.27</td>
    <td><b>64.32</b></td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td rowspan="4">50%</td>
    <td>OWL</td>
    <td><b>12.37</b></td>
    <td><b>18.53</b></td>
    <td><b>38.00</b></td>
    <td><b>70.30</b></td>
    <td><b>47.70</b></td>
    <td>72.10</td>
    <td>68.50</td>
    <td><b>59.22</b></td>
    <td>30m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>12.37</b></td>
    <td><b>18.92</b></td>
    <td>36.01</td>
    <td>67.34</td>
    <td>46.45</td>
    <td><b>72.91</b></td>
    <td><b>69.14</b></td>
    <td>58.37</td>
    <td>138m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>13.86</td>
    <td>21.43</td>
    <td>35.20</td>
    <td>69.70</td>
    <td>45.60</td>
    <td>72.20</td>
    <td>68.00</td>
    <td>58.14</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>13.29</td>
    <td>20.63</td>
    <td>36.18</td>
    <td>69.91</td>
    <td>46.03</td>
    <td>72.03</td>
    <td>68.51</td>
    <td>58.53</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td rowspan="4">70%</td>
    <td>OWL</td>
    <td>48.07</td>
    <td>52.32</td>
    <td>27.00</td>
    <td>54.90</td>
    <td>36.60</td>
    <td>65.10</td>
    <td>58.60</td>
    <td>48.44</td>
    <td>30m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>24.18</b></td>
    <td><b>31.38</b></td>
    <td><b>30.20</b></td>
    <td><b>61.95</b></td>
    <td><b>40.03</b></td>
    <td><b>68.72</b></td>
    <td><b>62.51</b></td>
    <td><b>52.68</b></td>
    <td>138m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>85.84</td>
    <td>98.35</td>
    <td>22.70</td>
    <td>49.90</td>
    <td>31.40</td>
    <td>62.10</td>
    <td>54.40</td>
    <td>44.10</td>
    <td><b>&lt;1s</b></td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>75.12</td>
    <td>96.82</td>
    <td>23.46</td>
    <td>53.66</td>
    <td>32.36</td>
    <td>63.11</td>
    <td>56.20</td>
    <td>45.76</td>
    <td><b>&lt;1s</b></td>
  </tr>
</table>

Table 4: Non-uniform pruning results of various methods on the Llama-3-8B model, evaluated at three sparsity levels with validation perplexity (PPL) and zero-shot accuracy. Avg. represents the averaged accuracy of five datasets.

results compared to the state-of-the-art method EvoPress but also demonstrates a significant advantage in terms of the time overhead required to handle compression requests over EvoPress.
<table>
  <tr>
    <th>Sparse</th>
    <th>Method</th>
    <th>Wik2 (J.)</th>
    <th>C4 (J.)</th>
    <th>ArcC (†)</th>
    <th>ArcE (†)</th>
    <th>HS (†)</th>
    <th>PiQA (†)</th>
    <th>WG (†)</th>
    <th>Avg. (†)</th>
    <th>Latency (J.)</th>
  </tr>
  <tr>
    <td rowspan="5">0%</td>
    <td>Dense</td>
    <td>4.57</td>
    <td>6.45</td>
    <td>49.23</td>
    <td>77.48</td>
    <td>79.37</td>
    <td>80.47</td>
    <td>72.22</td>
    <td>71.75</td>
    <td>–</td>
  </tr>
  <tr>
    <td>OWL</td>
    <td>5.61</td>
    <td>8.02</td>
    <td><b>45.22</b></td>
    <td><b>77.23</b></td>
    <td>56.20</td>
    <td>77.26</td>
    <td>72.85</td>
    <td><b>65.75</b></td>
    <td>114m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>5.45</b></td>
    <td><b>7.75</b></td>
    <td>42.75</td>
    <td>76.85</td>
    <td><b>56.49</b></td>
    <td>77.69</td>
    <td>71.90</td>
    <td>65.14</td>
    <td>139m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>5.54</td>
    <td>7.90</td>
    <td>43.17</td>
    <td>76.98</td>
    <td>55.98</td>
    <td>77.86</td>
    <td>73.09</td>
    <td>65.42</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>5.53</td>
    <td>7.89</td>
    <td>43.26</td>
    <td>76.94</td>
    <td>56.02</td>
    <td><b>77.91</b></td>
    <td><b>73.40</b></td>
    <td>65.51</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="4">50%</td>
    <td>OWL</td>
    <td>7.33</td>
    <td>10.08</td>
    <td><b>39.85</b></td>
    <td>72.69</td>
    <td><b>51.08</b></td>
    <td>75.35</td>
    <td>69.85</td>
    <td>61.76</td>
    <td>98m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>7.15</b></td>
    <td><b>9.98</b></td>
    <td>39.25</td>
    <td><b>73.11</b></td>
    <td>50.43</td>
    <td><b>75.52</b></td>
    <td><b>70.74</b></td>
    <td><b>61.81</b></td>
    <td>138m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>8.14</td>
    <td>11.29</td>
    <td>38.14</td>
    <td>72.05</td>
    <td>48.83</td>
    <td>74.59</td>
    <td>69.85</td>
    <td>60.69</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>7.60</td>
    <td>10.61</td>
    <td>39.33</td>
    <td><b>73.11</b></td>
    <td>49.92</td>
    <td>74.70</td>
    <td>70.56</td>
    <td>61.52</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td rowspan="4">60%</td>
    <td>OWL</td>
    <td>14.57</td>
    <td>17.95</td>
    <td>31.31</td>
    <td>65.28</td>
    <td>41.53</td>
    <td>70.57</td>
    <td>67.09</td>
    <td>55.16</td>
    <td>121m</td>
  </tr>
  <tr>
    <td>EvoPress</td>
    <td><b>10.24</b></td>
    <td><b>13.52</b></td>
    <td><b>33.87</b></td>
    <td><b>69.19</b></td>
    <td><b>44.31</b></td>
    <td><b>71.76</b></td>
    <td><b>67.25</b></td>
    <td><b>57.28</b></td>
    <td>138m</td>
  </tr>
  <tr>
    <td>Uniform</td>
    <td>40.33</td>
    <td>47.5</td>
    <td>24.06</td>
    <td>52.19</td>
    <td>32.17</td>
    <td>62.13</td>
    <td>57.14</td>
    <td>45.54</td>
    <td>&lt;1s</td>
  </tr>
  <tr>
    <td>UniCuCo</td>
    <td>15.85</td>
    <td>21.13</td>
    <td>31.23</td>
    <td><b>65.66</b></td>
    <td>39.71</td>
    <td>69.31</td>
    <td>64.64</td>
    <td>54.11</td>
    <td>&lt;1s</td>
  </tr>
</table>

Table 5: non-uniform pruning results of various methods on the Llama-2-13B model, evaluated at three sparsity levels with validation perplexity (PPL) and zero-shot accuracy.

![Contour lines of each scalarization function for a two-objective minimization problem. The while arrow represents the request. The green curve and green points represent the Pareto front and the models obtained with different scalarization functions, respectively.](page_1012_1012_388_246.png)

Figure 8: Contour lines of each scalarization function for a two-objective minimization problem. The while arrow represents the request. The green curve and green points represent the Pareto front and the models obtained with different scalarization functions, respectively.

B.2 Results of UniCuCo on Non-Uniform Pruning for Different LLMs

Tables 4 and 5 present additional results for non-uniform pruning. We observe that UniCuCo achieves the same time efficiency in handling compression requests as Uniform, while outperforming Uniform in terms of average accuracy. For instance, UniCuCo outperforms Uniform by approximately 1.6% on Llama-3-8 and by 9% on Llama-2-13B at a sparsity of 70%. Furthermore, although UniCuCo slightly lags behind high-demand time algorithms like OWL and EvoPress in terms of effectiveness, it offers an advantage in terms of time efficiency.

B.3 Impact of Scalarization Function

As mentioned in Section 3.1, our approach optimizes the StratNet using a weighted Tchebycheff scalarization function (Eq. (5)). In this subsection, we compare it with other scalarization functions, such as the weight sum function (Eq. (4)) and the Penalty-based Boundary Intersection (PBI) method. The PBI scalarization function is defined as follows:

\[
\min_x g_{pbi}(x \mid \lambda) = d_1 + \xi d_2,
\]

where \( d_1 = \frac{\|(z^* - f(x))^T \lambda\|}{\|\lambda\|} \) and \( d_2 = \|f - (z^* - d_1 \lambda)\| \). \( \xi > 0 \) represents a penalty parameter. \( d_1 \) geometrically represents the distance between \( f(x) \) and \( \lambda \), indicating the degree of alignment with the request. \( d_2 \) geometrically represents the distance between \( f(x) \) and the origin when \( z^* \) is set to the origin.

To facilitate the analysis of the properties of the above scalarization functions, we present their contour plots for a given white request \( \lambda \) in Fig. 8. In these plots, the bluer the color, the smaller the value of the scalarization function, indicating better performance, while the redder the color, the larger the value, indicating poorer performance. Theoretically, the weighted Tchebycheff function can find an optimal solution for any given request, regardless of the shape of the Pareto front. However, the model obtained by weighted sum and PBI (\( \xi = 0.1 \)) cannot accurately align request under concave Pareto fronts. For PBI (\( \xi = 5 \)), the emphasis on aligning requests (due to the large value of \( \xi \)) leads to neglecting the scalarization function value, i.e., the distance from the origin. Fig. 9 illustrates the adjusted perplexity (higher values indicate bet-
Figure 9: Effects of different scalarization functions on compression performance at five levels of sparsity.

ter performance), where the weighted Tchebycheff function achieves the best results, followed by the Weighted Sum and PBI (\( \xi = 0.1 \)), with PBI (\( \xi = 5 \)) yielding the poorest performance.

Reviewer paper summaries: This paper proposes UniCuCo, which leverages StratNet to map pruning requests to appropriate pruning strategies, resulting in greater efficiency compared to existing methods.

This paper introduces UniCuCo, a universal model for customized compression of large language models (LLMs). Unlike traditional pruning methods that handle each compression request independently—and thus lead to a linearly increasing time cost—UniCuCo employs a StratNet to map diverse compression requests to tailored pruning strategies in a single, unified framework. To overcome the inherent non-differentiability of the pruning process, the authors leverage a Gaussian Process (GP) surrogate to approximate the evaluation of pruning strategies (measured via a KL divergence metric), enabling efficient gradient-based updates. The result is a method that can process multiple compression requests simultaneously with dramatic improvements in inference speed while maintaining competitive performance.
Summary of strengths: - The problem of handling multiple pruning requests for LLMs presents an intriguing experimental setup.
- Estimating the performance of the pruned model using a Gaussian Process is a novel and efficient approach. Moreover, it addresses the issue of gradient propagation, making the optimization process differentiable.

- Efficient Multi-Request Handling: Produces pruning strategies in under a second for diverse requests.
- Innovative Integration: Combines StratNet with a GP surrogate to overcome non-differentiability and balance compression with performance.
- Solid Empirical Performance: Demonstrates significant speed gains and competitive accuracy on a range of LLMs.
Summary of weaknesses: - In the abstract, the authors argue "making them inefficient for real-world scenarios," but as far as I know,  batch pruning request scenario is rare in real-world applications. Additionally, it can be observed in the experiment section that EvoPress outperforms UniCuCo in most cases, further limiting the practicality of UniCuCo. In real-world applications, network performance is generally more critical than pruning time. Sacrificing a small amount of time in exchange for higher performance is often worthwhile.
- How is $\lambda$ mapped from textual requests of the  users?
- The network structures used in StratNet and the Gaussian Process are not explicitly listed in the paper. Are they simple MLPs or more complex architectures? Additionally, the authors only report the StratNet's runtime, without mentioning the training time. Furthermore, does this method require training a StratNet for each model? If so, considering that search-based methods like EvoPress do not require additional network training and are model-agnostic while also achieving better performance, they are clearly more practical than the proposed approach. Although this paper reduces the time required to generate pruning decisions, the cost of training StratNet is non-negligible and should be taken into account when evaluating its overall efficiency.
- Pruning aims to minimize parameter size while preserving model performance as much as possible. In the depth pruning setting of Table 1, UniCuCo only outperforms the baselines when the pruning rate exceeds 50%. However, at such a high pruning rate, the PPL exceeds 100, indicating that the model has already collapsed. Under these conditions, comparisons lose their significance.

- GP Scalability: Will the GP surrogate effectively scale to larger models with hundreds of layers?
Comments, suggestions and typos: - For author citations, such as 153 line on page 2 "Gromov et al.", it is recommended to use \citeauthor{}.
- On page 3, line 173, "Wandb" is a typo; it should be "Wanda".
- On page 3, line 184, the work of He et al. is not for LLM compression.
- When filling out the Checklist, it is essential to provide elaborations regardless of whether selecting "Yes" or "No". This demonstrates responsible research and helps reduce the risk of desk rejection. Since this paper studies pruning using open-source models, the response to "B. Use or Create Scientific Artifacts" should be marked as "Yes", with the corresponding section number explicitly stated.
- Finally, the figure in the paper is so cute and visually appealing. I really like it! :)

N/A
Overall assessment scores: 2.5

4