Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations (2024)

Abstract: Structured pruning fundamentally reduces computational and memory overheads of large language models (LLMs) and offers a feasible solution for end-side LLM deployment. Structurally pruned models remain dense and highprecision, highly compatible with further tuning and compression. However, as the coarsegrained structured pruning poses large damage to the highly interconnected model, achieving a high compression ratio for scaled-up LLMs remains a challenge. In this paper, we introduce a task-agnostic structured pruning approach coupled with a compact Transformer architecture design. The proposed approach, named TransAct, reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules, while preserving the inter-module activations that are sensitive to perturbations. Hence, the LLM is pruned into an intra-module low-rank architecture, significantly reducing weights, KV Cache and attention computation. TransAct is implemented on the LLaMA model and evaluated on downstream benchmarks. Results verify the optimality of our approach at high compression with respect to both efficiency and performance. Further, ablation studies reveal the strength of activation-guided iterative pruning and provide experimental analysis on the redundancy of MHA and MLP modules.

heading: 1 Introduction
text: Deploying large language models (LLMs) locally on edge devices instead of relying on remote APIs has been a pressing initiative. Local deployment of LLMs ensures independence from network conditions and enhances privacy at an advanced level (Ma et al., 2023a). Nevertheless, deploying a scaled-up LLM onto a resource-constrained end
† Work done during an internship at Xiaomi AI Lab. * Zheng Lin and Daren Zha are the corresponding au-
thors.
device poses multifaceted challenges, encompassing inference speed, memory footprint, and power consumption. Therefore, comprehensive optimizations on the efficiency of LLMs are imperative, including architecture design (Gu and Dao, 2023), model compression (Zhu et al., 2023), inference schemes (Leviathan et al., 2023; Cai et al., 2024), compilation and runtime (Lai et al., 2023).
Model compression emerges as the silver-bullet solution for reducing deployment costs given an accessible LLM. To essentially reduce model computation and memory overhead, pruning aims to discard weights with low salience to the LLM. Jaiswal et al. (2023) suggest that state-ofthe-art (SOTA) unstructured pruning approaches i.e., SparseGPT (Frantar and Alistarh, 2023) and Wanda (Sun et al., 2023), along with their semistructured variations, often underperform in downstream benchmarks. Zimmer et al. (2023) emphasize the significance of post-training after pruning to restore the capabilities of the LLM. However, the post-training and inference of a sparse model
9781
are notably inefficient. Also, an unstructured pruning with arbitrary sparsity pattern has no speedup or memory saving on the LLM, whereas a semistructured sparse model heavily relies on specific hardware (Frantar and Alistarh, 2023).
An alternative pruning category, i.e., structured pruning, has shown feasibility for LLMs. LLMPruner (Ma et al., 2023b), the pioneering structured pruning of LLM, incorporates the approximated Taylor series as the pruning metric. However, this approximation loses accuracy when pruning a large ratio of the model (LeCun et al., 1989). While Taylor expansion assumes small perturbations, it is not applicable when a large number of parameters are pruned (i.e., set to zero). The SOTA approach Sheared-LLaMA (Xia et al., 2024), on the other hand, completely transfers the evaluation of the pruning metric to supervised training with masks. However, training with masks poses much more computation and memory footprint at training time, as well as the training unstableness. Also, the pruned architecture of Sheared-LLaMA, as illustrated in the upper part of Figure 1, involves the unified pruning of layer normalization (LN) weights, disregarding the varying sensitivity of LN parameters to perturbation across layers (Zhao et al., 2023).
To address the challenges of efficient and effective LLM pruning, we propose TransAct, a transitional activation-based structured pruning approach. From the perspective of pruning architectural design, TransAct reduces intra-module activations, which prunes the MHA and MLP in LLM into low intrinsic dimension as depicted in Figure 1. TransAct pruning metric is inspired by the observation of Dettmers et al. (2022) that a small proportion of activations within the LLM exhibit outlier magnitudes, rendering them particularly sensitive to perturbations and need to be preserved. This approach effectively reduces the memory footprint of both model weights and KV cache, alleviating the memory constraints inherent in autoregressive generation on edge devices (Kwon et al., 2023). Specifically, the contributions of this paper are outlined as follows.
• We propose a co-design of pruning architecture and pruning metric named TransAct, which substantially compresses the KV cache as well as the model weights.
• TransAct pruning architecture achieves the fastest inference speed among SOTA pruned
models, while the pruning is efficient without gradients or masked training.
• Experiment results on downstream benchmarks verified the stableness of TransAct at a high compression ratio. Ablation studies on module redundancy provide insights for compact model design.

heading: 2 Related Work
text: Extensive works have been proposed to optimize the efficiency of Transformer-based LMs, covering pruning, quantization, dynamic acceleration, etc. However, to generalize these approaches to the continually scaling-up LLMs remains challenging.
Quantization, which reduces the bit representation of values, stands out due to its ease of implementation. Post-training quantization (PTQ) approaches, e.g., GPTQ (Frantar et al., 2022) and AWQ (Lin et al., 2023), are without any further tuning after the quantization. On the contrary, quantization-aware training (QAT) approaches train the model along with the quantization parameters and is still challenging when the LLM is scaled up (Liu et al., 2023). Quantizing an LLM from float16 to int3 with weight-only PTQ approaches like GPTQ (Frantar et al., 2022) can reach roughly 80% compression of model weights. However, the KV cache which contributes to a large amount of memory overheads is still in float16 and uncompressed. Also, obtaining an acceptable quantization precision with int3 weights remains a challenge. Xiao et al. (2023) proposed a W8A8 PTQ approach where both weights and activations are quantized to int8, saving 50% memory footprint. The lack of flexibility poses a significant limitation to quantization. Most general computing platforms and libraries primarily support low-bit representations such as int8 and int4 (Nagel et al., 2021). However, opting for representations lower than 4 bits necessitates dequantization back to the supported higherbit representations, thereby introducing additional computation and memory overheads.
Apart from quantization, unstructured pruning is also an efficient approach to obtain a sparse LLM. Frantar and Alistarh (2023) and Sun et al. (2023) enabled fully unstructured and semistructured N:M sparsity (i.e., N zeros in M consecutive weights) of LLM across different sizes. However, there are two major obstacles hinder-
ing the adoption of unstructured sparsity. (1) The pruned sparse LLM cannot be efficiently further trained. Although Sun et al. (2023) claimed to use LoRA (Hu et al., 2022) to train the compressed model, the LoRA modules cannot be merged into the sparse backbone LLM, which incurs additional overhead at inference time. (2) The sparsity is fixed at 50% with current hardware and platform affinity. While only NVIDIA Ampere and Hopper GPUs support the 2:4 sparsity pattern, achieving customized sparsity requires hardware co-design (Fang et al., 2022). This limitation restricts the broader application of unstructured pruning.
The approaches discussed above are static compression of LLM, where the computation at inference is fixed. On the contrary, dynamic acceleration at inference time speeds up LLM generation by selective computation. Early exiting approaches (Schuster et al., 2022; Corro et al., 2023) allow the LLM to finish the decoding of a token without passing all the layers. Mixture-ofExpert (MoE) architecture (Jiang et al., 2024; Lepikhin et al., 2021) incorporates multiple parameter shards in MLP as experts and selects experts to compute when facing different inputs. The dynamic approaches usually do not reduce parameters. Thus, the storage of the model is not reduced, while the runtime memory can be saved by fine-grained neuron-aware offloading (Song et al., 2023).

heading: 3 Methodology
text: In this section, we first recap the preliminaries of Transformer-based LLM architecture and introduce the transitional activations. Then, we propose our approach TransAct with the pruning metric and architecture design of the pruned model.

heading: 3.1 Transitional Activations in LLM
text: Transformer-based LLMs generally consist of embedding, MHA (multi-head attention), MLP (multi-layer perceptron), and LM head.
The majority of model weights lie in MHA and MLP, which exist in every Transformer layer of the LLM. Specifically, MHA has three matrices WQ, WK , WV with the shape of H × A, and one matrix WO of the inverted. The MHA mechanism splits the output dimension A into An × Ad (i.e., head number by head dimension), which forms An logical attention heads. The input activation hl of the l-th layer is projected by { WQ l k,WK l k,WV l k }An k=1
and split into An groups of query, key, and value {qlk,klk,vlk}Ank=1. Then the multi-head self-attention computation is as
actA l k = Softmax(q l kk l k ⊺ / √ Ad)v l k, (1)
where k is the attention head index counted from 1 to An, and l indicates the l-th layer. H ×Ad at the superscript is the shape annotation of the weight matrix. Then, the results are concatenated to shape A and projected back to shape H by WO.
hA l = Concat[actA l k] An k=1WO
lA×H . (2)
As a bound between the group of WQ, WK , WV and WO, we define actAl as the transitional activation of MHA module. By default, the transitional size A of MHA is the same as hidden dimension H , but A can be smaller than H by reducing An or Ad in the case of pruning.
The other module, MLP, has a pair of upcast and downcast phases. In the first phase, the input hidden state h is projected to a transitional state with larger dimension P through WU and an optional gate WG, the later phase consists of a downcast
WD that projects the transitional state back to the original shape H . We consider WG exists and formulate MLP as
actP l = σ(hA lWG lH×P )⊙ (hAlWU l H×P ), (3)
hP l = actP lWD lP×H . (4)
Similar to the MHA module, we define actlP as the transitional activation of the MLP module at the l-th layer. In case there is no optional gating in the model (e.g., OPT (Zhang et al., 2022), BLOOM (Le Scao et al., 2023)), the transitional activation of the MLP module can be viewed as actP l = σ(hA lWG lH×P ).

heading: 3.2 Pruning with Transitional Activations
text: Based on the model architecture, we identify the pruning target as the following. (1) An, i.e., the number of attention heads in MHA. On the other hand, Ad is kept intact, as reducing it incurs the adaption of RoPE (rotary positional embedding) (Su et al., 2024) used by a high quantity of LLMs and increases the training unstableness. (2) P , i.e., the transitional dimension of MLP. Studies (Mirzadeh et al., 2023) indicate that, with an activation function suppressing negative values, the transitional state of MLP is with high redundancy. It is worth mentioning that H , i.e., the hidden dimension throughout the model is not compressed. We justify the reason as compressing H incurs the unified pruning of layer normalization (LN) weights across layers, whereas the sensitivity of LN parameters to perturbation is not unified across layers (Zhao et al., 2023). Although further training can reconstruct the LN module from the damage of compressing, the significant training cost is contrary to efficiency.
With the definition of transitional activations and the pruning objects, we propose the transitional activation-based pruning approach to compress MHA and MLP modules into an intramodule low-rank architecture as depicted in Figure 2. For the MHA module, we define the pruning granularity (i.e., the least separable structure) to be the attention head, in turn reducing An while keeping Ad intact. Such an attention head pruning is unified on WQ, WK and WV because the selfattention calculation, as formulated in Equation 1, requires the aligned head index among the three matrices. Then, we can define the salience of all
heads in MHA as
S lAk = 1
Ad
Ad∑
i=0
∥∥∥actAlki ∥∥∥ 2 + αmax i ∥∥∥actAlki ∥∥∥ 2 ,
(5)
where α is a weight factor amplifying the maximum activation value in the k-th head. By Equation 5, we want to evaluate both the general and outlier values in the activations, so that we can precisely prune out the most insignificant head. For MLP, we can simply use the corresponding value of actP to represent the salience of MLP transitional dimension as S lP i = ∥∥actP li ∥∥ 2 .
With the salience SA and SP formulated, we can model the activation-based structured pruning of a weight matrix W as
prune(W ,K,S) = Concat[W i]i∈arg topK(S). (6)
Specifically, the pruning dimension of WQ, WK , WV , WG (optional) and WU is the output, while the pruning dimension of WO and WD is the input as depicted in Figure 2.
Obtaining the salience of the source LLM requires only forward passes with a small amount of calibration samples. Hence, the pruning procedure is efficient in both memory and computation. To avoid a single shot pruning to compression ratio R posing unrecoverable damage to the model, we provide an enhanced implementation where the model is iteratively pruned to the target size. A set of pruning ratios is defined as R = {r1, r2, · · · , rn}, where the i-th shot prunes the model to the size of (A′i, P ′ i ) subject
to ∑
ri∈R = R, and A ′ i mod Ad = 0. During the
interval of two pruning steps, full fine-tuning (FT) is performed on the model to recover the pruning damage.

heading: 4 Experimental Evaluation

heading: 4.1 Experiment Setup
text: Model and Datasets Settings In this paper, we select the representative LLaMA2-7B-base (Touvron et al., 2023) as the source model to prune, as its size is suitable for experiments and has shown important features of LLMs. We also use the pre-trained OPT-1.3B and OPT-2.7B (Zhang et al., 2022) as the baseline of the pruned models.
We use subsets of RedPajama-V1 (Together Computer, 2023), a 1 trillion-token corpus, as the
training dataset. Specifically, a subset of 800 million tokens are randomly sampled in the iterative pruning process, while 50 billion tokens are randomly sampled in post-training. For evaluation, we select held-out downstream tasks from Huggingface open LLM leaderboard1, LLaMA2 paper (Touvron et al., 2023), and Sheared-LLaMA paper (Xia et al., 2024). The tasks include zeroshot ARC-E (Clark et al., 2018), BoolQ (Clark et al., 2019), LogiQA (Liu et al., 2020), OpenbookQA (OBQA) (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017) and few-shot ARC-C (Clark et al., 2018), HellaSwag (Zellers et al., 2019), TriviaQA (Joshi et al., 2017), TruthfulQA (Lin et al., 2022) and WinoGrande (Sakaguchi et al., 2020). Details of the evaluation tasks can be found in Appendix B.
Baselines and Implementations We compare the following baselines. (1) LLM-Pruner (Ma et al., 2023b), a structured purning approach with Taylor expansion-based metrics. We reproduce LLM-Pruner with the same architecture as TransAct implementation. (2) Sheared-LLaMA (Xia et al., 2024), a masked training-based approach for LLM pruning. We use the open-sourced pruned models and post-train with the same data as TransAct implementation.
The finalized architectures of the pruned models are shown in Table 1 as well as the pruning and training paradigm. LLM-Pruner and TransAct are implemented in iterative pruning mode, where pruning take place at certain fine-tuning steps. Sheared-LLaMA is reproduced without dynamic batch loading to expose the real performance of pruning without adding influential factors of training. Our implementation is with DeepSpeed on 8 NVIDIA A100 80G GPU, while the sequence length is 4096. Please refer to Appendix A for more implementation details2.

heading: 4.2 Experiment Results

heading: 4.2.1 Efficiency Metrics
text: Aiming at efficient deployment of LLMs, the major objective is to reducing inference overheads. In this section, efficiency metrics of FLOPs and endto-end (E2E) latency of original and pruned models are reported.
1 https://huggingface.co/spaces/HuggingFaceH4/ open_llm_leaderboard
2 Code available at https://github.com/sbwww/ TransAct-pruning
FLOPs Reduction To verify the therotical computation reduction, we profiled the pruned models using PyTorch profiler. Specifically, the profiling is conducted on a single model forward with input length of {256, 512, 1024, 2048, 4096} tokens and a single output token. Figure 3 elaborates the FLOPs saving of pruned models, where TransAct1.3B achieves -83% FLOPs of the original LLaMA model and achieves 20% addtional FLOPs saving compared to Sheared-LLaMA with similar parameter size. Also, as the context length increases, the FLOPs growth of TransAct remains more steady compared to Sheared-LLaMA. The gradual increase in computation against context length is essential for prevailing LLM applications such as retrieval-augmented generation (RAG) and agent, where the context length commonly exceed 4K, even if the user queries are not necessarily long.
End-side E2E Latency Beyond therotical analysis, we deploy the models and test the endto-end inference latency of models. In Figure 4, TransAct-1.3B, Sheared-LLaMA-1.3B and LLaMA2-7B are quantized to W4A16 and deployed on a Xiaomi 14 mobile phone using MLCLLM (team, 2023). TransAct-1.3B has 75%-80% time saving compared to the original LLaMA2-7B and 15%-25% time saving compared to ShearedLLaMA, which is consistent to the therotical FLOPs saving and reduced size of model and KV cache. Compressing the MHA module and KV cache size is crucial on resource-constrained endside devices as well as the server side. This importance stems from the prevailing deployment
approach, which prioritizes weight-only quantization over activation quantization. Weight-only quantization (e.g., W4A16) offers reduced degradation and requires smaller storage compared to activation quantization (e.g., W8A8), making it more favorable for potential mobile applications. Hence, despite the small number of the KV cache compared to model weights, the memory footprint of the 16-bit KV cache is comparable to that of the 4-bit model weights with 4 times amplified.
Server-side E2E Latency We deploy TransAct2.6B, Sheared-LLaMA-2.7B and LLaMA2-7B on a single NVIDIA A100 GPU. The models are tested in the original bfloat16 precision. We report the end-to-end generation latency with the batch size of {4, 8, 16}, context length of {1024, 2048} tokens, and generation length of 2048 tokens. As shown in Table 2, TransAct continually outperforms Sheared-LLaMA in variable batch size and context length. Notably, an NVIDIA A100 GPU has high bandwidth memory (HBM)
of 2TB/s, which makes the acceleration of TransAct architecture not obvious in streaming mode (i.e., B=1). However, the inference bottleneck switch from memory bandwidth to computation with larger batch size and context length in serverside applications, and TransAct largely benefits from the reduced computation in the MHA module.

heading: 4.2.2 Performance Metrics
text: The evaluation results of pruned models on heldout benchmarks are listed in Table 3 while the perplexity of language modeling tasks are in Appendix C. On few-shot tasks, TransAct2.6B achieves the best performance performance compared to SOTA approaches. TransAct exhibits a significant leap over LLM-Pruner and Sheared-LLaMA on TriviaQA and TruthfulQA, which evaluate the truthfulness and world knowledge of the LLM. Whereas, the pre-trained OPT models achieve the highest metric on the two tasks although other abilities are inferior to the
pruned models. We interpret that TransAct better preserved the world knowledge of the original LLM, which is much harder than preserving language modeling and commonsense reasoning capabilities. At 80% compression, TransAct-1.3B achieves 78.0% performance of LLaMA2-7B on average, addressing the effectiveness of TransAct at highly compressed settings. Whereas LLMPruner fails at most few-shot tasks. Thereby, we address the inapplicability of structured pruning with the Taylor expansion-based metric. LLMs are fundamentally pre-trained on a large corpus to obtain world knowledge. However, the Taylor expansion-based metric, which guides the pruning by minimizing the approximated language modeling loss on a small calibration set, fails to preserve knowledge and degrade the pruned LLM. Amplifying the calibration set by a significant order of magnitude is an intuitive solution. However, the computation of Jacobian and Hessian matrices of LLM weights on a large calibration set is enormous.
Notably, the reproduced LLM-Pruner-2.6B with iterative pruning reaches 83.6% performance of the uncompressed LLaMA2-7B. Whereas in its original paper, the performance at 50% compression ratio can barely reach 78% of the original model (Ma et al., 2023b). The results strengthen the necessity of iterative pruning at LLM structured pruning. Specifically, iterative pruning is gradual and conservative at each step, lessening the approximation error of pruning metrics.
Figure 5 illustrate the zero-shot LAMBADA language modeling performance at each checkpoint of the pruned model post-training. Although TransAct-2.6B has a clear advantage between 10b to 30b tokens trained, the gap between different pruning approaches diminishes as the pruned
model is gradually recovered by post-training. Notably, the result of LLM-Pruner-2.6B exhibits the lowest perplexity in Figure 5. However, it does not necessarily indicate the highest accuracy on LAMBADA, nor the performance on other tasks.

heading: 4.2.3 Ablation Studies
text: We conduct a comprehensive evaluation of the pruned LLM, considering factors of pruning shots, calibration samples, and the pruning ratio of each module. The findings provide insights for the further development of compact LMs.
Impact of Iterative Pruning While LLMPruner has demonstrated a close performance gap to the original model at a moderate ratio of 20%, the significant performance degradation observed at over 50% pruned is far from acceptable in the original implementation (Ma et al., 2023b). However, the results in Table 3 indicate that LLMPruner achieves comparable performance to the SOTA approach Sheared-LLaMA even at a compression ratio of 85%. This achievement can be attributed to our iterative implementation of pruning.
To further verify the effectiveness of iterative pruning, we conduct experiments on LLM-Pruner2.6B and TransAct-2.6B with different numbers of pruning shots. Specifically, we explore pruning shots ranging from {1, 2, 4, 8, 16}. Except for single-shot pruning, all others have a total of 800 million tokens throughout the iterative pruning stage. After the final pruning, all models undergo full fine-tuning with 200 million tokens. Sheared-LLaMA is considered an ∞-shot pruning approach with all the parameters trained and is not compared.
Results in Figure 6 indicate the relationship between pruning shots and performance on LAMBADA language modeling. Although iterative pruning is beneficial, the pruning shots need to be controlled with a total number of tokens is fixed. The performance of 2.6B models degrades when the pruning shot is increased from 4 to 8. The rationale of this phenomenon is that when training is insufficient between two pruning shots, the pruning would be misguided and the pruned model would exhibit a degradation. Whereas, for 1.3B models, the performance exhibits a slight degradation at 16
shots, indicating the benefit of increased shots has not yet been overwhelmed by the insufficiency of training data. LLM-Pruner has a slight advantage over TransAct at 16 shots pruning, as fewer parameters pruned at each shot reduce the approximation error of loss with Taylor expansion.
Impact of Calibration Samples To evaluate the sensitivity of pruning approaches to calibration samples used in the pruning process, we conduct single-shot pruning experiments on different numbers of calibration samples. 200 million tokens are used for the restoration after pruning.
The results in Figure 7 indicate that increasing the sample size can bring gains, but the marginal benefits decrease after increasing to 128 samples. When leveraging 256 samples, the performance of both TransAct and LLM-Pruner degrade. Also, the degradation trend is more obvious on LLMPruner than on TransAct. We attribute this to early overfitting of calibration samples, where the pruning guided by Taylor expansion of loss quickly overfits on the calibration set, and the calibration samples are not large enough to exhibit diversity. As pruning is efficient in our implementation, we prefer using 128 samples for the pruning metric, which can be computed in less than 1 minute on a single A100 GPU to prune LLaMA2-7B.
Analysis on Module Redundancy To validate the redundancy of pre-trained models and help future compact model design, we conduct experiments on different compression ratios of MHA and MLP modules. Specifically, after single-shot pruning with TransAct and post-training on 200 million tokens, the accuracy of LAMBADA language modeling is evaluated. Specifically, using the shape of our TransAct-2.6B as the center point, we vary the MHA dimension A ranged from {512, 1280, 2048, 2816, 3584} with the head dimension of 128. And, the MLP dimension P is set to {1024, 2048, 3072, 4096, 5120}. These configurations resulted in 25 distinct models obtained by pairwise combinations. Notably, the 25 models are organized into 9 groups, each containing an e number of parameters. These groups are visually distinguished by color in Figure 8.
The results presented in Figure 8 reveal a clear trend that, the models at the center exhibit the best performance within each group, and in some cases, even surpass models of larger sizes. For instance, the combination of 2048A-3072P (i.e., TransAct2.6B) model surpasses both 3584A-2048P and
1280A-5120P (2.9B) models. Also, when pruning the MHA intermediate size to 512, the performance drops to the worst within each group. We interpret that MHA functions as the crucial module of Transformer-based LLMs while MLP has a larger redundancy that can be compressed. Further, the findings indicate that models with a uniform MHA and MLP size generally outperform the others. For 2048A-3072P, an MHA module has 33.5 million parameters and an MLP module has 37.7 million parameters. On the contrary, extreme pruning of either MHA or MLP alone leads to severe performance degradation. Hence, the collaborative compression of both MHA and MLP is encouraged.

heading: 5 Conclusion
text: In this paper, we introduce TransAct, an effective and efficient pruning approach coupled with an architecture designed for pruned LLMs. TransAct compresses the original LLM into a compact dense model with an intra-module low-rank architecture, achieving the fastest inference speed and lowest overheads compared to models of similar sizes. The compression is guided by the magnitudes of the transitional activations within the MHA and MLP modules. Specifically, intramodule dimensions with small activations are structurally pruned out, while inter-module dimensions are preserved. Experiments on open-source LLMs and downstream benchmarks demonstrate the strength of our approach, particularly at high compression rates. Also, we thoroughly evaluated the pruned LLM with respect to calibration samples, pruning ratio, and pruning shots. The results provide insights and experimental results for further development of compact yet powerful LMs.
Limitations
Although TransAct is found effective in the experiments, some points are not fully covered in this paper. We list the limitations and future directions as follows. (1) TransAct is a static pruning approach where the computation of the pruned LLM is irrelvant to input instances. However, recent research progress in MoE (Jiang et al., 2024) indicates that dynamically compressed models are model powerful than statically compressed ones. Hence, a pruning approach integrating static and dynamic compression with approporate ratio can be further studied. (2) TransAct is targeted to Transformer-based LLMs. Different architectures including RWKV (Peng et al., 2023), Mamba (Gu and Dao, 2023) are not yet investigated. (3) The pruning of TransAct is conducted on base models Structurally pruning a human-aligned LLM still remains challenging, mainly because of the inconsistency in training data paradigm between pre-training and alignment.

heading: A Details of Training Arguments
text: The training arguments are listed in Table 4. The experiments are conducted on Huggingface Transformers with DeepSpeed and FlashAttention2 integration. We set the training arguments based on accessible computational resources and setting of Xia et al. (2024). There is no hyperparameters searching or tuning in this work, and we believe it is potentially beneficial to tune the hyperparameters with sufficient resources.

heading: B Details of Evaluation Tasks
text: The downstream tasks used for evaluation are listed in Table 5. The evaluations are conducted based on lm-evaluation-harness 3 repository with MIT license. In Table 5, "acc_norm" stands for accuracy after normalization by byte-length, "em" stands for exact match, and "mc2" stands for the normalized probability assigned to all true answers in multiple choices (Lin et al., 2022).

heading: C Perplexity of Language Modeling
text: To evaluate the basic ability of language modeling, we test perplexity of models on WikiText and LAMBADA corpus, and the results are in Table 6. WikiText contains long documents that exceed the maximum length of LLaMA2 (i.e., 4K), and the documents are truncated into three set of maximum length, {1K, 2K, 4K}. Samples in LAMBADA test set are below 1K tokens, so the above three set of maximum length does not effect the results.
3https://github.com/EleutherAI/ lm-evaluation-harness

heading: Future Work
text: We list the limitations and future directions as follows. (1) TransAct is a static pruning approach where the computation of the pruned LLM is irrelvant to input instances. However, recent research progress in MoE (Jiang et al. , 2024) indicates that dynamically compressed models are model powerful than statically compressed ones. Hence, a pruning approach integrating static and dynamic compression with approporate ratio can be further studied. (2) TransAct is targeted to Transformer-based LLMs. Different architectures including RWKV (Peng et al. , 2023), Mamba (Gu and Dao, 2023) are not yet investigated. (3) The pruning of TransAct is conducted on base models Structurally pruning a human-aligned LLM still remains challenging, mainly because of the inconsistency in training data paradigm between pre-training and alignment