Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Authors: Chaojun Xiao
Yuqi Luo
Wenbin Zhang
Pengle Zhang
Xu Han
Yankai Lin
Zhengyan Zhang
Ruobing Xie
Zhiyuan Liu
Maosong Sun
Jie Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: Large language model
model compression
plugins
TL;DR: We propose a parameter-efficient acceleration method, named as Variator, which can save 53% computational costs with only 0.9% additional paramters

Abstract: Large language models (LLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs. In this paper, we propose Variator, a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins. Compression plugins are designed to reduce the sequence length via compressing multiple hidden vectors into one and trained with original LLMs frozen. Different from traditional model acceleration methods, which compress LLMs to smaller sizes, Variator offers two distinct advantages: (1) In real-world applications, the plug-and-play nature of our compression plugins enables dynamic selection of different compression plugins with varying acceleration ratios based on the current workload. (2) The compression plugin comprises a few compact neural network layers with minimal parameters, significantly saving storage and memory overhead, particularly in scenarios with a growing number of tasks. We validate the effectiveness of Variator on seven datasets. Experimental results show that Variator can save 53\% computational costs using only 0.9\% additional parameters with a performance drop of less than 2\%. Moreover, when the model scales to billions of parameters, Variator matches the strong performance of uncompressed LLMs. Our code and checkpoints will be released to facilitate future work.

Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Chaojun Xiao', Yuqi Luo', Wenbin Zhang', Pengle Zhang', Xu Han 1.3, Yankai Lin
Zhengyan Zhang', Ruobing Xie Zhiyuan Liu 1.3, Maosong Sun 1.3, Jie Zhou
NLP Group, DCST, LAI, BNRIST, Tsinghua University, Beijing
ZZhili College, Tsinghua University, Beijing *Quan Cheng Laboratory
"Gaoling School of Artificial Intelligence, Renmin University of China, Beijing
"Beijing Key Laboratory of Big Data Management and Analysis Methods "Tencent Inc.
xiaocj200mails, wtstimaiiisssssituaaaaaa..aaaa edu. cn, hanxu2022@tsingghe edu. cn, mr1yk423@gmail. com
Abstract
PLM
PLMM
Pre-trained language models (PLMs) have achieved remarkable results on NLP tasks but the expense huge parameter sizes and the consequent computational costs. In this pa- per, we propose Variator, a parameter-efficient acceleration method that enhances computa- tional efficiency through plug-and-play com- pression plugins. Compression plugins are de- signed reduce the sequence length via pressing multiple hidden vectors into one and trained with original PLMMs frozen. Different from traditional model acceleration methods, which compress PLMs to smaller sizes, Varia- tor offers two distinct advantages: (1) In real- world applications, the plug-and-play nature of our compression plugins enables dynamic selection of different compression plugins with varying acceleration ratios based on the cur- rent workload. (2) The compression plugin comprises few compact neural network layers with minimal parameters, significantly saving storage and memory overhead, particularly in scenarios with a growing number of tasks. We validate the effectiveness of Variator on seven datasets. Experimental results show that Vari- ator can save 53% computational costs using only 0.9% additional parameters with a per- formance drop of less tham 2%. Moreover, when the model scales to billions of param- eters, Variator matches the strong performance of uncompressed PLMs. Our code and check- points can be found in https: //gi thub. com/ hhunlp/Compressioon- -Pluuii........nnnn...
2x
4x
(a) Traditional Model Acceleration
PLM
1x
2X
4X
(b) Accelcration with Compression Plugins
Figure I: Mlsstratioon model acceleration with com- pression plugins.
enhanced downstream performance. Nevertheless, the expansive parameter scale intrinsic to PLMs demands significant computational and storage re- sources. Such formidable overheads necessitate investigating alternative strategies maintain per- formance while reducing costs.
Many efforts have been devoted to improving the training and inference efficiency of PLMs (Sun et al., 2019; Liu et al., 2022: Fan et al., 2020; Xia al., 2022: Stock al., 2021). These meth- ods compress PLMs into fixed smaller sizes, and cannot fulfill the following requirements: (1) Dy- namic Workload. In real-world scenarios, the sys- tem workload varies dynamically over time, while the computational resources are fixed. This im- plies that we can use more resources for higher performance when the workload is low, and en- response efficiency when the workload is high. (2) Storage Efficiency. These methods typically depend on large number of additional parame- ters to construct compressed models, which require amounts of memory space for model training and storage across various tasks and acceleration ratios.
1 Introduction
Large pre-trained language models (PLMs) have made significant advancements in natural language processing tasks (Han et al., 2021; Brown et al., 2020; Qiu et al., 2020; Bommasani et al., 2021; OpenAI, 2023). It is widely observed that ampli- fying the model scale correlates positively with
To address these issues, we propose novel plug- and-play acceleration framework named Variator, As shown in Figure 1, different from compressing
"Corresponding authors.

---

PLMs into smaller sizes, Variator enables PLMs ac- celeration via devising compression plugins, which can be inserted into PLMs to enhance the inference speed. Various plugins entail different acceleration ratios, and the system can dynamically choose the appropriate one to trade off response speed and model performance depending on the workload. Moreover, Variator only necessitates plugins with minimal parameters and freezes the original pa- rameters of PLMs, which substantially lowers the memory and storage requirements.
When the model scales billions of parameters. Variator can achieve nearly no performance drop. We also examine the effectiveness of Variator on a decoder-only LLM, LLaMA (Touvron et al., 2023). In addition, we conduct neuron-level analysis for compression plugins, and find that compression plugins can effectively store important information in the compressed vectors to achieve satisfactory performance with limited computational costs.
Related Work
To achieve plug-and-play acceleration, there are two main challenges: (1) Plugin Architecture: Compression plugins do not modify PLMs scales, and how devise plugins to reduce the inference time is challenge. (2) Plugin Training: Compres- sion plugins only contain limited parameters, and how to effectively train plugins so that they can enhance the model speed while preserving down- stream performance.
2.1 Model Acceleration
Improving the computational efficiency of PLMs has been widely studied in recent years (Gupta and Agrawal, 2022: Zhang et al., 2022a). The related work can be divided into four categories: knowledge distillation, which guides the training of compressed models with the output or middle states of original PLMs (Hinton et al., 2015; Sanh et al., 2019; Sun et al., 2019, 2020); model pruning, which removes unimportant parameters layers from PLMs (Fan al., 2020; Michel al., 2019; Chen al., 2020; Xia al., 2022); model quantiza- tion, which converts model parameters into low-bit precision values, thus achieving accelcration on compatible devices (Stock et al., 2021; Xiao et al., 2022); and conditional computation, which only selects parts of parameters to compute outputs for each input (Zhang et al., 2022b; Xin et al., 2020). Some researchers make preliminary exploration for dynamic acceleration, such as early exit (Xin et al., 2020; Matsubara al., 2023), which attempts skip layer computation based on instance complex- ity. But these works rely heavily on confidence judgment and are thus only applicable to specific tasks and model architectures. Our model, which focuses on dynamic acceleration based on system workload, parallels these works and can be intu- itively combined with them to reduce computa- tional costs further.
As for plugin architecture, inspired by previous findings about redundancy in hidden vectors (Goyal al., 2020; Ye et al., 2021), we design compression plugins for data compression rather than parame- ter compression. Specifically, compression plugins consist of hidden compression layers and hidden decompression layers. The goal of the hidden com- pression layers is to compress multiple hidden vec- tors into one, thereby diminishing the sequence length for PLMs and enabling model acceleration. Simultaneously, to preserve token-level informa- tion, we also devise decompression layers that re- cover the processed shorter sequence to the original length. Compression plugins can be applied in any layer of PLMs, cnabling various levels of acceler- ation. As for plugin training, we adopt a two-step training strategy. Firstly, we train compression plu- gins on pre-trained PLMS with pre-training corpus. Then the compression plugins trained in the first step are used as initialization for task task-specific mod- els. In both steps, we apply knowledge distillation objectives to train the compression plugins not to alter the hidden vectors produced by PLMs.
Besides, within the realm of conditional com- putation there are a line of reasearches find out the redundancy of hidden vectors and focus on dis- carding tokens at each layer of PLMs to accelerate model inference (Goyal et al., 2020; Ye et al., 2021; Kim and Cho, 2021; Kim et al., 2022; Dai et al., 2020; Murahari al., 2022), which inspire the de- sign of our compression and decompression layers. But these works require to retrain the whole PLMs to achieve accleration, while Variator focuses on the parameter-efficient acceleration setting and thus
To verify the effectiveness of Variator, we con- duct experiments with widely-used pre-trained backbone, T5 (Raffel et al., 2020), seven widely- used language understanding benchmarks. The experimental results show that Variator can save 53% computational costs using only 0.9% param- eters with absolute average performance drops of 2% compared to original downstream PLMs.

---

enable dynamic acceleration ratio selection with minimal additional memory requirements. In ad- dition to merge tokens, the compression layer can also be designed to dynamically prune the parame- ters, which we leave for future work.
rived from M, Variator aims construct com- pression plugin P. which can be inserted into Mr to improve the computational efficiency. That is, given an input squence 8, the computation costs of (MT + (s) should be lower than Mr(s). Varia- tor is designed for dynamic workload, which means plugins with different acceleration ratios can be ap- plied in the same downstream model Mr. There- fore, the original Mr should be frozen during the training of the compression plugin P.
2.2 Parameter-Efffiicee Learning
The huge parameter scale of PLMs imposes sub- stantial costs on model training and storage. To alleviate this problem, parameter-efficient learning, also known as delta tuning, is proposed to perform task adaptation via tuning small portion of pa- rameters and keep other parameters frozen (Liu et al., 2021; Ding et al., 2022; He al., 2022). According to the operation of tunable parameters, delta tuning methods can be divided into: addition- based models, which introduce additional layers into PLMs (Houlsby et al., 2019; Lester et al., 2021); specification--asee models, which specify existing weights of PLMs tunable (Zaken et al., 2022: Guo et al., 2021); and reparameterization- based models, which rewrite the computation pro- cess of specific layers into parameter-efficies man- ners (Hu et al., 2021). In addition, some researchers attempt to construct plug-and-play modules for re- trieval augmentation (Shi et al., 2023; Yu et al., 2023), knowledge injection (Wang et al., 2021; Zhang et al., 2023; Xiao et al., 2023), controllable text generation (Pascual et al., 2021; Madotto et al.. 2020), and model debiasing (Lauscher et al., 2021). In this paper, we propose a parameter-efficient ac- celeration model with hidden vector compression, which can save the memory and storage costs com- pared to traditional compression methods.
3.2 Overall Framework
Previous researches find out the redundancy in hid- den vectors, which means eliminating hidden vec- tors is a promising direction for acceleration (Goyal et al., 2020; Ye et al., 2021). Inspired by these works, our compression plugins are designed to compress hidden vectors, and thus the sequence length is reduced to speed up inference.
As shown in Figure 2, compression plugins con- sist of two layers: hidden compression layer and hidden decompression layer, which are inserted before and after a vanilla neural layer, respectively. In this way, the compression layer can reduce com- putational overhead for the following neural layer and the decompression layer aim to restore token- level information into the output vectors. Then we will introduce these two layers in detail.
Hidden Compression Layer. Hidden compres- sion layers aim reduce the sequence length. Pre- vious token pruning methods assign importance scores for each hidden vector and discard hidden vectors with low scores, which may suffer from loss of useful information when the required compres- sion ratio is high. Different from directly dropping hidden vectors, our hidden compression layer is designed to merge multiple vectors into one.
3 Methodology
In this section, we first describe the paradigm and basic annotations for our plug-and-play model ac- celeration. Then we present the framework and training recipes of Variator to accelerate model in- ference with minimal additional parameters. To showcase the efficiency of Variator, we also con- duct an analysis of the computational and storage complexity.
Specifically, given the input vector sequence with tokens, H {ho, .... h J.. we first split the sequence into several groups with each group containing kt vectors, Di = (hik.... b((4))-1)-
Then the compressed vector is calculated as the weighted average of input vectors:
a Softmax Softmax(W.Comee (W.Concat(g)) + be).
3.1 Preliminary
g: a h
Our primary goal is to design plug-and-play ac- celeration framework, which can dynamically im- prove the computational efficiency with multiple compression plugins. Specifically, given an PLM M, and the fine-tuned downstream model My de-
-=
where W. R < d) and b E R are trainable parameters, and d is the dimension of hidden vectors. Then the compressed vectors are fed into the original neural layers.

---

De- De-Compress
Adapter
Att / FFN
Att / FFN
Compression Plugin Insertion
Compress
Transformers
Figure 2: Illustration of Variator, which improves the computational efficiency via compressing the hidden vectors.
Hidden Decompression Layer. Compression layers merge multiple hidden vectors into global vector with information for all tokens in the corre- sponding group. To preserve the ability to solve token-level tasks, design hidden decompression layers, which are inserted after the original neural layer, to restore token-level information into the output vectors.
tional requirements of (ced-forweld layers in Trans- former (Zhang et al., 2022b), we attempt to apply compression plugins in feed-forward layers in most of our experiments.
3.3 Plugin Training
To mitigate information loss during the sequence compression of Variator, we design two-step train- ing strategy with plugin pre-training and plugin adaptation.
Given the output of the original neural layer g generated from the compressed vector gi we need to compute the output vectors for all k vec- tors in Gt- We first concatenate the original vector ha+j and the compressed output vector g to com- bine the token-level and group-level information. Then, instead of applying linear projection layer with high computation complexity, we adopt an Adapter (Houlsby et al., 2019) layer and resid- ual layer to project the concatenated vector to the output vector Osk+j
Plugin Pre-training. Plugin pre-training aims to learn general information compression ability and obtain good initialization of compression plugins for downstream models. In this step. compression plugins trained mitigate redundancy in the original input text. Specifically, we insert the com- pression plugins into the original PLM M, and train compression plugins on a pre- pre-training corpus. Notably, the pre-training process is task-agnostic. It is conducted only once and caters to the require- ments of all downstream tasks, which make com- pression plugins pratical even when PLMs scale to billions of parameters.
hnx+)) + bb)) + ba Oik+j = 8. + + "i+jj'
Mu+; (W
Here, WT Rrx2d by R". W Rdxr, by Rd are trainable parameters, and r & refers to the bottleneck dimension of adapter layers.
Plugin Adaptation. Plugin adaptation is de- signed to drive compression plugins to preserve task-specific information during compression. Dif- ferent tasks tend to pay attention to different infor- mation in the sequence. For example, sentiment analysis tasks usually need to maintain the infor- mation contained in emotional words, while read- ing comprchension tasks usually need to maintain information about the question. Therefore, it is im- portant for compression plugins learn different task information preferences in plugin adaptation. During plugin adaptation, compression plugins are inserted into downstream model My, and trained
Both two layers only involve minimal additional parameters and computation overhead and can sig- nificantly reduce the sequence length. Besides, our proposed compression plugins can be flexibly ap- plied in any neural layers, such as self-attention layers and feed-forward layers, allowing for dif- ferent acceleration ratios. Notably, the compres- sion and decompression layers can be implemented with other efficient operations including convolu- tional neural networks. Due to the high computa-

---

with task data.
4 Experiments
Both steps adopt knowledge distillation as the training objectives, guiding the compression plu- gins not to modify the output distribution. Given output vectors of the model without compression plugins O', and output vectors of the model with compression plugins O, the final training loss is computed as the mean squared error (MSE) be- tween O' and O: C C= C=110'-01
4.1 Datasets
To evaluate the effectiveness of Variator, seven typical NLP datasets evaluation bench- marks, including text classification and sequence- to-sequence generation tasks. Specifically, we adopt three natural language inference datasets, MNLI-m (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Wang et al., 2019), two sen- tence similarity datasets, QQP (Wang et al., 2019), MRPC (Dolan and Brockett, 2005), a sentiment analysis dataset, SST-2 (Socher al., 2013), and reading comprehension dataset, SQuAD (Ra- jpurkar et al., 2016). We apply Fl scores for MRPC, Fl scores and exact match scores (EM) for SQuAD, and accuracy for other datasets as evaluation met- rics. We also present the average scores on these seven datasets, where we use EM scores of SQuAD for average. Please refer to Appendix for statistics.
(1)
3.4 Complexity Analysis
In this section, we provide a complexity analysis of computational and storage overhead. Here we present the analysis with compression plugins ap- plied in feed cord-forwrrddd networks (FFNs), with the input length as n, hidden vector dimension as d, and the middle dimension of FFNS as 4d. As men- tioned in previous sections, & and r refer to the compression ratio and bottleneck dimension of de- compression layers.
4,2 Implementation Details
Computational Complexity, Compression and decompression layers involve several linear pro- jections with tiny matrices. Therefore, our com- pression plugins only require minimal computation costs. For each token, compression plugins contain three linear projection operations and two addition operations. The floating point operations (FLOPs) required by the compression and decompression layer are (kd+ 2dd+2+++3) +d)+ and (3rd+ 2d +r),,,, respec- tively. In contrast, the FLOPs of the feed-forward network are 8nd2. The computation costs of com- pression plugins are only about (4 + h + 3r) of FFN, where k, r < And compression plu- gins can reduce the computation costs of FFN to Therefore, compression plugins can achieve significant inference speed-up for PLMs.
We adopt the widely-used pre-trained model, T5- base and Ta-large (Raffel et al., 2020), as our model backbone. Please refer to Appendix for results of compression plugins on BERT backbone (Devlin et al., 2019). For the main experiments, we only insert the compression plugins around the feed- forward network layers in the encoder, which counts for the majority of computational require- ments. As for the training objective, the MSE loss with the output vectors from the last layers. The compression ratio is set as for the main experiments and the bottleneck dimension r of the Adapter layers is set as 64.
compute
For plugin pre-training, we apply the widely- used Wikipedia corpus. The learning rate is set as 10-3 and batch size is set as 256. We pre-train compression plugins for 60k steps. For plugin adap- tation, we apply grid search for Myper-paramen selection. We select batch size in (16, 32), learn- ing rate {10-4,5 x 10-53. The total training steps for each task are set as 26k, and we evalu- ate the models every 1k steps. We train all mod- els with half precision floating-poin ppint on NVIDIA A100 GPUs. For both plugin pre-training and adap- Adam for parameter optimization. Please refer Appendix for more details.
Storage Complexity. Different from training the entire models accelerate model inference, Variator relies on two projection layers to compress hidden vectors. Compression and decompression layers consist of three linear projection layers, with only Add + kt and 3rd + r + d parameters, respec- tively. In contrast, an FFN layer consists of 8df parameters.
To demonstrate the effectiveness of our parameter-efffcient compression plugins more in- 64, and
tation,
tuitively, we assume that k d 768. In this way, compression plugins can save 71.7% computational costs with only 3.4% additional parameters for FFNs.
4.3 Baselines
In this paper, we compare Variator with several competitive baseline models, including: (1) The

---

MNLI-m Acc.
QNLI
QQP Acc.
SST-2
MRPC
SQuAD EM/FI
Dataset
Avg.
Para.
FLOPS
Acc.
Acc.
T5 15-Basc
Original
86.7
93.0
91.2
82.9
94.3
92.6
82.8/90.0
89.1
Distillation
84.6
91.8
89.3
81.4
93.1
93.1
81.1/89.3
87.8
61.9%
44.3%
LTP
84.0
91.7
86.5
76.8
92.6
92.5
81.1/89.2
86.4
100.0%
44.3%
Variator
84.6
91.5
88.4
81.1
93.6
93.8
80,4/88.1
87.6
0,9%
46.8%
T5-Large
Original
88.9
94.0
91.5
88.6
95.4
93.0
85.3/92.5
91.0
Distillation
88.4
94.2
90.4
84.3
94.5
91.9
81.3/90.9
89.3
59.1%
52.5%
ITT
87.0
93.1
88.0
82.5
94.4
93.3
84.3/91.7
88.9
100%
52.5%
Variator
87.1
93.5
89.4
85.4
93.7
92.8
83.1/90.7
89.3
0.7%
54.1%
Table Comparison results between Variator and baseline models. Here Avg, refers to the average scores on seven datasets. Para. and FLOPs refer to the ratio of the number of additional parameters and floating point operations required by the compressed methods to the original PLMs.
original fine-tuned downstream PLMs without ac- celeration, which are also used as teacher models to guide the training of other compressed models. (2) The widely used model compression method, model distillation (Sanh et al., 2019). (3) Our method aims to reduce the sequence length for PLMs. which is inspired by previous token pruning models. Therefore, also compare Variator with typical token pruning model, LTP (Kim al., 2022), which adopts the attention scores im- portance scores, and only keeps tokens with the most scores for each layer. Notably, original token pruning models directly discard tokens for entire Transformer layers, and our models in main exper- iments focus on the acceleration of FFNs. There- implement fore, to make fair comparison, token pruming models with only skipping computa- tion of FFNS and only keeping 25% tokens in each layer. We apply knowledge distillation objectives to train all downstream tasks a fair comparison.
MNLI-m
SST-2
SQuAD EM/F1
Dataset
Acc.
Acc.
Variator
84.6
93.6
80.4/88.1
w/o PT
84.1
92.7
79.3/87.3
w/o PA
59.6
87.8
11.6/19.4
w/o Com
83.5
92.3
79. 19.4//7.1
w/o DeCom
73.8
86.1
38.2/50.6
Table 2: The results for ablation study.
from scratch, which require amounts of additional parameters. Limited by amounts of parameters, switching traditional methods between different compression ratios requires large memory space or repeatedly loading compressed models from the disk. (2) Compared to the widely-used model dis- tillation, our parameter-efficien model accelera- tion method achieve competitive performance with much fewer parameters, which indicates the po- tential of parameter-efficiee model compression. (3) Compared the token pruning baselines, our models can achieve better performance with even a small portion of parameters, which proves that merging tokens can better preserve sequence infor- mation compared to directly dropping them.
4.4 Main Results
The comparison results shown in Table To further demonstrate the effectiveness of Variator, we show the additional parameters, and FLOPs for each input required by compressed models. Here we assume the input length is 512 and batch size is 1 for calculating FLOPs. From the results, we can observe that: (1) Variator can achieve com- parable results with the original PLMMs using min- imal additional parameters with absolute perfor- mance drops of 2%. Specifically, Variator can save 53.2% and 45.9% computation costs for T5- base and Ta-large, using only 0.9% and 0.7% addi- tional parameters. In contrast, traditional accelera- tion methods need to construct compressed models
4.5 Ablation Study
To verify the effectiveness of each component Variator, we conduct an ablation study in this sec- tion. Specifically, we show the results of compres- sion plugins without plugin pre- vee-training (w/o PT) or plugin adaptation (w/o PA). Besides, we also examine the effectiveness of compression and de- compression layers in the ablation study. We show the model performance with compression layers replaced with mean-pooling operation (w/o Com) or decompression layers replaced with a copy op- eration (w/o DeCom). We run w/o Com and w/o

---

94
Dataset
MNLI-m
SST-2
Acc.
Acc.
Original
86.7
94.3
90
Variator (FFN)
84.1
92.7
Variator (Atl)
81.0
91.9
88
MALI
55T-2
Variator (Att-KV)
83.1
92.1
Table The results for compression plugins inserted around the self-attention layers.
84
5
10 15
30
the compression ratio increasing, the model per- formance decreases expected. But the decline rate is becoming slow, which indicates the poten- tial for Variator achieve higher compression ra- tios. (2) Variator can achieve competitive perfor- mance even when the compression ratio reaches 32, where Variator maintains 95.4% and 96.7% ac- curacy scores of original PLMs for MNLI-m and SST-2, respectively, while reducing 69% computa- tional costs. The satisfactory performance ensures the response speed of real-world applications when the system load is high.
Figure 3: Model performance with different compres- sion ratios. The horizontal lines indicate the perfor- mance of original PLMs without compression plugins.
DeCom without plugin pre-training to speed up experiments. We select three tasks for the abla- tion study, including sentence classification, SST-2, sentence-par paire classification, MNLI-m, and reading comprehension, SQuAD.
The results are shown in Table 2. From the re- sults, we can observe that: (1) Both two training steps contribute to the main model, when any- step is missing. the model performance drops significantly. (2) Plugin adaptation is important for all tasks. Plugin pre-training guides compression plugins to discard general redundant information contained the input text. Therefore, for SST-2, which usually only focuses on parts of important words, compression plugins without task-specific adaptation can also achieve satisfactory results. In contrast, for SQuAD and MNLI-m, which require models to collect information from entire contexts, plugins without adaptation lead to large perfor- mance drop. (3) Compression and decompression layers play an important role in selecting informa- tion for hidden merging and restoring token-level information, as without anyone of them, model per- formance drops significantly. Especially, decom- pression layers are quite important for preserving token-level information, and training compression plugins without decompression layers lead to large drops for the span extraction task, SQuAD.
4.7 Compression for Attention Layers
In our main experiments, we insert the compression plugins around the FFN layers. In this section, we examine the performance of Variator when we in- sert compression plugins around the self-attention layers. Here we not perform plugin pre-training. The results are shown Table 4. For comparison, we also present the results of original models and Variator with plugins in FFN layers.
From the results, we can observe that Variator with plugins in self-attention layers perform worse than plugins in FFN layers. That is because self- attention layers are designed to fuse token-level information, and inserting hidden compression lay- ers before self-attention layers would lead the loss of token information. Thus in the self-attention layers, only the k-gram information integration is performed, resulting in a significant performance drop. To address this issue, we improve compres- sion plugins for self-attention layers by only com- pressing key and value vectors, denoted as Variator (Att-KV). With compression only for key and value vectors, Variator (Att) can achieve comparable sults with Variator (FFN). And compressing key and value vectors can be further adopted in decoder- only models to reduce the sequence length of past key-value vectors, which we leave for future work.
4.6 Effects of Compression Ratios
Variator apply compression plugins to compress multiple hidden vectors into one, thus achieving inference speedup. In this section, we explore the effects of compression ratios for compression plugins. We construct compression plugins with compression ratios as {2, 4, 8, 16, 32). The results are shown Figure
From the results, we can find that: (1) With

---

Tosk
Rotio of Activated Neurans (56)
3.0
S5
2.5
Accurso)
95
2.0
1.5
93
Taakk
SS7-2 557-2-bbrittrrr
0.5
10
15
20
25
30
Parameters
Figure 4: Performance with different backbone sizes,
Figure 5: The ratio of activated neurons with different compression ratios on two datasets.
4.8 Scaling to PLMs with Billions of Parameters
approach can reduce the computational overhead while maintaining comparable performance with the original model for decoder-only LLMs.
In this section, attempt to apply compres- sion plugins to PLMs with billions of parameters. We adopt four variants of T5 our backbones. including T5-base (200 million parameters), T5- large (700 million parameters), T5-XLarge (3 bil- lion parameters), and T5-XXe (11 billion pa- rameters). Following the main experiments, for each model, we conduct the two-step training pro- cess with 6k-step plugin pre-training and 26k-step plugin adaptation. We apply parameter-efficien learning method, LoRA (Hu al., 2021), to train the task-specific downstream models to speed up the experiments. We show the results of the SST-2.
4.9 Neuron-Level Analysis
Our compression plugins enable the feed-forward layers to process information from multiple tokens simultaneously save computational costs. In this section, attempt to explore the computational mechanism of our compression plugins from the perspective of activated neurons. Previous works find out that FFNs can be regarded as memory net- works (Geva et al., 2021; Dai et al., 2022), and the activated neurons can be used as indicators to reflect what information is preserved in the input hidden vectors. T5 adopts ReLU (Nair and Hin- ton, 2010) as the activation function, and following Zhang al. (2022h), define activated neurons as ones with positive (non-zero) activation values. We present the average ratio of activated neu- rons with different compression ratios, {1, 2, 4, 8, 16, 32), in Figure 5. From the results, we can observe that the ratios of activated neurons drop with the increase in compression ratios. When the compression ratio reaches 32, only less than 2he neurons activated process sequence informa- tion. In indicates that compressed hidden vectors only contain the necessary information for the quences and discard unimportant ones. Besides, the low activated ratios also indicate the potential of the combination of Variator and neuron pruning methods (Zhang et al., 2022b) to further improve the computational efficiency.
As shown in Figure 4, the performance continues to improve with the increasing of backbone model sizes. Similar to previous parameter-efficient learn- ing methods (Lester et al., 2021; Ding al., 2022), the performance gap between Variator and original PLMs becomes small when the model scales to billions of parameters. It shows the potential of Variator to be applied in nowadays general PLMS with more than 100 billion parameters, such as ChatGPT and GPT-4 (OpenAI, 2023).
LLaMA-7B
Variator (w/o PT)
SST-2
97.3
96.3
Table 4: The results for compression plugins in LLaMA. Besides, to present the effectiveness of Variator on decoder-only LLMs, we evaluate Variator with recent popular backbone LLaMA (Touvron et al., 2023) with 7 billion parameters. Variator can be used for the input encoding acceleration and reduce the service latency in real-world applications. We conduct experiments with compression ratio of on the FFN layers and without plugin-pretraining pretraining to accelerate experiments. The results suggest that our
Then we further explore the relationship between the activated neurons of FFNs with compression plugins and the activated neurons of original FFNs. Specifically, we denote the intersection set and union set of activated neurons & hidden vec-

---

4
16
62236004), T Joint Research Fund, and Institute Guo Qiang Tsinghua University.
CNZ
0.89
0.85
0.79
0.66
0.61
cnzt/el
0.88
0.89
0.93
0.98
0.99
Author Contributions In the preparation and discussion of the project, Chaojun Xiao, Yuqi Luo, and Xu Han designed the model architec- tures. Chaojun Xiao and Yuqi Luo wrote the code and conducted the experiments. Besides, Wenbin Zhang and Pengle Zhang wrote the code for baseline models and ablation study. Chao- jun Xiao wrote the initial draft. Xu Han, Yankai Lin, Zhengyan Zhang, Ruobing Xie, and Zhiyuan Liu significantly edited and improved the paper. Maosong Sun and Jie Zhou provided valuable ad- vice to the research.
Table 5:
The relationship between activated neurons of Variator and original models.
tors I and U. The set of activated neurons of compressed vector C. The intersection set Z can be regarded as important global information for hidden vectors, and U can be regarded as all infor- mation contained in the & hidden vectors. Compres- sion layers are used to select important information and feed them into neural layers. Therefore, we hope that I is approximately a subset of C and C is upproximately subset 2.. In Table 5, present what fraction of neurons in I are in C and what frac- tion of neurons in C are in U. From the results, we can observe that when the compression ratios are no more than 8, the relationship between the three sets approximately satisfies the abovementioned inclusion assumption. It proves the effectiveness of our compression plugins in preserving global information. When the compression ratios become larger (such as 16, 32), only more than 70% neurons in I contained in C. That is because with the increase of compression ratios, selecting global important information from multiple vectors becomes challenging for compression layers with limited parameters. It also shows the potential to add regularization from the neuron level for com- pression plugs to preserve important information.
Limitations
We discuss the limitations of Variator in this sec- tion: (1) In the experiments, we implement Variator with T5 as backbone. It is worth exploring ap- plying Variator in other large-scale decoder-only pre-trained models. (2) In this paper, we mainly fo- cus on accelerating the encoding process of PLMs. Language decoding also plays an essential role in real-worke world applications. show the potential of Variator to compress key and value vectors for accelcration. We believe Variator can also serve as flexible framework to speed up decoding. (3) Our plug plug-and-plaa and-play compres- sion framework parallels other model compression methods. It is worth exploring the combination of multiple acceleration methods to achieve more efficient and effective model inference frameworks.
the experiments, we
5 Conclusion
In this paper, explore the parameter-efficient acceleration setting and propose Variator, which reduces the computational costs with compres- sion plugins. The extensive experiments on seven datasets show that we can reduce 53% computa- tional costs with only 0.9% additional parameters. In the future, we will explore more effective token- merging frameworks to improve compression plug- ins. Besides, we will further decouple compression plugins from specific tasks, thus we can construet compression plugins once and for all with transfer- ability across multiple tasks.
References
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dal- las Card, Rodrigo Castellon, Niladri S. Chatterji. Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha. Tatsunori Hashimoto, Peter Henderson, John He- witt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keel- ing, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and al. 2021. On the opportunities and risks of foundation models. CoRR, abs/2108.07256.
Acknowledgement
This work is supported by the National Key R&D Program of China (No.2022ZD01 16312), Na- tional Natural Science Foundation of China (No.

---

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vore Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Firic Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are sho-shot learners. In Proceedings of NewrIPS.
Manish Gupta and Puneet Agrawal. 2022. Compression of deep learning models for text: A survey. TKDD, 16(4):61:1-61:55
Xu Han. Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yugi Huo. Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang. Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song. Jie Tang. Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past, present and future. Al Open, 2:225-250.
Tianlong Chen, Jonathan Frankle, Shiyu Chang. Sijia Liu, Yang Zhang. Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained BERT networks. In Proceedings of NerulPS.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards unified view of parameter-efficien transfer learning. In Proceedings of ICLR.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of ACL. pages 8493-8502.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in neural network. CoRR, 0t//1503302531
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, 2019. Parameter-eeffcient transfer learning for NLP. In Pro- ceedings of ICMI,, volume 97, pages 2790-2799.
Zihang Dai, Guokun Lai. Yiming Yang, and Quoc Le. 2020. Funnel-trante tranfforrr:: Filtering out sequential redundancy for efficient language processing. In Proceedings of NewrIPS.
Jacob Devlin. Ming- Wei Chang, Kenton Lee, and
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, thss2100.09688.
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Proceedings of NAACL-HLT, pages 4171-4186.
Gyuwan Kim and Kyunghyun Cho. 2021. Length- adaptive transformer: Train once with length drop, use anytime with search. In Proceedings of ACL- IJCNLP, pages 6501-6511.
Ning Ding, Yujia Qin. Guang Yang, Fuchao Wei, Zong- han Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang. Juanzi Li, and Maosong Sun. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. CoRR, abs/2203.0699)
Sehoon Kim, Sheng Shen, David Thorsley, Amir Gho- lami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. Learned token pruning for transform- ers. In Proceedings of KDD, pages 784-794.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of IWP@ IJCNLP
Anne Lauscher, Tobias Lüken, and Goran Glavas. 2021. Sustainable modular debiasing of language models. In Findings of ACL: EMNLP, pages 4782-4797.
Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with struc- tured dropout. In Proceedings of ICLR.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of EMNLP, pages 3045- 3059.
Mor Geva, Roei Schuster, Jonathan Berant, and Oner Levy. 2021. Transformer feed-forward layers are key- value memories. In Proceedings of EMNLP, pages 5484-5495.
Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. 2022. Multi granuligr structural knowledge distillation for language model compression. In Pro- ceedings of ACL. pages 1001-1011.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sabhar- wal, and Ashish Verma. 2020. Power-bert: Accel- erating BERT inference via progressive word-vector climination. In Proceedings of ICML, volume 1 119, pages 3690-3699.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107. 13586.
Demi Guo, Alexander M. Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff prun- ing. In Proceedings of ACL-IICNLP, pages 4884
Andrea Madotto, Etsuko Ishii Zhaojiang Lin, Sumanth Dathathri, and Pascale Fung 2020. Plug-and-play conversational models. In Findings ACL, volume EMNLP 2020, pages 2422-2433.
4896.

---

Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. 2023. Split computing and early exiting for deep learning applications: Survey and research challenges. CUSR, 19:0::00
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of EMNI.P-IJCNIP, pages 4322-4331.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than onc? In Proceed- ings of NeurlPS, pages 14014-14022
Zhiqing Sun, Hongkun Yu, Xiaodan Song. Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilehert: a compact task-agnostic BERT for resource- limited devices. In Proceedings of ACL, pages
Vishvak Murahari, Carlos E. Jimenez, Runzhe Yang. and Karthik Narasimhan. 2022. Datamux: Data mul- tiplexing for neural networks. In Proceedings of NeurlPS.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, ahs/2302. 13971.
Vinod Nair and Geoffrey E. Hinton, 2010. Rectified linear units improve restricted boltzmann machines. Proceedings of ICML, pages 807-814.
OpenAl. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Proceed- ings of ICLR.
Damian Pascual, Beni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer, 2021. A plug- ald-play method for controlled text generation. In Findings of ACL, volume EMNLP 2021, pages 3973- 3997.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang. Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou, 2021. K-adapter; Infusing knowledge into pre-trained models with adapters. In Findings of ACL, volume ACL/ICNLPP 2021, pages 1405-1418, 1418.
Xipeng Qiu, Tianxiang Sun. Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey, CoRR, abs/2003.08271.
Adina Williams, Nikita Nangia, Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Pro- ceedings of NAACL-HLT, pages 1112-1122.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. JMLR, 21: -0:1-64
Mengzhou Xia, 7exuan Zhong, and Danqi Chen. 2022.
Structured pruning learns compact and accurate mod- els. In Proceedings of ACL, pages 1513-1528.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383-2392.
Chaojun Xiao, Zhengyan Zhang, Xu Han, Chi-Min Chan, Yankai Lin. Zhiyuan Liu, Xiangyang Li. Zhonghua Li, Zhao Cao. and Maosong Sun. 2023. Plug-and-play document modules for pre-trained models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- 1: Long Pupers), ACL. 2023, Toronto, Canada, July 9- 9-14, 2023, pages 15713-15729. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR. abs/1910.01108.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. REPLUG: retrieval-augmenned black-box language models. CoRR, abs/2301.12652.
Guangxuan Xiao, Lin, Mickaël Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large lan- guage models. CoRR, abs/221 1.1048..
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Mamning, Andrew Y. Ng. and Christopher Pouts. 2013. Recursive deep models for semantic compositionality over a sentiment tree- bank. In Proceedings of EMNLP, pages 1631-1642.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating BERT inference. In Proceedings of ACL, pages 2246-2251.
Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. 2021. Training with quantization noise for ex- treme model compression. In Proceedings of ICLR.
Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. 2021. TR-BERT: dynamic token reduction for accelerating BERT inference. In Proceedings of NAACL-IHLT pages 5798-5809.

---

Zichum Yu. Chenyan Xiong, Shi Yu, and Zhiyuan Liu. 2023. Augmentation-adapter retriever improves gen- eralization of language models generic plug-in. In Proceedings of the 6lst Annual Meeting the Association for Computational Linguistics ( (Volume 1: Long Pupers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pages 2421-2436. Association for Com- putational Linguistics.
A Training Details
In this section, we describe some training details, including the datasets and in our experiments.
used
A.1 Datasets
As for the plugin pre-training corpus, adopt widely-used Wikipedia corpus for pre-training. To facilitate the pre-training process, we split each document into several paragraphs with 128 tokens.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-basee masked language-models In Proceedings of ACI,, pages 1-9.
As for the plugin adaptation datasets, we adopt seven widely used language understanding datasets as our evaluation benchmarks. As we use T5 (Raf- fel et al., 2020) as our backbone, we formalize all these tasks into sequence- toaeeettsssooo to-sequence formats. The detailed statistics and the input template are shown in Table 6.
Zhengyan Zhang, Baitao Gong, Yingfa Chen, Xu Han, Guoyang Zeng, Weilin Zhao, Yanxu Chen, Zhiyuan Liu, and Maosong Sun. 2022a. Bmcook: task- agnostic compression toolkit for big models. In Pro-
ceedings of EMNLP, pages 396-405.
Zhengyan Zhang, Yankai Lin. Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022b. Moefication: Transformer feed-forward layers are mixtures of ex- perts. In Findings of ACL, pages 877-890.
A.2 Implementation Details
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang. Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li. Maosong Sun. and Jie Zhou. 2023, Plug- Plug-and-play and-play knowledge injection for pre-trained language models. In Proceedings of ACL, pages 10641-10658. Association for Computational Lin- guistics.
In this subsection, we describe the implementation details used in our experiments.
As for plugin pre-training, we use 8 A100 (80G) GPUs to train Variator on T5-base for 4.9 hours and T5-large for 9.9 hours. We adopt the knowledge distillation objectives to pre-train plugins. Follow- ing settings in Raffel et al. (2020), the mean length of the masked span set as 3, and the mask ratio is set as 0.15.
As for baseline implementation, we fine-tune the original T5 with lcarning rate searched from [10 5 x 10 5 10-5] and batch size searched from {16, 32). The checkpoints with the best vali- dation performance are used as the teacher models to distill all other baselines. For distillation models, we first conduct task task-agnostic distillation for 10k: steps on the Wikipedia corpus, where the learning rate is set as 10 and the batch size set as 256. For both distilled models and token pruning mod- els, we fine-tune them on downstream data using distillation objectives, with learning rate searched from {10-3,6 5 x 10-5), and batch size searched from {16, 32).
B Training Objectives
In this paper, we adopt knowledge distillation to guide the training of compression plugins to pre- serve token-level and sequence-level information. is intuitive adopt the task-specific loss func- tion to optimize the parameters of compression plugins. In this section, we explore the effects task-specific objectives and nowllege-dissillatin

---

Dataset
Train
Validation
Input Template
MNLI-m
393k
9.8k
Sentence 1: PREMISE Sentence 2: HYPOTHESIS Does sentence
entails sentence 2? <extra_id_0"
QNLI
105k
5.3k
Question: QUESTION Sentence: SENTENCE Does the sentence con-
tains the answer the question? <extra_id__ 0_>>
QQP
364k
40.4k
Question 1: QUESTION1 Question 2: QUESTION2 Are the ques-
tions paraphrase of each other? <extra_id_@t
RTE
2.5k
277
Sentence SENTENCE1 Sentence SENTENCE2 Does sentence
entails sentence 2? <extra_id_0"
SST-2
67.3k
872
Sentence: SENTENCE Does this sentence express positive or nega-
tive emotions? <extra_id_@)
MRPC
3.7k
408
Sentence 1: SENTENCE1 Sentence 2: SENTENCE2 Are the two sen-
tences paraphrase of each other? rextra_id___
SQUAD
87.k
10.6k
Question: QUESTION Context: CONTEXT Answer: <extra_id_@>
Table 6: The statistics and input templates of downstream datasets. In the templates, the task-specific inputs are denoted monospaced font, and gextra_id_@e refers to special mask token TS.
MNLI-m
SST-2
C Compression Plugins for BERT
Dataset
Ace.
Acc.
Our compression plugins can be applied in Transformer-based pre-trained models. In this sec- tion, we explore inserting compression plugins into the widely-used encoder-only pre-trained model, BERT (Devlin et al., 2019). We adopt the 100- million million-parameter parameter version, BERT-base, as our back-
= 0
84.6
93.6
= 0.1
83.3
92.2
A = 0.5
83.1
92.4
Table 7: The performance with different training objec- tives.
bone. Following the main experiments, set the compression ratio as 4 and the bottleneck dimen- sion as 64. We conduct plugin pre-training for 24k steps. The results are shown Table 8. From the results, we can observe that Variator on BERT can also show competitive results, and longer plugin pre-training is supposed to lead to better perfor- mance.
MNLI-m
SST-2
Dataset
Acc.
Acc.
Original
83.3
93.0
Variator (BERT)
80.0
90.3
Table The performance with compression plugins in BERT.
objectives. Here, we denote the task-specific loss L, and the distillation loss as La. The final loss is calculated = XL, + La- We present the performance of Variator with compression ratio as on T5-base.
As shown in Table 7, we can find that training with task-specific objectives leads to performance drop on both MNLI-m and SST-2 datasets. That is because task-specific loss is usually easier to opti- mize than distillation loss, and adding task-specific loss functions makes our compression plugins more likely to fall into the local optimum of the model. Therefore, other experiments, we only utilize the distillation loss functions to optimize compression plugins.


Reviews:
[{"review_id": "z7JdhPo75J", "reviewer": "Reviewer_rvsY", "summary": "", "questions": "1. What if you use CNN as the module to shorten the sequence length?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestion and positive comments.\n\n**R1: Comparison with convolutional layers**\n\nThank you for your suggestion. We explored the use of a convolutional neural network as a compression layer in preliminary experiments, and the results were comparable to our current compression approach. We will address this and provide additional discussion on this aspect in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is smart and interesting, with very limited additional overhead.\n2. The experimental results demonstrate its effectiveness across many datasets.", "reasons_to_reject": "1. The method reminds me of using a convolutional layer with stride > 1 (e.g., =k) to process the input sequence. The author should compare to this.", "questions_to_authors": "1. What if you use CNN as the module to shorten the sequence length?", "paper_topic_and_main_contributions": "This paper presents a parameter-efficient acceleration approach called Variator, which reduces the computational costs with compression plugins by compressing input tokens/hiddens into a compressed representation to shorten the length. The extensive experiments on seven datasets show that Variator can reduce 53% computational costs with only 0.9% additional parameters.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9uvGCsMx7q", "reviewer": "Reviewer_prtK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestion and positive comments.\n\n**R1: Advantages of Variator**\n\nThe primary objective of this paper is to enhance the flexibility of compression algorithms while reducing the additional storage overhead required by model acceleration techniques.\nSpecifically, this work focuses on a novel plug-and-play acceleration paradigm, which aims to introduce only a small number of additional parameters to achieve model inference acceleration, enabling dynamic selection of acceleration ratios, and reducing storage overhead.  The experimental results in Table 1 demonstrate that Variator can achieve plug-and-play model acceleration with only 0.9\\% additional parameters, and the performance is comparable to traditional acceleration methods which require large additional storage for the compressed models trained from scratch.\n\n**R2: Model size compression**\n\nAs disscussed in Section 3.1, Variator is not designed for compressing the model size; rather, it introduces a minimal additional module to achieve inference acceleration. The traditional approaches, such as distillation, necessitate constructing different models for different tasks and acceleration ratios, incurring substantial storage overhead. For instance, considering an LLM with 10 billion parameters, obtaining compressed models for N tasks at computation reduction of 1/2, 1/3, and 1/4 would require creating 3N compressed models, each ranging from 2.5 to 5 billion parameters. In contrast, Variator achieves acceleration with less than 1\\% additional parameters for each task and acceleration ratio, significantly reducing storage overhead and the plug-and-play characteristic enables dynamic selection of acceleration ratios.\n\n**R3: Latency reduction**\n\nWe further conduct latency testing to assess the reduction brought about by Variator. Our supplementary tests reveal that for T5-large, Variator reduces the FFN computation latency from 0.18s to 0.103s (including compression, decompression, and FFN computation). This demonstrates a tangible enhancement in model computational efficiency. We will incorporate a discussion on latency in the revised version.\nMoreover, as part of our ongoing efforts, we plan to integrate specific efficient computational operators that align with the Variator framework. This step is anticipated to result in even more pronounced improvements in latency reduction. \n\n**R4: Comparison with token merging**\n\nThank you for your valuable suggestion. As previously mentioned in Q1, Variator aims to enhance the flexibility and storage efficiency of acceleration algorithms. To this end, we utilize a token merging algorithm to implement compression plugins. Token merging techniques commonly applied in computer vision could potentially be adapted for implementing compression and decompression layers, which we leave for future work. In the revision, we will provide a discussion on the differences between Variator and token merging algorithms in CV community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written, making it easy to comprehend and follow. The figures' visual quality is exceptional, with the teaser figure being a particular highlight. The proposed method is technically sound and achieves decent empirical performance. The authors have thoughtfully included ample implementation details, enabling other researchers and practitioners in the community to reproduce their work successfully. Furthermore, the authors' commitment to releasing their codebase upon paper acceptance adds to the paper's transparency and facilitates further exploration and validation of their findings.", "reasons_to_reject": "My major concern with this paper centers around its experimental evaluation, which requires further clarification and elaboration.\n\n* Firstly, upon reviewing Table 1, I observed that the proposed Variator achieves comparable performance to the other two baselines, particularly Distillation, with similar accuracy and #FLOPs. This similarity raises questions about the distinct advantages of Variator over existing methods.\n* Secondly, I find it challenging to grasp why the proposed Variator claims to compress the model size. From my understanding, Variator introduces additional transformation layers to the original model, which should theoretically increase the model size instead of reducing it. The paper needs to provide more clarity on this aspect.\n* Thirdly, it is important to note that #FLOPs (floating-point operations) represent a hardware-agnostic metric. A reduction in #FLOPs may not necessarily equate to reduced latency on specific hardware. To address this limitation, I recommend the authors report measured latency on hardware in both tables and figures, providing a more comprehensive evaluation of the proposed method's real-world performance.\n\nAdditionally, I noticed that the proposed method bears similarities to token merging, where similar tokens are merged to save computation, as effectively demonstrated in the computer vision community. In light of this resemblance, the authors should offer both verbal and quantitative comparisons with token merging to highlight the uniqueness and advantages of Variator.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces Variator, a parameter-efficient acceleration method designed to enhance the computational efficiency of large language models (LLMs) without sacrificing performance. The proposed approach employs compression plugins that consolidate multiple hidden vectors into a single one, offering dynamic selection of various plugins with different acceleration ratios, depending on the workload. The effectiveness of Variator is validated on seven datasets, demonstrating a 53% reduction in computational costs with only 0.9% additional parameters and less than a 2% performance drop.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o9ezUQosBE", "reviewer": "Reviewer_3yEr", "summary": "", "questions": "1. The proposed method seems only suitable for language understanding tasks since it introduces hidden vector compression within a group (i.e., future information). So, how to utilize this method into generation model, especially current LLMs usually refer to generation models, rather than understanding.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestion and positive comments.\n\n**R1: More backbones**\n\nIn the appendix, we present experimental results on BERT, which similarly confirm that significant inference acceleration can be achieved with a minimal number of additional parameters. In the revised manuscript, we will incorporate experiments using LLaMA as the backbone to further validate the effectiveness of our method. \nBesides, as shown in Section 4.8, we provide experimental results on widely-used LLMs, T5-XLarge (3 billion parameters) and T5-XXLarge (11 billion parameters). The results demonstrate that Variator can achieve promising performance when the model scales to billions of parameters.\n\n**R2: Missing references**\n\nThe primary contribution of our paper lies in exploring a novel, plug-and-play approach for model inference acceleration with minimal additional parameters. Sequence compression is one of the mechanisms used to achieve this goal within this specific context. Thus, the distinction between our work and [a] is as follows: [a] focuses on a more efficient pre-training architecture and requires retraining the model from scratch. In contrast, we concentrate on accelerating the inference of existing pre-trained models for downstream tasks, accomplished by employing exceedingly small-scale plugins while preserving the integrity of the original model.\nWe will include a discussion on the differences between Viarator and [a] in the revision.\n\n**R3: Plugin pre-training**\n\nAs discussed in Line 291-294, the pre-training process in our method is task-agnostic and serves the purpose of finding an appropriate initialization point for the plugin. It is conducted only once and caters to the requirements of all downstream tasks. This process shares similarities with task-agnostic model distillation methods, which is widely employed for LLMs. These methods first obtain a distilled version of a pre-trained model on task-agnostic pre-training data and then adapt it to specific tasks. \nMoreover, the overhead introduced by plugin pre-training is minimal: during this phase, we keep the parameters of the original LLM unchanged and only update the plugin's parameters (which constitute less than 1\\% of the LLM's parameters). This reduction in parameter count contributes to a decrease in memory consumption and computational burden. Notably, the plugin pre-training requires only 60k steps, significantly fewer than the nearly one million steps needed for traditional language model pre-training (such as masked language model and autoaggressive language model).\n\nAs shown in Table 2, Variator without pre-training can also achieve good performance, which indicates the flexibility and effectiveness of Variator.\n\nIn the future, we will release the pre-trained compression plugins to promote future research and application.\n\n**R4: Comparison with other compression models**\n\nThe main focus of our paper is on the context of lightweight acceleration achieved through minimal parameter addition. It is foreseeable that traditional compression setups generally involve building a compression model from scratch, demanding substantial additional storage space. This diverges from the specific focus of our work.\nFurthermore, the proposed algorithm can be seamlessly integrated with conventional compression techniques such as distillation, pruning, and quantization, as they serve complementary purposes. In the revised manuscript, we will include additional results that encompass a broader spectrum of traditional compression techniques to offer a comprehensive comparative analysis.\n\n**Q1: Generation tasks**\n\nThe algorithm we propose is not limited to language understanding tasks, and it can also be effectively applied to sequence-to-sequence tasks. For instance, in our experiments, we evaluate Variator on question-answering tasks, SQuAD, which we formalize as sequence-to-sequence generation task. This demonstrates the framework's compatibility to reduce the computational overhead of the input encoding process.\nBesides, Table 3 displays the effectiveness of key and value vector compression within attention layers. This approach can also be extended to the decoding process by compressing key and value vectors, thereby reducing the computational burden of attention mechanisms. Consequently, we believe our algorithm possesses a robust versatility that lends itself to various types of tasks, including generation tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method attempts to accelerate model speed by compressing hidden vectors and thus reducing sequence length.", "reasons_to_reject": "1. The experiments are only conducted on T5, which lacks enough convincing since the authors demonstrate their experimental settings are for LLMs. Therefore, more LLMs (e.g., LLaMa) should be provided. Besides, it will be better if authors can also provide results on BERT to demonstrate the generalization of the proposed method.\n2. The idea of the proposed method (compress sequence) is similar to [a], but this paper does not cite or discuss this paper. Authors should give some discussions about the difference of [a].\n3. The proposed method introduces (plugin) pre-training into the training, hence I do not think the proposed method is suitable for large language models (at least 7B like LLaMa) and only suitable for some pre-trained models. Since large language models usually cost enormous memory, it is impractical for LLMs to pre-train. As the target of this paper is for acceleration, using pre-training has violated this policy.\n4. Experimental results only compared with distillation and LTP, without any comparison of other compression models (e.g., TinyBERT, MiniLM, etc)\n\n[a] Zihang Dai, Guokun Lai, Yiming Yang, Quoc Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing", "questions_to_authors": "1. The proposed method seems only suitable for language understanding tasks since it introduces hidden vector compression within a group (i.e., future information). So, how to utilize this method into generation model, especially current LLMs usually refer to generation models, rather than understanding.", "paper_topic_and_main_contributions": "Although Large language models have achieved significant performance in many tasks, the enormous computations and memory cost are still a challenging problem. In this paper, the authors propose Variator, a parameter-efficient acceleration method that enhances computational efficiency via a plug-and-play way. Here, they introduce a compression plugin, which reduces the sequence length by compressing multiple hidden vectors into one and then training with frozen LLM.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The emergence of \"compression plugin'' or \"plugin\" is confusing or sudden. Maybe first giving some descriptions of this noun."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces Variator, a parameter-efficient acceleration method for large language models (LLMs) that aims to reduce computational costs and enhance efficiency. Variator utilizes compression plugins to compress multiple hidden vectors into a single representation, shortening the sequence length. Experiments conducted on seven datasets show a 53% reduction in computational costs with only a 0.9% increase in parameters and less than a 2% performance drop.

Strengths:
* The idea is novel and interesting, with limited additional overhead.
* The method is technically sound and demonstrates effectiveness across various datasets.
* The paper is well-written and provides sufficient details to reproduce the results.

Weaknesses:
* The method has similarities to existing approaches, such as convolutional layers with stride > 1 and token merging, and requires comparisons with these methods.
* The experimental evaluation needs further clarification and elaboration, including comparisons with other compression models and exploring the method's applicability to a wider range of LLMs.
* The proposed method may not be suitable for large language models requiring pre-training, as it may violate the acceleration objective.
The method's suitability for generation models needs to be addressed.

Overall, the paper has merits and potential and the weaknesses and concerns raised by the reviewers are partially addressed.
Decision: Accept-Findings