The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
Authors: Satya Sai Srinath Namburi GNVV
Makesh Narsimhan Sreedhar
Srinath Srinivasan
Frederic Sala
Venue: EMNLP 2023 Findings (2023)
Keywords: Pruning
Quantization
Language Models
TL;DR: The paper is about understanding the impact of various compression techniques (pruning, quantization) on parametric knowledge and the analysis is done on all families of transformers/LLMs at various scales.

Abstract: Compressing large language models (LLMs), often consisting of billions of parameters, provides faster inference, smaller memory footprints, and enables local deployment. 
The standard compression techniques are pruning and quantization, with the former eliminating redundant connections in model layers and the latter representing model parameters with as little as 4 bits. The key tradeoff is between the degree of compression and the impact on the quality of the compressed model. Existing research on LLM compression primarily focuses on performance in terms of general metrics like perplexity or downstream task accuracy. More fine-grained metrics, such as those measuring parametric knowledge, remain significantly underexplored. To help bridge this gap, we present a comprehensive analysis across multiple model families  using the LAMA and LM-Harness benchmarks in order to systematically quantify the effect of commonly employed compression techniques on model performance. A particular focus is on tradeoffs involving parametric knowledge, with the goal of providing practitioners with practical insights to make informed decisions on compression.

The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, Frederic Sala
University of Wisconsin Madissn (sgnamburi, msreedhar, srinivasan32, fsala)@wisc.ed
Abstract
focused on developing novel compression tech- niques that enable efficient local deployment and inference. Notable examples of such techniques include SparseGPT (Frantar and Alistarh, 2023) and LLM.int80 (Dettmers et 2022).
Compressing large language models (LLMs), often consisting of billions of parameters, pro- vides faster inference, smaller memory foot- prints, and enables local deployment. Two standard compression techniques are pruning and quantization, with the former eliminating redundant connections in model layers and the latter representing model parameters with fewer bits. The key tradcoff is between the degree of compression and the impact on the quality of the compressed model. Existing research on L.LM compression primarily fo- performance in terms of general met- rics like perplexity or downstream task accu- racy. More fine-grained metrics, such as those measuring parametric knowledge, remain sig- nificantly underexplored. To help bridge this gap, present comprehensive analysis across multiple model families (ENCODER, ENCODER-DECODEER and DECODER) using the LAMA and LM-HARNESS benchmarks in order to systematically quantify the effect of commonly employed compression techniques model performance. A particular focus is on tradeoffs involving parametric knowledge, with the goal of providing practitioners with practical insights to help make informed deci- sions on compression. We release our code- base to enable further research.
The tradeoff between model compression and quality is typically studied either through general metrics like perplexity (See et al., 2016; Michel et al., 2019) standardized benchmark task accu- racy (Liang et al., 2021; Du al., 2021) on, e.g.. GLUE (Wang et al., 2018). Furthermore, much of the literature studies such tradeoffs for one model or a particular class of models. Unfortunately, as a result, practitioners do not have access to reliable insights or rules-of-thumb to ensure they can make an informed decision for compression in their own models. This is because
Metrics like perplexity too general, while benchmark prediction metrics are easy fool. For example, recent findings sug- gest that distilled versions of foundational LLMs, known as imitation models, may ex- hibit stylistic similarities but potentially lack knowledge when compared to the models they seek to imitate (Gudibande et al., 2023), Most recent research compression tech- niques has primarily focused on DECODER models. The applicability and effectiveness of such techniques for large ENCODER and RUCODER-DECODEE models (Chung et al., 2022) has yet to be extensively studied.
1 Introduction
Large language models (LLMs) have demon- strated exceptional performance across diverse tasks. However, their deployment in real-world ap- plications is hindered by their substantial size and the associated costs, even for inference (Schwartz et al., 2020; Strubell et al.. 2019). For in- stance, the LLama-65B model (Touvron et al., 2023), pioneering open-sourced LLM, uses ap- proximately 130GB of RAM for 16-bit inference. To address this challenge, recent research has
These difficulties suggest that there is a need for a more fine-grained understanding of the effects of compression schemes, comparing a variety of model families, compression techniques, and spe- cialized measurements.
We address these challenges, specifically focus- ing on the preservation of parametric knowledge, i.c., knowledge acquired during pretraining, that stored in model weights. This is particularly

---

crucial for tasks involving reasoning and for spe- cialized applications. Concretely, we examine the impact of different compression schemes on parametric knowledge across multiple model fam- ilies (ENCODER, ENCODER-DECOME CODER) where we apply pruning and quantization approaches and analyze the performance of such techniques downstream reasoning tasks. To the best of our knowledge, this work represents of the first large-scale investigations this direction. Among the erucial observations resulting from this study include:
and DE-
Figure An illustration of unstructured (left) struc- tured (right) pruning.
sive structured pruning may disproportionately im- pact the model's performance (Yao et al., 2019).
Choosing Saliency of Weights: When choos- ing the criterion to determine saliency, various fac- tors can be taken into account, such as weight mag- nitude, importance the overall network function- ality, or contribution to specific tasks. Typically, the saliency of weights determined based on their magnitudes when selecting which ones to re- move during pruning. A sparsity of k% means that the least salient k% connections are removed. The most commonly used pruning types are:
Pruning all modules together has the most significant impact on parametric knowledge, compared to pruning specific modules, At pruning levels of >50%, the parametric knowledge of all the models declines rapidly, Quantizing attention modules has less impact on performance compared to quantizing feed- forward networks for all the models. Across all models, structured pruning at the final layer has detrimental effects compared to unstructured pruning.
1. A--Unstruttured: Connections between neurons are eliminated individually, and their saliency is determined by their Lı-norm, i.c.. the smallest weights are removed.
2
Background
In this section, we briefly discuss the various com- pression techniques use in study.
2. Lp-Structured: Connections are climi- nated in structured way, i.e., an entire layer/channel is removed, and saliency is de- termined by their Ly-norm where p is a hy- perparameter.
2.1 Pruning
Pruning involves reducing the model size by eliminating unnecessary or redundant connections between neurons or entire neurons altogether. Broadly speaking, pruning approaches can be clas- sified into two types (Fig. I):
2.2 Quantization
Model parameters be categorized into weights and activations, which typically represented us- ing 32 bits. Quantization aims reduce the num- ber of bits used for representing these parameters. popular choice for this mapping is?:
Unstructured Pruning: Each connection is treated individual entity, and sparsity is attained by climinating connections with lower saliency. Although this approach enables the re- moval less important connections without com- promising performance, leads to sparse matrix operations, which may not be optimal for certain hardware accelerators (Buluc and Gilbert, 2008; Gale et al., 2019).
Q(r) = Int(r/S) Z.
where @ is the quantization operator, r is a real- valued input (weight or activation), is a real- valued scaling factor, and Z is an integer zero- point. An important factor in mapping to an in- teger is the scaling factor S. This is usually given by
Structured Pruning: This involves removing group of connections. such channels entire neurons, instead of individual connections. Unlike unstructured pruning, this approach avoids intro- ducing sparse matrix operations. However, aggres-
3
S=
(1)
2b
Thhe current landscape is evolving as advanced accelera- tors are emerging that provide support for sparse multiplica- tions.
Uniiform quantizationn maps real values to equally spaced integers

---

Here [a, 3] denotes the clipping range and b is the quantization bandwidth. The process of de- termining the clipping range is known as calibra- tion. Extensive research has been conducted to de- termine the optimal range to reduce the bit repre- sentation while balancing accuracy, computational efficiency. and inference speed (Gholami et al.. 2021). In most cases, statisties for weights precomputed they remain constant during in- ference. Often, may be necessary to fine-tune the quantized model parameters to enhance perfor- mance on lask-specific datasets. Taking these fac- tors into account, various methods have been pro- posed (Nagel et al.. 2021):
Output: The capital of France is Paris
Final Dense Layer
Feed-forward network
Attention Module Q, K, layers
Embedding layer
Post Training Statie Quantization (PTSQ):
Text: The capital of France is [MASK]
The clipping range for activations is pre-calculated using representative dataset, which is a small subset derived from the task-specific dataset. Us- ing this clipping range, the activations are quan- tized in advance and thus remain static during in- ference.
Figure 2: Block diagram of a simplified Transformer describing modules compressed in our experiments.
3.1 Settings Under Consideration
Post Training Dynamic Quantization (PTDQ): The clipping range is dynamically calculated for each activation during inference. Although this introduces additional computational overhead during inference, it yields improved performance compared to Post Training Static Quantization (PTSQ) as the signal range is exactly calculated for each input.
The general transformer block consists of an atten- tion module followed by feed-forward network. As result, we consider three choices for compres- sion: compress the attention module alone $3.2. compress the ferd-fo network alone $3.3, compress both together $3.4. Figure contains a visual representation of these modules.
Our chosen compression techniques include pruning, quantization, and combination of prun- ing and quantization. Following the methodology proposed in Han et al. (2015), we adhere the sequential order of pruning the selected group of modules first and then applying quantization. In addition, we also investigate the impact on dis- tilled models and explore the effects of employing various combined compression techniques.
Quantization Aware Training (QAT): The model undergoes process known as fake- quantization, i.e., during training all the calcula- tions involving forward and backward passes performed full- full-prrcision. Subsequently, after updating the weight parameters through gradient descent, the weights are quantized to a lower bit. While this approach achieves the highest perfor- mance, it requires finetuning the model.
3.2 Attention-only Global Compression
We note that while huge diversity of often so- phisticated and specialized compression methods have been proposed, we focus a subset of stan- dard approaches. This enables seek more general insights on compression tradeoffs.
We include all the linear layers within all the attention modules of the model. For encoder- decoder models, we also consider the cross- attention blocks.
Attention-only Global Pruning, (Atter):
We apply pruning to all the linear layers within the attention modules.
3 Experimental Setup
Attention-only Global Quantization, (Atteq): We quantize all the linear layers within the attention modules.
this section. we present comprehensive overview of our experimental setup, including the rationale behind our design choices, along with the selection of models and datasets.
Attention-only Global Pruning + Quantiza- tion, (Attgro): We prune the linear layers

---

the attention modules and subsequently quantize them.
that permit normalized quantities that can be used by practitioners who seek directly apply our work and refer the readers to Sec. A.2 for more details.
3.3 Feed-forward-only Global Compression We include all the linear layers within all the feed- forward networks of the model.
3.6 Design Choices
Weed-forward-one Global Pruming, (F FGP) We employ pruning to all the linear layers within the feed-forward networks.
In global pruning experiments (Overaller, Aller, FFcr), we use LI-Unstructured and apply pruning per- centages ranging from 10% to 90% with increments of 10%.
Feed-forward-only Global Quantization,
(F FcQ): We quantize all the linear layers within the feed-forward networks.
For quantization experiments, as we seek to investigate the zero-shot capabilities of LLMs, we select post-training dynamic quan- tization $2.2, eliminating the need for finetun- ing (unlike quantization-aware training; QAT 22.2) calibration of the model to repre- sentative dataset (unlike post-training static quantization: PTSQ $2.2) and quantize to 8 bits (int8).
Feed-for worddddoll Global Pruning + Quan- tization, (F FcrQ): We prune all the linear lay-
ers from fordrdorward networks and subsequently quantize them.
3.4 Overall Global Compression
We specifically target the linear layers within the attention and feed-forward network. Under this compression, the different setups are:
Since the quantization of activations occurs during inference, which is dynamic in nature, the order of inputs within a batch has a minor impact on the final accuracy (< 1%). There- fore, we seed the experiments to ensure con- sistent and reproducible results ($A.1).
Overall Global Pruning, (Overallap) :: We employ pruning to all the linear layers (except the final dense layer).
Overall Global (Uartaat
+ We apply quantization to all the linear lay- ers (including the final dense layer).
Previous studies (Gordon al., 2020; Michel et al., 2019) suggest that pruning levels of 30%-40% do not affect the model on down- stream tasks. Such rules-of-thune may may not hold for parametric knowledge. our experimental settings (GPQ, FLp), we select 20% and 40% as the levels to under- stand when a similar result holds.
Overall Global Pruning + Quantization
(Overallaro) ++:: We first apply pruning to all the linear layers (except the final dense layer), and subsequently. we quantize all the linear layers.
3.5 Final Dense Layer Pruning, (FLp)
Recent studies (Mitchell et al., 2021, 2022: Meng al., 2022) provide evidence suggesting that the final layers of a language model play a signifi- cant role in storing information. Given its im- portance, we focus on understanding how knowl- edge is encoded in the final layer. Therefore, we treat the final layer as individual module in our experimental setup and prune it. We consider Ll-structured and Ll-unstructured pruning as out- lined in 1.
3.7 Model Zoo
We consider the following models for our study. Where available, we choose both the base and large versions of the model to understand if larger models exhibit different behavior.
3.7.1 Encoder-only:
BERT (Devlin et al., 2019): Pretrained on masked language modeling (MLM) and next sentence prediction (NSP) objective.
We note that the number of parameters com- pressed differs for different settings. We record all of the values required for normalizing mea- surements. However, our focus is predominantly aimed understanding the effects of compressing modules and their combinations rather than pre- senting normalized results, and our insights reflect this framing. We provide full parameter counts
RoBERTa (Liu et al., 2019): Similar to BERT with different training choices (larger training dataset and removed NSP)
DistilBERT (Sanh et al., 2020): Distilled ver- sion of BERT whose training objective in- cludes MLM, a distillation loss, and a cosine embedding loss.

---

ALBERT (Lan al., 2019): Parameter- reduced version of BERT using cross-layer parameter sharing and factorized embedding parameterization.
dataset (Bisk et al., 2020), and the Winogrande dataset (Sakaguchi al., 2021). These datasets provide range challenging prompts for each model type. We refer the reader to Table for ex- amples of samples from each dataset.
3.7.2 Encoder-Decoder:
Flan-T5 (Chung et al., 2022): Instruction- finetumed encoder-decoder model with masked span corruption objective.
Experimental Results and Insights
To facilitate our discussion, we categorize pruning levels as follows:
Lamini-Flan-TS (Wu et al., 2023): Flan- T5 model finetuned on LaMini instruction dataset* which is generated and distilled us- ing ChatGPT output.
Plow: Sparsity levels of 10-30%
Pmediumi Sparsity levels of 30-50%
Plight Sparsity levels of >50%
3.7.3 Decoder only:
For encoder-only models, we report the % drop in top-1 accuracy, averaged across all the probes in LAMA. For the decoder-only and encoder- decoder models, we report the drop in accuracy, averaged across BoolQ, PIQA and Winogrande. the decoder-only and encoder-decoder plots, the majority -easoiitbbaeelinee baseline indicates the accuracy when all the predictions are assigned to the majority class.
Vicuna-7B (Chiang et al., 2023): An iitruction-basee LLama derived model fine- tuned on user-shared conversations collected from ShareGPT.
WizardLM-7B (Xu et al., 2023): An instruction- LLama derived model with instructions generated by LLMs (rather than
humans) using the Evol-Instruct mechanism.
4.1 Global Pruning
3.8 Datasets
We observe that for encoder-only models (Fig. 3, 19), there is minimal decline in performance at Plaw- At Pmediums the drop in performance is more significant for pruning feed-forward net- works (FFGP) as compared to attention modules (Allop).
We use the following datasets for our empirical analysis:
LAMA: To examine the effects compression on encoder-only models, we use the LAMA (LAn- guage Model Analysis) benchmark (Petroni et al., 2019). LAMA assesses the factual and common- sense knowledge of language models. Each ex- ample in LAMA is formulated as cloze-style question, where either the subject object is masked. By predicting the masked word, we evaluate the model's ability recover real-world facts. Specifically, we probe the encoder-only models with LAMA to investigate the impact of compression on various knowledge tasks. This benchmark consists of four datasets, namely TRex, Go, ConceptNet, and SQUAD, each de- signed to assess specific types of relational knowl- edge. These datasets provide valuable insights into the model's performance and its understanding of different types of information.
Finding: At Pmedium- for encoder- only models, pruning attention modules (Attep) has smaller impact compared to pruming feed-forwan networks (F (FFGP).
We observe that for encoder-decoder (Fig. 5, 13) and decoder-only models (Fig. 4), there is a minimal decline in performance at Plou- However, at Pmedium the drop in performance is more significant for pruning attention modules (Attap) compared to feed-forward networks (F (FFG)).
Language model evaluation harness: To
examine the effects of compression on encoder- decoder and decoder-only models, we use sub- set of evaluation harness tasks (Gao et al., 2021): the BoolQ dataset (Clark et al., 2019), the PIQA
Finding: At Pmeediu-- for encoder- decoder and decoder-only models, pruning the attention module (Attgp) has more im- pact performance compared to pruning feed-forward networks (FFGP).
instruction

---

BERT-base
BERT-large
DistilBERT --aas
04
20
-0
loss
-00
-60
-100
20
% pruned
% pruned
% pruned
Overallgp
AllGP
FFGP
Figure 3: Averaged drop in Top-1 accuracy for encoder-only models for global pruning.
M-7rl
Flee
Faa-15-arp
3
prued Mijeryy Backo:
Ouzilla
Ovutle
Tasita
Figure 4:
Averaged drop in accuracy for decoder-only Figure 5: Averaged drop in accuracy for encoder-decoder models for global pruning.
models for global pruning-
We note that the number of parameters in the feed-forward networks is significantly higher than the number of parameters in the attention mod- ules for all these models (Table 3). This observa- tion provides likely explanation for the pattern observed in encoder-only models, where pruning more parameters results in higher loss of para- metric knowledge. However, this finding is coun- terintuitive for encoder-decoder and decoder-only models, as we would expect that pruning the larger feed-forward networks would have a more signif- icant impact on the parametric knowledge. We suspect that the feed-forward networks parameterized and thus they be pruned without significant drop in performance.
the implications of employing global pruning tech- niques. We additionally note that at Phigh, the per- formance goes to zero as expected.
Additional results for global pruning individ- ual datasets for encoder-only models are shown in Fig 20, 21, 22; for decoder-only models Fig 23; for encoder-decoder models at Fig 13, 25.
4.2 Global Quantization
We observe that across all the models (Fig. 6. 15, 16), the performance drop is less significant when quantizing attention modules (Attgq) com- pared quantizing feed-forward networks alone (FF))). This contrasts with the results from global pruning ($4.1), where pruning attention- only modules had more detrimental effect on encoder-decoder and decoder-only models.
Finding: For all the models, pruning all the modules together (OverallGp) has the most significant impact on performance.
For all the models, quantizing attention modules (Atteq) has lesser im- pact compared to quantizing feed-forward networks (F (FFG)).
Finding:
Among all the models analyzed, pruning all modules together (Overallçp) has the most sig- nificant negative impact performance. This finding suggests that when compressing models. pruning all modules simultaneously leads to greater loss parametric knowledge compared to pruning specific modules or components individu- ally. Therefore, crucial to carefully consider
We hypothesize that in the of quantiza- tion where all connections are preserved, the para- metric knowledge cross-attention modules may remain relatively intact. However, in pruning, as connections are eliminated, there may have

---

FI
Smrp in
BERT
DERTTooge
RoBERTi-bsse
ALBERT-NEE
DomMay
Alo
Figure 6: Averaged drop in Top-1 accuracy for encoder- only models global quantization.
S-lane
Milecty Bautlo
BURT
BOBURTS-
Al ALBUTT-pp
Figure 8: Averaged drop in accuracy for encoder- decoder models for different attention modules compression. Self Compressing only self-attention modules, CrossAttarg' Compress- ing only cross-attention modules, Encoder Atte Compressing attention modules in encoder only, Decoder Auaaro Compressing attention modules in decoder only.
17a0
Figure 7: Averaged drop in Top-I 1 accuracy for encoder- only models global gruning-quantization
greater impact on the parametric knowledge in cross-attention modules, thereby affecting the overall capabilities of the model. is also interest- ing to observe that the performance drop during quantization almost similar to that of Pmedium-
pruning and quantization the cross-attention modules could be responsible for this change in the order of impact. To test this hypothesis, we lectively prune and quantize the self-attention and cross-attention modules separately and found out that indced the case (Fig. 8) and aligns with the claim made in Michel et al. (2019). Additional results for compressing attention-only modules are shown in Fig 12, 24. For fine-grained analysis on individual datasets, refer the interested reader Table 4, 5. 6.
Finding: For the models, quantizing all the modules together (Overallgg) hurts the most.
It is intuitive that quantizing all the modules to- gether (Overallgq) has the most significant nega- tive impact. Additional results shown in Tables.
4, 5, 6.
4.3 Global Pruning + Quantization
4.4 Final Dense layer Pruning
For all the models (Fig. 7, 17. 18), 20% sparsity, compressing attention modules (Attaro) results a smaller performance drop compared to compressing feed-forward networks (F FGPQ). At 40% sparsity, the same trend is observed for encoder-only and decoder-only models. However, we notice the reverse for EXCODER-DECODEE models i.c., that compressing feed-forward net- works affects performance less than compressing the attention modules 40% sparsity.
For encoder-only models (Fig. 9). LI- unstructured pruning has smaller impact compared to Ll-structured pruning. We hypoth- esize that the final layer of the encoder-only models might encode knowledge in structured or modular manner, and any form of structured compression would disrupt this encoding, result- ing in a larger performance drop. Such result would be consistent with existing approaches that enable editing knowledge in language models and rely on structure (Mitchell et al., 2021).
Finding: For all the models, 20% spar- sity level, Attoro hurts less compared ForrQ.
For encoder-only models, L1- unstructured leads to a smaller decrease in performance than L1-structured.
Finding:
We hypothesize that the sequential effects of

---

cus has been to understand the importance of the attention mechanism (Voita al., 2019; Michel et al., 2019) by pruning the heads. a similar fashion, works such as Zafrir et al. (2019); Bai et al. (2020); Zadeh et al. (2020); Tao et al. (2022); Prato et al. (2019); Frantar et al. (2022); Dettmers et al. (2023) pushed the limits of quantization on language models. Most of these works have fo- model class
300
000
Figure 9: Averaged drop in Top-1 accuracy for encoder- only models for final layer pruning.
particular metric. In another line work, a variety of approaches (Li and Liang, 2021; Hu et al., 2021; Liu et al., 2021; Lester et al., 2021) focus on alternatives to traditional finetuning of the model due to its scale. In contrast to these works, our paper primarily fo- cuses on the in-built parametric knowledge present in the model. This means we do not finetune and instead seek to understand whether some of the previously described phenomenona are applicable to other models as well.
cused
30
-30
350
Vatza
M 15
Figure 10: Averaged drop in Top-1 accuracy for decoder-only models for final layer pruning.
Also connected to this work are techniques that edit factual knowledge in models. The goal for such works is avoid retraining even finetun- ing models, instead seeking to directly change pa- rameters connected to certain facts (Mitchell al., 2021, 2022; Meng et al., 2022). However, given our focus on compression, the main theme of our work differs. Nevertheless. it would be interesting to understand the impact of relying on compressed models when using such editing techniques.
For decoder-only (Fig. 10) and encoder-decoder (Fig. 14) models, even at a sparsity level of 20%, the predicted accuracy very close to the major- ity baseline. This finding aligns with the claims made Mitchell et al. (2022) that final layers en- code significant amount of information. The dras- tic performance drop observed suggests that the fi- nal layers play crucial role in encoding knowl- edge. Additional results for pruning the final layer are shown in Fig. 26, 27, 28.
6 Conclusion
5 Related Work
Compression is crucial in deploying and using large language models. Despite its importance. existing empirical studies predominantly rely on generic measurements such as perplcxity or stan- dardized benchmark metrics when investigating the effects of compression. These coarse measure- ments are challenging to interpret. As result, it is difficult to use them to develop meaningful heuris- tics for practitioners.
Early works seeking understand large language model behavior focused on contextual represen- tations and how such models gain linguistic ca- pabilities (Goldberg, 2019; Ettinger et al., 2018; Jawahar et al., 2019). More recently, some lines of work have steered towards understanding how these models acquire factual and commonsense knowledge. Techniques such as probing evolved a way understand the knowledge capabili- ties of these models (Petroni al., 2019; Kassner and Schütze, 2020; Talmor al., 2020; Weir et al., 2020; Wallat et al., 2021).
To address these limitations, we provided a large-scale study that focused on fine-grained ef- fects on quantities like parametric knowledge. We studied a variety of compression choices across multiple model families, providing usable insights into what types of compression schemes have the least and most significant impact on models. We hope this work serves as a useful step towards de- veloping users' intuition for rules-of-thumb when selecting appropriate compression techniques large language models. For future work, we hope
Previous works including Gordon et al. (2020); Michel et al. (2019) pruned BERT and showed that it is resilient medium level of pruning. For ex- ample, Michel al. (2019) showed that after fine- tuning for particular downstream task, it is pos- sible to prune about 40% of the attention weights without any loss in performance. A particular fo-

---

add additional, more specialized techniques for large language model compression.
Toutanova, 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905. 1004..
7 Limitations
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 O: 8-bit matrix mul- tiplication for transformers at scale. arXiv preprint 2008::208.07339,
Our research has tackled diverse combination of models, compression schemes, and compression targets within the vast large language model re-
search area. We note that sophisticated and spe- cialized compression techniques tailored to spe- cific objectives for a particular class of models may exhibit distinct behavior compared to the find- ings presented in this study. Hence, our work does not aim to present an exhaustive set of findings that universally characterize the impact parametric knowledge across all conceivable models and com- pression approaches. We believe that our study serves as a valuable starting point, offering nu- anced examination of prevalent methodologies.
Tim Dettmers, Ruslan Svirschevski. Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Al- istarh. 2023. Spqr: A sparse-quantize representa- tion for near-lossless IIm weight compression. arXiv preprint arXiv:2306.0307
Jacob Devlin, Ming-Woo Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, and Ahmed Hassan Awadallah. 2021. What do compressed large language models forget? robustness challenges in model compression. arXiv e-prints, pages arXiv-2110.
We note, additionally, that we do not directly address the tradeoff between wall-clock inference time versus compression. While this is also an important tradeoff, the impact of compression on inference time contains many intricacies that are best treated with separate large-scale study.
Allyson Ettinger. Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sen- tence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790-1801, Santa Fe, New Mex- ico, USA. Association for Computational Linguis- tics.
References
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2020. Binaryhert: Pushing the limit bert quanti- zation. arXiv preprint arXiv:2012. 157..7
Elias Frantar and Dan Alistarh, 2023. Massive lan- guage models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.0077
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piga: Reasoning about physical commonsense natural language. In Proceedings the AAA/ conference on artificial intelligence, vol- ume 34, pages
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 22Xiv:2210.17323
Aydin Bulue and John R Gilbert. 2008. Challenges and advances in parallel sparse matrix-matrix multiplica- tion. In 2008 37th International Conference on Par- allel Processing, pages 503-510. IEEE.
Trevor Gale, Frich Elsen, and Sara Hooker. 2019. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.0993
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. 2021. A framework for few-shot language model evalua- tion.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ton Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-fietuueed language mod- cls. arXiv preprint arXiv:2210.1144 14416.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 2021. A survey of quantization methods for efficient neural network inference.
Christopher Clark, Kenton Lee. Ming- Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina
Yoav Goldberg. 2019. Assessing bert's syntactic abili- ties. arXiv preprint arXiv:1901.05220

---

Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing bert: Studying the effects of weight pruning on transfer learning, arXiv preprint arXiv:2002.08307
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Arnav Gudihande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary Ilms.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowl- edge in gpt. arXiv preprint arXiv:2202.0522
Paul Michel, Omer Levy, and Graham Neubig, 2019.
Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huff- man coding. arXiv preprint arXiv: 151.00099.
Are sixteen heads really better than one? Advances in newral information processing systems, 32.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv: 10. 1339..
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Ahle Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.0900 09685.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christo- pher D Manning, and Chelsea Finn. 2022. Memory- based model editing at scale. In International Con- ference on Machine Learning, pages 15817-15831. PMLR.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 3651-3657, Florence, Italy, Associa- tion for Computational Linguistics.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. 2021. A white paper on neural network quantization.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Se- bastian Riedel. 2019. Language models as knowl- edge bases?
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Compu- lational Linguistics, pages 7811-7818, Online. As- sociation for Computational Linguistics.
Gabriele Prato, Ella Charlaix, and Mehdi Reza- gholizadeh. 2019. Fully quantized trans- former for machine translation. arXiv preprint 1910.1991.0488
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv: 11942.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavat- ula. and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commu- nications of the ACM, 64(9):99-106.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parfiiter-efffcientt prompt tuning. arXiv preprini arXiv:21044 arXiv:2104.08691.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Xiang Lisa Li and Percy Liang. 2021. Trefix-tuning
Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63(12):54-63.
Abigail See, Minh-Thang Luong, and Christopher D Manning. 2016. Compression of neural machine translation models via pruning. arXiv preprint 16Xiv:1606.099274
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu. Pengcheng He, Tuo Zhao, and Weizhu Chen. 2021. Super tickets in pre- trained language models: From model compres- sion to improving generalization. arXiv preprint arXiv:2105. 12002.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019, Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang. and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103. 10385.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. olmpics - on what language model pre-training captures.

---

Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang. Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre-trained lan- guage models via quantization. arXiv preprint arXiv:2203.10705 10705.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar. al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2xXi::2302.13977
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads the heavy lift- ing, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2021. Bertnesia: Investigating the capture and for- getting of knowledge in bert.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint
arXiv:1804.07461.
Nathaniel Weir, Adam Poliak, and Benjamin Van Durme. 2020. Probing neural language models for human tacit assumptions. arXiv preprint arXiv:2004.4887.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muham- mad Abdul-Mageed, and Alham Fikri Aji. 2023, Lamini-Im: A diverse herd of distilled mod- els from large-scale instructions. arXiv preprint arXiv:2304. 14402.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang, 2023. WizardIm: Empowering large lan- guage models to follow complex instructions. arXiv preprint 2304. 12244.
Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. 2019. Balanced sparsity for ef- ficient dnn inference on gpu. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 5676-5683.
Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. 2020. Gobo: Quantiz- ing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium Microarchi- tecture (MICRO), pages 1-24. IEEE.
Ofrr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019, Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Effficient Machine Learning and Cognitive Nemrusing-NuurIPS Edi- tion (EMC2-NIPS), pages 36-39. IEEE.

---

A Appendix
The appendix contains all of the results we could not include in the body of the paper. We first discuss the statistical approach of the experiments and the performance drop against compression ra- tio. Then, we show individual plots for set of ex- periments that track decrease in accuracy for sev- eral types of compression and models. Next, provide table that contains information on the datasets used in our experiments. Afterwards, we provide tables with model details, including pa- rameter counts, and explicit results for compres- sion results across model families. Afterwards, we present large-scale comparison across datasets for encoder-decoder models under various atten- tion module compression approaches. We pro- vide LAMA probe results and finally, change-in- accuracy plots a variety datasets for different model classes.
BERt base TREx
A.1 Experimental Approach
Tor.t---aggcccoooo BoolQ
Alte
Our experiments fall into two categories: deter- ministic and stochastic. Our experiments on prun- ing are deterministic we used Ll-unstructured pruning. On the other hand, our quantization ex- periments have element of randomness. This is due our use of PTDQ, which computes dynamic clipping range. We deliberately struck a balance between the number of trials per set- ting and the overall number of settings studied. Consequently, we ran experiments with multiple seeds and recorded confidence intervals, as demon- strated in Table 1.
ABo
Marrly Bextre
A.2 Performance drop against compression ratio
Comprasion Rato
Normalizing the x-axis account for the param- ratio results in the same plots, but with skewed x-axis (Fig. 11). Given a specific perfor- mance drop percentage, it is highly likely that we can achieve greater parameter compression by tar- geting feedforward modules rather than attention modules. It is worth noting that across all the models studied, feedforward modules have more parameters than attention module
Figure 11: Performance drop vs compression ratios for different model families. Models and Datasets are ran- domly chosen for representation

---

Table 1: Top-1 Accuracy from quantizing BERT on SQUAD. Left: BERT-Base, Right: BERT-Large. Baseline for BERT-Base is 12.987 and BERT-Large is 15.909
Overallag
Attee
F Fag
Overallaq
AttG
FFao
Seed
5,844
13:322
5890
Seed 40
2,273
16,83
2.597
Seed
5,844
13.312
Seed
2,/22
14,286
3,896
Soed 60
6.169
12.87
4.221
Seed 6X
2,922
15,00
4,545
Mean
2:706
3,679
Standard Deviation (sumple)
0.188
0.188
0.676
Stundard Deviation (sample)
0.375
1.312
0.992
Standard Deviation (population)
Standard (population)
Table 2: Datasets in our experiments (we use dev sets for BoolQ, PIQA, and Winogrande)
Probe
Type
#Fgs
Questtoo
Answer
Factual
Hartolomco
[MASK]]
Google-RE
Faciual
5.5k
Mareva [MASK] actress & former bomnt Queen
French Polynesia
Squad
Newton played [MASK] during Super
Quantebback
ConcepiNet
Commonsense
TIK
Jake would make want to (MASK).
laugh
any hill higher Goal: "loe hox
PIQA
Commonsense
cooler you add Solm2; will turn into a cooler you add soda to
Solni:
Solal
Winogrande
Commonsennse
1.2k
suitcuse The troply doesat lit into the brown sulicase because its too large.
trophy
trophy
Table 3: Number of parameters (in million) across all the models
Model Name
Overallar
Attap.Gy.com
F FFanno. GPG
FL
Total trainable parumeters (for Overatlagg (PG)
Bert hase
21
64
109
Bert large
303
75
226
334
Roberta-buse
21
64
124
Roberma-lage
903
75
226
51
355
buse
Albort-bise-22
85
28
57
4
89
Aarert-lage--2
302
211
4
306
Flan SauuuS--buss
198
85
113
25
223
Distil-Plan TS-hase
198
85
223
FlanTS large
717
302
311
750
Distil-Hani I-llage
717
311
750
Vicuna
6476
2147
4329
131
6607
Wizard-I.M
6476
2147
4329
131
6607
Table Results from compressing different modules for encoder-only models (numbers represent top-1 accuracy)
Overalloq
Attac
FFcq
Overaliarq
Attaro
F'FPPQ
Model
Dutasel
Baseline
20%
40%
20%
40%
20%
40%
TRFx
30.27
11.536
29:903
16.614
5,552
4,233
29.675
29.217
14.691
14,628
GoogleRe
4.374
BERT-base
Squad
12.987
5.844
13.512
3.896
2.507
0.974
12,538
10.39
4.87
4.87
Conceptnet
16.33
6.02
15.897
8.677
3,919
3.69
16.224
15.553
8.262
8.324
30.485
4.376
29.624
BERT-large
GooglaKe
10.472
3.829
10. 309
3.612
1,434
0.436
10.018
9.964
4.247
3.376
Squad
15,909
2,273
16.883
2,597
1,948
0.325
15.584
14.935
3.247
2,597
Conceptnet
19.534
4.652
19.048
5.649
1,818
1.342
18.801
18.086
5.164
4.864
TRFx
11.9
3,566
8.014
8.421
1.424
0.006
11,529
4.663
6.489
0.048
GooglaRe
4,102
1.143
2.668
1.234
0.617
3.249
1.779
1.053
0.091
Squad
8.442
0.325
4.545
2.922
5.195
1.623
1.299
Conceptnet
17.036
3,769
14.467
7.247
0.83
0.026
14,865
8.147
5.746
0.856
TREx
16,862
6,264
16.159
11,885
0.175
0.019
15.845
15.714
$.358
0.057
RoaeRTaalarge
GoogleRe
3
1,488
3,829
2.868
0.036
2.196
1.869
0.926
0.054
Squad
13.636
4.87
13.312
7,468
12:013
10.714
1,299
0
Conceptnet
0.494
TREx
28.082
14.186
28.184
20.383
18.285
12.673
27.021
25:229
20.367
18.593
DistiBBERTT-bae
GoogleRe
10.181
4.791
10.073
8.766
5,681
3.92
9.111
8.403
8.113
7.241
Squad
5.195
Conceptnet
14.308
6.391
14.132
9,101
9.339
5.976
14.105
13.346
9.727
7.238
TREs 13.016
9.213
0.003
0.003
7.595
0.945
0.003
GongleRe
ALBERT-huse
Squad
3,896
0
1.623
0
0
1,299
0.649
0
Conceptnct
0.018
6.682
0.009
1.077
0.018
22.057
0.013
0.006
ALBERT-larg
GuogleRe
2.686
0
0
0
0
0.018
0
Squad
9.74
Conceptnet
14.794
0.009
0.071
0.132
0.009
0.026
0.044
0.009

---

Table 5: Results from compressing different modules decoder-only models (numbers represent accuracy). Majority baselines
BoolQ: 0.621, PIQA: 0.504, Winogrande: 0.504
Overallag
Affoc
Foc
Overallapg 20%
AMGrQ
F FoPQ
40%
20%
40%
20%
40%
Boolg
0.7657
0.5211
0.7645
0.3437
0.5272
0.4398
0.7125
0.556
0.5346
0.4495
Vieuna-7B
0.778
0.6866
0.5352
0,7421
0.6257
Winogrande
0.6725
0.5391
0.678
0.5825
0,5043
0,4925
0.663
0.6298
0.5612
0.5367
Boolg 0.7844
0,6073
0.7841
0.6003
0.5817
0.445
0.7514
0.6801
0.6141
0.5547
Mi-r
0.6654
0.6371
Winogrande
0.6646
0.5517
0.6638
0.588
0.5312
0,5193
0.6622
0.6346
0.5738
0.5817
Table 6: Results from compressing different modules of encoder-decoder models (numbers represent accuracy).
Majority baselines are BoolQ: 0.621, PIQA: 0.504, Winogrande: 0.504
Overallaq
Alae
FFoo
Overalle
Algro
F'Frr
Model
Dataset
Raseline
20%
40%
20%
40%
20%
40%
Boolg
0.7887
0,618
0.7841
0.6352
0.5049
0.482
0.7609
0,5058
0.6275
0.6125
FlunTS-Buse
Piqa
0.6251
Winegrande
0.5422
0.4862
0,5359
0.5138
0.5051
0.4949
0.5272
0.5241
0,5075
0.498
Boolq
0.8645
0.8034
0.8615
0,8165
0.6498
0.5618
0.856
0.4107
0.19
0.7969
Piga
0.7138
0.6638
0,716
0.6942
0.6181
0.5555
0,7214
0.6616
0,6953
0.6921
Winegrande
Boolq
0.7982
0.7297
0.667
0.6366
0.7315
0,7263
Lamini-Flan-1 Flan-T5-24MM
Piga
0.6676
0.6393
0.6594
0.6507
0.6208
015462
0.6627
0.6208
06334
0.6338
Winegrande
0.5304
0,5257
0,5083
0.513
0.543
0.5051
0.5099
0,5004
0.4964
0,5028
Boolg 0.8306
0,7982
0.8294
0.7979
0.7916
0.6211
0.8226
0.6783
0,7994
0.7982
Lamini-Flan -lanii-la-----MMMMM
Winegrande
0.5549
0.5241
0.5454
0.5517
0.5193
0.4878
0.5478
0.5114
0.5288
0.5201
LaMai-Fla 15:3RM
LaMini-laa Fwa---l------- 75-246M
30%
Maystry Bunches
Panit-Lagee
12-1000
10% Pemin
Milett Buntin
Figure 14:
Averaged drop in accuracy for encoder- decoder models for various local pruning ($4.4)
56 00g
LaMini Flan 75 T33M
LaMini Slan T5 TETM
2854
SelGrne
Exndeanus
Majrlly Beadiar
Daod
Figure 12: Averaged drop in accuracy for Lamini mod- els under various attention modules compression ($4.3)
LaMiin-Flae-S--soM
Viltna 13
MESNNL M XI
preaa
peased Mijnity Baddne
Figure 15: Averaged drop in accuracy for glohal quan- tization for decoder-only models ($4.2)
Ownlig
Mids
Figure 13: Averaged drop in accuracy for global prun- ing 1)

---

Crop 800
11
Fan-155 1aa--15-use luse
Pan: 155lang
Overilgg
Figure 16: Averaged drop in accuracy for global quanti-Fiiuure 17: Averaged drop in accuracy for global prun- zation for encoder-decoder models ($4.2).
ing+quantization for decoder-only models ($4.3).
F
M
% drop in
-20
Plan- 13-base
Flan- 15-large
LaVini-Flas- 15-24KM
LaMini-Flee T5-783M
Phoro
Allaro
20%
222 47%
Najanny Souetiee
Figure 18: Averaged drop in accuracy for global pruning-quammizatioon encoder-decoder models ($4.3).
Oceedla
Arre
TTae
Afgo
1Foor
Figure 19: Averaged drop in Top-1 accuracy for encoder-only models ($4.) I).
SEXT-
-20
Quirdg
Figure 20: Drop in Top-1 accuracy for respective LAMA probes. Left-to-Right, Top-to-bottom: TREx, Google- RE, SQUAD, Conceptnet (54.1).

---

Ru6EXTT-ba:
RodEiti-bee
RuBEKTe-lange
Ovanllar
Alge
Ohandgr
Maa
1Tar
RedoroCe-laaree
peused
Orenili
Adi
17ce
Oharskg
AFG
17
Figure 21: Drop in Top-1 accuracy for respective LAMA probes. Left-to-Right, Top-to-bottom: TREx, Google- RE, SQUAD, Conceptnet 1).
Al BLB87--eas
ALBERTIER
ALBERTher
ALaERT-ame
54 preeed
% pann
ponned
ALBERT
ALBERTiexc
ALBERTba:
ALaERT-ane
Ovenlly
A
1Yar
Figure 22: Drop in Top-1 accuracy for respective LAMA probes. Left-to-Right, Top-to-bottom: TREx, Google- RE, SQUAD, Conceptnet 1).
Voen 7B
WaandfMM M.
Vaus 7B
Wizrdi.M 7B
35 grust
prized
Synne
prece Nevrty Budas:
Ovaslo
Meeatt Boxine
Ovasler
Viemx-19
WonniLMM-B
% parsed
graned
Figure 23: Drop accuracy for decoder-only models. Left-to-Right, -optto-Bottom: BoolQ, PIQA, and Wino- grande 1)

---

%
FasT S-boe
S-bae
LaMini-laa-TS-000
LaMini- 2---3--66M
Safeage
Incadetange
Maurry Dandin
Seltazoo
Dacedetappe Desddrraus
Maurr Baudia:
4%%
Beasiirzas
4%%
lm
Flati S-tae
Plan S-bae
LaMine Pea-- 153400
LaMini 3-241M
27%
Incoddleaps
Maamy Uniiie
27%
Scllago
Marrt Saadia:
6%%
Demiiiiam
47%
N
Flan75-lange
LaMini-las-TSSii
LaMini Beulis:
Baolie:
43%
So in
Flan15-huu
Flat TS-leac
LoMii-Fa--TS-24MM
LaMini-Sland T-24M
30%
Selaru
Doodrraw
Mijusty Rodite
26%
Migusly Budin
700 45%
(iag75-trng
Hast S-
LaMaa- 18-88M
LaMini- Plan-
Majert
SelGrge
Mijilly
Comaange
Figure 24: Drop in accuracy across various datasets for encoder-decoder models under various attention modules compression. -ootto-Bottom:: BoolQ, PIQA, Winogrande ($4.3)

---

Flan75 hode
FlonTS lage
LaMinl Floo 3AM
Lobin Tlao 15 78.0
prunce
pussss
peasee
priseed
Ovarlle>
Micerty
Ovealee
Vion-TS-tore
phadd
pratal
peaned
Mijecly
Majely
large
peanci
Ovasig
Mlpe
PPa
Minerly Businc
Ovenler
PPg
Kijurly Btadde
Figure 25: Drop in accuracy across various datasets for encoder-decoder models. Top-to- Bottto- BoolQ, PIQA, Winogrande ($4.1)
M0
JEA
100
Baro
van
20 INPPOR
460
<10
130
366
466
KBI Bay
lage
Figure 26: Drop in Top-1 accuracy across various datasets encoder-only models for FLp. Left-to-Right, Top- to-Bottom: TRex, Google-RE, SQUAD, ConceptNet ($4.4)
150
330
35/
Vourant
Moast-M-1
WERSELN-10
Lodeecard
20h Nemer
2002 Tri.raee
Mastrr lnc
Indencand
3%% Peree
40% Pretag
Mastt Tnc
7.
250
Fania
Paaliig
Male Blacko
Figure 27: Drop in accuracy across various datasets for decoder-only models for FLp. Light Top-to- Bottom: BoolQ, PIQA, and Winogrande ($4.4)

---

Srakow!
11 Ritenneen
Figure 28: Drop in accuracy across various datasets for encoder-decoder models for FLp. Rightto-Rihht: BoolQ, PIQA, and Winogrande ($4.4)


Reviews:
[{"review_id": "fbULv0JS01", "reviewer": "Reviewer_syfP", "summary": "", "questions": "1. The models being compared in different categories vary largely in terms of model sizes. Decoder-only models are considerable larger than those of encoder-decoder and encoder-only. It is likely that the behaviour is different with different model sizes. Had you verified if the same findings hold with smaller decoder-only models? or with larger encoder-decoder models such as flan-T5-XXL? \n\n2. As being pointed out, attention contains more parameters than FF in some cases. It is not very informative to only look at the pruning levels. Would it be more meaningful to also plot the performance drop against compression ratio? \n\n3. For encoder-decoder models, self-attention and cross-attention might have different functions, and thus, it might be more favourable to study them separately. Also, attentions in encoder and in decoder should be examined in the same way. \n\n4. For the hypotheses made in Line 433-436, can it be verified by reducing the number of parameters in FF layers (e.g., by using a low rank approximation)? ", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### On comparison with other sized models:\n\nThank you! We have observed similar findings when examining larger encoder-decoder models, specifically the FlanT5-XL with 3B parameters. Our observation highlights that compressing Attention (Att) modules results in a more significant drop compared to Feed-Forward (FF) modules. The rationale behind this observation is discussed in lines 428-433 of our work.\n\n#### Attention GP:\n\n|  | Baseline | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% |\n|---------|----|----|----|----|----|----|----|----|----|----|\n| BoolQ   | 0.8945 | 0.8633 | 0.8584 | 0.8339 | 0.6073 | 0.4682 | 0.3838 | 0.3786 | 0.382 | 0.3789\n| PIQA    | 0.7568 | 0.7584 | 0.753 | 0.7437 | 0.6915 | 0.6376 | 0.5914 | 0.5571 | 0.5457 | 0.537\n| Winogrande | 0.693 | 0.6953 | 0.6961 | 0.674 | 0.5446 | 0.4996 | 0.4988 | 0.5264 | 0.4964 | 0.5012\n\n#### Feed-Forward GP:\n\n| | Baseline | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% |\n|---------|----|----|----|----|----|----|----|----|----|----|\n| BoolQ   | 0.8945 | 0.8642 | 0.8618 | 0.8596 | 0.8498 | 0.8428 | 0.8214 | 0.5486 | 0.3789 | 0.481\n| PIQA    | 0.7568 | 0.7557 | 0.7557 | 0.7546 | 0.7552 | 0.7367 | 0.7116 | 0.6045 | 0.518 | 0.5201\n| Winogrande | 0.693 | 0.6953 | 0.7032 | 0.6851 | 0.6961 | 0.6867 | 0.6361 | 0.5201 | 0.5004 | 0.4822\n\nNote: Majority baselines are - BoolQ: 0.621, PIQA: 0.504, Winogrande: 0.504\n\nCritically, our study focuses on **analyzing individual model families independently** rather than comparing encoder-only, encoder-decoder, and decoder-only models. This approach is reflected in the way we present our findings. \n\n### On plotting the performance against compression ratio:\n\nThank you! We have generated plots for all three model families, **yielding insights consistent with the discussion** in lines 428-433. The plots visually resemble those presented in our paper, albeit with a skewed $x$-axis. Consequently, given a specific performance drop percentage, it is highly likely that we can achieve greater parameter compression by targeting FF modules rather than Att modules. It is worth noting that across all the models studied, FF modules have more parameters than attention modules.\n\nOur plot construction is straightforward; we describe it as follows (line 286) in the submitted draft: \"We note that the number of parameters compressed differs for different settings. We record all of the values required for normalizing measurements. However, our focus is predominantly aimed at understanding the effects of compressing modules and their combinations rather than presenting normalized results, and our insights reflect this framing. We provide full parameter counts that permit normalized quantities that can be used by practitioners who seek to directly apply our work.\"\n\n### On studying attentions separately:\n\nThank you for the feedback! We acknowledge the potential divergence in the functions of self and cross attention, as well as their roles within encoder and decoder modules. We will address this aspect in the updated draft. We are conducting additional experiments to enable doing so. Additionally, our analysis in Figure 8 delves into the combined impacts of pruning and quantization on different types of attention modules.\n\n### On low-rank approximation to verify claims in lines 433-436:\n\nWe appreciate your suggestion! We will certainly incorporate the investigation of low-rank approximation in our updated draft through supplementary experiments. We also believe that a dedicated study on addressing the over-parametrization of LLM modules is a great topic for a future paper, and we plan to pursue this goal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper focused on the preservation of knowledge which has been somehow overlooked in existing study of compressing LLMs. Experiments are done with many mainstream LLMs, and the writing is  overall easy to follow. Compression methods for LLMs might potentially benefit from the findings made in this paper. ", "reasons_to_reject": "Findings seem a bit superficial. The author has provided a set of hypotheses, but few of them have been verified or examined. It could be more insightful with additional experiments. I will elaborate more on the my doubts in the following section. ", "questions_to_authors": "1. The models being compared in different categories vary largely in terms of model sizes. Decoder-only models are considerable larger than those of encoder-decoder and encoder-only. It is likely that the behaviour is different with different model sizes. Had you verified if the same findings hold with smaller decoder-only models? or with larger encoder-decoder models such as flan-T5-XXL? \n\n2. As being pointed out, attention contains more parameters than FF in some cases. It is not very informative to only look at the pruning levels. Would it be more meaningful to also plot the performance drop against compression ratio? \n\n3. For encoder-decoder models, self-attention and cross-attention might have different functions, and thus, it might be more favourable to study them separately. Also, attentions in encoder and in decoder should be examined in the same way. \n\n4. For the hypotheses made in Line 433-436, can it be verified by reducing the number of parameters in FF layers (e.g., by using a low rank approximation)? ", "paper_topic_and_main_contributions": "The main contribution made by this work is an experimental study and evaluation protocol to investigate the loss of knowledge when compressing the learned weights of mainstream pre-trained LMs. Specifically, it studies models of various structures, i.e., encode-only, encoder-decoder, and decoder-only.  Compression on different modules (FF and attention) are also done with interesting findings. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "02J332zk9L", "reviewer": "Reviewer_xMEC", "summary": "", "questions": "* The contemplation of distillation is somewhat of an afterthought. I would query if this was a last minute addition. I would personally be in favor of removing references to this understudied technique to report stronger findings on better-scoped set of experiments.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### On statistical validity:\nOur experiments fall into **two categories**. Some are deterministic, such as pruning, where we used l1-unstructured pruning (see, for example, line 299 of the submitted draft). In this setting, there is no concept of statistical significance as the experiment results are consistent across multiple runs.\n\nIn the second case, such as for our quantization experiments, there is an element of randomness. For quantization, this is due to our use of PTDQ, which computes a dynamic clipping range (described in line 310). We deliberately **struck a balance** between the number of trials per setting and the overall number of settings studied. Consequently, we ran experiments with multiple seeds and recorded confidence intervals, as demonstrated in the table below. This approach aligns with the standard procedure in large-scale machine learning, where numerous settings for large models need to be tested. **Importantly, we observed relatively minor variation** (standard deviation ranging from 1-5%). For this reason, we are confident in the meaningfulness of our findings.\n\n### BERT-base Experiments on Squad Dataset:\nThe results are as follows:\n\n|       | Overall | Attention only | Output only |\n|-------|---------|----------------|-------------|\n| Seed 40 | 5.844 | 13.312 | 3.896 |\n| Seed 50 | 5.844 | 13.312 | 5.195 |\n| Seed 60 | 6.169 | 12.987 | 4.221 |\n| **Mean** | 5.952 | 13.204 | 4.437 |\n| **Standard Deviation (sample)** | 0.188 | 0.188 | 0.676 |\n| **Standard Deviation (population)** | 0.153 | 0.153 | 0.676 |\n\n### BERT-large Experiments on Squad Dataset:\nFor this setting, we have:\n\n|       | Overall | Attention only | Output only |\n|-------|---------|----------------|-------------|\n| Seed 40 | 2.273 | 16.883 | 2.597 |\n| Seed 50 | 2.922 | 14.286 | 3.896 |\n| Seed 60 | 2.922 | 15.909 | 4.545 |\n| **Mean** | 2.706 | 15.693 | 3.679 |\n| **Standard Deviation (sample)** | 0.375 | 1.312 | 0.992 |\n| **Standard Deviation (population)** | 0.306 | 1.071 | 0.992 |\n\n### On \"best practices\" section:\n\nThank you for the feedback! We entirely agree with the reviewer's suggestions and will incorporate these changes in the updated version of our paper.\n\n### On distillation experiments:\n\nIn order to avoid potential variation due to chosen settings when distilling from scratch, we used standard pre-distilled models (such as distilBERT and Lamini-T5) in our study. Notably, **we found that distillation exhibited similar effects** on compression as seen with base models. This discovery suggests an intriguing possible approach: distilling a model before applying compression techniques like pruning and quantization.\n\n### On formatting suggestions:\n\nThank you for the formatting suggestions. We appreciate the input and have already made the necessary adjustments to our updated draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper presents a solid and thoughtful study on different methods of pruning or quantization for LLMs. This work will be helpful in community discussion of efficient methods from LLMs. \n\n* The presentation of Findings in this paper is an exemplar for the *CL community. Everyone should follow their lead.\n\n* Ideas are clear and well presented. This paper will help researchers gain an intuition on methods for compression (but see below for how to maybe improve this).", "reasons_to_reject": "* The core text is lacking in quantitative analysis. Conclusions are drawn based on numbers being \"higher\" or \"lower\" than others with minimal contemplation of (a) scale of similarity or (b) statistical validity. The only quantitative mention in figure axes which are sometimes missing or carelessly annotated.\n The authors could improve this work by testing the statistical significance of their claims and analyzing if each compression technique has a *significant* change on performance. While I want to believe the claims made in the \"Finding\" box, and the presentation is convincing, it would be better received if approached with more statistical rigour. \n\n* Conclusions and recommendations could be better summarised as \"best practices\" for future researchers. At present each finding is largely disconnected from each other. This makes a summary judgement from a lay researcher difficult to extract.", "questions_to_authors": "* The contemplation of distillation is somewhat of an afterthought. I would query if this was a last minute addition. I would personally be in favor of removing references to this understudied technique to report stronger findings on better-scoped set of experiments.", "paper_topic_and_main_contributions": "This paper presents a thorough decomposition of different pruning and quantization techniques for making LLMs more efficient. This work considers different ways to remove layers, connections or numerical precision for LMs ranging from BERT scale to larger >7B parameter models. The contribution of this paper is to study if different granularities of compression can help or hinder extrinsic performance. Each technique is described sufficiently and then different ablations / styles of compression are described and contrasted between.\n\nThe authors propose a series of Findings which may assist future researchers in developing a \"rule of thumb\" for evaluating how to shrink an LLM.\n\n> I have read the authors rebuttal and acknowledge their additional details to their contributions. I'm concerned that the approach to statistical rigour has been \"run many times for consistency\" and not \"run the Kolmogorov–Smirnov / Wilcoxon / Sign test to validate how different populations of models perform\". Statistical rigour here means rejecting the null hypothesis of your outcomes, not consistency between random seeds.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Perhaps highlight recent effort on direct distillation from LLMs https://arxiv.org/abs/2305.14864"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Recommend rendering your figures in .eps format and directly rendering in LaTeX for better readability, no text warping and a consistent aspect ratio\n\n* Fig 6 - 10 would benefit from bar labels with the actual numbers. Similar for any bar chart in the appendix.\n\n* Recommend a clearer split between main text figures and appendix figures. At line 451-452, please state that the Figures are not in the main text and can be found in the appendix. The paper must be airtight without the appendix which reviewers are not required to read."}}, {"review_id": "0tVke41at7", "reviewer": "Reviewer_E2eZ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Pruning and FF vs. Attention Modules:\n\nWhile seemingly intuitive, we have found that the notion that pruning FF modules has a more significant effect than pruning attention modules (due to FF modules containing more parameters), **does not always occur**. Our findings indicate that the outcome depends on the model class and the compression percentage, as elaborated in Section 4.1. The overparameterization of FF modules in certain model families, such as encoder-decoder and decoder-only models, might shed light on this behavior, as mentioned in lines 429-433. \n\nInterestingly, even though FF modules have more parameters in each model we studied, there appears to be a consistent ratio (Appendix, Table 2) between attention and feed-forward parameters:\n\n- Encoder-only: Attention to Feed-Forward ratio of 1:3 for BERT, RoBERTa, and 1:2 for ALBERT, DistilBERT\n- Encoder-decoder: Attention to Feed-Forward ratio of 1:1\n- Decoder-only: Attention to Feed-Forward ratio of 1:2\n\nThis pattern could account for some of our observations, where although FF modules contain more parameters, compressing them might not significantly impair performance in specific scenarios.\n\n### Pruning + Quantization vs. Only One Module:\n\nWe agree that the combined pruning and quantization of both FF and attention modules can yield worse results compared to the compression of only one of these modules. This is an intuitive insight **but nevertheless crucial**, since practitioners will likely rely on it. We have ensured that our findings align with logical expectations and have verified that they hold across various settings. Fortunately, our observations support these expectations.\n\n### Substantial Contribution:\n\nWe believe our contributions are substantial; these include:\n\n- Introducing an alternative metric for understanding LLM knowledge instead of the commonly-used metrics like BLEU score and perplexity.\n- Conducting a comprehensive study involving different model classes and various compression techniques, an area that has not been thoroughly explored before.\n- Modularizing LLMs to analyze the impact of different compression techniques on specific modules. This approach enhances our understanding of the significance of various blocks within an LLM and the consequences of compressing these blocks.\n\n### Analyzing Individual Layers:\n\nFor the BERT-base experiments with 20% pruning using l1-unstructured, we found that compressing final layers leads to a greater performance drop compared to initial layers. This discovery motivated us to conduct a separate analysis on final-layer compression, as described in Section 4.4. Our results align with similar findings from other research papers like [Mitchell, Eric, (2021)](https://arxiv.org/abs/2110.11309), [Mitchell, Eric, (2022)](https://proceedings.mlr.press/v162/mitchell22a.html), and [Meng, Kevin, (2022)](https://arxiv.org/abs/2202.05262). Importantly, we did not explore a bottom vs. middle/top layers analysis due to computational feasibility constraints for very large models (such as Vicuna-7B).\n\n|  Layer-num  | TRex | Google-Re | ConceptNet | Squad |\n|----|------|-----------|------------|-------|\n| 0 | 30.282 | 10.272 | 16.365 | 12.662 |\n| 1 | 30.285 | 10.29 | 16.347 | 13.312 |\n| 2 | 30.272 | 10.327 | 16.374 | 12.987 |\n| 3 | 30.18 | 10.272 | 16.321 | 13.312 |\n| 4 | 30.269 | 10.309 | 16.4 | 12.987 |\n| 5 | 30.199 | 10.272 | 16.347 | 12.662 |\n| 6 | 30.297 | 10.327 | 16.374 | 12.987 |\n| 7 | 30.307 | 10.309 | 16.383 | 12.987 |\n| 8 | 30.228 | 10.29 | 16.339 | 12.662 |\n| 9 | 30.297 | 10.29 | 16.4 | 13.312 |\n| 10 | 30.301 | 10.236 | 16.303 | 12.987 |\n| 11 | 30.326 | 10.2 | 16.374 | 12.987 |\n| 12 | 30.065 | 10.272 | 16.374 | 12.662 |\n| 13 | 29.903 | 10 | 16.392 | 12.338 |\n\n\n### Representation of Results as Average Across All Tasks:\n\nWe acknowledge that **in general**, averaging across different tasks might not be the optimal way to represent our results. However, **for the selected individual tasks**, there is no significant change in the observed patterns. Therefore, we opted to present average results in the main paper and included results for individual datasets in the appendix. This approach is similar to the approach taken in [Fig. 1](https://arxiv.org/abs/2208.07339).\n\n### Experiments on More Pruning and Quantization Techniques:\n\nWe appreciate the reviewer's suggestion regarding the generalizability of our findings to other compression techniques. We do acknowledge this limitation in our study. The reason we included these techniques is their standard nature---and the **critical gap in the literature** concerning comprehensive modular compression studies for various standard techniques across different model families. Our intent was to **fill this gap and initiate a direction** for understanding the effects of modular compression on model families.\n\n### Reference for Encoder-only Models:\n\nThank you for pointing out the need for a reference regarding the compression of encoder-only models. We have updated our draft accordingly. While there is a considerable body of research in the field of encoder-only model compression (as mentioned in lines 537-538), the recent compression techniques we explored (LLM-int8, SpQR, GPTQ) have not been extensively studied on large encoder-only models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors presented a comprehensive experiment on LLM compression varying in the methodologies and model architectures.\nThe reading was smooth and the paper is constructed elegantly. For example, I liked the fact that they included their main findings in textual boxes in order to ease the reading.\n\nWhile most of the findings were quite intuitive and easy to understand, some of them were interesting to read, especially that pruning the attention modules for encoder-decoder models has a stronger impact than feed-forward modules in some cases, or that for encoder-only models, L1-unstructured leads to a smaller decrease in performance than L1-structured.\n\nBesides that, I can't really point out any major contribution to literature.", "reasons_to_reject": "As I mentioned above, most of the findings were not exciting and included statements such as:\n- pruning FF modules has more effect than pruning attention modules since the FF module contains more parameters\n- pruning + quantization of both FF and attention modules is worse than the compression of only one of these modules.\netc.\n\nBesides this paper, there are already various papers that analyze the importance of compression techniques on LLM, and my feeling is that this paper does not substantially contributes to the work of others.\n\nIn my opinion, a much more interesting analysis will be to investigate the importance of specific layers/neurons (e.g.,  bottom layers vs. middle and top layers, attentions that are responsible for X vs. attention heads that are responsible for Y) or the importance of different compression techniques given the specific type of data domain (and not just showing average results across all tasks as the authors chose to do).\n\nAlthough the authors supplied an explanation as to why choosing only one type of technique per method, I would like to see experiments on more pruning and quantization techniques as there are numerous of them. This fact really harms the generalizability of the reported findings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a large experimental study on two compression techniques for LLM: pruning and quantization.\nThe authors focus on structured (L1) and unstructured (Lp) pruning for either attention modules, feed-forward modules, or the final dense layer.\nFor quantization, the authors focus on Post Training Dynamic Quantization (PTDQ), for either attention modules, feed-forward modules, or the final dense layer.\n\nThe authors conduct experiments on the LLM architectures:\nEncoder-only models: BERT, RoBERTa, DistilBERT, and ALBERT.\nEncoder-decoder models: Flan-T5, Lamini-Flan-T5.\nDecoder-only models: Vicuna-7B, WizardLM-7B.\n\nThe authors reach several important findings within their comprehensive analysis.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "in line 70 the authors stated, \"The applicability and effectiveness of such techniques for large ENCODER and ENCODER-DECODER models (Chung et al., 2022) has yet to be extensively studied.\"\n\nThis is certainly not the case, especially for encoder-only models where there is a vast amount of research that discusses the classification impact of many compression techniques."}}]

Author rebuttal:

Meta-review: This work investigates the effect of compressions such as pruning or quantization on the parametric knowledge learned by different types and sizes of pre-trained language models. 

As the reviewer mentioned, the paper presents a solid study on compression of LLMs by focusing on the preservation of knowledge which has been somehow overlooked in existing studies in the compression context. Furthermore, the findings are helpful for the researchers and practitioners for both further compression studies and the use of such methods. Finally, the presentation of the findings demonstrates a nice example and makes the paper easier to follow.  

However, the reviewers also noted that the paper has limited contribution to the existing literature. An important part of the paper focuses attention and the feedforward modules. More fine-grained analysis such as layers and attention heads are suggested. Furthermore, the issues such as varying model size in different model families (note that as mentioned in LLM-int8 some LLMs larger than 6B face a stronger quantization challenge), and separate analyses for different attention modules remain to be investigated within the context of this paper.
Decision: Accept-Findings