Compressing LLMs: The Truth is Rarely Pure and Never Simple
Authors: AJAY KUMAR JAISWAL
Zhe Gan
Xianzhi Du
Bowen Zhang
Zhangyang Wang
Yinfei Yang
Venue: ICLR 2024 poster (2024)
Keywords: Compression
Large Language Models
Pruning
Quantization
Primary area: datasets and benchmarks
TL;DR: We developed LLM-KICK to holistically access compressed LLMs’ ability for language understanding, reasoning, generation, in-context retrieval, long-context summarization, etc.

Abstract: Despite their remarkable achievements, modern Large Language Models (LLMs) encounter exorbitant computational and memory footprints. Recently, several works have shown significant success in *training-free* and  *data-free* compression (pruning and quantization) of LLMs achieving 50-60\% sparsity and reducing the bit-width down to 3 or 4 bits per weight, with negligible perplexity degradation over the uncompressed baseline. As recent research efforts are focused on developing increasingly sophisticated compression methods, our work takes a step back, and re-evaluates the effectiveness of existing SoTA compression methods, which rely on a fairly simple and widely questioned metric, perplexity (even for dense LLMs). We introduce **K**nowledge-**I**ntensive **C**ompressed LLM Benchmar**K** **(LLM-KICK)**, a collection of carefully-curated tasks to re-define the evaluation protocol for compressed LLMs, which have significant alignment with their dense counterparts, and perplexity fail to capture subtle change in their true capabilities. LLM-KICK unveils many favorable merits and unfortunate plights of current SoTA compression methods: all pruning methods suffer significant performance degradation, sometimes at trivial sparsity ratios (*e.g.*, 25-30\%), and fail for N:M sparsity on knowledge-intensive tasks; current quantization methods are more successful than pruning; yet, pruned LLMs even at $\geq 50$\% sparsity are robust in-context retrieval and summarization systems; among others. LLM-KICK is designed to holistically access compressed LLMs' ability for language understanding, reasoning, generation, in-context retrieval, in-context summarization, *etc.* We hope our study can foster the development of better LLM compression methods. The reproduced codes are available at https://github.com/VITA-Group/llm-kick.

Published as a conference paper at ICLR 2024
COMPRESSING LLMs: THE TRUTH IS RARELY PURE AND NEVER SIMPLE
Ajay Jaiswal', Zhe Gan², Xianzhi Du", Bowen Zhang?, Zhangyang Wang', Yinfei Yang² University of Texas at Austin, ? AApple
ABSTRACT
Despite their remarkable achievements, modern Large Language Models (LLMs) face exorbitant computational and memory footprints. Recently, several works have shown significant success in training-free and data-free compression (prun- ing and quantization) of LLMs that achieve 50 - 60% sparsity and reduce the bit width to 4 bits per weight, with negligible degradation of perplexity over the uncompressed baseline. As recent research efforts are focused on de- veloping increasingly sophisticated compression methods, our work takes a step back and re-evaluates the effectiveness of existing SoTA compression methods, which rely on a fairly simple and widely questioned metric, perplexity (even for dense LLMs). We introduce Knowledge-Intensee marK (LLM-KICK), a collection of carefully curated tasks to redefine the eval- uation protocol for compressed LLMs, which have significant alignment with their dense counterparts and perplexity fail to capture subtle change in their capabilities. LLM-KICK unveils many favorable merits and unfortunate plights of current SoTA compression methods: all pruning methods suffer sig- nificant performance degradation, sometimes at trivial sparsity ratios (e.g., 25- 30%). and fail for N:M sparsity in knowledge-inensiiee tasks; current quanti- zation methods more successful than pruning; yet, pruned LLMs even 2 50% sparsity are robust in-context retrieval and summarization systems; among others. LLM-KICK is designed to holistically access compressed LLMs' abil- ity for language understanding. reasoning, generation, in-context retrieval, in- context summarization, etc. We hope our study can foster the development of better LLM compression methods. The reproduced codes are available at https::/ www.comm/////// Bww.oo///V-----o//////aa Group/ 11mm kick
Compressed LLM Bench-
INTRODUCTION
Large Language Models (LLMs) omnipresent, profoundly influencing not only the landscape
of NLP RRam et al. 2023 Liu et al. 2023a (Swadda et al. 2023) Qin et al. 2023 Zhuo 2023
Lee al. (023)), but also recently buttressing numerous computer vision (Lian al.
2023
Wang
et al.) 2023 Lai et al. 2023 Lu et al.] 2023) and graph neural networks Ye et al.
2023 Chen
al.
2023 Qian et al. 2023 Duan et al. 2023) algorithms; achieving steller performance across
various task leaderboards. Despite their numerous unprecedented capabilities, their democratization
primarily restricted by the presence of billions of parameters, which depends on astonishingly high computational and memory requirements. For example, GPT-175B requires 325 GB of GPU memory simply to load its model weights, and at least five A100 (80GB) GPUs with sophisticated parallelism techniques (Sheng et al. 2023).
To democratize LLMs, considerable efforts have been taking to mitigate their high computational cost, mainly divided into two research directions: network pruning, and weight quantization. The former shrinks network sizes by removing specific weights from the model essentially setting them to zero, while the latter aims quantize parameters into lower bit-level representations. Sev- eral recent success in network pruning (Sun et al. 2023 Frantar & Alistarh 2023 JJiswal et al.]
2023a Ma et al. [2033
2023) and quantization (Liu al. all.|2222cc Kim et [2223] Dettmers 2023a Frantar et al. 2022 Lin et al. 2023a Dettmers et al. (023c) (detailed related work
et al.
Work done during internship Apple.

---

Published as a conference paper at ICLR 2024
Miugh duul SeCiiny Weurdd
Mageiaat:
Magribatc
Vousa
Spasecer
Wardda
Wotck
Spaniy Fait
saurpp Raalo
spanty raata
PROMPT 3> Ploase provide answer following, Cuestion: Which 1959 Alfiod Hichcook film had the sagline "It deadly game sag and Cary Grant i ' The answer
The antaar Vodth Northweed
arswver Cary Grant, who playedd the characher of film
The aroower Thke for Murrer (1854)
arrower (bear Window
The anaower le Catch
Thatt
Mega Sanddd.Se 50% Cuseprovol Veaso-B8
PpassCPP 59% Conpecone Vieaso7B
lmms
Wark 50% Coepeosee
4-58 GPTQ Cheanoool
Viaan-10
PROMPT >>> Pressa de arreeet the following Cuvotion:: By ishal rame Allen Kariggsburg bellee knosen? The arsower
The anewar be Weady Aler.
The aenoae 1963, wnd Ronwll Reagan
The arooerri Alen Karggatweng in beter known Keareg
The anoworls 100% corrot.
The aroowerte R00%
Alggimndee 50% Comppseeed SpanscIFP 55% Copperosed Vi
Wark 50% Oomproved Vicano 18
4-52 GPTQ Congpoooal
Vicrso-11
Figure I: True Merits of So Compression. Top row indicates marginal increase in perplexity via using So'T compression methods, when compared with simple magnitude-based pruning. Bottom row indicates the failure of compressed Vicuna-7B (Chiang et al.] 2023) (via Magnitude, Wanda,
SparseGPT, GPTQ) to respond correctly knowledge-intensive factoid-based questions.
discussion in Appendix A.I1) claim to retain the uncompressed LLM's performance while achieving 50-60% sparsity or up to cxtreme 2-3 bit quantization. Although these advancements look fascinat- ing, in most (if not all) cases, they heavily rely on perplexity as their primary metric evaluate the performance claims. Such relatively restricted evaluations limit the scope for developing new compression methods, and are potentially ill-suited to identifying new and unexpected capabili- ties/limitations of compressed LLMs.
Perplexity, even the case of dense LLMs, has been questioned as an unsatisfactory measure for comparing the true potential of LLMs, despite significant variations in model scales, training strate- gies, and architecture choices (Muhlgay et al. (023). It is important to note that all compressed models derived from the same dense counterpart with high similarity, and aforementioned dif- ferences don't exist, making their evaluation more challenging. In this work, we revisit a widely known yet under-explored question: How well does perplexity capture the change in capabilities of compressed LLMs that have significant alignment with their dense counterpart? We focus on the case of compressed LLMs, because we observe comparatively more serious failure of perplexity to capture the delicate performance variations incurred across varying compression stages of LLMs, demanding more ine-grained investigation.
In this work, we attempt to investigate the true promises and limitations of state-of-the-an com- pression algorithms for L.Ms.. We assemble the first comprehensive and diverse collection of tasks with varying difficulty levels to thoroughly study compressed LLMs under quantization and network pruning (structured and unstructured sparsity patterns). More specifically, we consider a broad range of tasks to evaluate subtle changes in pruned and quantized LLMs' ability for language understand- ing, reasoning, generation, in-context retrieval, long-context summarization, etc. Note that none of the datasets in our dineti-ddmesiionall study of compressed LLMs created from scratch, but rely on existing datasets as they have been widely accepted by researchers, but unfortunately yet not been adopted to study the effect of compression. We rigorously measure the performance of SoTA quantization and pruning approaches (in their most common, default settings), to understand their potential for our challenging and interesting tasks with high practical value.
Our key observations and contributions can be unfolded as:
We present Knowledge-Intensiv Compressed LLM BenchmarK (LLM-KICK), to re-define the evaluation protocols for compressed LLMs and facilitate a comprehensive assessment of SoTA compression algorithms. The premise of our work is to develop suite of challenging, realistic, and diverse tasks of high practical importance and datasets that can empower a systematic un- derstanding of how existing L.M compression strategies truly perform in preserving performance
2

---

Published as a conference paper at ICLR 2024
despite their similar perplexities, how they differ from each other, and how they compare against smaller LLMs of comparable parameter counts.
LLM-KICK unveils many interesting and critical observations, that perplexity- -hasit evaluations overlook. L Most SoTA pruning methods suffer significant performance degradation, sometimes trivial sparsity ratios (e.g., 25-30%), despite negligible changes in perplexity. ② All SoTA pruning methods do not work satisfactorily for structured N:M sparsity patterns on LLM-KICK. ② Current SoTA LLM quantization methods are more successful in perpetuating performance in comparison to So LLM pruning methods. (4) Compressed LLMs fail to generate knowledge- enriched and factually correct answers, despite the generated text is fluent, consistent, and coher- ent. 4 Compressed LLMs with larger architectures but same parameter counts perform poorer, which favors smaller dense models.
We further investigate compressed LLMs' ability for in-context settings, via adopting in-context retrieval augmented question answering (ICRA-QA) A-Q) (Ram et al. 2023), and text summarization with in-context learning (IC-Sum) (Jain et al. 2023). To our surprise, pruned LLMs, even at non- trivial sparsity ratios (e.g.. 250%), are robust retrieval systems, and can perform text summariza- tion while maintaining similar performance as their dense counterpart. However, with increasing compression degrees, their ability digest longer context affected more than smaller context.
2 SOTA LLM COMPRESSION: PERPLEXITY, OR WHAT'S MORE?
Scaling neural networks, now LLMs, have achieved astonishing performance benefits on a wide array of tasks, but at the cost of gigantic computational and memory footprints. Network pruning and weight quantization are two popular remedies to mitigate these overheads due to billions of parameter counts in current LLMs. Despite numerous existing algorithms for pruning (Singh & Alistarh 2020 Zhu & Gupta 2017; Gale et al. 2019 Jaiswal et al. 2022 Lin ct al. 2020 Liu ct al,,
202 Mostafa & Wangl 2019 [Dettmess & Zettlmmyerr 2019]] Evci et al.) 2020) and quantization
2/ (Dong et al.] 2022) Cardinaux al.) 2020 Kim et al. [022]] Liu et al. 2022b) Martinez et al.) 2020),
their ad-hoc adaptation for LLMs is restricted, due the lack of luxury to perform iterative training to regain any performance drop during compression. Recently, several works have shown significant success in training-free and data-free compression of LLMs achieving 50-60% sparsity and reducing the bit-width down to 3 or 4 bits per weight, with negligible perplexity degradation relative to the uncompressed baseline.
Perplexity is a statistical measure of how confident a language model predicts a text sample and quantifies the "surprise" encoded within language models (the lower the perplexity, the better the model). Despite its popularity, perplexity has been widely questioned as unsatisfactory measure to compare the true merits of two different LLMs (Muhlgay et al.] 2023). even for dense models although they significantly vary in model scale, training strategies, and design choices (encoder only, decoder only, etc.). To address this issue, several works (Li et al. 2023 Kaddour et al. 2023
Muhlgay et al. 2023 Zhang et al. 2023 Valmeekam et al.) 2022 Liu et al. 2023a (Sawada et al.
2023 Qin et al. 2023; Zhuo 2023 Lee et al. (023) attempt to go beyond perplexity, and evaluate the capabilities of dense LLMs across commonsense reasoning, language understanding, reading comprehension, programming, etc. However, it is critically important to note that all compressed models are derived from the same dense counterpart with high similarity sharing exactly the same scale, training strategies, design choices, etc. Surprisingly, unlike dense LLMs, no such effort has been carried out to understand subtle changes in the capabilities of compressed LLMs with varying compression strength. Orthogonal to the recent trend to develop new compression algorithms, our work provides the first attempt to assess the true merits and limitations of existing SoTA LLM compression algorithms, to provide a fair and detailed playground to develop better compression algorithms. We focus on the case of compressed I.LMs because we observe the profound failure of perplexity in capturing the delicate performance variations across varying LLM compressions.
Figure | Top) illustrates the change in perplexity of SoTA compression methods (pruning and quan- tization), such as SParseGPT. Wanda, GPTQ and baseline one-shot magnitude-based pruning on Vicuna-7B. 7B, 13B, and 33B (Chiang et al. 2023). Clearly, the perplexity (L) of all models does not show any significant variation up to 45-60%, with complete failure to capture subtle changes in the abilities of LLMs when compressed. is also interesting to observe that to a certain degree of spar- sity (~ 30%). all SoTA pruning methods have almost similar performance as the simple baseline of
3

---

Published as a conference paper at ICLR 2024
one-shot magnitude- pruning, which raises questions about their true merits within this sparsity range. Figure TiareeTBoooa Bottom) show the response of Vicuma-7B model when compressed with Magnitude, SparseGPT, and Wanda by 50% and quantized up to 4-bit. The uncompressed Vicuna-7B was suc- cessfully able to generate the correct answer, but all compressed versions failed to respond correctly, hallucinating with either wrong facts or irrelevant responses.
3 LLM-KICK: UNVEILING TRUE MERITS OF LLM COMPRESSION
LLM-KICK, short for Knowledge- Instensive Compressed LLM BenchmarK, is crafted to bring the attention of LLM compression community towards incompetence of perplexity to correctly reflect subtle changes in the ability of LLMs derived from dense counterparts with varying compression strength. LLLM-KCK consists of suite of challenging, realistic, and diverse task settings of high practical importance and datasets that can empower a systematic understanding of how existing LLM compression strategies truly perform in preserving performance despite having similar perplexity. Our work thoroughly investigates proclaimed merits/limirations of pruned and quantized LLMs for language understanding, reasoning, generation, in-context retrieval, in-context summarization, etc.
Specifically, LLM-KICK consists of broad task settings to study how compression impacts knowl- edge encoded during pre-training, how compressed LLMs perform tasks when required knowledge is augmented in-context, and how well compressed LLMs perform instruction following. To com- partmentalize task difficulty and diversity, we include factoid- QA, multiple-choice reasoning- based QA, in-context retrieval augmented QA, in-context text summarization, and instruction-based free-form text generation. Instead of creating new datasets, we carefully curate LLM-KICK from prior works and open-source GitHub repositories which have been widely accepted by researchers, but yet not explored by the LLM compression researchers. Our detailed prompt design strategies for different task settings can be found in Appendix A.2
To reduce the expense of redundant experiments and clutter in results, our work primarily focuses on the top-2 existing maining-free and data-free LLM pruning techniques (i.e., SparseGPT (Frantar & Alistarh 2023) and Wanda (Sun et al., 2023)) along with the baseline of One-shot Magnitude-
Pruning (Han et al. 2016), plus popular quantization technique (GPTQ) among recently available
choices (Lin et al. 2023a Frantar et al. 2022] Dettmers et al., 2023c We consider two types
of sparsities: (i) Unstructured Sparsity: individual model weights are zeroed out independently, leading to irregular zero patterns (LeCun et al. 1990, Han et al., 2016); and (ii) Structured N:M Sparsity: fine-grained sparsity pattern in which only N weights are non zero for every continuous M weights (Nvidia 2020; Zhou et al., (202]. We use Vicuna models for experiments, which are open-source chatbot models trained by fine-tuning ILLMA (Chiang et al.] [003) on user-shared conversations collected from ShareGPT, and have demonstrated impressive 90% quality of OpenAl ChatGPT and Google Bard. Note that the aim of this work not limited to identifying the failure cases of SoTA pruning methods, but instead provides in-depth lookup of LLM's ability under compression, and bring new insights which include highlighting observations that work in favor of current SoTA compression methods.
Formally, we study the performance drop of LLMs after compression (without fine-tuning) with respect their dense counterparts using compression algorithm C. For pre-trained LLM [a;; 0), a compressed LLM is a network fzamp (r;; fz), which a copy of f(; 8) with some weights fixed to 0 indicated by the pruning mask my the case of pruning, or quantized to ke-bit using a quantization algorithm. Next, we define matching compressed LLM.
Matching Compressed LLM: A compressed LLM Secmp (z; 0c) is matching for a com- pression algorithm on task T, if it results in performance no less than ep (compression tolerance regime) in comparison with f(;; T). In this work, we consider to be < 5% of the performance of (x; T)
Note that 6o is simple indicator of the tolerance level of performance drop when we start com- pressing any LLM. Many prior works (Chen et al., [020b] Jaiswal et al.) 2023a) consider matching thresholds to be the same as the dense subnetwork performance or within the margins of 1%. How- ever, in our work, carefully relaxed to 5% performance drop an acceptable tolerance (before
4

---

Published as a conference paper at ICLR 2024
Liara 28
Veane-T
Vours.r
PeSter DIV D.Oo Powaa
Speraedry
1c:te 52T0 G2T0
446 SPT
S4e GPTC
100
GOTO
1020222 20
NJV
Spandly
Sparsty
Figure 2: Compressed LLMs for Factoid-base based QA. Performance comparison of compressed LLMs on Factoid-QA task using FreebaseQA (Jiang et al., 2019). Results (average across 3 inde- pendent runs) presented are for structured (N:M sparsity), unstructured sparsity, and quantization.
calling the compressed model useless) keeping in mind that the performance of compressed LLM on any of our task categories/disciilines remains above the random guess.
3.1 SETTING 1: How WELL COMPRESSED LLMS ACCESS REMAINING KNOWLEDGE?
! Factoid- based Question Answering
Task Definition and Rationale. Factoid-based Question Answering (Factoid-QA) (Iyyer et al. 2014), which asks precise facts about entities, is long- long-standing problem in NLP. A typical Factoid- QA task aims to search for entities or entity attributes from a knowledge graph, and is widely used as tool in academia, commercial search engines, and conversational assistants. Modern LLMs trained on gigantic text corpora ingesting a large amount of world knowledge about entities and their relationships during pre-training, and have unique abilities generate factually correct responses to user queries. In this task setting, we aim to investigate how compression impacts LLMs' ability to answer natural language questions using facts, i.e., entities or attributes knowledge ingested within them during pre-training.
Dataset Details. We use FreebaseQA @Jang et al. 2019) which dataset for open-domain QA over the Freebase knowledge graph. The QA pairs are collected from various sources, including the TriviaQA dataset (Joshi al., 2017) and other trivia websites (QuizBalls. QuizZone, KnowQuiz), and are matched against Freebase to generate relevant subject-predicate predicateeoboeee triples that were fur- ther verified by human annotators. TriviaQA dataset shows rich linguistic variation and complexity, making it a good testbed for evaluating knowledge ingested within LLMs.
Results and Analysis. The results of various LLM compression methods are demonstrated in Figure 2 Our primary observations include: ! All SOTA LLM pruning methods seemingly fail to find matching sparse LLMs, even at trivial sparsities such as 30-35%. While several methods maintain the matching performance at 20-25% sparsity, their performance starts to drop significantly after that undergoing a catastrophic failure as sparsity ratio increases. This is in contrast with the claim made by SoTA pruning methods that pruning up to 50-60% of LLMs doesn't 't have any significant degradation on performance. 2 All pruning methods doesn't work for fine-grained structured N:M sparsity patterns with performance drop as severe as 25%. non-aggressive 8-bit quantization indicates that along with chasing for aggressive quantization levels (1-2 bits), it is also important to focus on yet unsolved 8-bit quantization.
8-00 drop in performance for
Multiple-Choice Reasoning based Question Answering
Task Formulation and Rationale. Multiple-Choice Reasoning based QA (MCR-QA) uses a natural prompting approach to present the question and answer options to the LLMs jointly, and have it output the symbol (e.g.. "A") associated with its chosen answer option. allows the model to explicitly compare answer options. In this setting, we aim to investigate compressed LLMs' ability to understand natural language questions, effectively reason using knowledge remaining within them, and successfully associate the correct answer among the given answer options with the symbols that represent them; potentially minimizing the effect of tokenization and exact answer generation.
Dataset Details. We use the popular MMLU (Massive Multitask Language Understanding) bench- mark which covers 50+ subjects across STEM, Humanities, Social Sciences, and more (Hendrycks) et al. 2020). It ranges in difficulty from elementary level to an advanced professional level, and it tests both world knowledge and groblem-solvingg ability of LLMs. The granularity and breadth of subjects make it ideal for fine-grained evaluation of compressed I.LMs" blind spots.
5

---

Published as a conference paper at ICLR 2024
Science
Olthars (Business, Heulth
N6 Parformrro (aon
Veurs.Th 7B
Mopei ste
Angeaude
Spcpecin
Vheda
Wanth
1i Partommnns momermancco Drop (Accunic))
Vicuue: 7B (NUM Sparsiyy
20
Mogotas
-00
-00
Werk
-100
to Performooo Dogg (ossumwa
-00
-12
Vicurna 1.3
Mogritue
Vage
Vageltude
400.GT
4id CITIC
4ULGrTQ
16 Plaffcr Parformninon maitos Dinin (Acssmay))
Vicurre 23B (NM Sparsity
40
-60
Figure 3: Compressed LLMs for Multiple-Choice Reasoning based QA. Performance compari- son of compressed LLMMs on MCR-QA tasks using the MMLU benchmark (Hendryeks al. 2020). Results (average across 3 independent runs) presented are for structured (N:M sparsity), unstruc- tured sparsity, and quantization.
Results and Analysis. The results of various LLM compression methods are demonstrated in Figure Our primary observations include: ① Despite a similar matching compression regime (~ 20- 40%) to Factoid-QA, the abrupt performance drop of all SoTA pruming methods for MMLU is comparatively subtle due to relaxing the task setting from exact answer generation to correct answer selection. 2 No matching compressed I.LMs are found for N:M structured sparsity. 3 SoTTA LIM quantization is seemingly more successful than SoTA pruning methods: we found 8-bit and 4-bit compressed LLM to be matching for Vicuna-7B and Vicuna-13B. respectively. ④ Interestingly, both quantization and pruning have comparatively higher performance drop for Humanities and Social Science wrt. STEM, which indicates compression impacts some disciplines more than others. 5 Surprisingly, within the compression tolerance regime, simple one-shot magnitude pruming seems perform quite well in comparison with SoTA pruning method, illustrating its high effectiveness.
3.2 SETTING 2: How WELL COMPRESSED LLMs SYNTHESIZE AUGMENTED KNOWLEDGE?
① In-context Retrieval Augmented Question Answering
Task Formulation and Rationale. In-context Retrieval-Augmented Question Answering (ICRA- QA) (Ram et al., (023) grounds the LLM answer generation by conditioning on relevant documents retrieved from an external knowledge source using retrieval algorithms like BM25. Our ICRA-QA evaluation system includes two high-level components: @ document selection, selecting the set of documents upon which to condition; and L document reading, determining how to incorporate the selected documents into the LLM answer process, which requires extracting correct answer phrases from conditioned documents. To discount the impact of the lost encoded knowledge during compression, ICRA-QA augments the required relevant knowledge for QA task directly within the

---

Published as a conference paper at ICLR 2024
Vicura-78 Closed Book
Vicune 781 Book
Vicuni-33R 138|
Viaine Open Book
Pafoormnnc Drop
Sperseti
-0
-00
10-a: gang
-00
10 paro
15 ERTO
<he OPTO
sbr OPEO
2015 a ae Sparsity Ratio
Sparsity
Rato
06 Performeoo Drop
-20
-20
-20
-00
-40
-40
-10
Magitade
MWN
Magn tall
-50
-60
-80
Wordh
1.2
1:2
Figure 4: Compressed LLMs for In-context Retrieval Augmented QA. Performance compari- son of compressed LLMs on ICRA-QA task. We present head-to-head comparison of closed- book evaluation (no external knowledge is augmented in-context) with open-book evaluation (external knowledge is augmented in-context). Results (average across 3 independent runs) presented are for structured N:M sparsity, unstructured sparsity, and quantization.
prompt context. In this task setting, we aim to evaluate compressed LLMs' ability to synthesize long in-context knowledge provided within input prompts, and locate and retrieve correct answers within it. We also present a head-to-head comparison of how augmented knowledge can work as a remedy to supplement the lost knowledge under compression.
Dataset Details. We use TriviaQA (Joshi et al. 2017) for evaluation, a popular reading comprehen- sion dataset which includes 95K question-answer pairs authored by trivia enthusiasts and indepen- dently gathered evidence documents, six per question on average, that provide high-quality distant supervision for answering the questions.
Results and Analysis. The results of various LLM compression methods are demonstrated in Figure 17 The closed-bood setting differs from ICRA-QA (i.e., using the open-book setting) only in terms of whether conditioning on relevant documents retrieved from an external knowledge source. Our key findings are: ! When compressed LLMs are conditioned on external knowledge (open book) and assigned the task of in-context retrievers, i.e,, extracting correct answer phrases from in-context knowledge, they perform significantly well even in extremely high compression regime. Vieuna- 7B can remain matching till ~40% sparsity and 8-bit quantization, while Vicuna-13B can remain matching up to ~50% sparsity and b-bit quantization. Our experimental results send a positive signal that even if high compression leads to significant knowledge loss, it doesn't 't leave L.MS completely useless, and they still work as robust in-context retrievers. 2 Despite we observe a significant benefit while conditioning external knowledge, no matching compressed LLM can be identified for N:M sparsity. a Again, we observe surprisingly good performance of simple one-shot unstructured magnitude pruning wrt. SparseGPT (second-order pruning) and Wanda factivation-based pruning) that rely on calibration data.
② In-Context Text Summarization
Task Formulation and Details. Modern LLMs have shown astonishing success in summarizing long-context documents in both abstractive and extractive settings. However, is yet not explored how compression impacts LLMs' capability for summarization. In this task setting, we aim to investigate compressed LLMs' ability to hold onto consistency. coherence, flunccy and relevance when prompted to summarize textual information of varying length (small, medium, and large) in abstractive setting (Jain el al.) (023). For evaluation, similar to ZZheng et al. (2023), we propose to use GPT-4 judge, which compares the compressed LLM generated summaries wrt. GPT-3.5 (text-davinci-003) generated summaries. Detailed evaluation settings can be found in Appendix A.3
Dataset Details. We use popular summarization dataset CNN/DailyMail (Chen et al. 2016 for evaluation, which is an
dataset containing just over 300k unique news articles written by journalists at CNN and DailyMail. We created 3 subset categories {small ($4400 words), medium (2470 and < 790 words), and large (2 790 words)) of stories, each with 100 articles reflecting word distribution of CNN//ailyMail to minimize OpenAl API costs.
7

---

Published as a conference paper at ICLR 2024
GPP14 ie Scoee Desp (Mag (Magritaah)
Coherence
Consistency
Fluency
Relevance
Beeal tiver
10
30
Stol
Sany
Mil Strey
Siny
-00
Larpe Slap-
-20
Larg Soay
-20
Luppt Sjay-
-20
Laoge
20
2015 20.25 3020-4 DO
1015.0 253032-4444 57538025.70
3023 32225 45523000085
Sparshy Raito
Sparsty Rabo
Sparsity Raz
Sportiy Railo
to Scaree Drop (Spansaan)
-10
Srall Sine
-10
Spay
-10
Siory
-10
Senry
Sory
Medeey
Mi Stady
Stay
120
Lorve
-20
Last sijey
-20
(IPT
pero
1038 20 0330-00 300000000000
1015.0 2010118- 4044 5ib08095
3039 2522 40303666099
3-3335 2012 a0
Sparshy Rato
Spartty Rono
Sparsiy Rato
Sporiyy Parto
Mayina
100 Mag
GPT-4
Smbll
M1C
Larg
5mlll
140
Large
Smal
M2
Large
S:12
MC
Large
Figure 5: Compressed LLMs for In-Context Summarization. Performance comparison of com- pressed Vicuna-7B for in-context summarization of small, medium, and large stories while preserv- ing coherence, consistency, fluency, and relevance. Results (average across 3 independent runs) presented are for structured (2:4 sparsity- Row 3), unstructured sparsity, and quantization.
Ch
OPE4 2.0g2 t4 Son'd Do00
Vigure
1o
Vayrbude SpacoeCPT
Avtige
Wird
Sparally Faalo
N:0 S3a
Spansity Rabio
Figure 6: Compressed LLMs for Instruction Following, LLM-as-a-Judge: GPT4 based evalua-
tion of compressed Vicuna-7B response wrt. ChatGPT (davici-003) (Left) unstructured spar- sity; (middle) structured N:M sparsity; (c) comparison of average unique token counts generated by compressed Vicuna-7B for 80 prompts across 10 different categories.
Results and Analysis. Results are summarized in Figure 3 We summarize our main observations as: ! All pruning and quantization methods tend to perform surprisingly well for in-context sum- marization, preserving high consistency, coherence, fluency. and relevance in generated summaries, which is an encouraging observation in favor compression. ? With increasing context length (i.e., long stories), we observe a sharper performance drop for compressed I.MMs, which highlights that compression impacts LLMs' ability to synthesize and summarize longer context lengths. 3 Quan- tization again seems to perform better than SoTA pruning methods, and surprisingly benefiting pos- itively over the dense model performance. ④ No matching compressed LLM can be identified for 2:4 structured sparsity.
3.3 SETTING 3: HOW WELL COMPRESSED LLMS PERFORM INSTRUCTION FOLLOWING?
Task Formulation and Rationale. In this task setting, we investigate compressed LLMs' ability to answer open-ended questions and evaluate their multi-turn conversational and instruction-folllvinng ability- two critical elements for human preference. Evaluating AI chatbots is a challenging task, it requires examining language understanding, reasoning, and context awareness. To compare the performance of compressed LLMs' responses, we closely follow the prompt design setting in MT-Bench (Zheng al. 2023) using GPT-4 as a judge. We prompt GPT-4 to the answers gen-
8

---

Published as a conference paper at ICLR 2024
erated by compressed LLMs wrt. GPT-3.5 (text-davinci-003) model based varying metrics (e.g., correctness, helpfulness, logic, accuracy, etc.) on a scale of [0-10] with detailed explanations.
Dataset Details. We rely on the 80 high quality multi-turn questions identified in MT-Bench (Zheng ct al. 2023). This setting covers common-use human-centric interaction with LLMs, and focuses on challenging questions to differentiate models. We used 8 common categories of user prompts to guide the prompt construction to interact with compressed LLMs: writing, roleplay, extraction, rea- soning, math, coding, etc. For each category, we adopted manually designed 10 multi-turn questions from MT-Bench to evaluate our compressed models. Details can be found in Appendix A.4
Results and Analysis. Results summarized in Figure 6 Our primary observations are: ! Unlike in-context text summarization, in this task setting, compressed LLMs have to access the knowledge respond conversations maintaining high helpfulness, relevance, accuracy, and detail. We again observe that compressed LLMs with various pruning methods are matching only up to sparsity ratio of 25%. 2 Surprisingly, in the matching regime, the simple baseline of one-shot magnitude pruning performs comparable matching subnetwork can be identified for N:M sparsity. Interestingly, our average generated unique token analysis in Figure ((c) illustrates that compressed LLMs lose the ability to generate distinct unique content, instead, they can only produce more repetitive texts.
slightly better than SoTA pruning methods. ① No
ADDITIONAL RESULTS AND DISCUSSIONS
Small-Dense Large-Sparse: which is favorable? We attempt to understand interesting question: if pruned LLMs with larger architecture (Large-Sparse) is better than smaller dense mod- els with similar parameter count SmaallDense)-- Pruning large LLMs doesn't 't come for free, and is important to investigate if the cost of pruning can be reflected in the performance benefit of Large- Sparse models. To our surprise, in comparison with dense Vicuna-7B (MMLU accuracy 46.7%) we found compressed Vicuna- with exactly similar parameter count (46.16% sparsity) of 7 billion using one-shot magnitude, Wanda, SparseGPT can only achieve MMLU accuracy of 31.7%, 45.3%, and 46.3%, respectively. This is clear indication that current sparsity algorithms are not yet up to stage where the cost of pruning can justified by performance benefits obtained from large-sparse compressed models.
How many calibration data samples are needed? We at- tempt to analyze how calibration dependent pruning methods (Wanda and SparseGPT) perform with varying amount of cali- bration samples. Figure illustrates the zero-shot performance of 50% & 70% pruned Vicuna-7B using Wanda and SparseGPT on knowledge-intensivg MMLU benchmark. observe that calibration sample count plays vital role in pre- serving the performance of SparseGPT unlike Wanda. Note that at high sparsity ratio (70%), Wanda cannot recover any performance: SparseGPT surprisingly benefits noticeably from calibration. This suggests that carefully selected calibration samples can play a vital role in designing better pruning algo- rithms to compress LLMs up to significantly high sparsity.
40
Average Accuracy
Wanta 50%
20
SpansoPT 50%
Wabca Z0M
SpanwGPP
interesting to
64
128
Wkiieett Caliberttion Samples
Figure 7: Zero-shot performance of 50% & 70% pruned Vicuna-7B wrt. calibration sample counts.
CONCLUSION AND LIMITATIONS
In this paper, we propose to explore the effectiveness of SoTA compression methods beyond per- plexity to address the inability of perplexity to capture the subtle variations incurred during the derivation of compressed LLMs from their dense counterparts. Our work introduces Knowledge- Intensive Compressed L.LM BenchmarK (ILLM-KICK) to facilitate a fair and holistic evaluation by unveiling many merits and pitfalls of SoTA compression methods. Our study reveals that compres- sion significantly impacts the knowledge encoded in LLMs during pre-training, compressed LLMs perform quite well with knowledge augmented in-context settings. We primarily restrict our eval- uation to Vicuna (decoder-only architecture) due to its open-source license, high performance, and instruction-followinn ability. For future work, aim to investigate how the lost knowledge due to compression can be recovered using parameter-effe -fffiiee-efficientt fine-tun mee-tunig methods, e.g., LoRA (Hu et al. 2021) and QLoRA (Dettmers et al. 2023b

---

Published as a conference paper at ICLR 2024
REFERENCES
Fabien Cardinaux, Stefan Uhlich, Kazuki Yoshiyama, Javier Alonso García, Lukas Mauch, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Iteratively training look-up tables for network quantization. IEEE Journal of Selected Topics in Signal Processing, 14(4):860-87, 2020.
Danqi Chen, Jason Bolton, and Christopher Manning. thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv: 02858, 2016.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834-15844 2020a.
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. arviv preprint arXiv:2101.00063; 2020b.
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al. Exploring the potential of large language models (Ilms) in learning on graphs. arXiv preprint arXiv:2307.03339, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//imsys. Linnsyy..ro/blooo/220000000---00 -0-viiiuan----
Tim Detimers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv: 1977::9907.04440 2019.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 40318-30333 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.
Qlora: Efficient
finetuming of quantized 1lms. ArXiv, abs/2305.14314, 2023a. URL https://api.
semant org/ http//CrpussDi : 258841328
Tim Dettmers, Artidoro Pagnoni. Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprins 230i::23005/43114 2023b.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized rep- resentation for near-lossless IIm weight compression. ArXiv, abs/2306.03073, 2023c. URL
Runpei Dong, Zhanhong Tan, Mengdi Wu, Linfeng Zhang, and Kaisheng Ma. Finding the task- optimal low-bit sob-distribution in deep neural networks. In International Conference on Machine Learning. pp. 5343 5359. PMLR, 2022.
Keyu Duan, Qian Liu, Tat-Seng Chua, Shuicheng Yan, Wei Tsang Ooi, Qizhe Xie, and Junxian He. Simteg: A frustratingly simple approach improves textual graph learning. arXiv preprint vrXiv:2308.02566 2023.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR. 2020.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot, 2023.
Elias Frantar, Eldar Kurtie, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems, 34:14873- 14886, 2021.
10

---

Published as a conference paper at ICLR 2024
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ArXiv, abs/2210. 17323, 2022. URL
https://api www.stecccnncola.......cc.c httpss///aab..emmmnncccccoooor.o ottp/orppss1::5222222222
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint wwxXiv:902.099777 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 0300000000 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09688 2021.
Mohit lyyer, Jordan L. Boyd-Graber, Leonardo Max Batista Claudino, Richard Socher, and Hal Daumé. A neural network for factoid question answering over paragraphs. In Confer- ence on Empirical Methods in Natural Language Processing, 2014. URL https://api.
semant wcscholacc......oo wwwpoolaa.rrgg/oooroo wotp/oorrrIIDao//o0oo 22634467226
Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint arXiv:2306.0122 2023.
Ajay Jaiswal, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. The emergence of essential spar- sity in large pre-trained models: The weights that matter. arXiv preprint arXiv:2306.0380 2023a.
Ajay Kumar Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pp. 9833-9844. PMLR, 2022.
Ajay Kumar Jaiswal, Shiwei Liu, Tianlong Chen, Ying Ding, and Zhangyang Wang. Instant soup: Cheap pruning ensembles in single pass can draw lottery tickets from large models. In Interna- tional Conference on Machine Learning, pp. 14691-14701. PMLR, 2023b.
Yupeng Ji, Yibo Cao, and Jiucai Liu. Pruning large language models via accuracy predictor. arXiv preprint w0XVv2230.005577, 2023.
Kelvin Jiang, Dekun Wu, and Hui Jiang. Freebaseqa: new factoid qa data set matching trivia-style question-answer pairs with freebase. In North American Chapter of the Association for Com- putational Linguistics, 2019. URL https:: http:://p....mmmasssss..oooooooe
supervised challenge dataset for reading comprehension. arXiv preprint wtt.://urr.r//ooouuuss....
174800890
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly 2017.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169 2023.
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language mod- els via sub-4-bit integer quantization. ArXiv, abs/2305. 14152, 2023. URL https://appi semant. i cscho l ar. www./orgpu.sssoooooo TD: 200821188110
Schoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer- only bert quantization. In International conference on machine learning, pp. 5506-5518, PMLR, 2021.
11

---

Published as a conference paper at ICLR 2024
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXi::2203.07259 2022.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint arXhv:2109.04838 2021.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. arXiv preprint arXiv:2308.0000 0at:::2388.0002,20000 2023,
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural information processing systems, pp. 598-605, 1990.
Noah Lee, Na Min An, and James Thorne. Can large language models infer and disagree like humans? ArXiv, 1bs/22305.1388, 2023. URL https: /ttp:://apiiie www.seeeatniccccceeeeoe org/
CorpusID:25884100
Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang. Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, et al. Flm-101b: An open Ilm and how to train with 100 budget. arXiv preprint 2.X.::2309.03852, 2023.
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern Felix Yu, Ruiqi Guo, et al. Large models are parsimonious learners: Activa- tion sparsity in trained transformers. arXiv preprint arXiv:2210.06313, 2022.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023.
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for I1m compression and acceleration. ArXiv, 00sp0300600978 2023a. wttp//oopusS::253995550
www.s:eeaannccccc.ac URL https:://// htt.pp/:/apl.eemmnnccccooo..aaa
Ji Lin, Jiaming Tang. Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for I1m compression and acceleration. arXiv preprint TrXiv:2306.00978, 2023b.
Tao Lin, Sebastian Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL https:
// //openreview. net / www.moiiumiioooo SJem81SF Stttt//ooommmiddSSmm_sssse--
Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang. Chenyu You, et al. Limrec: Benchmarking large language models on recommen- dation task. arXiv preprint arXiv:2308. 2tt.::2038.222.1... /2241, 2023a.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems (NeurlPs)., 2021a.
Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Jaiswal, and Zhangyang Wang. Sparsity may cry: Let fail (current) sparse neural networks together! arXiv preprint arXiv:2303.0214/ 2023b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LIm-qat: Data-fre free quantization aware training for large language models. arXiv preprint 2305 7888, 2023c.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quan- tization for vision transformer. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021b. URL hetps: 7/openrevi ew. wett/forrmmmd==77000000
12

---

Published as a conference paper at ICLR 2024
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-ple compositional reasoning with large language mod- cls. arXiv preprint 230.:2200409944,, 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprini urXiv:2305. 1627, 2023.
Julieta Martinez, Jashan Shewakramani, Ting Liu, Ioan Andrei Bårsan, Wenyuan Zeng, and Raquel Urtasun. Permute, quantize, and fine-tune: Efficient compression of neural networks. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15694-15703, 2020.
semant icscholar. ottp/Corpus1::2221133333
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1rXiv:16090078e0 2016.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning. 2019.
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton- Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. arXiv preprint arXiv:2307.06908 2023.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequenee rmns and beyond. arXiv preprint arXiv: 1r0v.:660.000023. 2016.
Nvidia. Nvidia a100 tensor core gpu architecture. oawp://eeeeeeeeeeedddeeeeeeeeeee
httpss//www.mdddi..ccooootttttto
2020.
Chen Qian, Huayi Tang, Zhirui Yang. Hong Liang, and Yong Liu. Can large language models empower molecular property prediction? arXiv preprint arXiv:2307.07443, 2023.
Chengwei Qin, Aston Zhang. Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general-purpose natural language processing task solver? arXiv preprint arXiv:2302.00446. 2023.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context retrieval-augmented language models. arXiv preprint 2.Xiv:3302.0088: 2023.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 20389. Curran Asso- ciates, Inc., 2020. URL httpsssspppccccedigss..... newreids wwt.annws.....ppeeee///e/eeeeeee hctp://ccccc/pppr////e1eee eael5aabaa 7
fine-tuning.
ppff
Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models. arXiv preprint 1xXiv:2307.13692, 2023.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E Gonzalez, et al. High-throughput generative inference of large language models with a single gpu. arXiv preprint arXiv:2303.0668 2023.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 13:18991-,,,9, 2020.
Mingjic Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. arXiv preprint 2323:2306..1695, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
13

---

Published as a conference paper at ICLR 2024
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). ArXiv, abs/2206.10495 2022. URL https:: segpsssshaanttccchlaaa.... ott.s org/
CorpusTD:249B834
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:230S.11177 2023.
Dongkuan Xu, lan EH Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning-under the pre-train and fine-tune paradigm. arXiv preprint urXiv: 1rXiv22104.08882 2021.
Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all graph needs. arXiv preprint arXiv:2308.07/34 2023.
Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trained language models. arXiv preprint arXiv:2111.0575% 2021.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference Machine Learning, pp. 36809-26823, PMLR, 2022.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori Hashimoto. Benchmarking large language models for news summarization. ArXiv, eschol
abs/2301.13848, 2023.
URL https:: //api semant ieeaantcscholaar wrw..oor////p/....
256416014
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05683, 2023.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning n: m fine-grat -re-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2 102.00000 2021.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 2017.
Terry Yue Zhuo Large language models are state-of-the-ar evaluators of code generation. arXiv preprint arXiv:2304. 14317, 2023.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
RELATED WORKS
A.1.1
SPARSITY IN LARGE LANGUAGE MODELS
The advent of large-scale pre-trained models has led to the development of advanced post-training pruning methods, aiming to enhance the cost-efiectiveness of these expansive models (Sanh et al,
2020
Chen et al., 2020a Jaiswal et al. 2023b Zafrir et al.
2021
Kurtic et al., 2022 Xu et al.
2021
Lagunas et al. 2021 Zhang et al. 2022 Frantar et al.) 2021 Jaiswal et al. 2023a Ma et al.
2023
et al. 2003)) Among them, Frantar al. (2021) extend second-order pruning to the BERT-
level scale, enabling the pruning of blocks of weights and achieving state-of-the- results for sparse Frantar & Alistarh (2023) introduce SparseGPT for pruning large language models (LLMs) in single shot without requiring re- re-training fine-tuning. They leverage column- wise second-order pruning, and successfully remove 100B weights from OPT-175B without a significant increase in perplexity. More recently, Sunn et al. | (2023) propose straightforward pruning method that takes both weights and activations into account, demonstrating comparable performance to Frantar & Alistarh 2023). Li et al. (2022) reveal that activation sparsity prevalent phenomenon in Transformers (90% of intermediate output), yielding another opportunity for acceleration. Liu et al. (2023b) in- troduce large-scale SMC-Bench. indicating that state-of-theear magnitude- and/or gradient- based sparse algorithms fall short when applied out-of-the-bos to larger-scale models and selected of complex downstream tasks.
Frantar&
A.1.2 QUANTIZATION IN LARGE LANGUAGE MODELS
With the recent open-source releases of language models like BLOOM, Vicuna, LLaMa, OPT, etc., quantization has emerged as a widely embraced technique to alleviate the storage and computa- tional overhead of deep learning models. Recent research endeavors have harnessed quantization to compress LLMs and they can be classified into the two mentioned approaches: Quantization- Aware Training (QAT), and Post-Training Quantization (PTQ). In QAT, the quantization objective is embedded into the LLM training process, enabling them to adapt to low- eew-rcciiion representa- tions and handle precision loss caused by quantization. LLM-QAT (Liu al.) (023c) proposes a data-free distillation method that leverages generations produced by the pre-trained model, pre- serving the original output distribution and allows quantizing LLaMa models independent of its training data. PEQA (Kim et al., 2023) operates through dual-stage process: initially, the param- eter matrix of each fully-connected layer undergoes quantization into a matrix of low-bit integers and scalar vector; subsequently, fine-tuning occurs on the scalar vector for each downstream task. OLoAA (Dettmers al.] (202a)) proposes an efficient finetuning approach that reduces memory us- age enough to finetune 65B parameter model on single 48GB GPU while preserving full 16-bit finetuning task performance by backpropagating gradients through a frozen, 4-bit quantized pre- trained language model into Low Rank Adapters (LoRA). PTQ involves quantizing the parameters of LLMs after the completion of the LLM's 's training phase. GPTQ (Frantar et al., (002)) proposes novel layer-wise quantization technique based approximate second-order information resulting a bitwidth reduction to 3 or 4 bits per weight, with minimal accuracy loss compared to the uncom- pressed version. AWQ (Lin et al., 2023a) based the observation that weights not equally important: protecting only 1% of salient weights can greatly reduce quantization error, employs an activation-aware approach by considering the significance of weight channels corresponding to larger activation magnitudes. SpQR (Dettmers et al., 2023c) works by identifying and isolating out- lier weights, which cause particularly-large quantization errors, and storing them in higher precision, while compressing all other weights to 3-4 bits, and achieves relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs.
A.1.3 LARGE LANGUAGE MODELS AND EVALUATION
Large language models (LLMs) are gaining increasing popularity in both academia and industry playing vital role in both research and daily use. With increasing popularity, several works (Li et al.)
2023
Kaddour et al. 2023 Muhlgay et al., 2023 Zhang al. 22023] Valmeekam et al.) [022 Liu
ct al. 2023a (aaaada et al. 2023) Qin et al.] 2023] Zhuo 2023; Lee et al., (023) attempt to go
beyond conventional perplexity evaluate performance of LLMS across factuality, commonsense
15

---

Published as a conference paper at ICLR 2024
reasoning, language understanding, reading comprehension, programming, instruction following abilities, etc. Muhlgay et al. (2023) propose new metric FACTOR to understand factuality correct information in the LLM generated text. It found that although FACTOR accuracy and LMM per- plexity tend to be highly correlated but sometimes induce different orderings between LMMs. They reported that pairs of models share similar perplexity but differ significantly in terms of FAC- TOR accuracy. [eee et al. (2023) evaluate the performance and alignment of LLM distribution with humans using two different techniques: Monte Carlo Reconstruction (MCR) and Log Probability Reconstruction (LPR); and found LLMs exhibit limited ability in solving NLI tasks and simulta- neously fail to capture human disagreement distribution, Zhang et al. (202)) attempt investigate promise for automatic summarization with respect to human summary writers and found that LMM summaries are judged to be on par with human written summaries, Valmeekam et al. (2022) propose an extensible assessment framework to test the capabilities of L.MSs on reasoning about actions and change, a central aspect of human intelligence and found that GPT-3 and BLOOM have dismal per- formance these benchmarks. Despite these efforts investigate the performance of dense LLMs comprehensively, it surprising that no such efforts have been yet carried out for a more daunting of compressed LLMs, which derived from dense counterparts sharing significantly high similarity with them. Our work is first attempt to address this gap and encourage sparse commu- nity researchers to go beyond perplexity evaluate the true merits and drawbacks of compression methods.
A.2 PROMPT DESIGN AND EXAMPLES FOR DIFFERENT TASK SETTINGS IN LLM-KICK
A.2.1 FACTOID-BASED QA
Prompt Design: Please give answer to this question: <QUESTION> The answer is Example: Please give answer this question: The film '10 things T hate about you' is based on which Shakespeare play? The an- swer
Model Response: Please give answer to this question: The film '10 things hate about you' based on which Shakespeare play? The answer is the taming of the shrew.
A.2.2 MULTIPLE-CHOICE REASONING-BASED QA
Prompt Design: The following are multiple choice questions (with answers) about <SUBJECT NAME>.\n QOQUSSIIONN MAMIT//
(nA. <OPTION 1>\nB. <OPTION
2>\nC. <OPTION 3> IDD <OPTION 4> Answer:
Example: The following are multiple choice questions (with answers) about algebra.)p)n Find the degree for the given field extension (2), sqrt (3), sgrt (18)) over Q. |nA. OBnB. 4)nC. 2),66))
Answer:
The following are multiple choice questions (with answers) about algebra.)n), Find the degree for the given field extension Q (sqrt (2), sqrt (3), (18) (18)) over Q. |nA. OBn 4)nCC 2).66))
Model Response:
Answer:
A.2.3 IN-CONTENT RETRIEVAL AUGMENTED QUESTION ANSWERING
! Closed Book Setting: For closed- book setting, we adopted the prompt from Touvron et al. 2023 follows.
16

---

Published as a conference paper at ICLR 2024
Prompt Design: Answer these questions:),n)y InQ: <QUESTION>>> In
Example: Answer these questions:) nn: Who was the man behind The Chi pmunks??
Model Response: Answer these questions:) ni who was the man behind The Chipmunks?|o A: The man behind The Chipmunks was David Sarge, who was the founder of the Alphaville Virtual Real Estate Company.
Open Book Setting: For open-book setting, we extend the above prompt as follows.
Prompt Design:
<EVIDENCE>> n Answer these questions: nQ: <QUESTION>>
A:
Example: AAlvin and the Chipmunks (2007) IMDb TMDb 17 January 2017 4:34 PM, UTC NEWS. struggl ing songwrited tee
named Dave Seville finds success ... "|n Answer these questions: Q: Who was the man behind The Chipmunks?\
Model Response: "Alvin and the Chipmunks (2007) IMDb IMDb 17 January 2017 4:34 PM, UTC NEWS. A struggling songwriter named Dave Seville finds success ... ...""n Answer these questions:),n Q: Who was the man behind The Chipmunks?|n A: Dave Seville.
A.2.4 IN-CONTEXT TEXT SUMMARIZATION
Prompt Design: A chat between curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's ques- tions. USER: Summarize the given story in less than 150 words while preserving high coherence, consistency, fluency, and relevance. \nnn <STORY>. ASSISTANT:
Example: chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers the user's questions. USER: Summarize the given story in less than 150 words while preserving high coherence, consistency, fluency, and relevance. hellevance.//ppiiyyyaaa \n/nLibyan and U.S. officials say the two governments held face-to-face talks in Tunisia ...have denied previous reports of talks with the government. AS- SISTANT:
Model Response: shown in Figure g
The model response of one-shot magnitude pruned Vicuna-7B ASSISTANT is
ca-led cald sod thit vape caace 18aa CEsGUCT neeeting H01d Me novery EnreetS snWaart one has tradiing the
tnseato servee stnge the Iul. snewary B.C. orrtriat Iequess she react and passimeny Sikyas vamila Lana any wat balle..
Figure 8: Output response of 10% compressed (unstructured one-shot) Vicuna-7b ASSISTANT.
17

---

Published as a conference paper at ICLR 2024
A.2.5 MULTI-TURN CONVERSATION AND INSTRUCTION FOLLOWING
Prompt Design: A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <QUESTION> ASSISTANT:
Example: A chat between curious user and an artificial intelligence assistant. The as- sistant gives helpful, detailed, and polite answers to the user's ' questions. USER: How can I improve my time management skills ASSISTANT:
Model Response: shown in Figure
The model response of one-shot magnitude pruned Vicuna-7B ASSISTANT
your
liss priceitieg you
lopa make anuecyy This Help you 1llx the your thee effictidts oulld pateepttaaion assential that denls Lil thase help you saap track your dead telonivie,
100s c
tiee Teary
Teiing trme you vhoy reciad distre haner olten
stoplons t9k8 Liee
timer
loin..
Laneber,, Slee masgeerss ia still that takas proction, be poslant with solf and keep sor sorting it.
Figure 9: Output response of 10% compressed (unstructured one-shot) Vicuna-7b ASSISTANT.
A.3 IN-CONTEXT SUMMARIZATION EVALUATION SETTINGS
For evaluating the performance of LLMs to generate high-quality in-context summarization, we cus consistency, coherence, fluency, and relevance metrics. We prompt GPT-4 which has been recently identified to be highly effective as an automated evaluation framework for benchmark gen- eration and performance assessments, to evaluate these metrics in comparison to the summaries generated by GPT-3.5. Examples of our prompts used for evaluating with GPT-4 Judge are shown in Figure TO
We also provide an example of GPT-4 Judge output in Figure 11
INCONTEET SUMMARIZTTON EVALUATION PROMPT "You are helph preces weestett checkin the quilty the summartaries Vaceuanacy
withe w30
Twonpt,
Anclesant ate Suarary, turaasy 211ain and would lske to zequess your deecback on the pazforannce of bwo Al suxiscanta An cosponse to the user requeated suarsey above, taPlease rase the saberence, conalatenny, ftumnyy and releranco aurascy gezerated. Rach sasletant pecsivas reore tines
2 Scanary Syyeaae pcopptt sote", "dafaclta"s ("pooppt i <No
forr rsparsted by sgace, to the oibseguent Tine, please provi de compretera eepl inition of your exn oat avolding potential and eroering that the order in witch the reaporoes were presented does vor affect your judgeent.") "1
veluro indleacing
Figure 10: Example of prompt used to evaluate the compressed LLM ASSISTANT wrt. GPT-3.5
ASSISTANT using GPT-4 as Judge on consistency, coherence, fluency, and relevance of generated summaries.
Reesssass anoridad comprnsenttin manory oF REATT, Enetaating Berm DACATTE AREEE the HIT ESSENT Oye and sSceAt cuass, and santees, the aesacn
ve sigt. expticed the mall pelece and providnd cleer ONEETIN AcconcLo..
Aasissuns commar
saeryy, was shrr Misbotie 1e05 bay pesoee. JOMENE, porsdded c. good ovsction main pocsss t30 st227.
Elceat, Iac loakee the deaall waree dnad aboe
dEat siocy Alighlly J0e2 caas ARSCREMO:
Figure 11: GPT-4 Judge Evaluation of responses generated by GPT-3 (ASSISTANT wt.. 10% compressed (unstructured one-shot) Vicuna-7b (ASSISTANT2).
A.4 INSTRUCTION FOLLOWING ABILITY EVALUATION SETTING
For evaluating the responses generated by compressed LLMs, we closely follow the prompt design settings of MT-Bench (Zheng et al. 2023 using GPT-4 as judge. We prompt GPT-4 to rate the answers generated by compressed LLMs wrt. GPT-3.5 (text-davinci-003) model based on varying metrics (eg. correctness, helpfulness, logic, accuracy, etc.) scale of [0-10] and provides
18

---

Published as a conference paper at ICLR 2024
detailed explanation behind the score. Examples of our prompts used during evaluation for questions as well as GPT-4 Judge response are as shown in Figure 12 and respectively.
GENERAL QUESTION PROMPT to You are a andaki and procis asastant for crecking the quility of amawar. "soossinnpkx""
[Qusation) int quaasion hain The stars Asstacant Assoeee] inseoee 21 hain (tt and Aastatats Acavee) tala [the Stact Asacatant Anawer) la lananwez 2/ The End Auaz stant Answes Eyaan Inipsseps Iaint, "Gatesita"s I"pocapt" "We would Like to request your beucback on the pactocmant
aosiacanta in respocse
user queation diaplayed absos berleus axstatent
rate bedgouinens, relevatees accuracys
detallyy tactses intormations and length their cesponses, pezfocmince, sespectirity. any possntinl ccsluation,
stauzing
zuaperanca
your sudgpent.
CODING QUES NIW PRODUPT to You nalpi and precsee senstent for checking the qualiy of the srrnect "proept Jampkear" "O "[onsttoon (ugaattioon Chhh Start of Assistant 1t Aroverl In inaaver 1o the the Asscatart Iha anower loka thh start Asslatars 21a
anower Ans lanswer Po the tho of assistant 2 bnowey
tyst Ietppropp Eyeset, "dofasltofi ("pronpt to thour task to eswtuate the ondingg ieter of the abows two sorfecante they howe nee adsed to tep' meent a proyroe to colue 1 given problew steade sect theie code ostnliatios, paylig clese actertion their problomsnaeting appinaah, code atrecture, rendibilsty. and the Inciustio helpfel comnente Anlani ansuce that the sacietinta' cubeisstanni Natal, cornectl teplomant the given probles atabement And, Contain uccuzace atclotent oode funesionnal Liy tok,
couss conotse commenss taut explate code's Logie practices Antoereee you have caretully reviesed sunetsstinn, any sogpestion tons reprommennt onderio dewney 10 perforts
to proper codinng standards
provtiee
feelback single thee contatiting
strongtha Linw.")
ourgut
respertively.
MATHS QUESTION PROMPT to You are heptal and presse asobtart for chooking the quality the answer". poong (Qunation)] tuuquaacion hnln (Te stave of Ansistant L' Anrwer 'o tooaee teeehhe and of saci etann Anrnnse incanve 21 21500n/ The and of Auclecant 2he kceve daaaee inlprenptjaaaart, *dadaubsa"" ("peoepet) f *th would Like to cecusat your taedback on the nathesstiosi prcfielancy of tao at sastacanta zegueding the qives uaez question. displayed above, lalieat, pleass aclie the prosias indapendenti, vitaout radeccing to the ansvarz provided by Asssatans and Asziacant 7, P. dd, glease exontre the probt oesanlatng grecnss Asslotann and bostatant stopety-rcop identitying say soccirect stepe if grosett, Your cvatiation should take tato account not only the arever but s160 the probler- roloing stope. tapeellli pleate astput lython tople contaising two noseriest soores for knsletare and aessatant 3, canging from 3 to 16, cospectivaly. if applisabis, axplain the ceasons for any vartaciona in thate scoces and deteraine which sactetant performed bettaz.")
Jnewne tointho Seart of Acriecant 2
ensure thesr connectetts
Figure 12: Examples of prompts used for different categories to evaluate the compressed LLM ASSISTANT wrt. GPT-3.5 ASSISTANT using GPT-4 as a Judge.
both armistante geovided coppeteenive and socurate resporans the question, They both identitled key challenges so sccoso to quality educstion, quality, the of
importance eprlty inclusion.
Assiatant i's
vie aligtnly
and organi providing ieportance addreszing
enuneration
explasation
touched
iemans education, which crcetsl arpect moders edxcation. conclusion about the
need for inn anony utshoboldecs van thouptciul oscition.
Assiatant also provided good zespasse, but alightly Leso detailee and arganized compared Aexistant 1. The poizta vere relevazt and accurate lut the axplanatioee vere not as comprehenaive. The ausiutars did, hovever, bring up the chillange of stodens engageneti,, which a nigniticant issoe today.
globalization
Orenat1, both ansistante did a good Job, Ansistant I'x zerpozze vae elightly more decasled and millaatrortured.
Figure 13: audde evaluation of responses generated by GPT-3 (ASSISTANT) 1) wrz. 10% compressed (unstructured one-shot) Vicuna-7b (A
A.5 USEFUL LINKS FOR LLM-KICK
Table 1: Dataset and code link used in our work.
Method / Dataset
Downkod URL
Freebas:QA Diang ot [209
bttea: REteas/hhuoline htt.///ooonnngneoooo face. coo aattt ht /resbaae
MMLU Benemmark Hendrycks al. 2020
Sttee: / hwgcing Lace co/ Jatasets Freckase
TriviQAA Joshi et
MT-Bench
RF hungi ng ane
axse Hogging aogeeag bench prompts
CNN/Daily Mail Summanzzzee
Nallagati et 2016
St 25 edu7 " scho//u MWW
WildTee MMeily al. 2016
Wanda Soi pr 31251
Hip3:/// h.....th....... Anny Torik Tab7 vanda
SparseGPT [Frantrr & Xlistar 2025
5t St /tttt/tttttt... con/ 118 SAStab/ spat
M.I.MJuddee (Zhas
GPTQ Frantar al
https::///tbbbb... con/ qwtpq//ppppp..... woopgwopzuu/200 CPCP 202-
L1253
A.6 COMPARSION WITH AWQ AND LLM-INT8
In this section, we considered evaluating AWQ (Lin et al. 2023b and LLM.int80 (Dettmers et al.
2022) across our different task settings and we summarize results on Vicuma-7B as in the follow- ing table. We observe that LLM.int8( despite its simplicity and case-of-use, achieves better results than AWQ (8-bit), and GPTQ (8-bit) across all listed tasks.
19

---

Published as a conference paper at ICLR 2024
Task
GPTQ
AWQ
LLM-int80
Factoid-QA
60.14%
60.31%
61.02%
MCR-QA (MMLU)
47.10%
47.18%
47.82%
Retrieval Augmented QA
75.55%
75.89%
75.91%
Instruction Following (GPT4-Score)
9.74
9.72
9.81
Table 2: Performance comparison of AWQ and LLM-in(80 on LLM-KICK.
A.7 UNDERSTANDING THE IMPACT OF K-SHOT FOR COMPRESSED LLMs
In this section, we aim to investigate how few-shot in-context
learning examples benefit SoTA pruning methods to pre- serve performance across various sparsity levels. Figure [4 illustrates the performance comparison of Vicuna-7B vary-
Average Accuracy
40
130
ing sparsity ratios when augmented with k-shot in-context ex- amples on MMLU benchmark. It is interesting observe that k-shot in-context learning examples have marginal im- pact on dense network performance, while they significantly help in preserving the performance at high sparsity. More- over, we found 2-3 examples are sufficient to retain perfor- mance, and supplementing additional examples doesn't 't neces- sarily provide further noticeable benefits.
20
D-shot
1-shot
20
3-ship
10
20
30
40
50
60
70
Sparsity
Figure 14: Vicuna-7B pruned with Wanda.
k-shot results of
A.8 SUMMARY OF VARIOUS PRUNING METHODS ON LLM-KICK
Task
Pruning Method
0%
10%
20%
30%
40%
50%
Factoid-QA
Magnitude
65.44
61.74
66.53
60.84
42.06
13.99
SparseGPT
65.44
63.84
62.44
58.54
55.54
42.86
Wanda
65.44
63.34
65.23
61.24
58.24
44.66
MCR-QA (MMLU)
Magnitude
0.471
0.466
0.455
0.422
0.339
0.050
SparseGPT
0.471
0.470
0,460
0.437
0.395
0.308
Wanda
0.471
0.469
0.460
0.455
0.425
0.386
In-context Retrieval (Long Story: Coherence)
Magnitude
5.883
6.112
5,855
5.567
4.329
1.233
SparseGPT
5,883
6.033
5.533
6.067
5.567
5.067
Wanda
5.883
6.0
5.783
5.933
5.267
5.033
Instruction Following (GPT-4 Score)
Magnitude
7.763
7.567
7.621
7.201
6.208
3.308
SparseGPT
7.763
7.645
7.50
7.188
6.905
6.206
Wanda
7.763
7.731
7.546
7.202
7.071
6.838
Table 3: Performance comparison of various pruning methods on Vicuna-7B with LLM-KICK
A.9 ADDITIONAL RESULTS ON LLAMA-2
LLamuu 2-78
LL7762277
LLare
A Diw (ooo Moue
Pettar TONTOO Drep Dasd 14
Nog Faxde SpanwiP7
Moggiauds Spars-GP7 Veacda
-10
Putl
74M Spuriiy
NUN Sparaiy
Figure 15: Compressed LLMS for Factoid-based QA. Performance comparison of compressed
LLMs (LLaMa & on Factoid-QA task using FreebaseQA (Ting et 2019). Results presented are for structured (N:M sparsity) and unstructured sparsity.
20

---

Published as a conference paper at ICLR 2024
(Dusiness,
LiaMa 78 (NMM Sperstty) 55 PPPtformano (A in Performeroo (A00
Vagelute Spara/2211 Aurr
Spanee
Wark
Wurds
Liaia 2-78
-10
-0
-20
-00
-39
No p-
-00
-00
2-4
4B
100
NM Soarsity
NMM Sparsity
NM Sparsit
NM Sparsity
Figure 16: Compressed LLMs for Multiple-Choice Reasoning based QA. Performance compari- son of compressed LLaMa-2 7B on MCR-QA tasks using the MMLU benchmark (Hendrycks al.) 2020 Results presented are for structured (N:M sparsity) and unstructured sparsity.
Obood Book LLann
Closed Book LLame 2.28
Open Book LLano 2.78
Open Book LLare 2-18
Miggtrude Spersetirr
Veeri
Mogetade
abdi SporSccPP
Wance
Wards.
2M0 Wards
Sparsiy Roto
NSM Sporsty
Spassyy Roto
NcM Sporsity
Figure 17: Compressed LLMs for In-context Retrieval Augmented QA. Performance compari- son of compressed ILaMa-2 7B on ICRA-QA task. We present head-to-head comparison of closed- book evaluation (no external knowledge is augmented in-context) with open-book evaluation (ex- ternal knowledge is augmented in-context). Results presented are for structured N:M sparsity and unstructured sparsity.
21


Reviews:
[{"review_id": "88m33eXZuL", "reviewer": "Reviewer_QryF", "summary": "The paper is very timely and identifies an important gap in evaluation of compression on LLMs. It points out how perplexity is not a correct metric to evaluate compression benchmarks (which is also previously observed in other contexts). They curate a set of datasets which can form a better representation of language model capabilities.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- timely ... with an array of papers on compressing LLMs with especially surprising results such as training free pruning coming out. It is important to enable researchers with better tools of evaluation\n- provides a decent array of dataset benchmarks that will be use ful in research.\n- clearly shows the gap between evaluation of perplexity and other proposed datasets.", "weaknesses": "Not weaknesses. but suggestions. \n1. add a summarizing table to list dataset statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Ad2wXRhPV", "reviewer": "Reviewer_PUep", "summary": "This paper benchmarks a few LLM compression methods based on quantization and pruning, on different datasets.", "questions": "The authors say that SparseGPT is data-free. Is that true?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think it is important to have a more fine-grained understanding of compression methods, specially to design new algorithms that can improve upon current weaknesses.", "weaknesses": "- This paper is essentially benchmarking a few algorithms on a few datasets. Although the insights are interesting, the paper does not include any new model, data or algorithm, which I'd say makes this paper more suitable for a workshop, not a full conference paper.\n\n- Some arguments are rather subjective. Why choose the 5% threshold? If we change the threshold to 10% it seems 4-bit quantization is then in the range in most cases, and sparse models can still be \"competitive\" for around 50% sparsity.\n\n- The loss of accuracy also has to be contextualized with the inference time speedups. If a 5% loss of accuracy leads to a 10% reduction in inference time, I'd call that successful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CH77B1FDYi", "reviewer": "Reviewer_b73d", "summary": "This paper revisits the efficiency of some compression (pruning and quantization) techniques for LLMs. It conveys that more than perplexity is needed for performance comparison among compressed and uncompressed models. It displays several tasks and benchmarks over which the compressed models exhibit performance degradation despite having similar perplexities. \n\nEssentially, the proposed benchmarks show that quantization if mostly more efficient than pruning where structured pruning appears to offer least ML performance. Interestingly, the paper also shows that even 8-bit quantization if not on par with the uncompressed baseline. \n\nImportantly, the paper states that their related codes are planed to be open-sourced", "questions": "See weakness 1 and 2. Also, does the authors have insights regarding why, in the paper's evaluation, quantization works better than pruning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compression of LLMs is very timely and important.\n\n2. The paper reveals new and yet widely unknown gaps in compressed LLMs in comparison to their uncompressed counterparts. \n\n3. The paper shows that compressed models may offer better performance in some tasks (e.g., In-Context Text Summarization) than others (e.g., Factoid-based Question Answering)\n\n4. The authors plan to release their code which may be help in the development of future compression techniques.", "weaknesses": "1. It would make the conclusions more robust and convincing if the evaluations use more than a single family of LLMs (i.e., Vicuna).  Why not repeat these experiments with, e.g., Llama 2 and Falcon?\n\n2. Regarding the observation that even 8-bit quantization has evident gaps with respect to uncompressed models, have the authors considered evaluating LLM.int8()?  (https://arxiv.org/pdf/2208.07339.pdf) \n\n3. It would help the reader to have a table summarizing all the tasks' performance over the different architectures, compression techniques and the their resulting perplexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ot5bsHf75X", "reviewer": "Reviewer_X7p4", "summary": "The paper benchmarks compressing techniques, pruning and quantization, on various datasets and metrics, highlighting that a common evaluation metric, perplexity, does not always translate into real-world values.", "questions": "Why do we see these performance degradations? Any way to explain why some methods work and some do not? \nAre there are any counter forces one can use to mitigate performance degradation while still preserving the quality?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written, all the plots are clear and comprehensive. It's easy to understand information in most of the cases (see weaknesses for improvement). \nAuthors correctly pinpoint that most of the compression algorithms do not validate extensively on real-world scenarios and show some interesting insights. For example, there is evidence that pruning may not work as well as quantization methods. Or that 8-bit quantization works well on majority on datasets. These insights are easy to use and can have a big impact to how practitioners use these models.", "weaknesses": "A few ways to improve this paper.\n1. More LLMs. Right now, it's only vicuna model (7B and 13B), having other architectures and perhaps bigger sizes (e.g. llama 70b) would add more evidence for the insights. \n2. Add prompt designs to the main body of the paper. This will make it easy to understand how each task is distinct. \n3. More quantization methods. AWQ/SmoothQuant are interesting to see.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)