Title: SIRIUS : Contexual Sparisty with Correction for Efficient LLMs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Contextual Sparsity

LLM inference

Knowledge Distillation
Abstract: With the blossom of large language models (LLM), inference efficiency becomes increasingly important. Various approximate methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without significant performance degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, it significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models and original models often share the general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces SIRIUS, an efficient correction mechanism, which significantly boosts CS models on reasoning tasks while maintaining its efficiency gain. SIRIUS is evaluated on 6 models with 8 difficult generation tasks in reasoning, deduction, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for SIRIUS and show that SIRIUS delivers theoretical latency reduction with roughly a 20% reduction in latency for 8B model on-chip and a 35% reduction in latency for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git.

SIRIUS: Contextual Sparsity with Correction for Efficient LLMs

Abstract

With the blossom of large language models (LLM), inference efficiency becomes increasingly important. Various approximate methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without significant performance degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, it significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models and original models often share the general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces SIRIUS 1 , an efficient correction mechanism, which significantly boosts CS models on reasoning tasks while maintaining its efficiency gain. SIRIUS is evaluated on 6 models with 8 difficult generation tasks in reasoning, deduction, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for SIRIUS and show that SIRIUS delivers theoretical latency reduction with roughly 20% reduction in latency for 8B model on-chip and 35% reduction in latency for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git .

1 Introduction

Large Language Models (LLM), such as OpenAI et al. (2024) (GPT-4), Team et al. (2024) (Gemini), and Touvron et al. (2023) (Llama) have demonstrated their proficiency in a wide range of natural language processing applications such as content creation, summarization, and impressive and complex reasoning tasks. However, their deployment is very challenging, especially in latency-sensitive settings (Kaplan et al., 2020). Exploiting the model sparsity is a natural way to reduce the model parameter size and computational cost with a long history (LeCun et al., 1989; Tibshirani, 1996). More recently, many studies have shown that contextual sparsity (Liu et al., 2023b; Li et al., 2022; Dong et al., 2024; Lee et al., 2024), which highly correlates to the prompt or the context, can greatly speed up LLM inference without quality degradation.

However, in this paper, we first demonstrate a critical and fundamental problem with contextual sparsity (CS): while generally robust in classification tasks and generation tasks that mainly rely on prompt understanding (e.g., summarization, chat question-answering), we found that CS models struggle at high-level reasoning and understanding tasks.

(a) Comparing Sparsity Method

on CNN/DailyMail and GSM8K

(b) Contextual Sparsity applying

to Llama-3-70B-Instruct

(c) Correcting only 11% of tokens

brings Csparse to full performance

Llama-3-8B-InstructwithCoarse-GrainedSparsity

Llama-3-70B-InstructonGSM8K-COT

CorrectingCSparseLlama-3-8B-Instruct

0.7

0.12

0.9

0.92

0.75

CNN/DailyMailUnitxtRouge

0.6

0.10

GSM8K-COTAccuracy

0.8

0.70

GSM8KAccuracy

0.5

0.08

0.7

0.75

(Strict)

0.65

0.60

0.4

0.06

0.6

0.55

0.3

0.04

0.5

Coarse

0.50

0.2

Contextual Sparsity

0.4

Fine

Performance

0.1

Contextual Sparsity

CNN/DailyMail

0.02

0.00

0.3

Full

L-3-8B-1

0.45

0.40

Full ModelScore

withCorrection

0.5

0.6

0.7

0.8

0.9

8

20

30

40

50

60

70

0.115

0.4

0.6

0.8

1.0

Density Ratio

EffectiveParameter(B)

PercentageofCorrectedTokens

For example, in Figure 1 (a), we contrast between the Text Summarization task (CNN/DailyMail) and Arithmetic Reasoning (GSM8K) with contextual sparsity methods on Llama-3-8B-Instruct. Varying sparsity levels, Llama-3-8B-Instruct with contextual sparsity performs consistently worse on GSM8K than CNN/DailyMail. With roughly 50% sparsity globally, the sparse model degradation is still reasonable on text summarization (right axis in color coral) compared to almost collapsed on arithmetic reasoning (left axis in color green). However, for reasoning tasks, can we simply live with a higher density ratio to preserve more performance? Unfortunately, the answer is NO. Besides the significant efficiency loss, shown in Figure 1 (b), the Llama-3-70B-Instruct model with contextual sparsity also crashes at around 50% sparsity globally. The 50% sparse model still has 4 × the parameter size compared to the smaller dense model (Llama-3-8B-Instruct), while still performing worse on GSM8K-CoT, rendering contextual sparsity utterly not useful for complex reasoning tasks.

We conduct an in-depth study on the CS model failure cases and notice that the overall reasoning pathway of these sparse models is usually sound and adheres to the full model. The fatal mistakes are always caused by some middle tokens and propagate towards the end, examples can be seen in Figure 4. Following this observation, we conduct a simple experiment with 65% Contextual Sparse Llama-3-8B-Instruct on GSM8K as presented in Figure 1 (c). We run both the sparse and the full models together for the same prompt and compare two generation output token-by-token.

Surprisingly, the trend increases steeply with the percentage of corrected tokens. Correction of only 6%tokens in the sparse model's generation recovers most of GSM8K accuracy ( < 5%to full), and 11% to recover the full performance. The results show potential for an efficient and powerful correction mechanism to maintain the sparse efficiency while boosting its performance. Contextual sparsity uses a dynamic sparsity pattern and naturally requires the full model to be in GPU memory during runtime, allowing the full model to be used efficiently for infrequent correction. Even though only very few need to be corrected, locating these mistaken tokens efficiently turns out to be challenging.

Ideally, we want a correction system to have the following properties: 1) Effective , the sparse model quality degradation can be improved to the full model vicinity; 2) Cheap , the full model only gives minimal intervention; 3) Adaptive , the system is efficient across various reasoning datasets.

In this paper, we carefully analyze and formulate correction efficiency in Section 2. We extensively categorize the strengths and weaknesses of CS in Section 3. In Section 4, We systematically design SIRIUS, a correction method covering all three desired properties. ( When? ) In Section 4.1, we show that the sparse model can be both confident or uncertain when making mistakes, rendering the signal from sparse unreliable for determining when to correct. SIRIUS is a period-based method with the period as a hyperparameter. ( How? ) In Sections 4.1 and B.2, we introduce novel KV Cache direct rewriting, minimal rollbacks, and hardware-efficient tree building to help increase the effective period of full model correction, thus, ensuring the correction efficiency.

Wrong KV Cache

(Call infrequently, doing correction)

(Sparse Pattern Is Contextually Aware)

+

New Input

Intermediate mistakes eventually

leads to wrong end-result

Intermediate mistake

Wrong Result

Rewriting the KV Cache

Correcting tokens Full Model deemed Unlikely

+

Interleaving with high-quality token

Infrequently, we use the in-memory

Full Model to check the past chunk

Correct Result

With correction,

Sirius gets it correct

In Section 5, we empirically evaluated SIRIUS on 6 different models with 8 different reasoning tasks and showed that SIRIUS is generally effective and efficient. On GSM8K and Llama-3-8B-Instruct specifically, we boost the fine-grained sparsity from 58% to 72% with 4% increase in effective parameter size and coarse-grained sparsity from 38% to 70% with the cost of 5% effective parameter size. We also show that SIRIUS delivers the promised efficiency on mainstream GPUs in both on-chip and offloading settings.

2 Related Works and Problem Formulation

In this section, we first present the classification of the prior Contextual Sparsity methods and narrate important efficiency metrics in. Also, we present careful analysis and quantitative comparisons on why Speculative Decoding is inefficient in recovering contextual sparsity. Due to space constraints, we refer to Appendix A.2. For extended related works on model compression, contextual sparsity, and speculative decoding, we present in Appendix A.1.

Contextual Sparsity Classification - Contextual sparsity (CS) methods are usually training-free, easy to use, and seemingly effective, making them highly attractive to ML practitioners looking to reduce LLMinference costs. CS exists naturally in MLP layers of the LLM, which occupies roughly 70% of the LLM total weights (Dong et al., 2024; Lee et al., 2024)). The contextual sparsity selection is as follows: given the context, only a limited number of the most relevant neurons are selected based on the input activation. The rest contributed to the output far less is discarded. We refer to two main directions of contextual sparsity methods as Coarse-grained Sparsity (CSparse) Methods (Dong et al. (2024)) - that within the same input prompt, the sparsity pattern is fixed for all tokens generated. Fine-grained Sparsity (FSparse) Methods (Lee et al. (2024)) - that exploits the per-token sparsity to save resources.

Average Parameters Used Per Token - A key metric is used to evaluate the efficiency of our proposed method, the Average Parameter Used per token decoded (later referred to as APU). LLM inference is memory I/O bound (Leviathan et al., 2023; Kim et al., 2023). The latency of generating every single token is dominated by the memory loading time from the GPU HBM to SRAM. On the other hand, SIRIUS relies on full model parallel verifying a chunk of tokens. Although from the FLOPs standpoint, the amount of compute performed per evaluation step is the number of input token times of a single token input process, the latency of parallel verification is still roughly the same as taking a single token (Verified further in 10, length 64 is only 1.1 ms longer than length 1), because the inference is memory bound.

SIRIUS operates in the memory-bound regime (single inference sequence length smaller than or equal to 64). Thus, the average parameter count of a model gives us a rough judgment of the latency of inference. Formally, for a full LLM to have C full number of parameters, and its sparse counterpart of a certain predetermined sparsity C sparse . The average advancement length (later we refer to as AAL) in the number of tokens between two consecutive LLM corrections can be represented as n AAL .

The average parameters used per token (APU) are the following

Wewant the metric to be as small as possible, and obviously, we want n AAL to be as large as possible.

Another thing to note is that we always compare the system's APU against the full model's APU, which is C full . If we divided the above equation by C full , we can have an equivalent parameter density of the system defined based on I globalsparsity , which is C sparse /C full .

Later, if we use period n period , the equation can be rewritten as

Later when presenting SIRIUS, we mainly specify n period with n AAL to evaluate its efficiency. Notice that I globalsparsity is determined by the sparsity method, SIRIUS cannot change it anymore.

3 Observations

In this section, we present a detailed study of the strengths and weaknesses of Contextual Sparsity (CS). 3.1 presents the strengths of CS. 3.2 presents the weaknesses of CS. Additionally, we show that given the similar parameter size, the more well-trained the model is, the more CS degradation will be for the model. Due to limited space, we present the details in Appendix B.1. 3.3 shows our findings when looking into the failure cases of the CS model in complex reasoning generation tasks.

In the following series of experiments, we build our implementation 2 of fine-grained sparsity based on Lee et al. (2024) and coarse-grained sparsity based on Dong et al. (2024). The default sparsity for both methods is 50% for the MLP component of the model (whole MLP for coarse-grained sparsity and Up and Down linear layers only for fine-grained sparsity). We mainly use this default setting in most experiment tables in the paper without explicitly mentioning it. Otherwise, we will explicitly specify the different sparsity levels we used.

3.1 Contextual Sparsity: Where Does It Succeed?

For tasks on prompt understanding, CS generally performs well and gives consistent and strong output. We evaluate CS models on machine summarization (CNNDailyMail, See et al. (2017)), and Conversational Question Answering (CoQA, Reddy et al. (2019)). The results show that the correctly selected contextual sparsity in the MLP layers and the full attention layers can fully extract and understand the local prompt information. More details are presented in Figure 3, where we show that by varying the sparsity level, the language model's performance on CNN/DailyMail is robust even when the activation sparsity drops to below 20%, which translates to around 44% global density.

For tasks accessing factuality and hallucination, we select the generation portion of the TruthfulQA dataset (Lin et al., 2022). Results are shown in Table 1, where we evaluate the techniques on 5 different LLMs. Interestingly, we find that the Fine-grained sparsity is often better than the dense model baseline across different models. This finding is consistent with previous works Laser (Sharma et al., 2023) and Dola (Chuang et al., 2024). They both observed that compressing the original LLM in a carefully designed way would lead to improvement in factuality and better de-hallucination. Laser comes from the low-rank approximation of the MLP layers, while Dola proposes a factuality-aware layer-skipping algorithm. Based on their findings, hallucination occurs when parts of the weights aren't as well-versed in the given input as the other parts. They expose the 'averaging' effect that blurs the factuality of the output. Removing these neurons gives rise to better facutality and less hallucination. Our studies look at the same problem from a neuron sparsity standpoint.

MMLU is a classification task, not generation tasks. We use MMLU-FLAN-COT

SparsityonCNN/DailyMail

Sparsity on GSM8K(30%)

SparsityonHumanEval

0.12

0.7

Fine-Grained

Sparsity

0.5

Fine-Grained

Sparsity

Coarse-Grained

Coarse-Grained

0.10

0.6

Sparsity

Sparsity

Score

Full

0.4

Full

0.5

Rouge

0.08

0.4

0.3

0.06

Unitxt

0.3

0.2

0.04

Fine-GrainedSparsity

0.2

Pass@1

0.02

Coarse-GrainedSparsity

0.1

0.1

===

Full

0.00

0.5

0.6

0.7

0.8

0.9

0.0

0.5

0.6

0.7

0.8

0.9

0.0

0.5

0.6

0.7

0.8

0.9

Non-ZeroParameterRatio

Non-ZeroParameterRatio

Non-ZeroParameterRatio

3.2 Contextual Sparsity: Where Does It Fail?

Onthe other hand, contextual sparsity severely struggles when the generation tasks rely solely on the model's own reasoning and world knowledge understanding ability. Here we show the Llama-3-8BInstruct and the Llama-2-7B-Chat models in Table 1, refer to Table 12 for evaluations on more models. Notice that since fine-grained sparsity method needs the activation from Gate MLP for selecting sparsity, while coarse-grained sparsity has a predetermined pattern after prefilling and can sparsify the Gate MLP. Even though both are at 50% activation sparsity, the coarse-grained sparsity method effectively achieves higher parameter savings than fine-grained sparsity in practice. Here we evaluate the sparse techniques using 5-shot CoT on the GSM8K dataset (Cobbe et al., 2021). We found that across all the models we evaluated, both sparsity methods lead to significant accuracy degradation. We include HumanEval (Chen et al., 2021), a coding task that requires complex reasoning and planning ability. Wefound that both sparsity methods exhibit similar performance degradation when it comes to coding. Shown in Figure 3, two tasks see sparsity significantly drop performance after 50% activation sparsity.

For knowledge recall and world knowledge understanding, we specifically test on MMLU-Flan-CoT (Chung et al., 2022) the CoT text generation version of the MMLU dataset (Hendrycks et al., 2021). Table 1 shows the results. Stronger models like Llama-3-8B-Instruct suffer from significant

degradation too. Furthermore, we found that given the similar parameter size, the more well-trained the models are, the higher its degradation from the contextual sparsity, more details in Appendix B.1.

Question:

Marie ordered one chicken meal that costs $12, 5

packs of milk that costs $3 each, 4 apples that cost $1.50

each, and some boxes of pizza. Marie paid a total of $50.

How many boxes of pizza did Marie order if each box costs

$8.50?

Answer:

The total cost of the chicken meal, milk, and

apples is 12 + 5 * 3 + 4 * 1.5 = <<12+5*3+4*1.5=

27>>27

dollars

(33 dollars)

……

Miscalculation

Question

: A merchant wants to make a choice of

purchase between 2 purchase plans: jewelry worth $5,000

or electronic gadgets worth $8,000. His financial advisor

speculates that the jewelry market will go up 2.5% while

the electronic gadgets market will rise 1.2% within the

same month. If the merchant is looking to maximize profit

at the end of this month by making a choice, how much

profit would this be?

Miscalculation

Answer

: The jewelry will increase in value by 2.5% which

is 5,000 * 0.025 =

$1250

($125)

……

Question:

Janet's ducks lay 16 eggs per day. She eats three

for breakfast every morning and bakes muffins for her

friends every day with four. She sells the remainder at the

farmers' market daily for $2 per fresh duck egg. How much

in dollars does she make every day at the farmers' market?

Answer (Sparse):

Janet eats 3 + 4 = <<3+4=7>>7 eggs per day.

She lays 16 - 7 = <<16-7=9>>9 eggs per day.

She sells 9 - 7 = <<9-7=2>>2 eggs per day.

She makes 2 * 2 = <<2*2=4>>4 dollars per day.

#### 4

Different Reasoning Path

Answer (Full):

Janet eats 3 eggs and bakes 4 eggs, so she has 16 - 3 - 4 =

<<16-3-4=9>>9 eggs left.

She sells 9 eggs for $2 each, so she makes 9 * 2 =

<<9*2=18>>18 dollars.

#### 18

Question:

Raymond and Samantha are cousins. Raymond

was born 6 years before Samantha. Raymond had a son at

the age of 23. If Samantha is now 31, how many years ago

Answer:

Samantha is 31,

so she is 31 - 6 = 25 years old

.

Raymond is 25 - 23 = 2 years older than Samantha, so he is

25 years old ......

Unreasonable Statement

Question

: A candle melts by 2 centimeters every hour

that it burns. How many centimeters shorter will a candle

be after burning from 1:00 PM to 5:00 PM?

Unreasonable Statement

Answer

: The candle will be 4 centimeters shorter after

5:00 PM because it will be 4 hours x 2 centimeters =

<<4*2=8>>8 centimeters shorter.

#### 4

Remarks:

There is conflicting statement in reasoning,

leading to the wrong end result.

3.3 ACloser Look on GSM8K Quality Degradation

To study the inability of the sparse model in deduction, we conduct a case study on the sequence-level coarse-grained sparsity methods Dong et al. (2024) with the Llama-3-8B-Instruct model. We visually inspect extensive cases where the sparse model and dense differ in answers. Generally, the sparse model always produces highly similar answers to the dense model: the similar approach or logic flow when approaching the same problem and even the same number of sentences before the first mistake occurs or in success cases. However, the key differences are usually caused by the following three categories of small token-level mistakes: (1) frequent miscalculation in the intermediate steps, (2) wrong reasoning in intermediate steps, and (3) insensible and random statements. For each of the above-summarized cases, we find failure question-answer pairs provided in Figure 4. These mistakes happen in the middle of arguments and propagate to the wrong end result.

Similar observations can also be found for fine-grained sparse methods with different model types. Interestingly, we find that even with these mistakes, the sparse model can still fully generate coherent tokens and make further reasoning assuming their prior steps are correct.

We hypothesize that the gap between the full model and these sparse counterparts is at these key tokens. The following simple experiment is conducted to further verify our hypothesis. We run the coarse-grained sparse model and the full model with the same input prompt and for every token the sparse model generates, the full model is used to check the likelihood of these decoded tokens, mainly removing tokens with low likelihood. By varying the likelihood threshold, we can control the frequency of the correction. The experiments are conducted for both Llama-3-8B-Instruct and Llama-2-7B-Chat Touvron et al. (2023) models with coarse-grained sparsity. The results are shown in Figure 1(c). In both cases, we found that a very small amount of correction would drastically improve the sparse model performance, showing a steep gradient when the percentage of corrected tokens is small. With merely 10% of tokens needing to be corrected, the sparse model can completely match the full model's performance. The experiment verifies our hypothesis that by correcting the small portion of key tokens, the sparse model can meet the large model's performance.

4 Methods

Though we find a minor portion of tokens needed to be corrected for the contextual sparsity model to fully recover performance, the challenge remains: how to locate these mistaken tokens with the minimal number of parallel verification rounds of the full model? In this section, we show that the sparse model provides signals that cannot be trusted 4.1. Then, we describe in detail the various correction techniques in 4.1. Because of the space limit, we put how to boost the sparse generation with hardware-efficient tree building B.2.

(b)

(c)

(a)

Entropy of the Generated Text

Llama-3-8B-Instruct CSparse+Sirius

Llama-3-8B-InstructFSparse+Sirius

2.00

0.7

1.75

0.7

0.6

1.50

GSM8KAccuracy

0.5

Performance

0.6

1.25

0.4

1.00

0.3

0.5

0.75

0.2

0.50

GriffinSparse

0.4

CATS (Top-k)

0.1

GriffinPlusSirius

CATSPlusSirius

0.25

GriffinPlus Sirius

CATSPlus Sirlus

0.00

AC

customer

custome

0.0

0.5

AverageParameterUsed

0.6

0.7

0.8

0.3

ParametersUsedper token generated

0.75

0.80

0.85

0.90

4.1 Sparse Model's Self-Awareness Cannot Be Trusted

Intuitively, rather than fixing the n sparse number, letting the system decide when to call the LLM for evaluation would then give more flexible n sparse . Nevertheless, we argue that the sparse model's output probability distribution cannot be used as a metric for accuracy decisions. We empirically experiment with various methods to utilize the information contained in the sparse model's output distribution. However, varying the threshold leads to n sparse being too short when the threshold is strict or failing to correct when the threshold is lenient. We then discovered that the sparse model has very limited self-awareness of its own mistakes. To make the observation concrete, we present a small example in Figure 6 a piece of text where the sparse model makes a mistake while the full model succeeds. The red bars signify the error location. The token entropy is neither high nor at zero, making it impossible to effectively use a threshold to control the number n sparse .

4.2 Howto Correct the Sparse Output Tokens

The full overview of SIRIUS is presented in Figure 2 and Algorithm 1. The full model is called once every kernel size. The KV cache is shared between the sparse and the full model. The KV cache is mostly populated by the sparse model, which is called for every token. During correction, the full model takes in the last kernel size of tokens and generates its KVs for the past kernel size tokens in parallel, these KVs are directly written to their corresponding positions in the shared KV Cache. Empirically, we found that full model's KV helps the sparse model's output. When LLM is called to evaluate the sparse model's output, it uses its own predicted likelihood to determine whether to accept or reject the sparse model's past output. The decision is based on comparing the likelihood against a preset threshold. Detailed ablation for threshold is in B.3. Besides the above-mentioned techniques, we also found that the "second/third" choices of the sparse models' rejected token position offer > 80% coverage of the LLM accepted tokens. The observation motivates us to build a hardware-friendly tree on the sparse model generating side that doesn't sacrifice the performance while significantly boosting the n AAL or efficiency. Due to the space limit, a great amount of details is in Appendix B.2.

5 Experiments

In this section, we empirically evaluate SIRIUS to correct CS models on various generation tasks in complex reasoning. We show that SIRIUS is consistent in various tasks, effective in helping CS models recover their performance, and efficient in correction with low additional overhead.

In 5.1, we evaluated SIRIUS on six models with 8 different datasets. SIRIUS is consistently effective and efficient. Specifically, on GSM8K, SIRIUS corrects FSparse Llama-3-8B-Instruct from 58% accuracy to 72% with only 4% increase in parameter density and corrects CSparse model from 38% to 70% with 5% density.

In 5.2, we presents more details on our system implementation for SIRIUS. We show that SIRIUS delivers its theoretical efficient promise, achieving roughly 20% reduction in latency compared to full on-chip on various hardware. SIRIUS further achieves 35% speedup to full in offloading settings.

Algorithm 1 Sirius

Require: Prompt [ x 1 ,...,x t ], full model M F , and sparse model M S sharing weights and sharing KV Cache C , cache _ pos is the location where new k and v are written to C , kernel size n

Require: forward function FORWARD, threshold r , which is a value used by M F to judge whether the token occurs likely enough

Require:

StoppingCriteriaMet() downstream task-specific, returns a boolean

1: while not StoppingCriteriaMet() do

We also present ablation with rich details on how each component of SIRIUS contributes to its performance and how threshold is used to trade off efficiency and performance. Due to space limit, we place it in Appendix B.3.

5.1 Sirius Significantly Recovers CS Degradation with Low Cost

Models and Datasets - To comprehensively evaluate SIRIUS performance, we deploy six mainstream LLMswith sizes ranging from 7B to 13B: Llama-2-7B, 13B, and Llama-3-8B with their instruction finetuned counterparts, all from Llama family. Following Wei et al. (2022) in LLM reasoning, we also tested CS models on two popular types of reasoning generation tasks: arithmetic and commonsense reasoning. On the Arithmetic side, besides GSM8K, we also evaluate CS models on AQuA-RAT. On the Common Sense side, we use CSQA Saha et al. (2018), StrategyQAGeva et al. (2021), Date, and Sports, last two from Big Bench Suite bench authors (2023). Most of these tasks are originally classification tasks. Following the instruction in Wei et al. (2022), we manually compose COT prompts to transform these into logic argument generation tasks. Besides, CS models do not perform well in coding, which requires forming logical arguments and planning. We select HumanEval Chen et al. (2021) and MBPP+ Liu et al. (2023a) to evaluate SIRIUS.

For arithmetic reasoning and coding, we use 50% neuron sparsity for both CSparse and FSparse. FSparse relies on the gate layer to be dense, leading to higher global density than CSparse. Since commonsense reasoning tasks are generally less logically challenging comparatively, we lowered the neuron sparsity level to 40%.

Main Results - Due to space limits, we only select the best treewidth of SIRIUS for GSM8K, CSQA, and HumanEval for the main results in Table 2. Extensive studies on the rest 5 datasets with different treewidth are presented in the Appendix C. From Table 2, we can see that SIRIUS is consistently effective and efficient across all different classes of tasks. Specifically for Llama-3-8B-Instruct, besides GSM8K, SIRIUS corrects FSparse and CSparse, on CSQA, from 61% and 64% accuracy to 70% with cost only 3% sparsity for FSparse and 7% for CSparse respectively. On HumanEval, SIRIUS corrects FSparse from 45% to 61% with 4% sparsity overhead even surpassing the full model's performance,

and from 20% to 52% with 8% sparsity as cost. Besides, Llama-3-8B-Instruct, SIRIUS corrects all 6 models with additional sparsity overhead smaller than 10% across these three datasets, further showing its strong efficiency. Besides results in Table 2, in Appendix C, we show that SIRIUS consistently shows great effectiveness with high efficiency across the rest of the 5 datasets.

5.2 Wallclock Speedup

Here we show that Sirius delivers its promised efficiency claim for on-chip and offloading settings. Because the fine-grained sparsity Lee et al. (2024) relies on a custom CUDA kernel to achieve the target generation speedup not open-sourced, we focus on coarse-grained sparsity on GSM-8K COT, and the input sequence length with average prefill length 900.

Firstly, we consider the on-chip setting running Llama-3-8B-Instruct on a single GPU. The sparse model (APU 0.65) achieves 36.01% accuracy on GSM8K-COT, while the full model achieves 76.12% accuracy on GSM8K-COT. With kernel size 10, SIRIUS achieves 0.74 APU with accuracy 71.27% accuracy. We use torch compile to optimize the inference latency and limit the overhead other than running model in-

ference. The average latency generated per token is used to compute latency. Results are shown in Table 3. On average, SIRIUS delivers the promised latency reduction from APU calculations. The speedup ratio on A40 and L40 closely aligns with the theoretical APU reported. On the other hand, A100 and H100 compute MLP more efficiently than it compute attention, making the latency ratio between computing MLP and attention not perfectly aligned with their ratio in parameter size. Therefore, we see that even the sparse model baseline has slightly higher latency as expected. We increase the kernel size from 10 to 16 for these two devices, where accuracy reaches 0.7089 and the AAL reaches 13.67. For A100 and H100, building a hardware-efficient tree is nearly free of cost and highly effective. For numbers in Table 3, we use the width 4 tree that boosts the AAL to 15.01 out of 16. More details are in Appendix B.2.

Secondly, we consider the offloading setting which is the only way for resource-limited users to run 70B models by loading only the weights in use to GPU memory, while the others are offloaded to the CPU. Results are shown in Table 4. We use a single L40 48GB with a PCIe bus bandwidth of 25 GB/s to run Llama-3-70B-Instruct with batch size 1. Llama-3-70B-Instruct has roughly 80% of parameters to be MLP, which gives the theoretical APU for Griffin to be 0.6. Sparse + Sirius gives 0.649 APU, which is roughly what our system achieved.

6 Conclusion

Weobservethat contextual sparse methods significantly degrade reasoning and deduction tasks. SIRIUS, an efficient correction mechanism, enables accurate LLM inference with contextual sparsity. With roughly 11% to 18% sparsity increase, SIRIUS improves fine-grained and coarse-grained sparsity significantly in their performance while maintaining their efficiency gain.

Further, SIRIUS is still relying on rollback to correct the tokens that are deemed unlikely, which is inefficient. On the other hand, making the weak-strong model synergy systems that match the performance of the strong while keeping the efficiency of the weak, without strictly matching the strong models' output distribution remains an interesting and unsolved problem. We leave these topics to future works.

References

bench authors, B. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research .

Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. (2023). Accelerating large language model decoding with speculative sampling.

Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code.

Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., and He, P. (2024). Dola: Decoding by contrasting layers improves factuality in large language models.

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V ., Huang, Y ., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V ., and Wei, J. (2022). Scaling instruction-finetuned language models.

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems.

Dong, H., Chen, B., and Chi, Y. (2024). Prompt-prompted mixture of experts for efficient llm generation.

Frantar, E. and Alistarh, D. (2023). Sparsegpt: Massive language models can be accurately pruned in one-shot.

Freitag, M. and Al-Onaizan, Y. (2017). Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806 .

Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. (2021). Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL) .

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding.

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 .

Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M. W., Gholami, A., and Keutzer, K. (2023). Speculative decoding with big little decoder.

LeCun, Y., Denker, J., and Solla, S. (1989). Optimal brain damage. Advances in neural information processing systems , 2.

Lee, J.-Y., Lee, D., Zhang, G., Tiwari, M., and Mirhoseini, A. (2024). Cats: Contextually-aware thresholding for sparsity in large language models.

Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding.

Li, Z., You, C., Bhojanapalli, S., Li, D., Rawat, A. S., Reddi, S. J., Ye, K., Chern, F., Yu, F., Guo, R., and Kumar, S. (2022). The lazy neuron phenomenon: On emergence of activation sparsity in transformers.

Lin, S., Hilton, J., and Evans, O. (2022). Truthfulqa: Measuring how models mimic human falsehoods.

Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. (2017). Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146 .

Liu, J., Xia, C. S., Wang, Y ., and Zhang, L. (2023a). Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems .

Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y ., Re, C., and Chen, B. (2023b). Deja vu: Contextual sparsity for efficient llms at inference time.

Ma, X., Fang, G., and Wang, X. (2023). Llm-pruner: On the structural pruning of large language models.

OpenAI, Achiam, J., et al. (2024). Gpt-4 technical report.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67.

Reddy, S., Chen, D., and Manning, C. D. (2019). Coqa: A conversational question answering challenge.

Saha, A., Pahuja, V., Khapra, M. M., Sankaranarayanan, K., and Chandar, S. (2018). Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph.

See, A., Liu, P. J., and Manning, C. D. (2017). Get to the point: Summarization with pointer-generator networks.

Sharma, P., Ash, J. T., and Misra, D. (2023). The truth is in there: Improving reasoning in language models with layer-selective rank reduction.

Song, Y., Mi, Z., Xie, H., and Chen, H. (2023). Powerinfer: Fast large language model serving with a consumergrade gpu.

Sun, H., Chen, Z., Yang, X., Tian, Y., and Chen, B. (2024a). Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.

Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. (2024b). A simple and effective pruning approach for large language models.

Team, G., Anil, R., et al. (2024). Gemini: A family of highly capable multimodal models.

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology , 58(1):267-288.

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y ., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V ., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837.

Xia, M., Gao, T., Zeng, Z., and Chen, D. (2024). Sheared llama: Accelerating language model pre-training via structured pruning.

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V ., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. (2022). Opt: Open pre-trained transformer language models.

Appendix

Appendix Table of Contents

Additional Background

Extended Related Works

WhyNotUsing Speculative Decoding to Correct the Sparse Model?

Supplemental Experiments

Given Similar Parameters, More Well-trained Parameters Suffers More

Hardware-Friendly Tree Building Process

Ablation: Various Aspects of SIRIUS is Tested and Challenged

Large Model Experiments

Variable Sequence Length with Batch Size One

Error Occurs at Which Position inside a Chunk

Miscellaneous Results

Llama-2 and Llama-3 Models on GSM8K-COT

Additional Results on Reasoning

Arithmetic Reasoning

CommonSense Reasoning

Code

A Additional Background

A.1 Extended Related Works

Pruning in LLM Sparsity in neural networks has been widely studied. In the context of LLM, sparsity is studied under two branches - unstructured and structured. On the unstructured sparsity side, Frantar and Alistarh (2023) (SparseGPT) is a ground-breaking work that formulates pruning as a solving a series of sparse regression problems and proposes a fine solver for the problem. Sun et al. (2024b) (Wanda) introduces input activations into the pruning decision and achieves strong results in inducing LLM sparsity. On the structured side, LLMPruner Ma et al. (2023) and Sheared Llama Xia et al. (2024) each proposes different meticulous pruning algorithms and restoring weights through either parameter-efficient finetuning or efficient full weights training.

Contextual Sparsity Many recent works on LLM sparsity notice that the sparse pattern is highly related to the input or context. Deja Vu Liu et al. (2023b) revealed that for OPT models Zhang et al. (2022) the contextual sparsity is as high as 85%, meaning that 80% of the parameters can be pruned that won't hurt the token decoded quality given the prompt. Deja Vu formulates the problem of neuron selection as a near-neighbor search problem: finding neurons that are the most similar to the input activations. PowerInfer Song et al. (2023) extends the contextual sparsity to benefit the heterogeneous setting. Compared to the rest of the model, MLP layers tend to possess significant contextual sparsity and can be effectively exploited in a training-free manner. Concurrently, Griffin Dong et al. (2024) discovers the phenomenon of flocking, where MLP neurons have temporal locality, where given a fixed prompt, similar neurons tend to get activated throughout the following generation. Flocking is shown to occur in most activation types and open-source LLMs. Griffin selects the same set of heated neurons with 50% sparsity throughout the generation of each input prompt, which we refer to as coarse-grained sparsity. CATS Lee et al. (2024) successfully exploits per-token contextual sparsity in the MLP layers for inference latency reduction. They resample a new set of neurons per every new input token, which we categorize it as fine-grained contextual sparsity. Our paper mainly focuses on the training-free MLP sparsity techniques. Although these recent works show minimal accuracy degradation in classification and easy text summarization tasks, they both severely degrade in generation quality under tasks that require high-level reasoning and understanding ability. Our work serves as a low-cost complementary tool, aiming to push these elegant and promising techniques for mainstream use cases.

Also, previous contextual sparsity methods haven't fully and exhaustively evaluated their benefits and limitations in downstream generation tasks. To fully study this technique, we extensively go through

open-source LLMs in diverse performance and sizes on diverse generation tasks and datasets to locate where these sparse models maintain the performance or fail.

Speculative Decoding Besides model compression techniques, Speculative decoding Leviathan et al. (2023), Chen et al. (2023), Kim et al. (2023) is another important LLM inference latency reduction method. Compared to LLM, small transformer models are much more computationally accessible and can effectively model short-range tokens. Therefore, smaller models are asked to speculate short-term future tokens, which the LLM takes in in parallel to trade in FLOPs with memory loading time. During verification, most speculative decoding methods pursue lossless acceleration, leading to frequent rollback during rejection. In contrast, Sirius solves a very different problem. Our method aims to maximally preserve the efficiency of sparse models while boosting its performance. Sparse models, pruned directly from LLM, are much stronger at modeling a longer range of text than draft models, thus requiring much less help from the LLM. Our work aims to find the minimum amount of LLM overhead while boosting its performance to the LLM level. Given the resemblance and relevance of Speculative Decoding to our method Sirius, we will elaborate more in-depth on their differences and Speculative Decoding's inefficiencies when it comes to helping the Sparse method in A.2.

A.2 WhyNotUsing the Speculative Decoding to Correct the Sparse Model?

When Speculative Decoding is used to correct sparse using the full model, we will show that the efficiency of the overall process will be largely limited. We followed the common practice from speculative decoding and measured the acceptance rate on different datasets C4 Raffel et al. (2020) and GSM8K Cobbe et al. (2021). Take the Coarse-grained sparse model as an example. For Llama-3-8B as the full model, the 50% sparse (APU 0.65) model will produce an acceptance rate of 0.71 on C4 and 0.89 on GSM8K. Speculative decoding also use parallel verification in the period-basis. Naturally, to keep the system efficiency high, we need to (1) enlarge the period and (2) increase the average number of tokens accepted (AAL) given the gamma (period - 1) value. Take the acceptance rate of 0.89 on GSM8K as an example, following the formulation in Leviathan

Speculative Decoding Efficiency

3.0

8.90

8.608.78

8.30

7.84

8

2.5

7.09

AdvanceLength

Density

5.91

6

2.0

1.93

Effective

4.01

1.69

4

1.5

1.45

Average

2.68

1.24

1.0

1.05

FullModelAPU

2

0.86

SparseModelAPU

0

02

24

8

12

16

20

24

28

32

Gamma

et al. (2023), we can calculate the expected number of accepted tokens for every gamma term in the Speculative Decoding literature. AAL = 1 -α ( γ +1) 1 -α . The trend (green) is plotted in Figure 6

Wecan notice the trend that the average advance length starts to plateau as the gamma becomes larger. Take the gamma of 16 as an example, the period is then 17. The average advance length is only 7.84. The APU is (16 * 0.65 + 1)/7.84 = 1.45, which is larger than the full model 1. The blue line in Figure 6 shows the relationship between APU and gamma.

Because of the plateauing effect, for an acceptance rate of 0.89, the best gamma is 2 (period = 3). The optimal APU is 0.86, compared with 0.65 coarse-grained sparse APU. A similar picture can be applied to Fine-grained sparsity as well. The key reasons for the observation are two-fold: (1) the contextually sparse models are too big to be the draft model of the speculative decoding system and to have a large period; (2) Speculative decoding preserves the original model's performance so that the acceptance criteria are usually very strict, which is also not suitable for large period and high average advance length. Following the same spirit, Sun et al. (2024a) also uses a large draft model to do self-speculation, but for them, the authors select gamma = 1 to achieve the optimal speedup of their system. In contrast, SIRIUS brings <0.76 APU in this case with period ≥ 10 . Specifically, with a threshold of 0.1, Sirius can correct Llama-3-8B coarse-grained sparsity from 20.85% to 43.9%, compared to the 49.66% full model. With a period of 16 tokens (gamma = 15), Sirius on average can accept 13.4 tokens out of a kernel size of 16 and over 9 tokens out of a kernel size of 10, translating to APU < 0.76, significantly lower than SD does.

B Supplemental Details in SIRIUS observations, Design, and Experiments

In this section, we provide several supplemental experiments to the picture. First, we run SIRIUS on Llama-3-70B. However, because of computational limits, we cannot run SIRIUS with the tree on Llama-370B with the scale we did for other models. Nevertheless, we do show that 70B has roughly the same pattern as we have seen before, large model sparsity also somehow struggles on reasoning tasks. Second, we provide additional proof for the parallel verification efficiency statement. After that, I show results on where the error is located in the chunk size of 16 tokens. The error is dis-

Neuron Index

Neuron Index

Tokens

Tokens

Prompt Understanding Tasks

Reasoning Tasks

selected sparse

(a)

(b)

tributed almost uniformly. Last but not least, we also apply SIRIUS on datasets that are reasoning. Lastly, weprovide more results on the comparison between models of similar size but have a huge performance gap. We show that given the similar parameter size, the trend is for a more well-trained, powerful model to degrade more from contextual sparsity. We present here more illustration on contextual sparsity and Llama-2-7B-Chat experiments with fixing only a minor portion of tokens is shown in Figure 7.

B.1 Given Similar Parameter Size Well-trained Models Suffer More

Weobserve another interesting phenomenon: given the similar parameter size, the more well-trained the model is, the more performance degradation contextual sparsity would make on the full models. Here we present two pairs of results. First, we look at the performance between Llama-3-8B-Instruct and Llama-2-7B-Chat with Llama-3-70B-Instruct and Llama-2-70B-Chat. All models are evaluated on GSM8K-COT. We draw these models in CSparse in Figure 8, and the readers can find more results in Appendix B.8. We can see figures from top to bottom, where even at lower density (more elements are not selected), Llama-2-7B-Chat and Llama-2-70B-Chat suffer from less performance degradation (blue) compared to the Llama-3-8B-Instruct and Llama-3-70B-Instruct models. Furthermore, suppose we focus on Llama-3-70B-Instruct for global density at 60% or lower. In that case, the performance (coral) is degraded significantly, which is comparable or even lower to Llama-3-8B-Instruct full model performance at 0.76, Even at 50% density, the 70B model still has more than 40B parameters, much more expensive than the 8B model. The observation fully manifests the difficulty of using CS in complex reasoning tasks.

B.2 Hardware Friendly Tree Building Process

In this section, we first look at the insights behind whether building the tree can help efficiency, then we detail the specific steps towards tree pruning.

The goal for the Sirius system is to make n AAL to be as large as possible. Despite the full model sharing KVs with the sparse model, Sirius still encounters costly rollbacks because of sparse greedily decoded tokens being rejected. Interestingly, we look closely into where the sparse model is likely to make a mistake

on GSM8K and AQuA-RAT-COT Ling et al. (2017) with Sirius on Llama-3-8B-Instruct and a kernel size of 16. More details are shown in Appendix B.6. The error distributes almost uniformly across all positions of the kernel size. Also, when the token makes the mistake, besides the greedily decoded

Density

CSparse Llama3-8B-In

CSparseLlama3-70B-In

Degradation

0.7

0.9

Degradation

0.7

0.61

0.6

0.56

0.6

0.8

0.6

Strict Accuracy

Strict Accuracy

0.5

0.5

Degradation

0.7

0.5

Degradation

0.4

0.38

0.4

0.4

0.6

Q.32

0.3

0.3

0.3

0.5

0.2

0.2

0.16

0.2

0.4

0.1

0.1

0.1

GSM8KACC

0.0

0.3

GSM8KACC

0.0

0.52

0.55

0.58

0.61

0.64

0.45

0.50

0.55

0.60

Density

Density

CSparse Llama2-7B-Chat

CSparse Llama2-70B-Chat

0.18

Degradation

0.7

0.450

Degradation

0.7

0.425

0.16

0.6

0.6

Strict Accuracy

0.400

0.5

Dearadation

0.5

0.14

0.375

GSM8KACC

0.4

0.4

0.12

0.350

0.3

0.3

0.325

0.10

0.2

0.18

0.2

0.11

0.300

0.11

0.08

0.08

0.1

0.06

0.1

0.03

0.275

GSM8KACC

0.0

0.0

0.475 0.500 0.525 0.550 0.575

0.45

0.50

0.55

0.60

Density

tokens, we find that other tokens of lower likelihood offer the potential to boost efficiency. Surprisingly, we found that out of the cases where the greedily decoded tokens are rejected, the probability that the second or third most likely tokens from the sparse being accepted by the full model is reasonably high.

ShowninTable 5, we test on part of the GSM8Krejected cases. The "Second Hit" is defined as the count of the second most likely tokens being accepted by the full model when the greedily decoded token is rejected, while the "Third Hit" is defined as the count of the third most likely token being accepted when the first two are rejected. Both sparsity method has a high acceptance rate, or "Coverage", from the second and third most likely tokens when the most likely token is rejected, showing huge potential for gains in efficiency.

0

th

Round

(token from

1

st

Round

(1) Each drafts

k

children

(2) Computes

Cumulative likelihood

for every path

tokens

and discard

the rest

keep k branches

2

nd

Round

Have

Need

[Space]

3

Will

Has

is

needs

Drafting and Filtering Process for

the nth round

to

#branches each round is fixed at treewidth (k), tree building continues

To capitalize the potential from the second to third tokens, we propose to build a tree during the sparse generating process (lines 6 to 11 in Algorithm 1. The tree algorithm is similar to Beam Search Freitag and Al-Onaizan (2017). However, to make sure that the tree building and tree parallel correction processes can achieve speedup over cases that don't build trees, we impose strong restrictions on the tree structure we build. For a fixed kernel size, we limit every step to having a fixed number of leaves, or treewidth, through tree pruning based on ranking the cumulative log-likelihood of the path. The resulting tree has a fixed shape for a given kernel size and tree width, but only the interconnection pattern between steps varies based on the pruning and ranking within each step. The details are illustrated in Figure 9. During verification, out of the treewidth complete paths, we select the one that reaches the longest advance length. In practice, we found that for kernel size 16, when the treewidth is increased to

Degradation

8, the optimal verification tree is around 64. From Section A.2, we see that the parallel verification of the tree of 64 roughly equals the time the full input 1 token.

Therefore, a treewidth of 8 is set as the maximum treewidth when building the tree for kernel size 16 for later. We show that building a tree makes the system significantly more efficient while retaining correction effects.

In Table 6, we present more details complementing Table 3 on the use of tree-building for A100 and H100. We ablate the use of a width 4 tree by not using the tree under kernel size 16. We observe that the speedup improvements solely from the increase in AAL from 13.67 to 15.01 for kernel size 16.

B.3 Ablation: Various Aspects of Sirius Are Tested and Challenged

Probing Components To understand the contribution and the utility of each component of Sirius, we ablate all components of Sirius in Table 7. We started by only letting the LLM correct the token it is evaluating (interleaving only). Then, we add on top of it the KV cache correction, and then the rollback. All these three techniques are effective when applied solely. Rollback seems to be the most effective technique. Even when applied alone, rollback asserts significant correction to both the CSparse and FSparse models. Interestingly, KV Cache is also effective alone, bringing a 12% increment for CSparse and an 11% accuracy increase for FSparse. Relatively, interleaving is the weakest. Surprisingly, adding both KV rewriting and rollback is only marginally better than rollback alone. Although it is tempting to think KV Cache rewriting is not useful with rollback, the improvement KV Cache Rewriting brings is a gain in efficiency. When adding the KV Cache Rewriting on top of Roll Back and interleave it significantly improves the efficiency of the correction. For CSparse, adding KV rewrite increases AAL from 12.77 to 13.80.

Likelihood threshold to balance Correction and Efficiency Wefound that the likelihood threshold is important for managing the Sirius correction and efficiency tradeoff. We present results in Table 8. Weablate this setting on a 30% subsampled GSM8K dataset, and only strict accuracy is reported. The

performance is the score, while the efficiency is measured by Average Advance Length (AAL). We can find that with the increase of threshold, the scores generally improve, while the efficiency metric decreases.

Building Wider Tree Westudy the effect of increasing the treewidth. In fact, for every number from SIRIUS in Table 2, we are selecting from a group of results by different treewidth. We present all of this treewidth and its corresponding accuracy and efficiency numbers in the Appendix C. Importantly, raising treewidth always improves AAL. Although different choices of treewidth usually give similar accuracy scores, there is hardly a pattern on which treewidth always gives the best accuracy. The optimal treewidth can only be found through empirical studies.

B.4 Large Model Experiments

To diversify the evaluation of Sirius, we also evaluate Sirius's Effectiveness on the Llama-3-70BInstruct model. MMLU is subsampled 10%, while CNN/DailyMail is subsampled 30%. The following table contrasts with Llama-3-8B-Instruct. We use strict match/flexible extract accuracy for GSM8K-COT, accuracy for MMLU, F1/EM score for CoQA, Rouge-1/2/L score for CNN/DailyMail, and Rouge-1/2 ACC for TruthfulQA.

B.5 Variable Sequence Length with Batch Size One

Here we show the benchmark latency on A100, where the input tensor to Llama-3-8B-Instruct has a shape of batch size 1 and a different input sequence length. To get the hardware optimal readings, we use torch compile to compile the whole forward pass of the model. We show that the latency only goes up insignificantly to 64, but the trend of increment to 96 is a bit steep.

B.6 Error Occurs At Which Position inside a Chunk

Welook at the distribution of where the error would be inside a kernel of 16 tokens. We run through Sirius with a kernel size of 16 on the entire GSM-8K and AQuA-RAT-COT dataset. The histogram is shown in Figure 10. We found that the error occurs in a uniform pattern, where it is hard to see any particular region where the tokens are likely to occur the most.

B.7 Miscellaneous Results

Besides, the results on the complex reasoning tasks, we evaluate Sirius on slightly more diverse tasks in Table 12.

AQuA-RAT-COT

Position inside 16

Position inside 16

120

50

100

40

80

FO

60

20

40

20

10

2

4

6

8

10

12

14

Q

2

4

6

8

10

12

14

B.8 Llama-2 and Llama-3 Models on GSM8K-COT

Here we present more experiments for the comparison between Llama-2 and Llama-3 family models, which is first mentioned in Section B.1, where we also include FSparse methods together with the CSparse method. The results are in Table 11.

C Additional Results on Reasoning

Due to page restrictions, we only show GSM8K, CSQA, and HumanEval in the paper. Below we show additional results to the numbers presented in the paper. Wepresent tables of a similar format. Please notice that the leftmost column writes a number that represents the treewidth in the given settings. Also, we show the results of SIRIUS on the other five datasets AQuA-RAT-COT (Arithmetic Reasoning), Sports (Commonsense Reasoning), Date (Commonsense Reasoning), and StrategyQA (CommonSense Reasoning), and MBPP+ (coding).

C.1 Arithmetic Reasoning

In this section, we present GSM8K and AQuA RAT COT evaluation results with the efficiency metric AAL. Sirius is shown to be effective on these two reasoning tasks about arithmetic. Below we show the raw AAL score associated with efficiency for all models and the performance of different treewidths.

C.2 CommonSense Reasoning

Wefollowed the COT paper and evaluated Sirius on CSQA, Sports, StrategyQA, and Dates. Sparse methods are capable of outputting high-quality output similar to the full model at the 0.5 mark, which is different than on other datasets. However, we tune the sparsity level to 0.4 (0.6 dense, 0.4 removed), and it starts to have performance degradation. Sirius can compensate them with relatively high efficiency)

C.3 Code

We also have a coding portion that evaluates Sirius on HumanEval. Sirius performs well similar to other datasets. Besides, we also have results on MBPP+. The results show SIRIUS effectiveness and efficiency again.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Donot modify the questions and only use the provided macros for your answers .

A. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

B. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

C. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

D. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

E. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

F. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

G. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

H. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

I. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

J. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

K. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

L. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

M. NewAssets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

N. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

O. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a sparse LLM correction mechanism (SIRIUS) designed to improve the inference efficiency of large language models through contextual sparsity. Contextual sparsity reduces computational cost by pruning parameters that are less relevant based on the input context. However, this approach degrades performance in tasks requiring high-level reasoning and deduction. SIRIUS addresses this by selectively correcting key tokens, using the full model to rewrite the sparse model's Key-Value cache. Experiments show improvements in strict-match EM scores, particularly on mathematical reasoning tasks.\n\nSTRENGTHS:\n1. The correction mechanism that selectively rewrites the KV cache is novel.\n2. The paper presents an interesting finding where sparse models fail on tasks that require reasoning and deduction.\n3. The paper demonstrates improvements in performance metrics on benchmarks like GSM8K.\n\nWEAKNESSES:\n- The KV cache rewriting mechanism requires an available full LLM to be of the same architecture as the sparse model.\n- The criteria for when to trigger KV cache rewriting are not clearly detailed, making it difficult to assess the practicality and reliability of the approach.\n- It is unclear whether using the full model’s KV cache to rewrite the sparse model guarantees that the rewritten KV cache can be accurately interpreted and utilised by the sparse model, due to the differences in the sparse and full model.\n- The hypothesis that the gap between the full and sparse models can be bridged by correcting a few key tokens (L44-45) is not sufficiently validated beyond the provided examples. It remains unclear whether this hypothesis holds across a broader range of tasks and datasets.\n- The writing needs to be improved for clarity, for example, \n     - there are a few missing citations (e.g., L114. L284)\n     - the tables can be clarified with the best performing variant per task in boldface, and a caption which describes the metrics in the table.\n     - Algorithm 1's functions can be better described, for example, the descriptions in \"normal forward function FORWARD, forward function with direct cache rewrite KVFORWARD, likelihood judge function of LIKELIHOOD\" are unclear.\n\nQUESTIONS:\n- How are the likelihood thresholds for detecting and correcting errors empirically tuned? Can the authors provide more details on this process and its impact on performance?\n- How does SIRIUS handle long-term dependencies and corrections over extended sequences of text? Are there scenarios where the KV cache rewriting might not be sufficient to correct extended errors?\n\nLIMITATIONS:\nnot observed negative societal impact to my understanding.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-26"}

comments: SUMMARY:
The paper introduces SIRIUS, a novel method designed to enhance the efficiency and accuracy of sparse Large Language Models in reasoning and deduction tasks. SIRIUS employs contextual sparsity to reduce computational costs and incorporates an efficient correction mechanism that recovers sparse model performance by correcting only a minimal number of key tokens. Experimental results demonstrate that SIRIUS significantly improves the performance of sparse models on complex reasoning tasks while maintaining efficiency.

STRENGTHS:
1.	The paper introduces SIRIUS, a novel and efficient correction mechanism specifically tailored for sparse models
2.	It provides a echnically rigorous approach to improving the inference efficiency of LLMs, supported by experimental validation.

WEAKNESSES:
1.	There are too few experiments,  so no experiments were conducted on other LLMs， and no experiments were conducted on more contextual sparsity methods
2.	The Mc annotation in pseudocode is unclear

QUESTIONS:
If the correction requires the use of a full model, won't the speed of correction slow down compared to directly using the sparse model? Is the memory usage the same as using a full model directly? If the memory usage is the same as using a full model directly, there is no advantage in memory savings

LIMITATIONS:
The author improved the performance of the sparse model in reasoning and understanding tasks by adding a full model correction, which requires the use of the entire full model. Although the average parameters used per token increase is minimal, the memory usage is the same as using the full model directly
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve contextual sparsity (CS) approaches: LLM sparsification / parameter pruning methods where the sparsification strategy is conditioned on the input sequence / prompt itself. The paper begins by reproducing contextual sparsity baselines, but applied to modern LLMs: LLama 2 and 3 (7B & 8B respectively), and verifying that their reproduction successfully replicates previous CS techniques on summarization and question answering tasks. The authors then demonstrate that the technique is much less effective on more challenging reasoning tasks such as GSM8K and MMLU-CoT. Through investigation the authors observe that often, by correcting a small amount of tokens during the generation (~10%), the sparse models can significantly improve their performance to be on-par with the full dense model. This motivates their main proposed technique: SIRIUS, which employs a speculative decoding like technique to use the full dense model to score every k tokens (k=16-24 in practice) during the decoding process in order to rollback and use the dense model to correct a token when necessary. By doing so, the method is able to mitigate the losses on reasoning tasks while maintaining most of the efficiency of the sparsification method.\n\nSTRENGTHS:\n1. The authors explore the viability of CS techniques on much more difficult reasoning tasks than previously in the literature, providing an important datapoint on whether CS approaches (at least ones studied in this paper) work in general. \n\n1. The authors run thorough quality experiments on modern LLMs, and demonstrate that their technique effectively rectifies the issues they first identify (significantly improving reasoning performance.)\n\n1. Using the full model to do something like self-speculative decoding with the drafting model being a sparse version of the same model is novel and promising. It aligns with contemporaneous works that explore this idea [1].\n\n1. The method is efficient w.r.t the number of parameters used on average during decoding. \n\n1. The paper is overall well structured and the investigation is easy to follow. \n\n1. The authors plan to release the code for their experiments.\n\n[1] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding\n\nWEAKNESSES:\n1. The main spirit of this paper is whether we can use some small amount of additional compute budget over the sparse model alone (e.g. via limited calls to the full model) to improve the decoding from the sparse model. However, the authors do not compare against any baseline approaches to do this same thing, such as speculative decoding. Even though the authors claim in Related Work that speculative decoding does not suit this problem since it’s not efficient enough, the authors do not provide any empirical evidence of such.\n\n1. This paper is very much an efficiency paper, which tries to introduce some small amount of corrections from the full model to the dense model, while not doing too much as to nullify the gains from the sparse method. However, the authors only analyze Average Parameters Used Per Token (APU), a sparsity measure, instead of any other efficiency metric (importantly wall-time). \n    - The authors make an argument for why this is in Section 4.1, using findings from previous papers. While this may be a reasonable intuitive argument, the method in this paper has significant differences like KV-cache correction that still makes this valuable to empirically verify. \n    - Moreover, even if I intuitively believe from the author’s argument that the wall time latency of this approach would be somewhere between the sparse and full model’s wall time, it is unclear where on this spectrum it lies, how it compares to other techniques like speculative decoding, or how the tradeoff plays with varying sparsity levels without such measurement and comparison. \n    - To place this into context, the CATS paper, which is one of the methods this work experiments with (FSparse) directly states “Activation sparsity of a model is not sufficient to directly enable wall-clock time inference\nSpeedups”, and provides wall-clock time analysis.\n\n1. The paper is poorly written, with various spelling errors (in the title), grammatical errors (for example: line 1, Section 4.2 title, and more), placeholders for citations (line 114), and difficult to read citation formatting. In many ways the presentation of this work is significantly not publication ready. \n\n1. Without the code release, the description of the method is not very detailed, even though Algorithm 1 is provided, and would make it very hard for the reader to understand how this method is implemented from reading the paper alone, which hurts reproducibility. \n    1. For example: the description of the correction algorithm is only described in one paragraph in Section 4.3 with insufficient detail on how the KV cache is updated. \n    1. Algorithm 1 has undefined variables with respect to the KV cache such as “cachetracker”. The description in 4.3 states that the KV cache is directly written to by the full model to correct the past, however Algorithm 1 does not describe this process other than  “Update C_s based on j” is unclear what kind of update is happening to the cache.\n    1. The important LIKELIHOOD function is undefined, which uses the full model’s likelihoods to judge whether the sparse model’s output is incorrect. There are no details on what threshold is used here. \n\n1. Some additional claims are not sound:\n    - Line 270: “KV Cache can also be looked as an implicit knowledge distillation method to pass dark information from the LLM to the sparse counterparts” What is dark information?\n\nQUESTIONS:\nWhat were the motivations behind the differences between Sirius (Algorithm 1 / Section 4.3) and speculative decoding? Why not directly use speculative decoding, with the sparse model being the drafting model? (which maybe has more guarantees)\n\nWhy is the method named Sirius?\n\nLIMITATIONS:\nThe authors briefly describe some limitations but the work is generally lacking discussion here. Mainly the authors mention that the method does not work well under extreme sparsity (to my knowledge almost no sparsity method does in the general case – this is more of an open question than a limitation.) The authors could be more forthcoming here. It would be valuable to elaborate on whether the authors believe the two CS approaches are representative of the space of approaches, whether the experiments are adequate, whether they expect the results to change at different model scales, or for tasks beyond the ones studied here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on enhancing the inference efficiency of large language models (LLMs) through contextual sparsity. While it identifies that contextual sparsity reduces hallucination, it also notes a significant impact on reasoning and deduction performance. To address these drawbacks, the paper introduces SIRIUS, a correction mechanism that effectively improves performance. Specifically, the correction mechanism uses additional parameters (i.e., full model) to check the output of the sparse model based on the input and the same instruction prompt.\n\nSTRENGTHS:\n1. The paper introduces SIRIUS, a correction mechanism that effectively enhances the performance of sparse models, particularly in complex reasoning tasks where traditional sparsity techniques falter.\n2. The paper includes many experiments across various datasets and tasks to reveal the problem in the existing contextual sparsity methods and sparse models.\n\nWEAKNESSES:\n**Method\n\n1. Based on the results in Tables 1 and 2, this paper claims that “the stronger the model, the larger the quality degradation would be”. However, the studies exclusively utilize relatively smaller models, Llama2-7B and Llama3-8B. Including results from larger LLMs, such as Llama3-70B, would provide a more comprehensive understanding of the impact across different model sizes.\n\n2. As the proposed method uses the probability from the full model to check the correctness of the sparse model, does this require forwarding twice for each token? If so, the cost would be double.\n\n3. It is unclear when to use the correction mechanism. Should it be used after the sparse model has generated all predicted tokens, or immediately after the sparse model predicts each token?\n\n**Experiment\n\n1. In Table 3, if the proposed method uses the full model to correct the output of the sparse model, why is there still a noticeable degradation compared to the full model itself?\n\n2. As the title highlights the efficiency of the proposed method, it would be better to provide some experiments about it like latency (ms) in DejaVu [1].\n[1] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML, 2023.\n\n**Minor Issues\n\nThere are many typos in this paper, e.g., the title “… Contexual Sparisty …” should be “… Contextual Sparsity …”.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nThere are no discussions about the proposed method's limitations and potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}