SIRIUS : Contexual Sparisty with Correction for Efficient LLMs
Authors: Yang Zhou
Zhuoming Chen
Zhaozhuo Xu
Xi Victoria Lin
Beidi Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Contextual Sparsity
LLM inference
Knowledge Distillation
Primary area: optimization_for_deep_networks

Abstract: With the blossom of large language models (LLM), inference efficiency becomes increasingly important. Various approximate methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without significant performance degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, it significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models and original models often share the general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces SIRIUS, an efficient correction mechanism, which significantly boosts CS models on reasoning tasks while maintaining its efficiency gain. SIRIUS is evaluated on 6 models with 8 difficult generation tasks in reasoning, deduction, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for SIRIUS and show that SIRIUS delivers theoretical latency reduction with roughly a 20% reduction in latency for 8B model on-chip and a 35% reduction in latency for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git.

SIRIUS: Contextual Sparsity with Correction for Efficient LLMs
Yang Zhou', Zhuoming Chen', Zhaozhuo Xu², Xi Victoria Lin", Beidi Chen 1.3
CCarnegie Mellon Univeristy
Steevens Institute of Technology
FAIR Meta
{yangzho6, zhuominc, beidic) Gandrew. cmu. edu 2xu79@stevens. edu victorialin@men com
Abstract
With the blossom of large language models (LLM), inference efficiency becomes increasingly important. Various approximate methods are proposed reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training- taining-frre free nature and its ability to reach a higher compression ratio scemingly without significant performance degradation. However, after comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, it significantly degrades tasks.
the model performance for reasoning, deduction, and knowledge-basee Despite the gap in end-to-end accuracy, we observed that sparse models and original models often share the general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces SIRIUS an efficient correction mechanism, which significantly boosts CS models reasoning tasks while maintaining its efficiency gain. SIRIUS is evaluated on 6 models with 8 difficult generation tasks in reasoning, deduction, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop system implementation for SIRIUS and show that SIRIUS delivers theoretical latency reduction with roughly 20% reduction in latency for 8B model on-chip and 35% reduction in latency for 70B model offloading. We open-source our implementation of Sirius at atlhttps://githubbcoo
Inf Aini-AA--ab//iaia
Introduction
Large Language Models (LLM), such as OpenAI ct al. (2024) (GPT-4), Team et al. (2024) (Gemini), and Touvron et al. (2023) (Llama) have demonstrated their proficiency wide range of natural language processing applications such as content creation, summarization, and impressive and complex reasoning tasks. However, their deployment is very challenging, especially in latency-sensitive settings (Kaplan et al.) 2020). Exploiting the model sparsity is natural way to reduce the model parameter size and computational cost with long history (LeCun al. [9989 Tibshirani
1996). More
recently, many studies have shown that contextual sparsity (Liu et al. [2023b]] Li et al.
2022 Dong
et al. 2024 Lee et al. 2024), which highly correlates the prompt the context, can greatly speed up LLM inference without quality degradation.
However, in this paper, first demonstrate critical and fundamental problem with contextual sparsity (CS): while generally robust in classification tasks and generation tasks that mainly rely on prompt understanding (e.g.. summarization, chat question-answering)) we found that CS models struggle at high-level reasoning and understanding tasks.
We draw inspiration from the astronomical concept, which SIRIUS refers two-body star system, where one is the brightest star ever detected, while the other dim star.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Llama 8B Instruct with Coarse Grained Spersity
Laama-3708-Inetuct GSMBK-COT
Correcting CSparse Llama-3-98-insruut
0.12
0.9
0.92
0.75
0.6
0.10
Chi--aayyaa Unbbr Rouge
GEMSK COT Accurao)
0.70
GRM9K Poou/o
0.5
0.7
6.75
Aocuracy (8ttc1
0.65
0.60
0.4
0.08
0.55
Full Motel
0.04
0.5
Coarse
0.1
Contastual Sparsity
Contestual Sparatty
0.4
L-5-684
0.45
0.40
Performance with Correction
Full Model
20
30
40
50
00
70
0.116
0.4
0.5
0.8
1.0
Denuity
Etfective Parameter (9)
Pencentage Gorrected lokens
(a) Comparing Spansity Method on CNN/DailyMaii GSM8K
(b) Contextuall Spuniity applying
Correcting only
to Allma-3-70R-Instmee
brings Csparse to full performance
Figure 1: Contextual sparse models struggle at challenging text generation tests that require high- level reasoning and understanding, e.g. GSM8K. On these tasks, contextually sparse models lead to significant quality degradation. In (a), we contrast CS Llama-3-SB-Inm Instruct on GSM8K (green) and CNN MailyMail (coral). (b) Contextual Sparsity Llama-3- 1--70BIInsttuct crashes at 50% global sparsity, making the smaller dense model -llama---B--ll 3-8B-Innstuuct (green star) a significantly more efficient choice than the sparse 70B model. (c) Sparse model crashing at reasoning tasks has patterns, and ideally only correcting 1% unlikely tokens recovers the sparse model performance fully.
For example, in Figure | (a), we contrast between the Text Summarization task (CNN/DailyMail) and Arithmetic Reasoning (GSM8K) with contextual sparsity methods on -aama------Instttee with contextual sparsity performs consistently worse
Varying sparsity levels, on GSM8K than CNN/Daily Mail. With roughly 50% sparsity globally, the sparse model degradation still reasonable on text summarization (right axis in color coral) compared to almost collapsed on arithmetic reasoning (left axis in color green). However, for reasoning tasks, can we simply live with higher density ratio to preserve more performance? Unfortunately, the answer is NO. Besides the significant efficiency loss, shown in Figure (b), the Llama-3-700 -anaa--------naaee 70B-Instruct model with contextual sparsity also crashes at around 50% sparsity globally. The 50% sparse model still has 4x the parameter while still performing worse on
size compared to the smaller dense model 3-lama-3-8B-Instin GSM8K-COT, rendering contextual sparsity utterly not useful for complex reasoning tasks.
We conduct an in-depth study on the CS model failure cases and notice that the overall reasoning pathway of these sparse models is usually sound and adheres to the full model. The fatal mistakes are always caused by some middle tokens and propagate towards the end, examples can be seen in Figure Following this observation, we conduct simple experiment with 65% Contextual Sparse Llama-3-8B- InSt-tnttuctt on GSM8K as presented in Figure] (c). We run both the sparse and the full models together for the same prompt and compare two generation output token-by-token.
Surprisingly, the trend increases steeply with the percentage of corrected tokens. Correction of only 6% tokens in the sparse model's generation recovers most of GSM8K accuracy (<5% to full), and 1 1% to recover the full performance. The results show potential for an efficient and powerful correction mechanism to maintain the sparse efficiency while boosting its performance. Contextual sparsity uses a dynamic sparsity pattern and naturally requires the full model to be in GPU memory during runtime, allowing the full model to be used efficiently for infrequent correction. Even though only very few need to be corrected, locating these mistaken tokens efficiently turns out to be challenging.
Ideally, we want a correction system to have the following properties: 1) Effective, the sparse model quality degradation can be improved to the full model vicinity: 2) Cheap, the full model only gives minimal intervention; 3) Adaptive, the system is efficient across various reasoning datasets.
In this paper, we carefully analyze and formulate correction efficiency in Section 2 We extensively categorize the strengths and weaknesses of CS in Section 3 In Section We systematically design SIRIUS, a correction method covering all three desired properties. (When?) Section[4.] we show that the sparse model can be both confident or uncertain when making mistakes, rendering the signal from sparse unreliable for determining when to correct. SIRIUS a period-based method with the period as hyperparameter. (How?) In Sections 4.1 B.2] we introduce novel KV Cache direct rewriting, minimal rollbacks, and hardware-efficien tree building to help increase the effective period of full model correction, thus, ensuring the correction efficiency.

---

Infrequently, we use the in-memory Full Model check the past chunk
Intermediate mistakes eventually leads to wrong ind-resuit
Correcting tokens Full Model deemed Unlikely
Intermediate mistake
Wrong Result
Interleaving with high-quality token
Sparse Model
Full Model (Call intreqnnttn doing correction)
(Sparse Pattern Is Contextually Aware)
With correction,
Sirius gats correct
Wrong KV Cache
New Input
Rewriting the KV Cache
Correct Result
Figure 2: Overview of Sirius. Contextual Sparsity requires full model weights to be placed on the GPU memory. While the sparse model doesn't 'tt perform well complex reasoning tasks, Sirius uses the Full Model to correct the Sparse model. The full model is called fairly infrequently. During the correction, the Full Model will rewrite the KV Cache, interleave with high-qualit tokens to the sparse outputs, and then roll back only when the token is deemed extremely unlikely by the Full Model.
In Section we empirically evaluated SIRIUS on 6 different models with 8 different reasoning tasks and showed that SIRIUS is generally effective and efficient. On GSM8K and specifically, we boost the fine-grained sparsity from 58% 72% with 4% increase in effective parameter size and coarse -oarseegrinedd sparsity from 38% to 70% with the cost of 5% effective parameter size. We also show that SIRIUS delivers the promised efficiency on mainstream GPUs in both on-chip and offloading settings.
2 Related Works and Problem Formulation
In this section, first present the classification of the prior Contextual Sparsity methods and narrate important efficiency metries in. Also, we present careful analysis and quantitative comparisons on why Speculative Decoding is inefficient in recovering contextual sparsity. Due to space constraints, we refer to Appendix A.2 For extended related works on model compression, contextual sparsity, and speculative decoding, we present in Appendix[A.. A.
Contextual Sparsity Classification Contextual sparsity (CS) methods are usually training-fres easy to use, and seemingly effective, making them highly attractive to ML practitioners looking to reduce LLM inference costs. CS exists naturally in MI.P layers the LLM, which occupies roughly 70% of the LLM total weights (Dong et al. [2024]] Lee et al. (024 The contextual sparsity selection is as follows: given the context, only a limited number of the most relevant neurons are selected based on the input activation. The rest contributed to the output far less is discarded. We refer to two main directions of contextual sparsity methods as Coarse-grained Sparsity (CSparse) Methods (Dong et al. (2024) that within the same input prompt, the sparsity pattern is fixed for all tokens generated. Fine-grained Sparsity (FSparse) Methods (Lee al. (2024)) that exploits the per-token sparsity to save resources.
Average Parameters Used Per Token A key metric is used to evaluate the efficiency of our proposed method, the Average Parameter Used per token decoded (later referred to as APU). LLM inference is memory 1/O bound (Ieeviathan et al.] 2023 Kim et al. 2023). The latency of generating every single token is dominated by the memory loading time from the GPU HBM to SRAM. On the other hand, SIR- IUS relies full model parallel verifying chunk of tokens. Although from the FLOPs standpoint, the amount of compute performed per evaluation step is the number of input token times of single token in- put process, the latency of parallel verification is still roughly the same taking single token (Verified further in 10 length 64 is only 1.1 ms longer than length 1), because the inference is memory bound.
SIRIUS operates in the memory bound regime (single inference sequence length smaller than equal to 64). Thus, the average parameter count of a model gives us a rough judgment the latency of inference. Formally, for a full LLM to have Cfull number of parameters, and its sparse counterpart of a certain predetermined sparsity C Csparse- The average advancement length (later we refer as AAL) in the number of tokens between two consecutive LLM corrections can be represented AAL-
3

---

The average parameters used per token (APU) are the following
APU Respare se Caparse + Cfull
(I)
RAAL
We want the metric to be small possible, and obviously, we want ALAA to be as large possible.
Another thing to note that we always compare the system's 's APU against the full model's APU, which is Cuu. If we divided the above equation by C' we can have an equivalent parameter density of the system defined based on gtobnlsparsiits which is Osparse/ Cfull
Effective Density Reparse se Ilobal +1
(2)
TLAL
Later, if we use period Rperiod, the equation can be rewritten as
Effective Density= (Aperiodd
1) globalspprssity
(3)
RAAL
Later when presenting SIRIUS, we mainly specify Rperridd with nALL to evaluate its efficiency. Notice that I globalsparsity is determined by the sparsity method, SIRIUS cannot change it anymore.
3 Observations
In this section, present a detailed study of the strengths and weaknesses of Contextual Sparsity the strengths of CS. c.s.3.2preseett the weaknesses of CS. Additionally, we show that given the similar parameter size, the more well-trained the model is, the more CS degradation will be for the model. Due to limited space, we present the details in Appendix B. J.ows our findings when looking into the failure cases of the CS model in complex reasoning generation tasks.
In the following series of experiments, we build our implementatior on Lee et al. (2024) and coarse-grained sparsity based on Dong et al. (2024). The default sparsity for both methods is 50% for the MLP component of the model (whole MLP for coarse coarse-grained sparsity and Up and Down linear layers only for fine-gr one-grained sparsity). We mainly use this default setting in most experiment tables in the paper without explicitly mentioning it. Otherwise, we will explicitly specify the different sparsity levels we used.
2 of fine-grained sparsity based
3.1 Contextual Sparsity: Where Does It Succeed?
For tasks on prompt understanding, CS generally performs well and gives consistent and strong output. We evaluate CS models on machine summarization (CNNDailyMail, Seee et al. (2077). and Conversational Question Answering (CoQA, Reddy et al. (2019)) The results show that the correctly selected contextual sparsity in the MIPP layers and the full attention layers can fully extract and understand the local prompt information. More details are presented in Figure where we show that by varying the sparsity level, the language model's 's performance on CNN/DailMMall is robust even when the activation sparsity drops to below 20%, which translates to around 44% global density.
For tasks accessing factuality and hallucination, we select the generation portion of the TruthfulQA dataset (Lin et al. 2022). Results are shown in Table where we evaluate the techniques on 5 different LLMs. Interestingly, we find that the Fine- tineggraneed sparsity is often better than the dense model baseline across different models. This finding is consistent with previous works Laser (Sharma et al. 2023) and Dola (Chuang al. 2024). They both observed that compressing the original LLM in a carefully designed way would lead to improvement in factuality and better de-hallucination. Laser comes from the low-rank approximation of the MI.P layers, while Dola proposes factuality -aware algorithm. Based on their findings, hallucination occurs when parts of the weights aren't 't as well-versed in the given input as the other parts. They expose the "averaging" effect that blurs the factuality of the output. Removing these neurons gives rise to better facutality and less hallucination. Our studies look at the same problem from a neuron sparsity standpoint.
"Since et al. 2024 doesn't 't oppe-sooure its implementation and relies on the threshold for determining the sparsity pattern, replicating the method isn't straightffrward. Using a threshold also increases the difficulty of determining the actual density of the sparse model. Our implementation uses topk on the Gate Layer activations. The rest is implemented as described in the original method.

---

Table We show the difference between cases when Contextual Sparsity (CS) succeeds or fails. CS is generally good at prompt understanding tasks and tasks that measure the trustworthiness of the language models while not good tasks that require reasoning and world knowledge understanding.
Where CS Succeeds
CNN/Daily Mail
CoQA EM/F1
TruthfulQA Rouge-1/2 ACC
Experiment Settings
Unitxt Rouge
Ulama-3-B-Innstre
0.1237
0.6153/0.7822
0.4945/0.3647
0.1144
0.6633/0.799 7977
0.4725/0.3400
0.1166
0.6625/0.7988 7984
0.5043/0.3305
Llama-2-7B-Chan
0.1489
0.5982/0.7588
0.4480/0.3831
0.1448
0.61 11/1770.7666 7639
0.4529/0.3849
Maima-2-B-Chhat-Sppm
0.1521
0.5898/0.7544
0.4565/0.3660
Where CS Fails
GSM8K
HumanEval
MMLU
Experiment Settings
ACC (strict/flexible)
Pass@1 (GD)
Accuracy
Jaam-338B-Insttee
0.7551/0.7544
0.560
0.6231
Alama-3--t--nnttuuc--SSppppip
0.3859/0.3874
0.207
0.5558
JJama-3-88-Instrrc--SSnnmp
0.5868/0.5891
0.457
0.5304
0.2396/0.2462
0.140
0.492
Seama-2-7-Chhat-CSppa
031334/0. 1380 0.1979/0.2017
0.067
0.4637
0.134
0.4768
MMLU is classification task, not generation tasks. We use MMLU-FLAN-COT
Sparsity CNN/DailyMall
Sparsity (30%)
Humar
0.12
0.7
Sime-Girained
PaAMg1 Axermm may (Cready (lecllllgg
0.5
sime-Gralaed
Coarse-
0.10
0.6
Sparsity
0.3
0.08
0.4
0.2
0.04
Fine-Graiind Spursity
0.02
Coarse Grained sparsity
0.00
0.5
0.6
0.8
0.5
0.66
0.7
0.8
0.5
0.7
Non-Zaru Nurameter
Mon-ZZaro Parumtetee:
Non- Piruaseter
Figure 3: We contrast between Contextual Sparsity on prompt understanding task and complex (a) Both CSparse and FSparse robust on CNN/DailyMail for various sparsity; (b) and (c) Show that both CSparse and FSparse crash on GSM8K and HumanEval the global sparsity that they are still robust in prompt understanding tasks.
generation tasks that require reasoning.
3.2 Contextual Sparsity: Where Does It Fail?
On the other hand, contextual sparsity severely struggles when the generation tasks rely solely on the model's 's own reasoning and world knowledge understanding ability. Here we show the Llama-3-8Bi Instruct and the Llama-2-7B-Char models in Table refer to T2 evaluations more models. Notice that since fine-grained sparsity method needs the activation from Gate MIP selecting sparsity, while coarse-grained sparsity has predetermined pattern after prefilling and can sparsify the Gate MLP. Even though both are at 50% activation sparsity, the coarse gaasee-raiied sparsity method effectively achieves higher parameter savings than fine-grained sparsity in practice. Here evaluate the sparse techniques using 5-shot CoT on the GSMSK dataset (Cobbe et al., 202T). We found that across all the models we evaluated, both sparsity methods lead to significant accuracy degradation. We include HumanEval (Chen et al. 2021 I ) a coding task that requires complex reasoning and planning ability. We found that both sparsity methods exhibit similar performance degradation when it comes coding. Shown in F3 two tasks see sparsity significantly drop performance after 50% activation sparsity.
For knowledge recall and world knowledge understanding, we specifically test MML-U-Flan-ool (Chung al. (202)) the CoT text generation version of the MMLU dataset (Hendrycks et al.) 2021). Table shows the results. Stronger models like Llama-3- 3-Bna B---BB-ttuue suffer from significant
5

---

degradation too. Furthermore, we found that given the similar parameter size, the more well-trained the models are, the higher its degradation from the contextual sparsity, more details Appendix B.I
Queestion: Horic ordered ond chiikke mee, that casts $12.3
Quaetton:
Quaettin:: Raymand and Samantha are causinx, Raymond was burn ycuro before Bemandroo Raymond had a She agoo 29. if Bammorth is now 81, how sass flaymnnn's son bois?
packa of milk- the cavtte 53 excn, hoplee that coer S1 .00
coolt, and some biowes pleza, Hor paid a tossl of 850.
friends
emmilider farmerrs mariest dislly for fresh egg, How much
four,
ago
how many brows or pizca did Maris oscer it each bes costs (80 607
in delliars doss she make every day at the farmere maiser?
Answen Sementio is S1, so als 3f d yours old. Rayyand 22 23 years older than Samanita, zo 26 years old coe..
and Answer: toral ocot the of chiclonn 1.52772277
Anseeer [Spamee:
Macalcuaatin
Jon colta S 5- oodsed o- 2003 oggo pe day.
Unioononible Biatement
She lays 10-77 en 1i egga per day.
Question: A merch ant wantts to make a choice of pusc hase betweee purcihson planss juuwiryy worth 56,000 or clscvoric gedgets worth 58 $8,000. His financiu, advisor screcuatee that jeauttry marker willigh go up 2.50: while the onio godgets marker willinse 2% withinn the same month. merchart ls lookinng to moximbee profit profr
She sells 7 ocdi 2002 aggs per day.
Quastion:
How by
She makes 2 daddaii doliara per day. anert
a candle
Answer: candie will cantimtera shorser after te whhr burning 00 5:00 PM1
Answrer (Fuld)
centimaters
Jenet eats 3 wags and bakea 4 eaga, so ahe has 18 30:3-44
making
ec16 3-4-go00 egga left.
She swile eage for 57 wich, no she makees R
Remaries: There is conficcting statemmntt seasoning, loodingio
Answar: The jewelry will incresse 5,000 -0000000000 0.025 - $1250 (8126)
2.5% whhich
ocgr2-1 1-972-1850-16 dollaro.
result. Unroosonubll Stutement
Rinzaicutation
Offferend
Figure 4: Examples of contextual sparse model making the identified three different types of mistakes. Most mistakes occur because the model makes calculation mistakes or has a wrong reasoning step compared to the full model. We also observe that there are rare cases where the model makes insensible statements in the middle that make the end result wrong.
3.3 A Closer Look on GSM8K Quality Degradation
To study the inability of the sparse model in deduction, we conduct a case study on the sequence-level coarse-grained sparsity methods Dong et al. (2024) with the L-ama-3-8B-Insstn inspect extensive cases where the sparse model and dense differ in answers. Generally, the sparse model always produces highly similar answers to the dense model: the similar approach or logic flow when approaching the same problem and even the same number of sentences before the first mistake occurs or in success cases. However, the key differences are usually caused by the following three categories of small token-level mistakes: (1) frequent miscaleulation in the intermediate steps, (2) wrong reasoning in intermediate steps, and (3) insensible and random statements. For each of the sooe-summarized cases, we find failure question-
model. We visually
guestion-answer answer pairs provided in Figure 4 These mistakes
happen in the middle of arguments and propagate to the wrong end result.
Similar observations can also be found for fine-grained sparse methods with different model types. Interestingly, we find that even with these mistakes, the sparse model can still fully generate coherent tokens and make further reasoning assuming their prior steps are correct.
We hypothesize that the gap between the full model and these sparse counterparts is at these key tokens. The following simple experiment is conducted to further verify our hypothesis. We run the coarse coarse-grained sparse model and the full model with the same input prompt and for every token the sparse model generates, the full model is used to check the likelihood of these decoded tokens, mainly removing tokens with low likelihood. By varying the likelihood threshold, we can control the frequency of the correction. The experiments Bha
conducted for both 1aama---8B-Inntruus and et al. (2023) models with coarse-grained sparsity. The results are shown in Figure In both cases, we found that very small amount of correction would drastically improve the sparse model performance, showing a steep gradient when the percentage of corrected tokens is small. With merely 10% of tokens needing to be corrected, the sparse model can completely match the full model's performance. The experiment verifies our hypothesis that by correcting the small portion of key tokens, the sparse model can meet the large model's 's performance.
4 Methods
Though we find minor portion of tokens needed to be corrected for the contextual sparsity model to fully recover performance, the challenge remains: how to locate these mistaken tokens with the minimal number of parallel verification rounds of the full model? In this section, we show that the sparse model provides signals that cannot be trusted 4.1 Then, we describe in detail the various correction techniques i Because of the space limit, we put how to boost the sparse generation with hardware- tree building B.2

---

[Jamo-3 3. Instridd CSparse Strius
1Jam--3 1-8m--1-8R... Instrict PSparse Strius
0.7-
0.7
0.8
0.3
0.6
0.4
0.0
0.2
Sparse
0.4
(lep
0.1
Grtdfli Sizlus
Strtns
Strlos
Stries
0.0
0.3
0.5
0.6
0.7
0.8
0.75
0.00
0.85
0,90
Avorago Parameter Liand
Parameters Uind per tolorn generated
(e)
(b)
[c]
Figure 5: In (a), we present an example that illustrates why the signals from the sparse model are unreliable. It is figure plotting entropy versus generated tokens. At the tokens where the sparse made the mistake (red), the entropy isn't t in large spikes which signifies chaos and low confidence, rather it is even quite low, compared nearby entropy spikes. In (b) and (c), we view Sirius compression method by itself. We compare Sirius with contextual sparse methods and show that given the same parameter used, Sirius performs better than Contextual Sparse Methods on GSM8K.
4.1 Sparse Model's Self-Awareness Cannot Be Trusted
Intuitively, rather than fixing the Nsparse number, letting the system decide when to call the LLM for evaluation would then give more flexible Raporse- Nevertheless, we argue that the sparse model's output probability distribution cannot be used as metric for accuracy decisions. We empirically experiment with various methods to utilize the information contained in the sparse model's output distribution. However, varying the threshold leads to nsparse being too short when the threshold is strict or failing to correct when the threshold is lenient. We then discovered that the sparse model has very limited self- awarenssssss of its own mistakes. To make the observation concrete, a small example in Figure [ piece of text where the sparse model makes a mistake while the full model succeeds. The red bars signify the error location. The token entropy is neither high nor zero, making it impossible to effectively use a threshold to control the number spparse
present
4.2
How to Correct the Sparse Output Tokens
The full overview of SIRIUS is presented in Figure Algorithm I The full model is called once every kernel size. The KV cache is shared between the sparse and the full model. The KV cache is mostly populated by the sparse model, which is called for every token. During correction, the full model takes in the last kernel size of tokens and generates its KVs for the past kernel size tokens in parallel, these KVs are directly written to their corresponding positions in the shared KV Cache. Empirically, we found that full model's KV helps the sparse model's output. When LLM is called to evaluate the sparse model's output, it uses its own predicted likelihood to determine whether to accept or reject the sparse model's 's past output. The decision is based on comparing the likelihood against preset threshold. Detailed ablation for threshold is in B.3 Besides the thoov-menniones techniques, we also found that the "sccond/third" choices of the sparse models' rejected token position offer > 80% coverage of the LLM accepted tokens. The observation motivates us build hardware-friende tree on the sparse model generating side that doesn't 't sacrifice the performance while significantly boosting the nALL efficiency. Due to the space limit, great amount of details is in Appendix B.2
Experiments
In this section, we empirically evaluate SIRIUS to correct CS models on various generation tasks in complex reasoning. We show that SIRIUS is consistent in various tasks, effective in helping CS models recover their performance, and efficient in correction with low additional overhead.
In 5.1 we evaluated SIRIUS on six models with 8 different datasets. SIRIUS is consistently effective and efficient. Specifically, on GSM8K, SIRIUS corrects FSparse Llama-3-8P 3-8--unttt from 58% accuracy to 72% with only 4% increase in parameter density and corrects CSparse model from 38% to 70% with 5% density.
5.2 we presents more details on our system implementation for SIRIUS. We show that SIRIUS delivers its theoretical efficient promise, achieving roughly 20% reduction in latency compared to full on-chip on various hardware. SIRIUS further achieves 35% speedup to full in offloading settings.
7

---

Algorithm 1 Sirius
Require: Prompt [et full model Mr, and sparse model Ms sharing weights and sharing KV
Cache C, eache__pe the location where new k and are written C, kernel size
Require: forward function FORWARD, threshold r, which is a value used by Mr to judge whether the token occurs likely enough
Require: StoppingCriteriaMet)) downstream task-specific, returns a boolean
while not StoppingCrilriaMMett do
2:
i<-0
3:
kernel povneleemmp empty
4: eache_pos +-
5:
i
6:
set FORWARD C, (el, I. Teti 1 cache_pos
7:
RRnning sparse model
8:
cache_pos +- sache_pos+t
9:
sample Ziti
10:
kernel 4 (4)( i+i+1
11:
>> Before exiting, kernel (ttnnn
12:
end while
13: cache_pos + cache cascheppssscceeeeeeeeeeeeee subtracts n
Enables Full to directly rewrites KV Cache
14:
set (4.....9+1.. FORWARD(MccCSs CO cache_ pose kernel)
15:
for j from 0, do
if (i+t <r then break end if end for
Full rejects
17:
b. j stores the first token position being rejected
18:
19:
cache_pos pos + j+1
Rollback
21:
kernel+ + empty
sample I+ts+1* ~ Pe+j
Interleaving Key Token
23:
end while
We also present ablation with rich details on how each component SIRIUS contributes to its performance and how threshold is used to trade off efficiency and performance. Due to space limit, we place it in Appendix[B.. B.3]
5.1 Sirius Significantly Recovers CS Degradation with Low Cost
To comprehensively evaluate SIRIUS performance, we deploy six mainstream LLMs with sizes ranging from 7B to 13B: Llama-2-7B, 13B, and Llama-3-8B with their instruction finetuned counterparts, all from Llama family. Following Wei et al. (2022) L.LM reasoning, we also tested CS models on two popular types of reasoning generation tasks: arithmetic and commonsense reasoning. On the Arithmetic side, besides GSM8K, we also evaluate CS models on AQuA-RAS On the Common Sense side, we use CSQA Saha et al. (2018), StrategyQAGevv et al. (2211. Date, and Sports, last two from Big Bench Suite bench authors (2023). Most of these tasks are ori originally classification tasks. Following the instruction in Wei et al. (2022), manually compose COT prompts to transform these into logic argument generation tasks. Besides, CS models do not perform well in coding, which requires forming logical arguments and planning. We select HumanEval Chen et al. (2021) and MBPP+ Liu et al. (2023a) to evaluate SIRIUS.
Models and Datasets-
For arithmetic reasoning and coding, we use 50% neuron sparsity for both CSparse and FSparse. FSparse relies on the gate layer to be dense, leading to higher global density than CSparse. Since commonsense reasoning tasks are generally less logically challenging comparatively, we lowered the neuron sparsity level to 40%.
Main Results- Due to space limits, we only select the best treewidth of SIRIUS for GSM8K, CSQA, and HumanEval for the main results in Table Extensive studies on the rest datasets with different treewidth are presented in the Appendix C From Table 2 we can see that SIRIUS is consistently effective and efficient across all different classes of tasks. Specifically for GSM8K, SIRIUS corrects FSparse and CSparse, CSQA, from 61% % and 64% accuracy to 70% with cost only 3% sparsity for FSparse and 7% for CSparse respectively. On HumanFval, SIRIUS corrects FSparse from 45% to 61% % with 4% sparsity overhead even surpassing the full model's performance,
1ama----B-Innsruuce besides

---

Table We show STRIUS effectiveness and efficiency in the following table. We select GSM8K for Arithmetic Reasoning, CSQA for Commonsense Reasoning, and HumanEval for code generation. Under the "SIRIUS Perf. column, A(B) is shown. A denotes the accuracy after SIRIUS correction in the dataset evaluated, while (B) represents the optimal treewidth selected under the current model dataset settings. Under the column of "AAL", X/Y shown, where X is the AAL, while Y the period.
GSM8K
CSparse Density
Effective Density
Model
Full Perf.
CSparse Perf.
SIRIUS Perf.
AAL
Llama-3-8H-Instmen
0.7536
0.3844
0.65
0.7051 (8)
15.22/16
0.706
Llama-3-8B
0.4966
0.2085
0.65
0.4177 (8)
15.29/16
0.703
Llama-2-7B-CUl
0.2403
0.1334
0.69
0.2244 (8)
15.00/16
0.757
Llama-2-7B
0.1357
0,0758
0.69
0.1183 (6)
15.87/16
0.715
Alama-221331 3hBtt
0.3548
0.2714
0.68
0.3381 (4)
15.34/16
0.730
Llama-2-11
0.2282
0.1759
0.68
0.2418
15.34/16
0.730
FSparse Density
Effective
Model
Full Perf.
ESparse Perf.
SIRIUS Perf.
AAL
Density
Llama-3-8B-Instiee
0.7536
0,5868
0.76
0.7278 (4)
15.37/16
0.807
Llama-3-88
0.4966
0.3199
0.76
0,4579 (2)
15.03/16
0.825
Llama-2-7B-Chn
0.2403
0.1971
0.79
0,2388
15.69/16
0.819
Llama-2-7B
0.1357
0.1137
0.79
(.1410
15.91/16
0.807
(lama-2-13--C--
0.3548
0.3222
0.78
0,3533 (1)
15.08/16
0.842
Llama-2- Hama-22133B 1-B
0.2282
0.2191
0.78
0.2372 (4)
15.92/16
0.797
CSQA
CSparse Density
Effective
Model
Full Perf.
CSparse Perf.
SIRIUS Perf.
AAT
Density
Jlama-3-88-Instmee
0.7073
0.6470
0.58
0.7076
14.76/16
0.657
Llama-3-8B
0.6437
0.5585
0.58
0,6429 (8)
15.43/16
0.628
Llama-2-7B-CC
0.6248
0.5200
0.62
0.6175
15.07/16
0,683
Llama-2-7B
0.4742
0.4414
0.62
0,4742 (8)
15.80/16
0.652
Llama-2-11 2-13B-Chat
0.6879
0.5536
0.61
0.6691 (4)
11.43/12
0.674
Llama-2-13B
0.6109
0.5601
0.61
0.6060 (4)
15.72/16
0.645
FSparse Density
Effective
Model
Full Perf.
FSparse Perf.
SIRIUS Perf.
AAL
Density
Llama-3-8P 3--ma-3--BIIsttuee
0.7073
0.6158
0.72
0.7043
15.66/16
0.753
Llama-3-8B
0.6437
0.533
0.72
0.6388 (1)
15.00/16
0.786
Llama-2-7B-Char
0.6248
0.6167
0.75
0.6380 (4)
15.09/16
0.811
Llama-2-7B
0.4742
0.4717
0.75
0.5012 (6)
15.89/16
0.771
Llama-2- 2-BB Chaa
0.6879
0.533
0.74
0,6691 (4)
14.30/16
0.846
Llama-2-
0.6109
0.5700
0.74
0.5864 (4)
15.72/16
0.770
Human
CSparse Density
Effective
Model
Full Perf.
CSparse Perf.
SIRIUS Perf.
AAL
Density
Llama-3-8B -nsssrrruut
0.561
0.207
0.65
0.524 (8)
14.67/16
0.733
Llama-3-8B
0.262
0.067
0.65
0.243 (8)
15.10/16
0.691
(lama-2-7B-Chi
0.140
0.067
0.69
0.159 (8)
10.88/12
0.789
Alama-2-78
0.116
0.079
0.69
0.128 (8)
14.84/16
0.765
Alama-2-133 -hatt
0.189
0.122
0.68
0.171 (8)
11.12/12
0.762
Llama-2-13B
0.262
0.067
0.68
0.244 (8)
15.10/16
0.741
FSparse Density
Effective
Model
Full Perf.
FSparse Perf.
SIRIuS Perf.
AAL
Density
Llama-3-81B-Instmncc
0.561
0.457
0.76
0.616 (6)
15.42/16
0.804
Llama-3-8B
0.262
0.189
0.76
0.298 (6)
15.54/16
0.797
Llama-2-7B-Cha
0.140
0.134
0.79
0.165 (6)
15.27/16
0.841
Llama-2-77
0.116
0.116
0.79
0.165 (6)
15.86/16
0.810
Llama-2- 2-13B Chat
0.189
0.146
0.78
0.183 (6)
15.34/16
0.827
Llama-2- 1-1B-2-13B
0.246
0.233
0.78
0.259 (4)
15.85/16
0.801

---

and from 20% to 52% with 8% sparsity as cost. Besides, Llama-3-8P ---n-----B--- --Intruutt,,, SIRIUS corrects all 6 models with additional sparsity overhead smaller than 10% across these three datasets, further showing its strong efficiency. Besides results in Table 2 in Appendix C we show that STRIUS consistently shows great effectiveness with high efficiency across the rest of the datasets.
5.2
Wallclock Speedup
Table 3: Performance and Speedup Ratios on GSM8K-COT with Different Hardware Configurations.
Settings
ACC
A40
Ratio
L40
Ratio
A100
Ratio
H100
Ratio
CSparse 0.3601
20.7 ms
0.66
15.6 ms
0.67
9.6 ms
0.72
6,6
0.76
Sirius
0.7127
24.1 I ms
0.78
18.2 ms
0.78
11.1 ms
0.83
7.7 ms
0.88
Full
0.7612
30.9 ms
1.0
23.2 ms
1.0
13.3
1.0
8.6 ms
1.0
Here we show that Sirius delivers its promised efficiency claim for on-chip and offloading settings. Because the -ra-graind sparsity [ee al. (2024) relies on a custom CUDA kernel achieve the target generation speedup not open-sourced, we focus on coarse- coarse-grained sparsity on GSM-8K COT, and the input sequence length with average prefill length 900.
Firstly, we consider the on-chip setting run- ning -lama-3-8B-Instruuu
single GPU. The sparse model (APU 0.65) achieves 36.01 % ac- curacy on GSM8K-CTT, while the full model achieves 76.12% accuracy on GSM8K-COT. With kernel size 10, SIRIUS achieves 0.74 APU with accuracy 71.27% accuracy. We use torch compile to optimize the inference latency and limit the overhead other than running model in- ference. The average latency generated per token used to compute latency. Results are shown Table 3 On average, SIRTUS delivers the promised latency reduction from APU calculations. The speedup ratio on A40 and L40 closely aligns with the theoretical APU reported. On the other hand, A A100 and H100 compute MLP more efficiently than it compute attention, making the latency ratio between computing MLP and attention not perfectly aligned with their ratio in parameter size. Therefore, we that even the sparse model bascline has slightly higher latency as expected. We increase the kernel size from 10 to 16 for these two devices, where accuracy reaches 0. 0.7089 and the AAL reaches 13.67. For A100 and H100, building hardware-efficieen tree is nearly free of cost and highly effective. For numbers in Table 3 we use the width 4 tree that boosts the AAL to 15.01 out of 16. More details are in Appendix B.2
Table 4: 1amma-3-70B-Instrucc with Offloading.
Settings
Sparse
Sirius
Full
Performance
0.7407
0.8719
0.9014
Latency (8)
3
3.68
3.72
Ratio to Full
0.6241
0.6434
1.0
Secondly, we consider the offloading setting which is the only way for resource-limited users to run 70B models by loading only the weights in use to GPU memory, while the others are offlo to the CPU. Results are shown in Table We use a single L40 48GB with a PCIc bus bandwidth of 25 GB/s to run Llama-3-70B Instruct with batch size 1. Llama-3-70BB 7 has roughly 80% of parameters to be MLP, which gives the theoretical APU for Griffin to be 0.6. Sparse + Sirius gives 0.649 APU, which is roughly what our system achieved.
6 Conclusion
We observe that contextual sparse methods significantly degrade reasoning and deduction tasks. SIRIUS, an efficient correction mechanism, enables accurate LLM inference with contextual sparsity. With roughly 11% to 18% sparsity increase, SIRIUS improves fine-grad grained and coarse-grained sparsity significantly in their performance while maintaining their efficiency gain.
Further, SIRIUS is still relying rollback to correct the tokens that are deemed unlikely, which inefficient. On the other hand, making the weak-strong model synergy systems that match the performance of the strong while keeping the efficiency of the weak, without strictly matching the strong models' output distribution remains an interesting and unsolved problem. We leave these topics to future works.
10

---

7 Acknowledgement
We would like to thank Feng Liang and Yunong Liu for their helpful feedback during the exploration and writing. We also want to give special thanks to Hanshi Sun for providing insights and suggestions for efficient implementation and speedup.
References
bench authors. B. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions Machine Learning Research.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L.,, and Jumper, J. (2023), Accelcrating large language model decoding with speculative sampling.
Chen, M., Tworek, J., Jun. H., Yuan. Q. Oliveira Pinto, P. Kaplan, J. Edwards. H.,, Burda, Y., Joseph. N., Brockman, G., Ray. A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder. N.. Pavlov, M., Power, A., Kaiser. L.. Bavarian, M.. Winter, C., Tillet, P., Such, F. P., Cummings. D., Plappert, M., Chantzis, F., Barnes, F., Vosss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S. Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, Achiim, J., Misra, V. M.Mooikww,, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code.
Chuang, Y-S., Xie, Y., Luo, H., Kim, Y., Glass, J., and He, P. (2024). Dola: Decoding by contrasting layers improves factuality large language models.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Wehson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., language models.
Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. (2022). Scaling cestruction-fineture
Cobbe, K., Kosaraju, V. Bavarian, M., Chen, Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers solve math word problems.
Dong, H., Chen, and Chi, Y. (2024). Prompt-prompted mixture of experts for efficient Ilm generation.
Frantar, E. and Alistarh, D. (2023). Sparsegpt: Massive language models can be accurately pruned one-shot.
Freitag, M. and Al-Onaizan, (2017). Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01880
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. (2021). Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL).
Hendrycks, D., Burs,, C.,, Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding.
Kaplan, J., McCandlish, S., Hemighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.0836.
Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M. W., Gholami, A., and Keutzer, K. (2023). Speculative decoding with big little decoder.
LeCun, Y., Denker. J., and Solla, S. (1989). Optimal brain damage. Advances neural information processing systems, 2.
Lee, J.-Y., Lee, D., Zhang, G., Tiwari, M., and Mirhoseini, A. (2024). Cats: Contextually-aware thresholding for sparsity in large language models.
Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding.
Li, Z., You, C., Bhojanapalli, S., Li, D., Rawat, A. S., Reddi, S.J. Ye, K., Chern, F, Yu, F, Guo, R., and Kumar, S. (2022). The lazy neuron phenomenon: On emergence of activation sparsity in transformers.
Lin, S., Hilton, J., and Evans, O. (2022). Truthfulga: Measuring how models mimic human falsehoods.

---

Ling, W., Yogatama, D., Dyer. C., and Blunsom, P. (2017). Program induction by rationale generation: Learning solve and explain algebraic word problems. arXiv preprint arXiv:1705.04145
Liu, J., Xia, S., Wang. Y., and Zhang, L. (2023a). Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems.
Liu, Z.. Wang, J., Dao, T., Zhou, T., Yuan, B., Song. Z.. Shrivastava, A., Zhang, C.,, Tian, Y., Re, C., and Chen, (2023b). Deja vu: Contextual sparsity efficient Iims inference time.
Ma, X., Fang. G., and Wang, X. (2023). Lim-pruner: On the structural pruning of large language models.
OpenAI, Achiam, J., et al. (2024). Gpt-4 technical report.
Raffel, C., Shazeer, N., Roberts, A., Lec, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. (2020). Exploring the limits of transfer learning with unified text-to-text transformrr. trntorrre.. Journal of machine learning research, 21(140):1-67.
Reddy, S., Chen, D., and Manning, C. D. (2019). Coqa: conversational question answering challenge.
Saha, A., Pahuja, V., Khapra, M. M., Sankaranarayanan, K., and Chandar, S. (2018). Complex sequential question answering: Towards learning converse over linked question answer pairs with knowledge graph.
See, A., Liu, J., and Manning, C. D. (2017). Get to the point: Summarization with pointer-gencrator generator networks.
Sharma, P., Ash, J. T., and Misra, D. (2023). The truth is there: Improving reasoning in language models with layer-selective rank reduction.
Song, Y., Mi, Z., Xie, H., and Chen, H. (2023). Powerinfer: Fast large language model serving with consumer- grade gpu.
Sun, H., Chen, Z., Yang. X., Tian, Y., and Chen, B. (2024a). Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. (2024b). A simple and effective pruning approach for large language models.
Team, G., Anil, R., et al. (2024). Gemini: A family of highly capable multimodal models.
Tibshirani, R. (1996), Regression shrinkage and selection the lasso. Journal of the Royal Statistical Society
Series B: Statistical Methodology, 58(1)2667-228.
Touvron, H. Martin, L., Stone, K., Albert, Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D.. Blecher. L.,, Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu,
W. Fuller, B., Gao. C., Goswami, V., Goyal, N., Hartshorn, A.. Hosseini, S.. Hou, R., Inan, H. Kardas, M.. Kerkez, V., Khabsa, M., Kloumann, I., Koreney, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Loe, J., Liskovich, Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E, M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M.. Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. (2023). Ilama 2: Open foundation and fine-tuned chat models.
D., Lu, Y., Mao, Martinet, X., Mihaylov,
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Lc, Q V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 24:24824-24837.
Xia, M., Gao, T., Zeng, 7.,, and Chen, D. (2024). Sheared Ilama: Accelerating language model pre-training via structured pruning.
Zhang. S., Roller, S., Goyal, Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li. X., Lin, X. V., Mihaylov, T., Out, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S.,, Sridhar, A., Wang, T., and Zettlemoyer, (2022). Opt: Open pre-trained transformer language models.
12

---

Appendix
Appendix Table of Contents
A. Additional Background
Extended Related Works
Why Not Using Speculative Decoding to Correct the Sparse Model?
B. Supplemental Experiments
Given Similar Parameters, More teal-t Parameters Sufters More
Hardware-Friendly Tree Building Process
3.
Ablation: Various Aspects of STRIUS is Tested and Challenged
4.
Large Model Experiments
5.
Variable Sequence Length with Batch Size One
Error Occurs at Which Position inside a Chunk
Miscellaneous Results
8.
Llama-2 and Llama-3 Models on GSM&K-CO1
C.
Additional Results on Reasoning
1. Arithmetic Reasoning
CommonSense Reasoning
3. Code
A Additional Background
A.1 Extended Related Works
Pruning in LLM Sparsity in neural networks has been widely studied. In the context of LLM. sparsity studied under two branches unstructured and structured. On the unstructured sparsity side, Frantar and Alistarh 2023) (SparseGPT) is a ground- boenddng work that formulates pruning as a sol ooinngg series of sparse regression problems and proposes a fine solver for the problem. [Sun
(2024b) (Wanda) introduces input activations into the pruning decision and achieves strong results in inducing LLM sparsity. On the structured side, LLMPruner Ma al. (2023) and Sheared Llama Xia al.) (2024) each proposes different meticulous pruning algorithms and restoring weights through either parameter-efficient finetuning or efficient full weights training.
Contextual Sparsity Many recent works on LLM sparsity notice that the sparse pattern is highly related to the input or context. Deja Lu et al. (2023)) revealed that for OPT models Zhang et al. (2022) the contextual sparsity is as high as 85%, meaning that 80% of the parameters can be pruned that won't hurt the token decoded quality given the prompt. Deja Vu formulates the problem of neuron selection as a neir-neighbor search problem: finding neurons that are the most similar to the input activations. PowerInferSSong et al. (2023) extends the contextual sparsity to benefit the heterogeneous setting. Compared to the rest of the model, MLP layers tend to possess significant contextual sparsity and can be effectively exploited in training-free free manner. Concurrently, Griffin GriffinDong et al. (2024) discovers the phenomenon of flocking, where MLP neurons have temporal locality, where given fixed prompt, similar neurons tend to get activated throughout the following generation. Flocking is shown to occur in most activation types and apen-source L.MMs. Griffin selects the same set of heated neurons with 50% sparsity throughout the generation of each input prompt, which we refer to as coarse-grain grannne sparsity. CATS Lee et al. (2024) successfully exploits por-token contextual sparsity in the MLP layers for inference latency reduction. They resample a new set of neurons per every new input token, which we categorize it as fine-grained contextual sparsity. Our paper mainly focuses on the training- free MLP sparsity techniques. Although these recent works show minimal accuracy degradation in classification and easy text summarization tasks, they both severely degrade in generation quality under tasks that require high-level reasoning and understanding ability. Our work serves as a low-cost complementary tool, aiming to push these elegant and promising techniques for mainstream use cases.
Also, previous contextual sparsity methods haven't 't fully and exhaustively evaluated their benefits and limitations in downstream generation tasks. To fully study this technique, extensively go through
13

---

open-source LLMs in diverse performance and sizes on diverse generation tasks and datasets to locate where these sparse models maintain the performance or fail.
Speculative Decoding Besides model compression techniques, Speculative decoding I Leviathan et al. (2023). Chen al. (2023), Kim et al. (2023) is another important LLM inference latency reduction method. Compared to LLM, small transformer models are much more computationally accessible and can effectively model short-range tokens. Therefore, smaller models are asked to speculate short-term future tokens, which the LLM takes in in parallel to trade in FLOPs with memory loading time. During verification, most speculative decoding methods pursue lossless acceleration, leading to frequent rollback during rejection. In contrast, Sirius solves a very different problem. Our method aims to maximally preserve the efficiency of sparse models while boosting its performance. Sparse models, pruned directly from LLM, are much stronger at modeling a longer range of text than draft models, thus requiring much less help from the L.M.M. Our work aims find the minimum amount of L.LM overhead while boosting its performance to the LLM level. Given the resemblance and relevance of Speculative Decoding to our method Sirius, we will elaborate more in-depth on their differences and Speculative Decoding's inefficiencies when it comes to helping the Sparse method in A.2
A.2 Why Not Using the Speculative Decoding to Correct the Sparse Model?
When Speculative Decoding is used to correct sparse using the full model, we will show that the efficiency of the overall process will be largely limited. We followed the common practice from speculative decoding and measured the accep- tance rate on different datasets C4 Raffel et al. (2020) and GSMSK Cobbe et al. (2021). Take the Coarse-grained sparse model as example. For Llama-3-8B as the full model, the 50% sparse acceptance
Speculative Decoding Efficiency
3.0
2.5
7.99
Averaage Advance Lengtt
Effective Density
2.0
1.93
4,97
1.60
1.5
(APU 0.65) model will produce rate of 0.71 on C4 and 0.89 on GSM8K. Specu- lative decoding also parallel verification in the period-basie Naturally, to keep the system efficiency high, we need to (1) enlarge the period and (2) increase the average number of tokens ac- cepted (AAL) given the gamma (period- 1) value. Take the acceptance rate of 0.89 on GSM8K as an example, following the formulation in Leviathan ct al. (202)), we can calculate the expected number of accepted tokens for every gamma term in the
2,68
L.
Full Model APU
0.55
Sparse Model APU
16
Gamama
Figure 6: Speculative Decoding has limitation in efficiency when correcting sparse models.
Speculative Decoding literature. AAL= = 1 1)
The trend (green) is plotted in Figure
I-a
We can notice the trend that the average advance length starts to plateau the gamma becomes larger. Take the gamma of 16 as an example, the period is then 17. The average advance length is only 7.84.
The APU is (16* 0.65 + 1)/7.84 = 1.45, which is larger than the full model 1. The blue line in Figure shows the relationship between APU and gamma.
Because of the plateauing effect, for an acceptance rate of 0.89, the best gamma is 2 (period= = 3). The optimal APU is 0.86, compared with 0.65 grarsee sparse APU, A similar picture can be applied to Fine-grained sparsity as well. The key reasons for the observation are two-fold: (1) the contextually sparse models are too big to be the draft model of the speculative decoding system and have large period: (2) Speculative decoding preserves the original model's performance so that the acceptance criteria are usually very strict, which is also not suitable for large period and high average advance length. Following the same spirit, Dpirit,SSun& et al. (2024a) also uses a large draft model to do self-speculation. but for them, the authors select gamma SIRIUS brings <0.76 APU in this case with period 2 10. Specifically, with a threshold of 0.1, Sirius can correct Llama-3-8B coarse-grained sparsity from 20.85 85% to 43.9%, compared to the 49.66% full model. With period of 16 tokens (gamma- = 15), Sirius average can accept 13.4 tokens out of kernel size of 16 and over 9 tokens out of a kernel size of 10, translating to APU < 0.76, significantly lower than SD does.
to achieve the optimal speedup of their system. In contrast,
14

---

B Supplemental Details in SIRIUS observations, Design, and Experiments
In this section, pro- vide several supplemental experiments to the picture. First, we run SIRIUS on Llama-3-70B. However, be- cause of computational lim- its, cannot SIRIUS with the tree on Llama-3- 70B with the scale did for other models. Neverthe- less, we do show that 70B has roughly the same pattern as have seen before, large model sparsity also some- how struggles on reasoning tasks. Second, we provide additional proof for the par- allel verification efficiency statement. After that, show results on where the error is located in the chunk size of 16 tokens. The eгror is dis- tributed almost uniformly. Last but not least, we also apply SIRIUS on datasets that are reasoning. Lastly, provide more results the comparison between models of similar size but have huge performance gap. We show that given the similar parameter size, the trend is for a more degrade from contextual sparsity. We present here more illustration contextual sparsity and -lma----BB-hhe experiments with fixing only a minor portion of tokens is shown in Figure
Prompt Understanding Tasks
Connecting CSparse Meta-l Lamm- 7B-Chet
Ressoning Tasks
aejected sperse
0.13
0.16
Takare
Nerformance with Correction
0.14
Nodal Score
Neuron Index
(a)
(b) Persntage Correctod Tokera
1.0
Figure 7: (a) Illustration on why Contextual Sparsity has uneven per- formance on different tasks. The activation heat map (red) has the brighter the color the larger in magnitude. On top, also show the neuron sparsity selected. The graph points signify that the pattern the prompt understanding task easier capture. (b) An additional graph of correcting Csparse Llama-2-7B-CC experiment on 8B. Only 10% tokens being corrected results in complete performance recovery.
is similar to the previous
powerful model
B.1 Given Similar Parameter Size Well-trained Models Suffer More
We observe another interesting phenomenon; given the similar parameter size, the more well-trained the model is, the more performance degradation contextual sparsity would make on the full models. Here present two pairs of results. First, we look at the performance between Llama-3-8B-Instee and Alamma-2-7B-hha with Llama-3-70B-Instee on GSM8K-CTT. We draw these models in CSparse in Figure g and the readers can find more results in Appendix B.8 We can see figures from top to bottom, where even at lower density (more elements not selected), Llama-2-7B- 7B-Chtt and Llama-2 20B Chat suffer from less performance degradation (blue) compared to the Mamma-3-B-- B-n-t---uttnnt and Llama-3-77 3-BB 70B-Instruce models. Furthermore, suppose focus on 1lama-3-70B 70B-Inntruct for global density 60% lower. In that case, the performance (coral) degraded significantly. which is comparable or even lower to Lama-3-8B-Instruus full model performance at 0.76, Even 50% density, the 70B model still has more than 40B parameters, much more expensive than the 8B model. The observation fully manifests the difficulty of using CS in complex reasoning tasks.
and Llama-2-4 2--02--77--h--- All models are evaluated
B.2 Hardware Friendly Tree Building Process
In this section, we first look at the insights behind whether building the tree can help efficiency, then we detail the specific steps towards tree pruning.
Table 5: The second and third most likely to- kens from sparse models offer potential for boosting efficiency.
The goal for the Sirius system is to make nAAL to be large as possible. Despite the full model sharing KVs with the sparse model, Sirius still encounters costly rollbacks because of sparse greedily decoded tokens being rejected. Interestingly. we look closely into where the sparse model likely to make mistake
Sparsity
2nd Hit
3dd Hit
Miss
Coverage
ESparse CSparse
79%
11%
9%
90%
65%
17%
16%
82%
on GSM8K and AQUA-RAT-COTIE
et al. (2017) with Sirius 1lma--38BB- SB-Insttuct and a kernel size of 16. More details are shown Appendix|B.6] The error distributes almost uniformly across all positions of the kernel size. Also, when the token makes the mistake, besides the greedily decoded
15

---

CSparse Llama3-83-In
CSparse Llama3-70B-In
Degradiation
0.9
Degradation
0.7
0.7
0.7
0.6
0.36
0.6
0.8
A61
0.6
Striel Aoburayy
Striet Accuracy
0.5
0.3
0.1
0.30
0.6
0.4
12
0.3
0.3
0.3
0.5
0.2
0.2
0.2
0.1
GSMBK ACC
GSMBK ACC
0.3
0.52
0.55
0.58 Deasity
0.61
0.64
0.45
0.50
0.55
0.60
Density
CSparse Llame2-7B-Chat
C.parse L'ama2-70B-Chat
0.18
Degradation
0.450
Degradation
0.425
0.6
0.6
Striet Arourany any
Acmuracy
0.400
Detradattin Costadaation
0.3
0.14
0.373
GSMBKK ACC
0.4
0.4
0.12
0.350
0.325
0.2
0.2
0.300
0.11
0.08
0.08
0.03
0.275
GSMBK 0.00
0.0
0.0
0.525
Deasity
Deusity
Figure 8: Given the similar model parameters, the well-trained the model is, the worse the degradation would be. (Compare the figures vertically between Llama-3 and Llama-2 family models).
tokens, we find that other tokens of lower likelihood offer the potential boost efficiency. Surprisingly, we found that out of the cases where the greedily decoded tokens are rejected, the probability that the second third most likely tokens from the sparse being accepted by the full model is reasonably high.
Shown in Table we on part of the GSM8K rejected cases. The "Second Hit" is defined as the count of the sec- ond most likely tokens being accepted by the full model when the greedily decoded token is rejected, while the "Third Hit" is defined as the count of the third most likely token being ac- cepted when the first two are rejected. Both sparsity method has a high ac- ceptance rate, or "Coverage", from the second and third most likely tokens when the most likely token is rejected, showing huge potential for gains ef- ficiency.
Drafting end Fittering Process for nth round
0 Round (token from Modell
1s Round
2d Round
(2) Computes
for likelihood
Will
Have
Has
Need
Kim
(3) Pick Top-k tokens and discard the keep branches
[Space]
[1) Each drafts
needs
children
Abrenches each round is fixed at treewidth (k), tree building continues
Figure Illustration of Tree Building Process.
To capitalize the potential from the second third tokens, we propose build a tree during the sparse generating process (lines 6 to 11 in Algorithm[] [ The tree algorithm is similar to Beam Search Freitag and Al-Onaizan (2017). However, to make sure that the tree building and tree parallel correction processes can achieve speedup over cases that don't 't build trees, we impose strong restrictions on the tree structure we build. For fixed kernel size, we limit every step to having a fixed number of leaves, or treewidth, through tree pruning based on ranking the cumulative log-likeling likelihoooo of the path. The resulting tree has fixed shape for given kernel size and tree width, but only the interconnection pattern between steps varies based on the pruning and ranking within each step. The details are illustrated in Figure During verification, out of the treewidth complete paths, we select the one that reaches the longest advance length. In practice, we found that for kernel size 16, when the treewidth is increased to
16

---

8, the optimal verification tree is around 64. From Section A.2 the tree of 64 roughly equals the time the full imput 1 token.
we see that the parallel verification of
Therefore, a treewidth of 8 is set as the maximum treewidth when building the tree for kernel size 16 for later. We show that building correction effects.
makes the system significantly more efficient while retaining
Table 6: Performance and Speedup Ratios on GSM8K-COT with tree building, latency measurement in millisecond.
Settings
ACC
A100
Ratio
H100
Ratio
CSparse
0.3601
9.6
0.72
6.6
0,76
Sirius (No Tree)
0.7127
11.8
0.88
8.2
0.95
Sirius (With Tree)
0.7089
11.1
0.83
7.7
Full
0.7612
13.3
1.0
8.6
1.0
In T we present more details complementing Ta the use of trieding for A100 and H100. We ablate the use of a width 4 tree by not using the tree under kernel size 16. We observe that the speedup improvements solely from the increase in AAL from 13.67 to 15.01 for kernel size 16.
B.3 Ablation: Various Aspects of Sirius Are Tested and Challenged
Table 7: Ablation on Components in Sirius.
CSparse
GSM8K 20%
FSparse
GSM8K 20%
Llama3-8B-Instruct
0.7538/0.7538
Mamma--B-Instruus
0.7538/0.7538
+ CSparse
0.3674/0.3674
+ FSparse
0.564470.5644
+ CSparse + Interleave
0.3826/0.3822
+ FSparse + Interleave
0.6288/0.68 62288
CSparse KV Rewrite
0.4735/0.4735
+ FSpparse KV Rewrite
0.6629/0.6299
+ CSparse + KV Rewrite
+ FSparse KV Rewrite
0.4886/0.4886
0.6780/0.6818
+ Interleave
+ Interleave
+ CSparse Roll back
0.6591/0.6591 + FSparse + Roll back
0.7273/0.7273
+ Interleave
+ Interleave
+ CSparse KV Rewrite
F'Sparse KV Rewrite
0.6667/0.6667
0.7273/0.7311
+ Interleave + Rollback
+ Interleave + Rollback
Probing Components To understand the contribution and the utility of each component of Sirius, we ablate components of Sirius in T We started by only letting the LLM correct the token it is evaluating (interleaving only). Then, we add on top of it the KV cache correction, and then the rollback. All these three techniques are effective when applied solely. Rollback seems to the most effective technique. Even when applied alone, rollback asserts significant correction to both the CSparse and FSparse models. Interestingly, KV Cache is also effective alone, bringing a 12% increment for CSparse and an 1 11% accuracy increase for FSparse. Relatively, interleaving is the weakest. Surprisingly, adding both KV rewriting and rollback is only marginally better than rollback alone. Although it is tempting to think KV Cache rewriting is not useful with rollback, the improvement KV Cache Rewriting brings a gain in efficiency. When adding the KV Cache Rewriting on top of Roll Back and interleave significantly improves the efficiency of the correction. For CSparse, adding KV rewrite increases AAL from 12.77 to 13.80.
Table 8: Ablation on the threshold for correction (FSparse Llama- (-8B--nntruct)
Threshold
Full
Sparse
0.05
0.1
0.3
0.5
0.7
0.9
Accuracy
0.7803
0.5884
0.7247
0.7399
0.7399
0.7677
0.7702
0.7652
AAL
N/A
N/A
15.2
14.6
11.6
8.5
6.2
4.2
I Likelihood threshold to balance Correction and Efficiency We found that the likelihood threshold important for managing the Sirius correction and efficiency tradeoff. We present results in Table We ablate this setting 30% subsampled GSM8K dataset, and only strict accuracy is reported. The
17

---

performance is the score, while the efficiency is measured by Average Advance Length (AAL). We can find that with the icreasee threshold, the scores generally improve, while the efficiency metric decreases.
Building Wider Tree We study the effect of increasing the treewidth. In fact, for every number from SIRIUS in T we are selecting from group of results by different treewidth. We present all of this treewidth and its corresponding accuracy and efficiency numbers in the Appendix Importantly, raising treewidth always improves AAL. Although different choices of treewidth usually give similar accuracy scores, there hardly a pattern on which treewidth always gives the best accuracy. The optimal treewidth can only be found through empirical studies.
B.4 Large Model Experiments
To diversify the evaluation of Sirius, also evaluate Sirius's Effectiveness the Llama-3-70B- Instruct model. MMLU is subsampled 10%, while CNN/Daily is subsampled 30%. The following table contrasts with Llama-3-8P 3-B----stttttttt.... We use striet match/flexible extract accuracy for GSM- 8K-COT, accuracy for MMLU, FI/EM score for CoQA, Rouge-1/211 score for CNN/DailyMail, and Rouge-1/2 ACC for TruthfulQA.
Table 9: Large model results on miscellaneous datasets.
GSM-SK-COT
MMLU
CoQA
CNN/Daily Mail
TruthfulQA
Llama-370B--nn
0.901470.9022
0.7456
0.6567/0.8066
0.1016/0 11006/0.0206600.. (2w.00000600000 0964
0.511670.4247
CSparse
0.7407/0.7483
0.7018
0,6497/0.004
0.1019/0. 00019/00020800..oo 0ww.000009999999 0,0677
0,4541/0.3807
FSparse
0.8726/0.8772
0.7193
0.6497/0.8035
R.001//0020660.000
0.4835/0.3905
Llama-3-8B-In
0.7612/0.7672
0.6272
0.6153/0.7825
00155/0.204/000090
0,4945/0.3647
CSparse
0.3601/0.3647
0.5307
0.6003/0.7735
0.1016/0.0206/00009
0.5067/0,3953
FSparse
0.6103/0.6202
0.4825
0.5828/0.7577
0.10170.022040.0995
0.5202/0.394T
B.5 Variable Sequence Length with Batch Size One
Table 10: A100 Latency versus Input Sequence Length.
Input Sequence Length
A100 Latency (ms)
0.0133
0.0135
0.0136
0.0138
16
0.0140
32
0.0149
64
0.0144
96
0.0171
Here we show the benchmark latency on A100, where the input tensor to
has
shape of batch size I and different input sequence length. To get the hardware optimal readings, use torch compile to compile the whole forward pass of the model. We show that the latency only goes up insignificantly 64, but the trend of increment to 96 a bit steep.
B.6 Error Occurs At Which Position inside Chunk
We look at the distribution where the error would be inside a kernel of 16 tokens. We run through Sirius with kemel size of 16 on the entire GSM-8K and AQUA-RAT-COO dataset. The histogram is shown in Figure 1O We found that the error occurs in a uniform pattern, where is hard to see any particular region where the tokens are likely to occur the most.
B.7 Miscellaneous Results
Besides, the results on the complex reasoning tasks, we evaluate Sirius on slightly more diverse tasks in T2ble
18

---

Figure 10: We look at the histogram of the number of errors the position among period of sixteen tokens on average. We have two different datasets of Arithmetic Reasoning GSM-8K and AQUA-RAT-COT. We can that the number of errors is distributed almost evenly for both datasets.
GSM-8K
AQUA-RAT-COT
120
40
80
40
Position inside 16
Position inside 16
B.8
Llama-2 and Llama-3 Models on GSM8K-COT
Table 11: Detail on Llama-2 and Llama-3 family models with CS.
Jamma3--7B-Instruct
Accuracy
Degradation
Llama---8-IIntrucc
Accuracy
Degradation
Full
0.9205
Full
0.7462
Csparse 60%
0.8144
0.1061
Csparse 50%
0.7652
0.1553
Csparse 50%
0.3636
0.3826
Csparse 40%
0.6023
0.3182
Csparse 40%
0.1856
0.5606
Csparse 30%
0.3144
0.6061
Csparse 30%
0.0644
0.6818
Fsparse 50%
0.8864
0.0341
Fsparse 50%
0.6477
0.0985
Esparss 40% %
0.8485
0.0720
Psparse 40 %
0.4053
0.3409
Esparse 30% %
0.7386
0.1819
Fsparse 30 %
0.0265
0.7197
Esparse 20%
0,2803
0.6402
Llama-2 2-7--hh
Accuracy
Degradation
-lama-2-7B--a Full
Accuracy
Degradation
Full
0.4508
0.1856
Csparse 50%
0.3939
0.0569
Csparse 50%
0.1515
0.0341
Csparse 40%
0.3447
0.1061
Csparse 40%
0.1098
0.0758
Csparse 30%
0.2689
0.1819
Csparse 30%
0.0720
0.1136
Fsparse 50% %
0.3864
0.0644
Fsparse 50%
0.1629
0.0227
Fsparse %
0.3902
0.0606
Fsparse %
0.1364
0.0492
Esparse 30 %
0.2689
0.1819
Psparse 30 %
0.1212
0.0644
Here present more experiments for the comparison between Llama-2 and Llama-3 family models, which is first mentioned in Section B.I, where we also include FSparse methods together with the CSparse method. The results are Table
19

---

Table 12: Miscellaneous Results: 5 models on different Three Different datasets.
Experiment Setting
CoQA
AGIEval (Math)
MMLU--LANNCOO
Llama-2-7B-CHh
0.5982/0.7580
0.072
0,4925
--B--Chat-FSpare
0.5898/0. 7544
0.077
0.4768
0.5908/0.7544
0.081
0,4670
0.6117/0.7633
0.065
0.4637
-lama-2-7B-Chan Bae--ChattCppppe
Siaat-CSparse-SSe SIRIUS
0.6117/0.7664
0.078
0.4794
1ainn-3--B-Inntruue
0.6153/0.7825
0,213
0.6231
0.5828/0.7577
0.172
0.5304
0.5868/0.7597
0.196
0,5709
Llama-3-8B-
0.6003/0.7735
0.154
0,5558
0.6005/0.7728
0.178
0.6003
Llama-2- 2-13--2111B---
0.6408/0.7896
0.092
0.5317
0.6320/0.7837
0.087
0,5082
Llama-2-11 1B-Chat-FSSarse SIRIUS
0.6340/0.7855
0.089
0,5219
1--3a-2-13B 1-13B-Chat- CB-Chat-CSparse
0.6350/0.7841
0.088
0.5127
Llama-2-13h Seaatt-Spprre-Sttee
0.6363/0.7847
0.1
0.5127
Llama-2-7B
0.6388/0.7733
0.101
0,4520
Llama-2 1omma-2-7B-a 2-7B-FSparse
0.6352/0.7699
0.09
0,4435
0.6352/0.7697
0.092
0,4415
1-ama-2-7B-CS5p -BB-CSpassee
0.6338/0.7700
0.086
0.4213
Llama-2-72 2-BB PBB-CSparee-Siii SIRIUS
0.6372/0.7709
0.093
0,4317
Llama-3-a 3-BB
0.6727/0. 8055
0.163
0,5754
Llama-3-888 -
0.6625/0.7988 7984
0.152
0.5349
0.6625/0.79K3
0.1.54
0.5532
Llama-3-8B-CS 3-8B-CSparse
0.6633/0. 7977
0.131
0,5049
Baama---SB-CSppase-SSnii
0.6670/0.7995
0.15
0.5428
C
Additional Results on Reasoning
Due to page restrictions, we only show GSM8K, CSQA, and HumanEval in the paper. Below we show additional results the numbers presented in the paper. We present tables a similar format. Please notice that the leftmost column writes a number that represents the treewidth in the given settings. Also, we show the results of SIRIUS on the other five datasets AQuA-RAT-COT (Arithmetic Reasoning). Sports (Commonsense Reasoning), Date (Commonsense Reasoning), and StrateyyQA (CommonSense Reasoning), and MBPP+ (coding).
C.1 Arithmetic Reasoning
In this section, we present GSM8K and AQuA RAT COT evaluation results with the efficiency metric AAL. Sirius is shown to be effective these two reasoning tasks about arithmetic. Below we show the raw AAL score associated with efficiency for all models and the performance of different treewidths.
C.2 CommonSense Reasoning
We followed the COT paper and evaluated Sirius CSQA, Sports, Strategy and Dates. Sparse methods are capable of outputting high-quality output similar to the full model at the 0.5 mark, which is different than other datasets. However, tune the sparsity level 0.4 (0.6 dense, 0.4 removed), and it starts to have performance degradation. Sirius can compensate them with relatively high efficiency)
C.3 Code
We also have coding portion that evaluates Sirius HumanEval. Sirius performs well similar to other datasets. Besides, we also have results on MBPP+. The results show SIRIUS effectiveness and efficiency again.
20

---

Table 13: SIRIUS Tree on GSM8K.
Experiment Settings treewidth
Glama-3-9B-Instruc--SS
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.7536/0.7544
N/A
0.7536/0.7544
N/A
Sparse Performance
0.5868/0.5891
N/A
0.3844/0.3867
N/A
0.7316/0.7324
14.5903
0.6983/0. 7005
13.1903
0.7172/0.7172
14.9554
0.7089/0. 7096
14.1517
0.7278/0.7309
15.3705
0.7119/0.7111
14.8393
6
0.7195/0.7187
13.5979
0.7081/0.7014.
13.0682
0.7202/0.7218
15,5548
0.7051/0.
15.2291
Llama-3-88- F'parse
Llama-3-8h -8-88-Spparee
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.4966/0.5042
N/A
0.4966/0.5042
N/A
Sparse Performance
0.3199/0.3260
N/A
0.2085/0.2163
N/A
0.4526/0.4572
14.6946
0.439/0.445
13.361
0.4579/0.4644
15.0355
0.4299/0.4367
14.3061
4
0.4579/0.4544
15.0355
0,4223/0.4300
14.9721
0.4450/0.44 4550
15.4834
0.4177/0.4233
15.1435
0.4352/0.4422
15.5863
0.4177/0.4238
15.2939
Llama-2-13B-C0 Chat-F'Sparse
Llama-2-13B-Chan Chat-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.354870.3647
N/A
0.3548/0.3647
N/A
Sparse Performance
0.3222/0.3275
N/A
0.2714/0.2767
N/A
0.3533/0.3472
15.085
0.3412/0.3472
14.0153
4
0.3412/0.3374
15.7576
0.3381/0.3412
15.3491
JJama-2-13B--Sparre
-B3--SS
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.2282/0.2312
N/A
0.2282/0.2312
N/A
Sparse Performance
0.2191/0.2229
N/A
0.1759/0.179
N/A
0.2328/0.2381
15.6759
0.2418/0.244 2472
15.3415
4
0.2372/0.2400
15.9283
0.2077/0.2100
15.825
Llama-2-788
1--7B--hht--SSppe Chat-FSSparse
Llama-2-7B 2-a--------------- Chat-CSparrs
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.2403/0.2426
N/A
1,2403/0.2221
N/A
Sparse Performance
0.1971/0.1994
N/A
0.1334/0.133 1372
N/A
0.2282/0.2312
14.8172
0.2214/0.277 2229
12.6888
0,2297/0.2305
15.1784
0.2252/0.2359
13,8875
0.2305/0.2282
15.5467
0.2183/0.2214
14.6751
0.2388/0.2411
15.691
0.2199/0.2200
14.8575
0.2312/0.2343
15.735
0.2244/0.225.
15.0017
Llama-2-7B 2-aa--------P-----
FSparse
Llama-2-77 2--B- 2-B-a---B-SSparss CSparse
Performance
AAL (out 16)
Performance
AAL (out of 16)
Original Performance
0.1357/0.144 1403
N/A
0.1357/0. 1403
N/A
Sparse Performance
0.1137/0.1168
N/A
0.0758/0.0804
N/A
0.1183/0.1209
15.6864
0.1152/0. 1168
15,1096
0.1334/0.1357
15.7893
0.113/0. 116
15.5358
4
0.1410/0.1448
15.9161
0.113/0.1 116
15,8558
(53.341)
0.1289/0.131
15.9662
0.1183/0. 12/5
15.8715
0.1236/0.1259
15.9568
0.1114/0.111 1145
15.8939
21

---

Table 14: STRIUS on AQuA-RAT-COT.
Experiment Settings
-ama-3-8B-Instruct-
treewidth
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0.515748
N/A
0.515748
N/A
Sparse Performance
0.42126
N/A
0.271654
N/A
0.429134
13.1945
0,468504
10,3322
0.488189
14,4722
0.44095
10.6228
0.496063
15.115
0.452756
10.8348
0.476378
15.3721
0.456693
11.0874
Llama-3-8B-
F-Fare
Llama-3-8h 3-B-
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0.456693
N/A
0.456693
N/A
Sparse Performance
0.287402
N/A
0.228346
N/A
0.377953
12.6665
0.377945
12.6665
0.385827
14.46
0.397638
10.2826
0.366142
15.0671
0.370079
10.6753
0.42126
15,0995
0.38189
11.0019
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0.232283
N/A
0.232283
N/A
Sparse Performance
0.251969
N/A
0.208661
N/A
0.275591
15.5163
0.26378
9.17465
0.279528
15.3995
0.259843
10.8726
Llama-2-131 1-------3--FF-P---
FSparse
Alama-2-13BB
13B-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0. 0.149606
N/A
0.149606
N/A
Sparse Performance
0.165354
N/A
0.149606
N/A
0. 085099
15,4652
0.161417
11.0874
0.220472
15.6346
0.192913
11.8238
Llama-2-7B: 11--7B-Chhtt-.
Llama-2-74
Chat- CSparsse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.251969
N/A
0.251969
N/A
Sparse Performance
0.283465
N/A
0.220472
N/A
0.248031
15,5294
0.251969
12.6424
4
0.259843
15.7254
0.244096
14.5794
Llama-2-
102--BB-Sppaps 1B-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.15748
N/A
0.15748
N/A
Sparse Performance
0.154543
N/A
0.177165
N/A
0.185039
15.423
0.141732
15.4122
0.161417
15.651
0.145669
15.3788
22

---

Table 15: SIRIUS CSQA.
Experiment Settings
Llama-3-8B-Imstrrut-SSSS
treewidth
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.707341
N/A
0.707341
N/A
Sparse Performance
0.615889
N/A
0.647011
N/A
0.699427
12.2108
0.724816
11.0512
4
0.687961
13.2734
0.709255
13.5876
0.714169
13.7842
0.720721
13.3097
0.710893
14.1173
0.707617
14.76893
Llama-3-88
88-C8B--CC CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.643735
N/A
0.643735
N/A
Sparse Performance
0.53317
N/A
0.558559
N/A
0.638821
15,0088
0.618346
12.7426
0,630631
14.6151
0.63964
14.8704
0.625717
14.9905
0.640459
15.1968
0.617527
15.2534
0.642916
15.4355
1lama----3B--Chat-PPppaae Caathhhtt-Spasee
-lama-2-131h
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0.687961
N/A
0.687961
N/A
Sparse Performance
0.53317
N/A
0,553645
N/A
0.657658
13.0868
0.649468
9,2183
0.669124
14.309
0.669124
11.438
-lama-2-13B-FS -3B-FSparre
Llama-2- 1aaaaa---1B-SSpaaaee
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.610975
N/A
0.610975
N/A
Sparse Performance
0.570025
N/A
0.560197
N/A
0.578215
15,2554
0.58231
14.7381
4
0.586405
15.7213
0.606061
15.7284
Llama-2-7B- Seama-2-7B-Chat-t'Spaa Chat-I'Sparse
Llama-2-7h Chat- -Sprree
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.624898
N/A
0.624898
N/A
Sparse Performance
0.616708
N/A
0.520066
N/A
0.632269
14.1015
0.608518
11.4607
0.638002
13.0978
0.605242
14.0366
0.605242
15.4365
0.611794
14.7197
0.621622
15.552
0.617527
15.0799
Llama-2-70 2-----2-------S-S
Llama-2- 2--22-7B-SSS
10--7BB-CSppas CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.474201
N/A
0.474201
N/A
Sparse Performance
0.471744
N/A
0.441441
N/A
0.488124
15.5119
0.461916
14.703
0.494676
15,9141
0.486486
15,5972
0.501229
15,8922
0.476658
15.7315
8
0.473382
15.9247
0.474201
15,802
23

---

Table 16: SIRIUS Sports.
Experiment Settings
Laama-3-8--Imstrrt-SSSSppp
treewidth
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.943299
N/A
0.943299
N/A
Sparse Performance
0.864948
N/A
0.879381
N/A
0.937113
12.3652
0.946392
9.95237
0.941237
14.5248
0.943299
1T.3858
0.942268
14.8651
0.943299
14.0954
0.939175
14.9832
0.941237
14.7718
LJama-3-8B-FSpaan
Llama-3-8B
-00--88--pppppp CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.898969
N/A
0.898969
N/A
Sparse Performance
0.748454
N/A
0.720619
N/A
0.86653
15.5259
0.845361
13.5897
0,849485
15.5917
0,847423
15.2325
0.863918
13.5256
0.843299
15.4376
0.869072
15.6014
0.841237
15.5023
Llama-2-13B-
Siamaa---3B-Chha--SSPee Chatt-SSapre
Llama-2-131 11-22-111--hate
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0.742268
N/A
0.742268
N/A
Sparse Performance
0.690722
N/A
0,584536
N/A
0.710309
13,9767
0.717659
7.72686
4
0.735052
14.9247
0.728953
10.7298
1aama-2-13B-FSSpaa
Llama-2-1 1-----2--1-B-CC 13B---------Sp
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.709278
N/A
0.709278
N/A
Sparse Performance
0.635052
N/A
0.558763
N/A
0.669072
15,4924
0.639175
14.3603
4
0.657732
15,9845
0.658763
15.48
Llama-2-7BB-Cha--SSparss
Llama-2-7B: Chat-CSparss
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.731959
N/A
0.731959
N/A
Sparse Performance
0.652677
N/A
0.596907
N/A
0.704124
14.3861
0.712371
11.1517
0.712371
13.5904
0.71134
13.7394
0.709278
15.7475
0.71134
13.9857
0.698969
15.9927
0.715464
14.3817
Llama-2-7B-Fi
B--FSarre
Llama-2-
2-nma----B-CSpaaaaa 1B-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0,545361
N/A
0.545361
N/A
Sparse Performance
0.536082
N/A
0.528866
N/A
0.524742
15.6754
0.536082
14.1031
0,547423
15.937
0.538144
15.6263
0.545361
15.9807
0.540206
15.7243
8
0.545361
15.9927
0.549485
15,811
24

---

Table 17: SIRIUS Date.
Experiment Settings treewidth
JJamm-3--B--nstruct--yy
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.657224
N/A
0.657222
N/A
Sparse Performance
0.518414
N/A
0.532578
N/A
0.688385
14.2885
0.671388
14.6771
4
0.671388
15.357
0.685552
13.6324
0.679887
15,2435
0.688385
15.1663
0.674221
15.2654
0.694051
15.4293
Llama-3-8B-FSp
Llama-3-B8 108---8--SSS BBCSSprre
Performance
AAL (out of 16)
Performance
AAL
(out of 16)
Original Performance
0.583569
N/A
0.583569
N/A
Sparse Performance
0.399433
N/A
0.424929
N/A
0.535014
15,4236
0.535411
14.4364
0.543909
15,4782
0.546742
15.606
0.546742
13.6365
0.526912
15.7718
0.549575
15.7159
0.541076
15.7997
Seama----i--Chaa--PSpaea Chatt-SSarre
Llama-2-13R Saa---hhat-rrsss
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.524079
N/A
0.524079
N/A
Sparse Performance
0.498584
N/A
0.419263
N/A
0.490085
13,9589
0.461756
14.1419
0.524079
15.432
0.478992
15.8545
1aana-2-13B-FSpamme
Llama-2- 111-2-1-----Spasss
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.501416
N/A
0.501416
N/A
Sparse Performance
0.464589
N/A
0.390935
N/A
0.447592
15,5992
0.461756
15,3896
4
0.492918
15,9129
0.484419
15.8357
Llame-2-7B-Chat--ISpars
Llama-2-7B: Chat-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.320113
N/A
0.320113
N/A
Sparse Performance
0.339943
N/A
0.3002823
N/A
0.31728
14.4663
0.305949
5.75938
0.345609
13.6588
0.325779
14.7519
0.342776
15.742
0.314448
14.5768
0.348442
15.7692
0.308782
14.3627
Llama-2-7B-Fi
7B-FFParse
Llama-2-
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.33711
N/A
0.33711
N/A
Sparse Performance
0.314448
N/A
0.235127
N/A
0.342776
15.5144
0.269122
15.3598
0.342776
15,9141
0.266289
15.8553
0.328612
15.943
0.274788
15.9266
8
0.322946
15.9671
0.271955
15.956
25

---

Table 18: SIRIUS on StrategyQA.
Experiment Settings
-lamas---B-Instruu--SSpppp
treewidth
Performance
AAL (out of 16)
Performance
AAL (out of 10)
Original Performance
0.770241
N/A
0.770241
N/A
Sparse Performance
0.713348
N/A
0.562363
N/A
0.741794
8.98893
0.737418
6.60992
0.741794
9.48412
0.746171
7.92521
0.743982
9.53292
0.728665
8.22667
0.743982
9.55946
0.708972
8.97268
Llama-3-8B--Sppe
Llama-3-8B
SB-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.649891
N/A
0.649891
N/A
Sparse Performance
0.599562
N/A
0.439825
N/A
0.623633
9,46018
0,531729
8.68633
0.623632
9.74383
0.560175
9.44497
0,632385
9.83975
0.560175
9.58122
0.680525
9.80493
0.555799
9.67198
Llama-2-13B- Chat-FSparse
-haama---3D--Chat-SSSasss
Llama-2-13B-Chall Chat-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.695842
N/A
0.695842
N/A
Sparse Performance
0.706783
N/A
0.634573
N/A
0.71116
9.48266
0.682713
6.74106
0.667396
9.83767
0.715536
8.09959
0.671772
9.88989
0.693654
8.82263
1lama-2-13B-FSparrt
Llama-2- 13B
3B-CBpaase
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.63895
N/A
0.63895
N/A
Sparse Performance
0.693654
N/A
0.533917
N/A
0.695842
9,8979
0.595186
8.77388
4
0.682713
9.96438
0.643326
9.47368
0.689278
9,9789
0.63895
9.56319
Llama-2-7B-
Llama-2-7B 1aiiaa-----Chh-a----- Chat-CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.654267
N/A
0.654267
N/A
Sparse Performance
0.678337
N/A
0.612691
N/A
0.684902
9.64754
0.669584
6,55818
0.691466
9.79539
0.671772
7.88988
0.68709
9.86474
0.643326
8.28982
0.689278
9.86488
0.66302
8.43513
Llama-2-7B-Fie BB-Free
Llama-2-78 BBCCSarse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.599562
N/A
0.599562
N/A
Sparse Performance
0.592998
N/A
0.538293
N/A
0.612691
9.73256
0.568928
8,38473
0.599562
9.93662
0.560175
9.36272
0.617068
9.95582
0.536105
9.35857
0.610503
9.96658
0.544858
9.4642
26

---

Table 19: SIRIUS on HumanEval.
Experiment Settings treewidth
Alama-3-8B-Instruue SSpare
Llama-3-8B- CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.5609756097556098
N/A
0.560975609756098
N/A
Sparse Performance
0.457317073170772
N/A
0.2073170731177732
N/A
0.585365853658533
14.7624
0.554878048780445
12.1326
0.5792668992682927
15.2299
0.5304878048780449
14.0546
0.615853658536585
15,4209
0.518292682926879
14.4431
8
0.585365553658337
13.5009
0.524390243902233
14.6725
Llama-3-8B- FSparse
Llama-3-8B CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
1,26119521195122
N/A
0.26219512195122
N/A
Sparse Performance
0.189024390243902
N/A
0.06707317073170733
N/A
0,2317073170731771
15.1878
0.109756997560976
12.1402
0.274390243902439
15.2827
0.219512195121951
13.7718
0.26219512195122
14.5355
0.207317073170772
14.7776
0.29268
15.6305
0.24390243902459
15.1074
Blama-2-1B-Chaa--Spae
Saama-2-33-Chhat-SS55
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0.
0.189024390243902
N/A
0.189024390233902
N/A
Sparse Performance
0.144341463414634
N/A
0.121951219512195
N/A
0.1707317073170733
14.3976
0.189024390243390
9.6447
4
0.182926829268293
15.1956
0.176829268292663
10.7946
0.182926829268229
15.3494
0.170731707317073
11.0149
8
0.176829268292683
15.4067
0.170731707317073
11.1252
1aama-2-13-FSSparre
-lama-2-13B-CSppp
Performance
AAL (out of 16)
Performance
AAL (out of I6)
Original Performance
0. 117622268292683
N/A
0. 0.176829262222683
N/A
Sparse Performance
0.158536585365854
N/A
0.097569975099756
N/A
0.146341463414634
15.2129
N/A
N/A
4
0.158536585365854
15,9093
0.146341463414634
14.1813
0.170731707317073
15.9211
0.13414634146341
14.5866
0.1 0.17682926829268:
15.9015
14.7508
Llama-2-7B-Chm 11--7B-ChatBB Chaa--Faaparss
-lama-2-7B-CC Cas--Cat-CCe Shaat-Spprrse
Performance
AAL (out of 16)
Performance
AAL (out 12)
Original Performance
0. 0.140243902439024
N/A
0.140243902430024
N/A
Sparse Performance
0.134146341466415
N/A
0.067073170731/000
N/A
0.134146341463415
14.055
0.140243902439024
8.83176
4
0.146341463414634
14.8504
0.146341466414463
10.1263
0.15243904390244
15.1924
0.152439024390244
10.5576
0.1646344146341463
15.2742
0.158536585365854
10.822
Llama-2-7B- 'Sppass
-lama-2-7B-SSppe CSaarse
Performance
AAL (out of 16)
Performance
AAL (out of 16)
Original Performance
0.13585365533588
N/A
0.11585365836685
N/A
Sparse Performance
0.115853658536585
N/A
0,0792682926829268
N/A
0.115853658536585
15.5268
0.121951219512195
12.6604
4
0.123048750487805
15.8167
0.121951219512195
14.4053
0. 0.16463414634144
15.8615
0.121951219512195
14.8296
0.109756097560976
15.9189
0.128048780487885
14.8443
27

---

Table 20: STRIUS on MBPP+
Experiment Settings
Siama----8--nsttut---33--
Llana-3--B-hnstrettSSprrm
treewidth
Performance
AAL (out of 16)
Performance
AAL (out of I6)
Original Performance
0.5846560884656885
N/A
0.584656084656685
N/A
Sparse Performance
0.5317460331746032
N/A
0.248677248677249
N/A
0.537037037037033
14.7267
0.563492063492064
11.5415
0.5634920034992044
15.2699
0.566137566137566
13.5896
0.552910052910053
15.3782
0.571428571428571
14.0547
8
0.552910052910053
15.4689
0.566137566137566
14.7648
Llama-3-88 -'Sparee
1aama-3-8--FSpaale
Samma-3-8B--Sparree
Performance
AAL (out of 16)
Performance
AAL (out of I6)
Original Performance
0.518518518518511
N/A
0.518518518518519
N/A
Sparse Performance
0,433862438624334
N/A
0.16137566137566
N/A
0.4894
14.8849
0,4153439153433115
12.7016
4
0,484126984126984
15.4346
0,407407407407407
14.0936
0.4735449735449974
15.3581
0.433862433862454
14.5662
0.468253968223366
15.6088
0.4100529100529)
14.4752
Llama-2-13B-
Llama-2-130
Chat-CSparrs CSparse
Performance
AAL (out of 16)
Performance
AAL (out of 16) N/A
Original Performance
0,23015873015873
N/A
0.27
Sparse Performance
0.19047619047619
N/A
0.1
N/A
0.201058201058201
13.8235
0.26
9,32827
4
0.2328042332002233
14.8394
0.26
10.7346
0.224867124867725
15.0801
0.26
10.8897
0.227513227513225
15.2373
0.25
11.0214
1lama-2-13B-FSaalle
Llama-2-1 1-33-B-pparee
Performance
AAL (out of 16)
Performance
AAL (out 16)
Original Performance
0.246031446031746
N/A
0.21
N/A
Sparse Performance
0,2328044328804233
N/A
0.13
N/A
0,214285714285714
14.8374
0.22
14.5174
4
0,259259259259259
15.8547
0.24
15.6461
0,235449735449775
15.9197
0.23
15.7174
0.246031746031746
15.9094
0.22
15.7179
Llama-2-7B-CHn SaauttFSpprrss
Saima---7B-Chat--ee
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
0,261904761904722
N/A
0,2619047619047766
N/A
Sparse Performance
0,224867724867725
N/A
0.100529100522101
N/A
0,238095238095228
14.1571
0.214285714285714
8.61325
0.26984126984127
14.9264
0.230158730155833
10.2517
0,238095238095228
15.2194
0,227513227513222
10.5845
0.272486772486777
15.3086
0.2254497355449735
10.7621
10
N/A
N/A
0.232804232804233
10.8962
Llama-2-7B.
2amaa---BBFSSparee
Llama-2-7B. CSppreseSSpaeeee
Performance
AAL (out of 16)
Performance
AAL (out of 12)
Original Performance
9,253968253982544
N/A
0.253968253668254
N/A
Sparse Performance
0,201058201058201
N/A
0.079365079365079.
N/A
0.2169331219931217
14.6103
0.171957671957772
10.6643
4
0,238095238095238
15.5672
0.1851851851885185
11.5561
0.224867724867725
15.6273
0.195767195767196
11.6547
0.240740740740741
15.5569
0.203703703703704
11.6753
28

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided proper justifictonn is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering binary way, we
"[No] or ][NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
A. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
B. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification: [TODO]
29

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
C. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
D. Experimental Result Reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims conclusoon the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
2. If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
3. If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
4. We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
E. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
wttpauammiiisioooooooooooip for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wwsssssoooooooooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
31

---

F. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
G. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
H. Experiments Compute Resources
Question: each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
Guidelines:
does The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The answer NA means that the paper does not include experiments.
The paper should provide the amount of compute required for each of individual experimental runs well as estimate the total compute.
32

---

The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary failed experiments that didn't make it into the paper).
1. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics Etticshtttps///eurrppp..o http://wwwpp..cco.o/oo.///ctaaar
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special considera- tion due to laws or regulations their jurisdiction).
J. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
K. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
33

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
L. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, datasets.papersst paperrsi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out the asset's ' creators.
M. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
N. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
34

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
O. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "CiijKMaIIT", "reviewer": "Reviewer_a8uY", "summary": "This paper focuses on enhancing the inference efficiency of large language models (LLMs) through contextual sparsity. While it identifies that contextual sparsity reduces hallucination, it also notes a significant impact on reasoning and deduction performance. To address these drawbacks, the paper introduces SIRIUS, a correction mechanism that effectively improves performance. Specifically, the correction mechanism uses additional parameters (i.e., full model) to check the output of the sparse model based on the input and the same instruction prompt.", "questions": "Please refer to Weaknesses.", "limitations": "There are no discussions about the proposed method's limitations and potential societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces SIRIUS, a correction mechanism that effectively enhances the performance of sparse models, particularly in complex reasoning tasks where traditional sparsity techniques falter.\n2. The paper includes many experiments across various datasets and tasks to reveal the problem in the existing contextual sparsity methods and sparse models.", "weaknesses": "**Method\n\n1. Based on the results in Tables 1 and 2, this paper claims that “the stronger the model, the larger the quality degradation would be”. However, the studies exclusively utilize relatively smaller models, Llama2-7B and Llama3-8B. Including results from larger LLMs, such as Llama3-70B, would provide a more comprehensive understanding of the impact across different model sizes.\n\n2. As the proposed method uses the probability from the full model to check the correctness of the sparse model, does this require forwarding twice for each token? If so, the cost would be double.\n\n3. It is unclear when to use the correction mechanism. Should it be used after the sparse model has generated all predicted tokens, or immediately after the sparse model predicts each token?\n\n**Experiment\n\n1. In Table 3, if the proposed method uses the full model to correct the output of the sparse model, why is there still a noticeable degradation compared to the full model itself?\n\n2. As the title highlights the efficiency of the proposed method, it would be better to provide some experiments about it like latency (ms) in DejaVu [1].\n[1] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML, 2023.\n\n**Minor Issues\n\nThere are many typos in this paper, e.g., the title “… Contexual Sparisty …” should be “… Contextual Sparsity …”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your attention and time taken to read the paper and your insightful comments. We are very thankful for you to point out the inadequacy of our method presentation and raising concerns on the diversity of the evaluation. We will try our best to address your concern. If you feel that your concern has been taken into account, please consider raising your score. \n\n1. Studies mainly use smaller models, which should include Llama-3-70B to verify the claim that \"the stronger the model, the larger the quality degradation would be\". \n\nIn light of your comment, we run the Llama-3-70B-Instruct model on GSM8K, MMLU-FLAN-COT, CoQA, CNN/DailyMail, and TruthfulQA. Due to running Llama-3-70B-Instruct under pipeline parallelism is slow on our hardware (8xA100), MMLU-FLAN-COT is subsampled 10%, and CNN/DailyMail is subsampled 30%. \n|\t|GSM-8K-COT|MMLU-FLAN-COT|CoQA|CNN/DailyMail|TruthfulQA| \n|---------|-----------|---------------------------|--------|-------------------|---------------|\n|Llama-3-70B-In|0.9014/0.9022|0.7456|0.6567/0.8069|0.101634/0.020614/0.096413|0.5116/0.4247|\n| + CSparse|0.7407/0.7483|0.7018|0.6497/0.8046|0.101922/0.020854/0.096703|0.4541/0.3807|\n| + FSparse|0.8726/0.8772|0.7193|0.6497/0.8035|0.101505/0.020623/0.096344|0.4835/0.3905| \n|Llama-3-8B-In|0.7612/0.7672|0.6272|0.6153/0.7825|0.101523/0.020481/0.096311|0.4945/0.3647|\n| + CSparse|0.3601/0.3647|0.5307|0.6003/0.7735|0.101681/0.020657/0.096432|0.5067/0.3953|\n| + FSparse|0.6103/0.6202|0.4825|0.5828/0.7577|0.101713/0.020448/0.096516|0.5202/0.3941|\nGSM-8K-COT, “strict match”/”flexible extract”; CoQA, EM/F1; CNN/DailyMail, Rouge-1/2/L, TruthfulQA gen, Rouge-1/2 ACC. \n\nIn the table, Llama-3-70B-Instruct still has features that we identified in the paper, sparse excels at prompt understanding (CoQA and CNN/DailyMail). Significant performance degradation occurring at GSM-8K-COT with coarse-grained sparsity. Even though the gap between sparse and dense for 70B is not as big as for smaller models, we don't think it contradicts the claim that the more powerful the model, the more degradation when doing sparse. First, even with Coarse-grained sparsity, the number of parameters is more than 45B, still colossal. Most tasks here are curated in pre-LLM era and are too easy for models now of huge parameters [1]. On the other hand, the claim is presented in the context of comparing Llama-2-7B Chat and Llama-3-8B-Instruct, two smaller but similar models. Llama-2-7B Chat has significantly worse performance in all different tasks. Contextual Sparsity causes much more damage to 8B than 7B. Nevertheless, trying larger parameter models does provide more insights to contextual sparsity characteristics. \n\n[1] Chiang, W. L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., ... & Stoica, I. (2024). Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132. \n\n2. Forwarding twice, cost doubled? \n\nThe tokens produced by the sparse model will be fed into the full model for verification, but the latency cost of full model verification will NOT be a lot, compared to the full model generating one new token. The cost in terms of latency won’t be doubled. The full model will verify the tokens generated by the sparse model periodically, usually in a period of 16. During full model verification, all tokens generated by the sparse model that haven’t been verified will be fed in one iteration. And, the full model will generate one more token in this run. Consider the following table of A100 latency of different sequence lengths with bsz=1. Only 1.1ms increase for 64 seqlen. The verification is as light as generating one token, without additional overhead. \n|Input Sequence Length|1|2|4|8|16|32|64|96|\n|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n|Latency (ms)|13.3|13.5|13.6|13.8|14.0|14.9|14.4|17.1|\n\n3. When to correct? \n\nCorrection happens periodically. Specifically, the sparse model would generate PERIOD -1 tokens, and the full model would take in these tokens, verify them based on likelihood thresholds, rollback, and then generate one more token. The cycle goes on. Sirius usually takes PERIOD of 16, sometimes 12. \n\n4. Degradation after correction? \n\nWe can answer by comparing Sirius with Speculative Decoding (SD). SD is lossless. Under the greedy setting, SD acceptance criteria guarantee that the two models' output is the same as the large model output. In comparison, here we effectively loosen up the acceptance criteria. We manually set a threshold on the full model's verification likelihood. i.e., If the full model thinks the token is likely to occur, it accepts. However, the token is not necessarily the one full will greedily decode, leading to divergence in generation and, thus degradation. However, the loosened criteria in turn boost the efficiency of the overall system by a margin to SD's. \n\n5. Hardware wall clock time speedup? \nSirius delivers the promised speedup. We present the speedup of Coarse-grained sparsity with Sirius in two settings. First, on-chip, we run Llama-3-8B-Instruct inference. Fine-grained sparsity prior works rely on a custom CUDA kernel, which we don't have access to. \n\n| Setting| GSM-8K-COT | A40 | Speedup | L40 | Speedup | A100 | Speedup |\n|-----|-----|-----|-----|-----|-----|-----|-----|\n| CSparsity | 0.3601 | 20.7ms | 0.66 | 15.6ms | 0.67 | 9.6ms | 0.72 | \n| Sirius (10 kernel size) |0.7309| 24.1ms | 0.78 | 18.2ms | 0.78 | 11.4ms | 0.85 |\n| Sirius (16 kernel size) | 0.7309 | 24.9ms | 0.80 | 18.8ms | 0.81 | 11.8ms | 0.88 | \n| Full | 0.7612 | 30.9ms | | 23.2ms | | 13.3ms | |\n\n\n\nSecond, Offloading Llama-3-70B-Instruct. We use a single L40 48GB with a PCIe bus bandwidth to be 25 GB/s. \n\n| | Sparse | Sparse + Sirius | Full |\n|---------------|----------------------|-------------------------------|--------------------|\n| Performance | 0.9014 | 0.8719 | 0.7407 | | \n| Latency (s) | 3.57s | 3.68s | 5.72s | 25.3 GB/s | \n| Ratio to Full | 0.6241 | 0.6434 | | |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YOaZdea9ZJ", "reviewer": "Reviewer_rbBk", "summary": "This paper aims to improve contextual sparsity (CS) approaches: LLM sparsification / parameter pruning methods where the sparsification strategy is conditioned on the input sequence / prompt itself. The paper begins by reproducing contextual sparsity baselines, but applied to modern LLMs: LLama 2 and 3 (7B & 8B respectively), and verifying that their reproduction successfully replicates previous CS techniques on summarization and question answering tasks. The authors then demonstrate that the technique is much less effective on more challenging reasoning tasks such as GSM8K and MMLU-CoT. Through investigation the authors observe that often, by correcting a small amount of tokens during the generation (~10%), the sparse models can significantly improve their performance to be on-par with the full dense model. This motivates their main proposed technique: SIRIUS, which employs a speculative decoding like technique to use the full dense model to score every k tokens (k=16-24 in practice) during the decoding process in order to rollback and use the dense model to correct a token when necessary. By doing so, the method is able to mitigate the losses on reasoning tasks while maintaining most of the efficiency of the sparsification method.", "questions": "What were the motivations behind the differences between Sirius (Algorithm 1 / Section 4.3) and speculative decoding? Why not directly use speculative decoding, with the sparse model being the drafting model? (which maybe has more guarantees)\n\nWhy is the method named Sirius?", "limitations": "The authors briefly describe some limitations but the work is generally lacking discussion here. Mainly the authors mention that the method does not work well under extreme sparsity (to my knowledge almost no sparsity method does in the general case – this is more of an open question than a limitation.) The authors could be more forthcoming here. It would be valuable to elaborate on whether the authors believe the two CS approaches are representative of the space of approaches, whether the experiments are adequate, whether they expect the results to change at different model scales, or for tasks beyond the ones studied here.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The authors explore the viability of CS techniques on much more difficult reasoning tasks than previously in the literature, providing an important datapoint on whether CS approaches (at least ones studied in this paper) work in general. \n\n1. The authors run thorough quality experiments on modern LLMs, and demonstrate that their technique effectively rectifies the issues they first identify (significantly improving reasoning performance.)\n\n1. Using the full model to do something like self-speculative decoding with the drafting model being a sparse version of the same model is novel and promising. It aligns with contemporaneous works that explore this idea [1].\n\n1. The method is efficient w.r.t the number of parameters used on average during decoding. \n\n1. The paper is overall well structured and the investigation is easy to follow. \n\n1. The authors plan to release the code for their experiments.\n\n[1] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", "weaknesses": "1. The main spirit of this paper is whether we can use some small amount of additional compute budget over the sparse model alone (e.g. via limited calls to the full model) to improve the decoding from the sparse model. However, the authors do not compare against any baseline approaches to do this same thing, such as speculative decoding. Even though the authors claim in Related Work that speculative decoding does not suit this problem since it’s not efficient enough, the authors do not provide any empirical evidence of such.\n\n1. This paper is very much an efficiency paper, which tries to introduce some small amount of corrections from the full model to the dense model, while not doing too much as to nullify the gains from the sparse method. However, the authors only analyze Average Parameters Used Per Token (APU), a sparsity measure, instead of any other efficiency metric (importantly wall-time). \n    - The authors make an argument for why this is in Section 4.1, using findings from previous papers. While this may be a reasonable intuitive argument, the method in this paper has significant differences like KV-cache correction that still makes this valuable to empirically verify. \n    - Moreover, even if I intuitively believe from the author’s argument that the wall time latency of this approach would be somewhere between the sparse and full model’s wall time, it is unclear where on this spectrum it lies, how it compares to other techniques like speculative decoding, or how the tradeoff plays with varying sparsity levels without such measurement and comparison. \n    - To place this into context, the CATS paper, which is one of the methods this work experiments with (FSparse) directly states “Activation sparsity of a model is not sufficient to directly enable wall-clock time inference\nSpeedups”, and provides wall-clock time analysis.\n\n1. The paper is poorly written, with various spelling errors (in the title), grammatical errors (for example: line 1, Section 4.2 title, and more), placeholders for citations (line 114), and difficult to read citation formatting. In many ways the presentation of this work is significantly not publication ready. \n\n1. Without the code release, the description of the method is not very detailed, even though Algorithm 1 is provided, and would make it very hard for the reader to understand how this method is implemented from reading the paper alone, which hurts reproducibility. \n    1. For example: the description of the correction algorithm is only described in one paragraph in Section 4.3 with insufficient detail on how the KV cache is updated. \n    1. Algorithm 1 has undefined variables with respect to the KV cache such as “cachetracker”. The description in 4.3 states that the KV cache is directly written to by the full model to correct the past, however Algorithm 1 does not describe this process other than  “Update C_s based on j” is unclear what kind of update is happening to the cache.\n    1. The important LIKELIHOOD function is undefined, which uses the full model’s likelihoods to judge whether the sparse model’s output is incorrect. There are no details on what threshold is used here. \n\n1. Some additional claims are not sound:\n    - Line 270: “KV Cache can also be looked as an implicit knowledge distillation method to pass dark information from the LLM to the sparse counterparts” What is dark information?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time taken to read the paper and your insightful comments. We are thankful that you point out that the comparison against speculative decoding is missing and we lack of convincing real efficiency metrics.  We will try our best to address your concern. If you feel that your concern has been taken into account, please consider raising your score. \n\n1. Differences to Speculative Decoding? Why Speculative Decoding is not efficient enough? \n\nSirius contains a stronger and a weaker model, the strong verifies the weaker's output, from which the resemblance can be easily drawn between Sirius and Speculative Decoding (SD). However, the important distinction is the problem setting. \n\nContextual sparsity users prioritize the tradeoff between the performance and the efficiency of the model. We show that the efficiency of the contextually sparse model corrected by SD will be largely limited. Take the Coarse-grained sparse model on Llama-3-8B as an example, the 50% sparse (APU 0.65) model will produce an acceptance rate of 0.89 on GSM8K. SD also runs the smaller model every iteration with verification of the larger model once a period. Naturally, to increase efficiency, we need to (1) enlarge the period and (2) increase the average number of tokens accepted in the period. \n\\frac{1 - \\alpha^{(\\gamma + 1)}}{1 - \\alpha}\nNow, given the acceptance rate of 0.89, following the formulation in Speculative decoding, we can calculate the expected number of accepted tokens for every gamma term in the Speculative Decoding literature, which is (period - 1). \n\n|   Gamma |   Advance Length |\n|--------:|-----------------:|\n|       4 |          4.01    |\n|       8 |          5.91    |\n|      12 |          7.09    |\n|      16 |          7.84    |\n|      20 |          8.30    |\n|      24 |          8.60    |\n|      28 |          8.78    |\n|      32 |          8.90    |\n\nWe can notice the trend that the average advance length starts to plateau as the period becomes larger. Take the gamma of 16 as an example, the period is then 17. If we use the formula in Section 4.1 of our paper, we can immediately see that the APU is (16 * 0.65 + 1)/7.84 = 1.45, even larger than full model 1. \n\n|   Gamma |   Advance Length |\n|--------:|-----------------:|\n|       1 |           0.87   | \n|       2 |           0.86   |\n|       4 |           0.90   |\n|       8 |           1.05   |\n|      12 |          1.24    |\n|      16 |          1.45    |\n|      20 |          1.69    |\n|      24 |          1.93    |\n\nBecause of the plateauing effect, for an acceptance rate of 0.89, the best gamma is 2 (period = 3). The best efficiency is 0.86, compared with 0.65 coarse-grained sparse APU. A similar picture can be applied to Fine-grained sparsity as well. The key reasons are (1) the contextual sparse model is too large to use a long period for SD, and (2) the acceptance criteria are too strict. \n\nIn contrast, Sirius gives a more flexible choice for contextually sparse model users. For a threshold of 0.1, Sirius can correct Llama-3-8B coarse-grained sparsity from 20.85% to 43.9%, compared to the 49.66% full model. Sirius on average can accept 13.4 tokens out of a kernel size of 16 and over 9 tokens out of a kernel size of 10, translating to APU < 0.76, significantly lower than SD does.\n\n2. Hardware speedup? \n\nWe show that Sirius delivers the promised speedup. \nWith the introduction of static cache from huggingface 4.38, kv cache is pre-allocated, making rewrite, expand, and rollback of very minimal overhead. Fine-grained sparsity speedup is via prior work [1]. However, CATS relies on a close-sourced custom CUDA kernel. We present the speedup of Coarse-grained sparsity with Sirius in two settings. \nFirst, on-chip, we run Llama-3-8B-Instruct inference. \n\n| Setting| GSM-8K-COT | A40 | Speedup | L40 | Speedup | A100 | Speedup |\n|-----|-----|-----|-----|-----|-----|-----|-----|\n| CSparsity | 0.3601 | 20.7 | 0.66 | 15.6 | 0.67 | 9.6 | 0.72 | \n| Sirius (10 kernel size) |0.7309| 24.1 | 0.78 | 18.2 | 0.78 | 11.4 | 0.85 |\n| Sirius (16 kernel size) | 0.7309 | 24.9 | 0.80 | 18.8 | 0.81 | 11.8 | 0.88 | \n| Full | 0.7612 | 30.9 | | 23.2 | | 13.3 | |\n\nSecond, Offloading Llama-3-70B-Instruct. We use a single L40 48GB with a PCIe bus bandwidth of 25 GB/s. \n\n| | Sparse | Sparse + Sirius | Full |\n|---------------|----------------------|-------------------------------|--------------------|\n| Performance | 0.9014 | 0.7407 | 0.7407/0.7483 | | \n| Latency (s) | 3.57 | 3.68 | 5.72 | 25.3 GB/s | \n| Ratio to Full | 0.6241 | 0.6434 | | |\n\n[1] Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., ... & Chen, B. (2023, July). Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning (pp. 22137-22176). PMLR.\n\n3. Criticism of writing quality \nWe do appreciate the suggestions on writing. In light of your precious effort put into reading, we carefully rewrote and proofread the entire methodology section that takes all your suggestions into account. \n\n4. KV Cache correction \n| Setting|GSM8K 20%                              |Setting| GSM8K 20%                                           |\n|-------------------------------------------|----------|-----------------------------------------------------|----------|\n| Llama3-8B-Instruct                        | 0.7538 | Llama3-8B-Instruct                                  | 0.7538 |\n| Llama3-8B-Instruct + CSparse             | 0.3674 | Llama3-8B-Instruct +     FSparse                       | 0.5644 |\n| Llama3-8B-Instruct + CSparse + KV Cache Correct | 0.4735 | Llama3-8B-Instruct + FSparse + KV Cache Correct        | 0.6629 |\nWe show that KV Cache correction contributes to correction. However, we agree that \"implicit distillation\" is unbased, which has been deleted in the latest version. \n\n5. Why Sirius? \nSirius is an astronomical term referring to a two-body star system. One is the brightest ever known, while the other is dim. We draw inspiration from that system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ak4RRRB2WN", "reviewer": "Reviewer_ArEw", "summary": "The paper introduces SIRIUS, a novel method designed to enhance the efficiency and accuracy of sparse Large Language Models in reasoning and deduction tasks. SIRIUS employs contextual sparsity to reduce computational costs and incorporates an efficient correction mechanism that recovers sparse model performance by correcting only a minimal number of key tokens. Experimental results demonstrate that SIRIUS significantly improves the performance of sparse models on complex reasoning tasks while maintaining efficiency.", "questions": "If the correction requires the use of a full model, won't the speed of correction slow down compared to directly using the sparse model? Is the memory usage the same as using a full model directly? If the memory usage is the same as using a full model directly, there is no advantage in memory savings", "limitations": "The author improved the performance of the sparse model in reasoning and understanding tasks by adding a full model correction, which requires the use of the entire full model. Although the average parameters used per token increase is minimal, the memory usage is the same as using the full model directly", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper introduces SIRIUS, a novel and efficient correction mechanism specifically tailored for sparse models\n2.\tIt provides a echnically rigorous approach to improving the inference efficiency of LLMs, supported by experimental validation.", "weaknesses": "1.\tThere are too few experiments,  so no experiments were conducted on other LLMs， and no experiments were conducted on more contextual sparsity methods\n2.\tThe Mc annotation in pseudocode is unclear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time taken to read the paper and your insightful comments. We are thankful that you suggested we add more experiment data to evaluate Sirius. We will try our best to address your concern. If you feel that your concern has been taken into account, please consider raising your score. \n\n1. Too few empirical experiments \n\nWe provide additional empirical results and analysis of Sirius on various models and different downstream datasets. \n\nAs discussed in the paper, the weakness of the contextual sparse models seems to occur upon difficult reasoning tasks. Besides, we collect experiment results on different reasoning tasks. In Arithmetic reasoning, besides GSM-8K, we also show a difficult AQuA-RAT COT from Google. In the Common Sense reasoning, we follow the COT paper to evaluate CSQA, StrategyQA, Sports, and Dates. Also, we found that contextual sparse models cannot do well in code generation. We then run Sirius on HumanEval. \n\nWe collect additional experiment results on six models (Llama-3-8B, Llama-2-7B, Llama-2-13B with their instruction fine-tuning counterparts). Due to the word limit, we only show AQuA, CSQA, and HumanEval on some models. Full results and Average advance length will be in the new version of the paper. \nL means Llama. \nO means Original \nS means Sparse \nSS means Sparse + Sirius \n\n| AQuA RAT                          | L-3-8B-I-FSparse | L-3-8B-I-CSparse | L-2-7B-Chat-FSparse | L-2-7B-Chat-CSparse |\n|-----------------------------------|-----------------------------|-----------------------------|-------------------------|-------------------------|\n| O               | 0.51                        | 0.51                        | 0.25                    | 0.25                    |\n| S                | 0.42                        | 0.27                        | 0.28                    | 0.22                    |\n| SS                            | 0.42                        | 0.46                        | 0.24                    | 0.25                    |\n| |L-2-13B-Chat-FSparse          | L-2-13B-Chat-CSparse    |         |      |                         \n| O             | 0.23                        | 0.23                        |                     |                     |\n| S               | 0.25                        | 0.20                        |                     |                     |\n| SS                            | 0.27                        | 0.26                        |                     |                    |\n\n| CSQA                                | L-3-8B-I-FSparse | L-3-8B-I-CSparse | L-2-7B-Chat-FSparse | L-2-7B-Chat-CSparse |\n|-------------------------------------|-----------------------------|-----------------------------|-------------------------|-------------------------|\n| O               | 0.70                        | 0.70                        | 0.62                    | 0.62                    |\n| S                 | 0.61                        | 0.64                        | 0.61                    | 0.52                    |\n| SS                              | 0.69                        | 0.72                        | 0.63                    | 0.60                    |\n|                                     | L-2-13B-Chat-FSparse    | L-2-13B-Chat-CSparse    |      |      |\n| O                | 0.68                        | 0.68                        |                     |                     |\n| S                 | 0.53                        | 0.55                        |                     |                     |\n| SS                              | 0.65                        | 0.67                        |                     |                     |\n\n| HumanEval                          | L-3-8B-I-FSparse | L-3-8B-I-CSparse | L-2-7B-Chat-FSparse | L-2-7B-Chat-CSparse |\n|------------------------------------|-----------------------------|-----------------------------|-------------------------|-------------------------|\n| O             | 0.56                        | 0.56                        | 0.14                    | 0.14                    |\n| S                 | 0.45                        | 0.20                        | 0.13                    | 0.07                    |\n| SS                             | 0.58                        | 0.55                        | 0.13                    | 0.15                    |\n|                                    | L-2-13B-Chat-FSparse    | L-2-13B-Chat-CSparse    |      |      |\n| O               | 0.18                        | 0.18                        |                     |                     |\n| S                | 0.14                        | 0.12                        |                     |                     |\n| SS                             | 0.17                        | 0.17                        |                     |                     |\n\nSirius can also work effectively on the Llama-3-70B-Instruct, model with large parameters. \n| L-3-70B-I                              |     GSM8KCOT           |\n|---------------------------------------------------|----------------|\n| L-3-70B-I                               | 0.90  |\n| L-3-70B-I CSparse         | 0.74  |\n| L-3-70B-I CSparse         | 0.87   (15.41) \nThe bracket is the average advance length of 16 kernel size.\n\n\n2. Other questions. \n\na. We appreciate the suggestions on Pseudo code, we have rewritten it in the newer version. \n\nb. Correction step by full model is slower than sparse, but the cost is amortized by the kernel size of 16, which is less significant. \n\nc. Contextual sparsity subsample sparse model from the full model dynamically according to input, so the full model weight will always be in GPU memory. Adding correction won't increase the memory usage of the sparse model. \n\nd. Please note that the latency is mainly affected by the memory loading not the shear memory size on VRAM. Full model is loaded only once per kernel size. Please refer to the hardware speedup on the new version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3fZsLuVYB", "reviewer": "Reviewer_BB9w", "summary": "The paper introduces a sparse LLM correction mechanism (SIRIUS) designed to improve the inference efficiency of large language models through contextual sparsity. Contextual sparsity reduces computational cost by pruning parameters that are less relevant based on the input context. However, this approach degrades performance in tasks requiring high-level reasoning and deduction. SIRIUS addresses this by selectively correcting key tokens, using the full model to rewrite the sparse model's Key-Value cache. Experiments show improvements in strict-match EM scores, particularly on mathematical reasoning tasks.", "questions": "- How are the likelihood thresholds for detecting and correcting errors empirically tuned? Can the authors provide more details on this process and its impact on performance?\n- How does SIRIUS handle long-term dependencies and corrections over extended sequences of text? Are there scenarios where the KV cache rewriting might not be sufficient to correct extended errors?", "limitations": "not observed negative societal impact to my understanding.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The correction mechanism that selectively rewrites the KV cache is novel.\n2. The paper presents an interesting finding where sparse models fail on tasks that require reasoning and deduction.\n3. The paper demonstrates improvements in performance metrics on benchmarks like GSM8K.", "weaknesses": "- The KV cache rewriting mechanism requires an available full LLM to be of the same architecture as the sparse model.\n- The criteria for when to trigger KV cache rewriting are not clearly detailed, making it difficult to assess the practicality and reliability of the approach.\n- It is unclear whether using the full model’s KV cache to rewrite the sparse model guarantees that the rewritten KV cache can be accurately interpreted and utilised by the sparse model, due to the differences in the sparse and full model.\n- The hypothesis that the gap between the full and sparse models can be bridged by correcting a few key tokens (L44-45) is not sufficiently validated beyond the provided examples. It remains unclear whether this hypothesis holds across a broader range of tasks and datasets.\n- The writing needs to be improved for clarity, for example, \n     - there are a few missing citations (e.g., L114. L284)\n     - the tables can be clarified with the best performing variant per task in boldface, and a caption which describes the metrics in the table.\n     - Algorithm 1's functions can be better described, for example, the descriptions in \"normal forward function FORWARD, forward function with direct cache rewrite KVFORWARD, likelihood judge function of LIKELIHOOD\" are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time and attention on the paper. We are very grateful for you to be interested in the KV Cache correction. We will try our best to answer your questions. \n\n1. Full and Sparse of the same architecture \n\nContextual Sparsity is the focus of Sirius. Contextually Sparse (CS) models are dynamically subsampled from the normal LLM. The subsampling process won't change the compatibility of the original model's cache with the newer subsampled sparse model. Therefore, the original cache is naturally usable by CS models. However, we are interested to see whether Sirius can be applied to model pairs outside the CS realm. \n\n2. When to rewrite KV Cache \n\nWe appreciate your suggestions, and we rewrote the method part of the paper to take into consideration KV rewriting confusion. The rewriting happens when the full model is correcting the sparse generated tokens. The full model is called once every kernel size, usually 16. Throughout the inference, the KV cache is shared between the sparse and the full model. The KV cache is mostly populated by the sparse model, which is called for every token. During correction, the full model takes in the last kernel size of tokens and generates its KVs for the past 16 tokens in parallel, these KVs are directly written to their corresponding positions in the shared KV Cache. \n\n3. Will the full model's KVs make semantic sense to the sparse model \n\nWe appreciate the brilliant question. Empirically, we found that the KV cache of the large model seems to always help the small model generation quality. Below shows one of the example on GSM8K (20% subsampled) \n| GSM8K 20%                                 |          | GSM8K 20%                                           |          |\n|-------------------------------------------|----------|-----------------------------------------------------|----------|\n| Llama3-8B-Instruct                        | 0.7538/0.7538 | Llama3-8B-Instruct                                  | 0.7538/0.7538 |\n| Llama3-8B-Instruct + Griffin              | 0.3674/0.3674 | Llama3-8B-Instruct + CATS                           | 0.5644/0.5644 |\n| Llama3-8B-Instruct + Griffin + KV Cache Correct | 0.4735/0.4735 | Llama3-8B-Instruct + CATS + KV Cache Correct        | 0.6629/0.6629 |\nYou can see that doing KV Cache correction alone brings substantial improvement to both settings. Empirically, the sparse model is clearly able to extract more insightful semantics from the full model's KVs. \n\n4. Whether correcting minor portion leads to performance recovery is applicable to more datasets \n\nWe appreciate the comments. Although we only show for GSM8K in the text. However, in the newer version of the paper. We evaluated Sirius across diverse datasets, ranging from arithmetic reasoning, common sense reasoning, and coding. Sirius managed to achieve competitive efficiency metrics on most of them. These results show that for general text generation that requires challenging step-by-step correlation, the minor portion is the only part that needs to be corrected to recover the original performance. \n\n5. Writing is unclear \n\nWe deeply appreciate the suggestions on improving the writing quality of the paper. All of the suggested parts are revised in the newer version of the paper. \n\n6. How to determine thresholds. \n\nThreshold determination is a crucial process of balancing the performance and efficiency of Sirius. Here we provide a much more in-depth ablation on how increasing the threshold exhibits the trend that improves the performance but hurts the efficiency. The efficiency of Sirius is measured by the average advance length out of a kernel size (usually 16 tokens), the higher the better. \n| Threshold                     | Performance Score | Efficiency Metrics |\n|-------------------------------|-------------------|--------------------|\n| Original Full Model           | 0.7803/0.7828     |                    |\n| No correction (threshold 0)   | 0.5884/0.5960     |                    |\n| 0.05                          | 0.7247/0.7247     | 15.2098            |\n| 0.1                           | 0.7399/0.7424     | 14.6451            |\n| 0.2                           | 0.7247/0.7273     | 13.2329            |\n| 0.3                           | 0.7399/0.7449     | 11.6134            |\n| 0.4                           | 0.7551/0.7601     | 10.0037            |\n| 0.5                           | 0.7677/0.7702     | 8.56022            |\n| 0.6                           | 0.7758/0.7778     | 7.44126            |\n| 0.7                           | 0.7702/0.7753     | 6.26547            |\n| 0.8                           | 0.7753/0.7803     | 5.25639            |\n| 0.9                           | 0.7652/0.7677     | 4.20542            |\n| 0.95                          | 0.7626/0.7652     | 3.56315            |\n| 1.0                           | 0.7601/0.7626     | 1.2685             |\nIn practice, threshold of 0.1 generally works well across various models and datasets. \n\n7. Long range dependencies \n\nSirius allows the large model to corrects the small model on a basis of kernel size. Besides KV rewriting, roll back mechanism is especially important for generating longer text. The rollback works as follows, when full model parallel verifies, If there is any token within the range of the kernel size that the full model regards as unlikely (rejects by likelihood threshold), then the token will be removed. Therefore, regardless of the generation length, all the verified tokens will be regarded by the full model as \"enough likely\" given the context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thanked all the reviewers [R1(a8uY), R2(rbBk), R3(ArEw), R4(BB9w)] for their attention and time put into reviewing the paper and also for the thoughtful and supportive comments. We are glad to see that the reviewers find the work interesting [R4(BB9w)] and effective [R1(a8uY)], consider the problem we are solving to be relevant [R2(rbBk)], and think that our proposed technique to be effective [R1(a8uY)]. Also, we are pleased to read that some find our overall presentation to be easy to follow [R2(rbBk)]. 

On the flip side, we want to make an additional effort to assure the reviewers that we take their suggestions and criticism seriously. Before diving into specific detailed questions, we rewrote the paper submitted considerably based on the precious feedback. The differences can be mainly summarized in the following bullet points. The revised new version of the paper can be accessed with the following link: 
https://drive.google.com/file/d/1mvrltX1vd4dlTOKaSyBK1BuXgNvZNYHd/view?usp=sharing.

 - Elaborate on the Difference in Setting between Sirius and Speculative Decoding [R2(rbBk)] 
Sirius is a method that seemingly involves two models, a powerful full model and a weaker sparse model. One can easily draw a resemblance with Speculative Decoding, a technique to speed up the large model decoding. The key difference in setting between Sirius and Speculative Decoding is that Speculative Decoding (SD) is LLM-centric, meaning that the key objective is to losslessly speed up the LLM inference. The restriction for whether to accept the weak draft model is high in order to preserve the LLM performance. However, for the users of Contextual Sparse (CS) models, they accept the potential performance degradation involved in the CS models and crave more on the efficiency-performance tradeoff. However, the CS model struggles at difficult tasks that require step-by-step reasoning. Applying Speculative Decoding directly to full and CS models would correct the CS model’s mistake but incur a large price. Sirius is a “small model-centric” technique aiming to still preserve the efficiency of the sparse models (substantially better than SD), but improve sparse model performance in the LLM’s vicinity. 

 - Refine the description of the Sirius [R1(a8uY), R2(rbBk), R3(ArEw), R4(BB9w)] 
We rewrote the algorithm and the method section in order to present the method in greater detail and address specific concerns about the method relating to the following major questions: When does the switch between full and CS models occur? How is the KV Cache correction implemented? How is the memory of the full with CS models managed, and why we don’t need to load more memory compared to full model inference? 

 - Present the Wallclock Time Speedup for Llama-3-8B-Instruct and Llama-3-70B-Instruct Models [R2(rbBk)] 
Sirius is about efficiency. We verify the paper’s claim on Average Parameter Used (APU) efficiency with two different high-quality models with different sizes (Llama-3-8B-Instruct and Llama-3-70B-Instruct) on two different settings: on-chip and offloading. Llama-3-8B-Instruct with its CS model can be placed on a single GPU with VRAM greater than 24GB. We implemented a system using Torch Compile and CUDA GRAPH to show that Sirius delivers its promise on three common high-end GPUs, Nvidia A40, L40, and A100. The llama-3-70B-Instruct model has a whopping 140GB of memory requirement in bfloat16, which cannot fit on any single Nvidia high-end graphic card. For normal users with limited resources, they still can run the model with CPU offload. Sirius is also evaluated and delivers the promised speedup. 

 - Add Evaluations of Sirius on Diverse Dataset and Finer Ablations on 7 models ranging from 7B to 70B in the Llama family [R3(ArEw)] 
We evaluate Sirius in more diverse settings following CS models that identified weaknesses in arithmetic reasoning. Besides GSM8K, we add other arithmetic reasoning datasets AQuA-RAT COT. Also, for commonsense reasoning, we evaluate Sirius' effectiveness on CSQA, StrategyQA, Sports, and Dates. Besides, CS models also struggle at coding, we also evaluate CS models boosted by Sirius on coding datasets HumanEval and MBPP, and show that Sirius can be effective in coding settings.

Meta-review: This paper investigates the contextual sparsity, by first studying the success and failure patterns of applying contextual sparsity method to modern LLMs. The study indicates that the contextual sparsity method often fails in reasoning and deduction tasks like GSM8K. The authors then propose a novel approach, SIRIUS, to improve the contextual sparsity according to the observation. The approach employed speculative decoding like technique to self-correct the decoding process. Experiments show the proposed change leads to a significant improvement in the GSM8K task while maintain the performance on other tasks. 

The authors are very engaged in the rebuttal process, although there are still some concerns like the improvement is relative light compared to the performance drop, most reviewers are positive about the paper. And the study of it is indeed motivated and the results supports most of the claims.
Decision: Accept (poster)