SIRIUS : Contexual Sparisty with Correction for Efficient LLMs

Authors: yang  zhou
zhuoming  chen
zhaozhuo  xu
xi victoria lin
beidi  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:57:38

Abstract: With the blossom of large language models (LLM), inference efficiency becomes increasingly important. Various approximate methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without significant performance degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, it significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models and original models often share the general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces SIRIUS, an efficient correction mechanism, which significantly boosts CS models on reasoning tasks while maintaining its efficiency gain. SIRIUS is evaluated on 6 models with 8 difficult generation tasks in reasoning, deduction, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for SIRIUS and show that SIRIUS delivers theoretical latency reduction with roughly a 20% reduction in latency for 8B model on-chip and a 35% reduction in latency for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git.

Main paper: 
 introduction: 
 Large Language Models (LLM), such as OpenAI et al. (2024) (GPT-4), Team et al. (2024) (Gemini), and Touvron et al. (2023) (Llama) have demonstrated their proficiency in a wide range of natural language processing applications such as content creation, summarization, and impressive and complex reasoning tasks. However, their deployment is very challenging, especially in latency-sensitive settings (Kaplan et al., 2020) . Exploiting the model sparsity is a natural way to reduce the model parameter size and computational cost with a long history (LeCun et al., 1989; Tibshirani, 1996) . More recently, many studies have shown that contextual sparsity (Liu et al., 2023b; Li et al., 2022; Dong et al., 2024; Lee et al., 2024) , which highly correlates to the prompt or the context, can greatly speed up LLM inference without quality degradation. However, in this paper, we first demonstrate a critical and fundamental problem with contextual sparsity (CS): while generally robust in classification tasks and generation tasks that mainly rely on prompt understanding (e.g., summarization, chat question-answering), we found that CS models struggle at high-level reasoning and understanding tasks. For example, in Figure 1 (a), we contrast between the Text Summarization task (CNN/DailyMail) and Arithmetic Reasoning (GSM8K) with contextual sparsity methods on Llama-3-8B-Instruct. Varying sparsity levels, Llama-3-8B-Instruct with contextual sparsity performs consistently worse on GSM8K than CNN/DailyMail. With roughly 50% sparsity globally, the sparse model degradation is still reasonable on text summarization (right axis in color coral) compared to almost collapsed on arithmetic reasoning (left axis in color green). However, for reasoning tasks, can we simply live with a higher density ratio to preserve more performance? Unfortunately, the answer is NO. Besides the significant efficiency loss, shown in Figure 1 (b), the Llama-3-70B-Instruct model with contextual sparsity also crashes at around 50% sparsity globally. The 50% sparse model still has 4× the parameter size compared to the smaller dense model (Llama-3-8B-Instruct), while still performing worse on GSM8K-CoT, rendering contextual sparsity utterly not useful for complex reasoning tasks.We conduct an in-depth study on the CS model failure cases and notice that the overall reasoning pathway of these sparse models is usually sound and adheres to the full model. The fatal mistakes are always caused by some middle tokens and propagate towards the end, examples can be seen in Figure 4 . Following this observation, we conduct a simple experiment with 65% Contextual Sparse Llama-3-8B-Instruct on GSM8K as presented in Figure 1 (c ). We run both the sparse and the full models together for the same prompt and compare two generation output token-by-token. Surprisingly, the trend increases steeply with the percentage of corrected tokens. Correction of only 6% tokens in the sparse model's generation recovers most of GSM8K accuracy (<5% to full), and 11% to recover the full performance. The results show potential for an efficient and powerful correction mechanism to maintain the sparse efficiency while boosting its performance. Contextual sparsity uses a dynamic sparsity pattern and naturally requires the full model to be in GPU memory during runtime, allowing the full model to be used efficiently for infrequent correction. Even though only very few need to be corrected, locating these mistaken tokens efficiently turns out to be challenging.Ideally, we want a correction system to have the following properties: 1) Effective, the sparse model quality degradation can be improved to the full model vicinity; 2) Cheap, the full model only gives minimal intervention; 3) Adaptive, the system is efficient across various reasoning datasets.In this paper, we carefully analyze and formulate correction efficiency in Section 2. We extensively categorize the strengths and weaknesses of CS in Section 3. In Section 4, We systematically design SIRIUS, a correction method covering all three desired properties. (When?) In Section 4.1, we show that the sparse model can be both confident or uncertain when making mistakes, rendering the signal from sparse unreliable for determining when to correct. SIRIUS is a period-based method with the period as a hyperparameter. (How?) In Sections 4.1 and B.2, we introduce novel KV Cache direct rewriting, minimal rollbacks, and hardware-efficient tree building to help increase the effective period of full model correction, thus, ensuring the correction efficiency. Contextual Sparsity requires full model weights to be placed on the GPU memory. While the sparse model doesn't perform well on complex reasoning tasks, Sirius uses the Full Model to correct the Sparse model. The full model is called fairly infrequently. During the correction, the Full Model will rewrite the KV Cache, interleave with high-quality tokens to the sparse outputs, and then roll back only when the token is deemed extremely unlikely by the Full Model.In Section 5, we empirically evaluated SIRIUS on 6 different models with 8 different reasoning tasks and showed that SIRIUS is generally effective and efficient. On GSM8K and Llama-3-8B-Instruct specifically, we boost the fine-grained sparsity from 58% to 72% with 4% increase in effective parameter size and coarse-grained sparsity from 38% to 70% with the cost of 5% effective parameter size. We also show that SIRIUS delivers the promised efficiency on mainstream GPUs in both on-chip and offloading settings. 
related works and problem formulation: 
 In this section, we first present the classification of the prior Contextual Sparsity methods and narrate important efficiency metrics in. Also, we present careful analysis and quantitative comparisons on why Speculative Decoding is inefficient in recovering contextual sparsity. Due to space constraints, we refer to Appendix A.2. For extended related works on model compression, contextual sparsity, and speculative decoding, we present in Appendix A.1.Contextual Sparsity Classification -Contextual sparsity (CS) methods are usually training-free, easy to use, and seemingly effective, making them highly attractive to ML practitioners looking to reduce LLM inference costs. CS exists naturally in MLP layers of the LLM, which occupies roughly 70% of the LLM total weights (Dong et al., 2024; Lee et al., 2024) ). The contextual sparsity selection is as follows: given the context, only a limited number of the most relevant neurons are selected based on the input activation. The rest contributed to the output far less is discarded. We refer to two main directions of contextual sparsity methods as Coarse-grained Sparsity (CSparse) Methods (Dong et al. (2024) ) -that within the same input prompt, the sparsity pattern is fixed for all tokens generated. Fine-grained Sparsity (FSparse) Methods (Lee et al. (2024) ) -that exploits the per-token sparsity to save resources.Average Parameters Used Per Token -A key metric is used to evaluate the efficiency of our proposed method, the Average Parameter Used per token decoded (later referred to as APU). LLM inference is memory I/O bound (Leviathan et al., 2023; Kim et al., 2023) . The latency of generating every single token is dominated by the memory loading time from the GPU HBM to SRAM. On the other hand, SIR-IUS relies on full model parallel verifying a chunk of tokens. Although from the FLOPs standpoint, the amount of compute performed per evaluation step is the number of input token times of a single token input process, the latency of parallel verification is still roughly the same as taking a single token (Verified further in 10, length 64 is only 1.1 ms longer than length 1), because the inference is memory bound.SIRIUS operates in the memory-bound regime (single inference sequence length smaller than or equal to 64). Thus, the average parameter count of a model gives us a rough judgment of the latency of inference. Formally, for a full LLM to have C f ull number of parameters, and its sparse counterpart of a certain predetermined sparsity C sparse . The average advancement length (later we refer to as AAL) in the number of tokens between two consecutive LLM corrections can be represented as n AAL .The average parameters used per token (APU) are the followingAPU = n sparse C sparse +C f ull n AAL (1)We want the metric to be as small as possible, and obviously, we want n AAL to be as large as possible.Another thing to note is that we always compare the system's APU against the full model's APU, which is C f ull . If we divided the above equation by C f ull , we can have an equivalent parameter density of the system defined based on I globalsparsity , which isC sparse /C f ull . Effective Density = n sparse I globalsparsity +1 n AAL (2)Later, if we use period n period , the equation can be rewritten asEQUATIONLater when presenting SIRIUS, we mainly specify n period with n AAL to evaluate its efficiency. Notice that I globalsparsity is determined by the sparsity method, SIRIUS cannot change it anymore. 
observations: 
 In this section, we present a detailed study of the strengths and weaknesses of Contextual Sparsity (CS). 3.1 presents the strengths of CS. 3.2 presents the weaknesses of CS. Additionally, we show that given the similar parameter size, the more well-trained the model is, the more CS degradation will be for the model. Due to limited space, we present the details in Appendix B.1. 3.3 shows our findings when looking into the failure cases of the CS model in complex reasoning generation tasks.In the following series of experiments, we build our implementation2 of fine-grained sparsity based on Lee et al. (2024) and coarse-grained sparsity based on Dong et al. (2024) . The default sparsity for both methods is 50% for the MLP component of the model (whole MLP for coarse-grained sparsity and Up and Down linear layers only for fine-grained sparsity). We mainly use this default setting in most experiment tables in the paper without explicitly mentioning it. Otherwise, we will explicitly specify the different sparsity levels we used.Where Does It Succeed?For tasks on prompt understanding, CS generally performs well and gives consistent and strong output. We evaluate CS models on machine summarization (CNNDailyMail, See et al. (2017) ), and Conversational Question Answering (CoQA, Reddy et al. (2019) ). The results show that the correctly selected contextual sparsity in the MLP layers and the full attention layers can fully extract and understand the local prompt information. More details are presented in Figure 3 , where we show that by varying the sparsity level, the language model's performance on CNN/DailyMail is robust even when the activation sparsity drops to below 20%, which translates to around 44% global density.For tasks accessing factuality and hallucination, we select the generation portion of the TruthfulQA dataset (Lin et al., 2022) . Results are shown in Table 1 , where we evaluate the techniques on 5 different LLMs. Interestingly, we find that the Fine-grained sparsity is often better than the dense model baseline across different models. This finding is consistent with previous works Laser (Sharma et al., 2023) and Dola (Chuang et al., 2024) . They both observed that compressing the original LLM in a carefully designed way would lead to improvement in factuality and better de-hallucination. Laser comes from the low-rank approximation of the MLP layers, while Dola proposes a factuality-aware layer-skipping algorithm. Based on their findings, hallucination occurs when parts of the weights aren't as well-versed in the given input as the other parts. They expose the "averaging" effect that blurs the factuality of the output. Removing these neurons gives rise to better facutality and less hallucination. Our studies look at the same problem from a neuron sparsity standpoint. 3.2 Contextual Sparsity: Where Does It Fail?On the other hand, contextual sparsity severely struggles when the generation tasks rely solely on the model's own reasoning and world knowledge understanding ability. Here we show the Llama-3-8B-Instruct and the Llama-2-7B-Chat models in Table 1 , refer to Table 12 for evaluations on more models. Notice that since fine-grained sparsity method needs the activation from Gate MLP for selecting sparsity, while coarse-grained sparsity has a predetermined pattern after prefilling and can sparsify the Gate MLP. Even though both are at 50% activation sparsity, the coarse-grained sparsity method effectively achieves higher parameter savings than fine-grained sparsity in practice. Here we evaluate the sparse techniques using 5-shot CoT on the GSM8K dataset (Cobbe et al., 2021) . We found that across all the models we evaluated, both sparsity methods lead to significant accuracy degradation. We include HumanEval (Chen et al., 2021) , a coding task that requires complex reasoning and planning ability. We found that both sparsity methods exhibit similar performance degradation when it comes to coding. Shown in Figure 3 , two tasks see sparsity significantly drop performance after 50% activation sparsity.For knowledge recall and world knowledge understanding, we specifically test on MMLU-Flan-CoT (Chung et al., 2022) the CoT text generation version of the MMLU dataset (Hendrycks et al., 2021) . Table 1 shows the results. Stronger models like Llama-3-8B-Instruct suffer from significant degradation too. Furthermore, we found that given the similar parameter size, the more well-trained the models are, the higher its degradation from the contextual sparsity, more details in Appendix B.1.Figure 4 : Examples of contextual sparse model making the identified three different types of mistakes.Most mistakes occur because the model makes calculation mistakes or has a wrong reasoning step compared to the full model. We also observe that there are rare cases where the model makes insensible statements in the middle that make the end result wrong.To study the inability of the sparse model in deduction, we conduct a case study on the sequence-level coarse-grained sparsity methods Dong et al. (2024) with the Llama-3-8B-Instruct model. We visually inspect extensive cases where the sparse model and dense differ in answers. Generally, the sparse model always produces highly similar answers to the dense model: the similar approach or logic flow when approaching the same problem and even the same number of sentences before the first mistake occurs or in success cases. However, the key differences are usually caused by the following three categories of small token-level mistakes: (1) frequent miscalculation in the intermediate steps, (2) wrong reasoning in intermediate steps, and (3) insensible and random statements. For each of the above-summarized cases, we find failure question-answer pairs provided in Figure 4 . These mistakes happen in the middle of arguments and propagate to the wrong end result.Similar observations can also be found for fine-grained sparse methods with different model types.Interestingly, we find that even with these mistakes, the sparse model can still fully generate coherent tokens and make further reasoning assuming their prior steps are correct.We hypothesize that the gap between the full model and these sparse counterparts is at these key tokens. The following simple experiment is conducted to further verify our hypothesis. We run the coarse-grained sparse model and the full model with the same input prompt and for every token the sparse model generates, the full model is used to check the likelihood of these decoded tokens, mainly removing tokens with low likelihood. By varying the likelihood threshold, we can control the frequency of the correction. The experiments are conducted for both Llama-3-8B-Instruct and Llama-2-7B-Chat Touvron et al. ( 2023) models with coarse-grained sparsity. The results are shown in Figure 1 (c). In both cases, we found that a very small amount of correction would drastically improve the sparse model performance, showing a steep gradient when the percentage of corrected tokens is small. With merely 10% of tokens needing to be corrected, the sparse model can completely match the full model's performance. The experiment verifies our hypothesis that by correcting the small portion of key tokens, the sparse model can meet the large model's performance. 
methods: 
 Though we find a minor portion of tokens needed to be corrected for the contextual sparsity model to fully recover performance, the challenge remains: how to locate these mistaken tokens with the minimal number of parallel verification rounds of the full model? In this section, we show that the sparse model provides signals that cannot be trusted 4.1. Then, we describe in detail the various correction techniques in 4.1. Because of the space limit, we put how to boost the sparse generation with hardware-efficient tree building B.2. At the tokens where the sparse made the mistake (red), the entropy isn't in large spikes which signifies chaos and low confidence, rather it is even quite low, compared to nearby entropy spikes. In (b) and (c), we view Sirius as a compression method by itself. We compare Sirius with contextual sparse methods and show that given the same parameter used, Sirius performs better than Contextual Sparse Methods on GSM8K.Intuitively, rather than fixing the n sparse number, letting the system decide when to call the LLM for evaluation would then give more flexible n sparse . Nevertheless, we argue that the sparse model's output probability distribution cannot be used as a metric for accuracy decisions. We empirically experiment with various methods to utilize the information contained in the sparse model's output distribution. However, varying the threshold leads to n sparse being too short when the threshold is strict or failing to correct when the threshold is lenient. We then discovered that the sparse model has very limited self-awareness of its own mistakes. To make the observation concrete, we present a small example in Figure 6 a piece of text where the sparse model makes a mistake while the full model succeeds. The red bars signify the error location. The token entropy is neither high nor at zero, making it impossible to effectively use a threshold to control the number n sparse .The full overview of SIRIUS is presented in Figure 2 and Algorithm 1. The full model is called once every kernel size. The KV cache is shared between the sparse and the full model. The KV cache is mostly populated by the sparse model, which is called for every token. During correction, the full model takes in the last kernel size of tokens and generates its KVs for the past kernel size tokens in parallel, these KVs are directly written to their corresponding positions in the shared KV Cache. Empirically, we found that full model's KV helps the sparse model's output. When LLM is called to evaluate the sparse model's output, it uses its own predicted likelihood to determine whether to accept or reject the sparse model's past output. The decision is based on comparing the likelihood against a preset threshold. Detailed ablation for threshold is in B.3. Besides the above-mentioned techniques, we also found that the "second/third" choices of the sparse models' rejected token position offer > 80% coverage of the LLM accepted tokens. The observation motivates us to build a hardware-friendly tree on the sparse model generating side that doesn't sacrifice the performance while significantly boosting the n AAL or efficiency. Due to the space limit, a great amount of details is in Appendix B.2. 
experiments: 
 In this section, we empirically evaluate SIRIUS to correct CS models on various generation tasks in complex reasoning. We show that SIRIUS is consistent in various tasks, effective in helping CS models recover their performance, and efficient in correction with low additional overhead.• In 5.1, we evaluated SIRIUS on six models with 8 different datasets. SIRIUS is consistently effective and efficient. Specifically, on GSM8K, SIRIUS corrects FSparse Llama-3-8B-Instruct from 58% accuracy to 72% with only 4% increase in parameter density and corrects CSparse model from 38% to 70% with 5% density. • In 5.2, we presents more details on our system implementation for SIRIUS. We show that SIRIUS delivers its theoretical efficient promise, achieving roughly 20% reduction in latency compared to full on-chip on various hardware. SIRIUS further achieves 35% speedup to full in offloading settings.Require: Prompt [x 1 ,...,x t ], full model M F , and sparse model M S sharing weights and sharing KV Cache C, cache_pos is the location where new k and v are written to C, kernel size n Require: forward function FORWARD, threshold r, which is a value used by M F to judge whether the token occurs likely enough Require: StoppingCriteriaMet() downstream task-specific, returns a boolean 1: while not StoppingCriteriaMet() do2: i ← 0 3: kernel ← empty 4: cache_pos ← 0 5: while i < n do 6: set pt+i ← FORWARD(M C , C, [x 1 ,...,x t ], [x t+1 , ..., x t+i-1 ], cache_pos) 7:▷ Running sparse model • We also present ablation with rich details on how each component of SIRIUS contributes to its performance and how threshold is used to trade off efficiency and performance. Due to space limit, we place it in Appendix B.3.Models and Datasets -To comprehensively evaluate SIRIUS performance, we deploy six mainstream LLMs with sizes ranging from 7B to 13B: Llama-2-7B, 13B, and Llama-3-8B with their instruction finetuned counterparts, all from Llama family. Following Wei et al. (2022) in LLM reasoning, we also tested CS models on two popular types of reasoning generation tasks: arithmetic and commonsense reasoning. On the Arithmetic side, besides GSM8K, we also evaluate CS models on AQuA-RAT. For arithmetic reasoning and coding, we use 50% neuron sparsity for both CSparse and FSparse. FSparse relies on the gate layer to be dense, leading to higher global density than CSparse. Since commonsense reasoning tasks are generally less logically challenging comparatively, we lowered the neuron sparsity level to 40%.Main Results -Due to space limits, we only select the best treewidth of SIRIUS for GSM8K, CSQA, and HumanEval for the main results in Table 2 . Extensive studies on the rest 5 datasets with different treewidth are presented in the Appendix C. From Table 2 , we can see that SIRIUS is consistently effective and efficient across all different classes of tasks. Specifically for Llama-3-8B-Instruct, besides GSM8K, SIRIUS corrects FSparse and CSparse, on CSQA, from 61% and 64% accuracy to 70% with cost only 3% sparsity for FSparse and 7% for CSparse respectively. On HumanEval, SIRIUS corrects FSparse from 45% to 61% with 4% sparsity overhead even surpassing the full model's performance, and from 20% to 52% with 8% sparsity as cost. Besides, Llama-3-8B-Instruct, SIRIUS corrects all 6 models with additional sparsity overhead smaller than 10% across these three datasets, further showing its strong efficiency. Besides results in Table 2 , in Appendix C, we show that SIRIUS consistently shows great effectiveness with high efficiency across the rest of the 5 datasets. With kernel size 10, SIRIUS achieves 0.74 APU with accuracy 71.27% accuracy. We use torch compile to optimize the inference latency and limit the overhead other than running model inference. The average latency generated per token is used to compute latency. Results are shown in Table 3 . On average, SIRIUS delivers the promised latency reduction from APU calculations. The speedup ratio on A40 and L40 closely aligns with the theoretical APU reported. On the other hand, A100 and H100 compute MLP more efficiently than it compute attention, making the latency ratio between computing MLP and attention not perfectly aligned with their ratio in parameter size. Therefore, we see that even the sparse model baseline has slightly higher latency as expected. We increase the kernel size from 10 to 16 for these two devices, where accuracy reaches 0.7089 and the AAL reaches 13.67. For A100 and H100, building a hardware-efficient tree is nearly free of cost and highly effective. For numbers in Table 3 , we use the width 4 tree that boosts the AAL to 15.01 out of 16. More details are in Appendix B.2.Secondly, we consider the offloading setting which is the only way for resource-limited users to run 70B models by loading only the weights in use to GPU memory, while the others are offloaded to the CPU. Results are shown in Table 4 . We use a single L40 48GB with a PCIe bus bandwidth of 25 GB/s to run Llama-3-70B-Instruct with batch size 1. Llama-3-70B-Instruct has roughly 80% of parameters to be MLP, which gives the theoretical APU for Griffin to be 0.6. Sparse + Sirius gives 0.649 APU, which is roughly what our system achieved. 
conclusion: 
 We observe that contextual sparse methods significantly degrade reasoning and deduction tasks. SIRIUS, an efficient correction mechanism, enables accurate LLM inference with contextual sparsity. With roughly 11% to 18% sparsity increase, SIRIUS improves fine-grained and coarse-grained sparsity significantly in their performance while maintaining their efficiency gain.Further, SIRIUS is still relying on rollback to correct the tokens that are deemed unlikely, which is inefficient. On the other hand, making the weak-strong model synergy systems that match the performance of the strong while keeping the efficiency of the weak, without strictly matching the strong models' output distribution remains an interesting and unsolved problem. We leave these topics to future works. 
acknowledgement: 
 We would like to thank Feng Liang and Yunong Liu for their helpful feedback during the exploration and writing. We also want to give a special thanks to Hanshi Sun for providing insights and suggestions for efficient implementation and speedup. 2023) is another important LLM inference latency reduction method. Compared to LLM, small transformer models are much more computationally accessible and can effectively model short-range tokens. Therefore, smaller models are asked to speculate short-term future tokens, which the LLM takes in in parallel to trade in FLOPs with memory loading time. During verification, most speculative decoding methods pursue lossless acceleration, leading to frequent rollback during rejection. In contrast, Sirius solves a very different problem. Our method aims to maximally preserve the efficiency of sparse models while boosting its performance. Sparse models, pruned directly from LLM, are much stronger at modeling a longer range of text than draft models, thus requiring much less help from the LLM. Our work aims to find the minimum amount of LLM overhead while boosting its performance to the LLM level. Given the resemblance and relevance of Speculative Decoding to our method Sirius, we will elaborate more in-depth on their differences and Speculative Decoding's inefficiencies when it comes to helping the Sparse method in A.2.A.2 Why Not Using the Speculative Decoding to Correct the Sparse Model?Figure 6 : Speculative Decoding has limitation in efficiency when correcting sparse models.When Speculative Decoding is used to correct sparse using the full model, we will show that the efficiency of the overall process will be largely limited. We followed the common practice from speculative decoding and measured the acceptance rate on different datasets C4 Raffel et al.(2020) and GSM8K Cobbe et al. (2021) . Take the Coarse-grained sparse model as an example. For Llama-3-8B as the full model, the 50% sparse (APU 0.65) model will produce an acceptance rate of 0.71 on C4 and 0.89 on GSM8K. Speculative decoding also use parallel verification in the period-basis. Naturally, to keep the system efficiency high, we need to (1) enlarge the period and (2) increase the average number of tokens accepted (AAL) given the gamma (period -1) value. Take the acceptance rate of 0.89 on GSM8K as an example, following the formulation in Leviathan et al. (2023) , we can calculate the expected number of accepted tokens for every gamma term in the Speculative Decoding literature. AAL = 1-α (γ+1) 1-α . The trend (green) is plotted in Figure 6 We can notice the trend that the average advance length starts to plateau as the gamma becomes larger. Take the gamma of 16 as an example, the period is then 17. The average advance length is only 7.84. The APU is (16 * 0.65 + 1)/7.84 = 1.45, which is larger than the full model 1. The blue line in Figure 6 shows the relationship between APU and gamma.Because of the plateauing effect, for an acceptance rate of 0.89, the best gamma is 2 (period = 3). The optimal APU is 0.86, compared with 0.65 coarse-grained sparse APU. A similar picture can be applied to Fine-grained sparsity as well. The key reasons for the observation are two-fold: (1) the contextually sparse models are too big to be the draft model of the speculative decoding system and to have a large period; (2) Speculative decoding preserves the original model's performance so that the acceptance criteria are usually very strict, which is also not suitable for large period and high average advance length. Following the same spirit, Sun et al. (2024a) also uses a large draft model to do self-speculation, but for them, the authors select gamma = 1 to achieve the optimal speedup of their system. In contrast, SIRIUS brings <0.76 APU in this case with period ≥ 10. Specifically, with a threshold of 0.1, Sirius can correct Llama-3-8B coarse-grained sparsity from 20.85% to 43.9%, compared to the 49.66% full model. With a period of 16 tokens (gamma = 15), Sirius on average can accept 13.4 tokens out of a kernel size of 16 and over 9 tokens out of a kernel size of 10, translating to APU < 0.76, significantly lower than SD does. In this section, we provide several supplemental experiments to the picture. First, we run SIRIUS on Llama-3-70B. However, because of computational limits, we cannot run SIRIUS with the tree on Llama-3-70B with the scale we did for other models. Nevertheless, we do show that 70B has roughly the same pattern as we have seen before, large model sparsity also somehow struggles on reasoning tasks. Second, we provide additional proof for the parallel verification efficiency statement. After that, I show results on where the error is located in the chunk size of 16 tokens. The error is distributed almost uniformly. Last but not least, we also apply SIRIUS on datasets that are reasoning. Lastly, we provide more results on the comparison between models of similar size but have a huge performance gap. We show that given the similar parameter size, the trend is for a more well-trained, powerful model to degrade more from contextual sparsity. We present here more illustration on contextual sparsity and Llama-2-7B-Chat experiments with fixing only a minor portion of tokens is shown in Figure 7 . 
 
 Appendix: We observe another interesting phenomenon: given the similar parameter size, the more well-trained the model is, the more performance degradation contextual sparsity would make on the full models.Here we present two pairs of results. First, we look at the performance between Llama-3-8B-Instruct and Llama-2-7B-Chat with Llama-3-70B-Instruct and Llama-2-70B-Chat. All models are evaluated on GSM8K-COT. We draw these models in CSparse in Figure 8 , and the readers can find more results in Appendix B.8. We can see figures from top to bottom, where even at lower density (more elements are not selected), Llama-2-7B-Chat and Llama-2-70B-Chat suffer from less performance degradation (blue) compared to the Llama-3-8B-Instruct and Llama-3-70B-Instruct models. Furthermore, suppose we focus on Llama-3-70B-Instruct for global density at 60% or lower. In that case, the performance (coral) is degraded significantly, which is comparable or even lower to Llama-3-8B-Instruct full model performance at 0.76, Even at 50% density, the 70B model still has more than 40B parameters, much more expensive than the 8B model. The observation fully manifests the difficulty of using CS in complex reasoning tasks. In this section, we first look at the insights behind whether building the tree can help efficiency, then we detail the specific steps towards tree pruning.The goal for the Sirius system is to make n AAL to be as large as possible. Despite the full model sharing KVs with the sparse model, Sirius still encounters costly rollbacks because of sparse greedily decoded tokens being rejected. Interestingly, we look closely into where the sparse model is likely to make a mistake on GSM8K and AQuA-RAT-COT Ling et al. (2017) with Sirius on Llama-3-8B-Instruct and a kernel size of 16. More details are shown in Appendix B.6. The error distributes almost uniformly across all positions of the kernel size. Also, when the token makes the mistake, besides the greedily decoded tokens, we find that other tokens of lower likelihood offer the potential to boost efficiency. Surprisingly, we found that out of the cases where the greedily decoded tokens are rejected, the probability that the second or third most likely tokens from the sparse being accepted by the full model is reasonably high. Shown in Table 5 , we test on part of the GSM8K rejected cases. The "Second Hit" is defined as the count of the second most likely tokens being accepted by the full model when the greedily decoded token is rejected, while the "Third Hit" is defined as the count of the third most likely token being accepted when the first two are rejected.Both sparsity method has a high acceptance rate, or "Coverage", from the second and third most likely tokens when the most likely token is rejected, showing huge potential for gains in efficiency.To capitalize the potential from the second to third tokens, we propose to build a tree during the sparse generating process (lines 6 to 11 in Algorithm 1. The tree algorithm is similar to Beam Search Freitag and Al-Onaizan (2017). However, to make sure that the tree building and tree parallel correction processes can achieve speedup over cases that don't build trees, we impose strong restrictions on the tree structure we build. For a fixed kernel size, we limit every step to having a fixed number of leaves, or treewidth, through tree pruning based on ranking the cumulative log-likelihood of the path. The resulting tree has a fixed shape for a given kernel size and tree width, but only the interconnection pattern between steps varies based on the pruning and ranking within each step. The details are illustrated in Figure 9 . During verification, out of the treewidth complete paths, we select the one that reaches the longest advance length. In practice, we found that for kernel size 16, when the treewidth is increased to 8, the optimal verification tree is around 64. From Section A.2, we see that the parallel verification of the tree of 64 roughly equals the time the full input 1 token.Therefore, a treewidth of 8 is set as the maximum treewidth when building the tree for kernel size 16 for later. We show that building a tree makes the system significantly more efficient while retaining correction effects. In Table 6 , we present more details complementing Table 3 on the use of tree-building for A100 and H100. We ablate the use of a width 4 tree by not using the tree under kernel size 16. We observe that the speedup improvements solely from the increase in AAL from 13.67 to 15.01 for kernel size 16. Probing Components To understand the contribution and the utility of each component of Sirius, we ablate all components of Sirius in Table 7 . We started by only letting the LLM correct the token it is evaluating (interleaving only). Then, we add on top of it the KV cache correction, and then the rollback. All these three techniques are effective when applied solely. Rollback seems to be the most effective technique. Even when applied alone, rollback asserts significant correction to both the CSparse and FSparse models. Interestingly, KV Cache is also effective alone, bringing a 12% increment for CSparse and an 11% accuracy increase for FSparse. Relatively, interleaving is the weakest. Surprisingly, adding both KV rewriting and rollback is only marginally better than rollback alone. Although it is tempting to think KV Cache rewriting is not useful with rollback, the improvement KV Cache Rewriting brings is a gain in efficiency. When adding the KV Cache Rewriting on top of Roll Back and interleave it significantly improves the efficiency of the correction. For CSparse, adding KV rewrite increases AAL from 12.77 to 13.80. Likelihood threshold to balance Correction and Efficiency We found that the likelihood threshold is important for managing the Sirius correction and efficiency tradeoff. We present results in Table 8 . We ablate this setting on a 30% subsampled GSM8K dataset, and only strict accuracy is reported. The performance is the score, while the efficiency is measured by Average Advance Length (AAL). We can find that with the increase of threshold, the scores generally improve, while the efficiency metric decreases.Building Wider Tree We study the effect of increasing the treewidth. In fact, for every number from SIRIUS in Table 2 , we are selecting from a group of results by different treewidth. We present all of this treewidth and its corresponding accuracy and efficiency numbers in the Appendix C. Importantly, raising treewidth always improves AAL. Although different choices of treewidth usually give similar accuracy scores, there is hardly a pattern on which treewidth always gives the best accuracy. The optimal treewidth can only be found through empirical studies.To diversify the evaluation of Sirius, we also evaluate Sirius's Effectiveness on the Llama-3-70B-Instruct model. MMLU is subsampled 10%, while CNN/DailyMail is subsampled 30%. The following table contrasts with Llama-3-8B-Instruct. We use strict match/flexible extract accuracy GSM-8K-COT, accuracy for MMLU, F1/EM score for CoQA, Rouge-1/2/L score for CNN/DailyMail, and Rouge-1/2 ACC for TruthfulQA. Here we show the benchmark latency on A100, where the input tensor to Llama-3-8B-Instruct has a shape of batch size 1 and a different input sequence length. To get the hardware optimal readings, we use torch compile to compile the whole forward pass of the model. We show that the latency only goes up insignificantly to 64, but the trend of increment to 96 is a bit steep.We look at the distribution of where the error would be inside a kernel of 16 tokens. We run through Sirius with a kernel size of 16 on the entire GSM-8K and AQuA-RAT-COT dataset. The histogram is shown in Figure 10 . We found that the error occurs in a uniform pattern, where it is hard to see any particular region where the tokens are likely to occur the most.Besides, the results on the complex reasoning tasks, we evaluate Sirius on slightly more diverse tasks in Table 12 . Here we present more experiments for the comparison between Llama-2 and Llama-3 family models, which is first mentioned in Section B.1, where we also include FSparse methods together with the CSparse method. The results are in Table 11 . • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification:Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We draw inspiration from the astronomical concept, in which SIRIUS refers to a two-body star system, where one is the brightest star ever detected, while the other is a dim star.Since Lee et al. (2024) doesn't open-source its implementation and it relies on the threshold for determining the sparsity pattern, replicating the method isn't straightforward. Using a threshold also increases the difficulty of determining the actual density of the sparse model. Our implementation uses topk on the Gate Layer activations. The rest is implemented as described in the original method. 


Decision: This paper investigates the contextual sparsity, by first studying the success and failure patterns of applying contextual sparsity method to modern LLMs. The study indicates that the contextual sparsity method often fails in reasoning and deduction tasks like GSM8K. The authors then propose a novel approach, SIRIUS, to improve the contextual sparsity according to the observation. The approach employed speculative decoding like technique to self-correct the decoding process. Experiments show the proposed change leads to a significant improvement in the GSM8K task while maintain the performance on other tasks. 

The authors are very engaged in the rebuttal process, although there are still some concerns like the improvement is relative light compared to the performance drop, most reviewers are positive about the paper. And the study of it is indeed motivated and the results supports most of the claims.

Reviews: review: paper_summary: paper_summary: This paper focuses on enhancing the inference efficiency of large language models (LLMs) through contextual sparsity. While it identifies that contextual sparsity reduces hallucination, it also notes a significant impact on reasoning and deduction performance. To address these drawbacks, the paper introduces SIRIUS, a correction mechanism that effectively improves performance. Specifically, the correction mechanism uses additional parameters (i.e., full model) to check the output of the sparse model based on the input and the same instruction prompt.
strength_weakness: strength_weakness: **Method

1. Based on the results in Tables 1 and 2, this paper claims that “the stronger the model, the larger the quality degradation would be”. However, the studies exclusively utilize relatively smaller models, Llama2-7B and Llama3-8B. Including results from larger LLMs, such as Llama3-70B, would provide a more comprehensive understanding of the impact across different model sizes.

2. As the proposed method uses the probability from the full model to check the correctness of the sparse model, does this require forwarding twice for each token? If so, the cost would be double.

3. It is unclear when to use the correction mechanism. Should it be used after the sparse model has generated all predicted tokens, or immediately after the sparse model predicts each token?

**Experiment

1. In Table 3, if the proposed method uses the full model to correct the output of the sparse model, why is there still a noticeable degradation compared to the full model itself?

2. As the title highlights the efficiency of the proposed method, it would be better to provide some experiments about it like latency (ms) in DejaVu [1].
[1] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML, 2023.

**Minor Issues

There are many typos in this paper, e.g., the title “… Contexual Sparisty …” should be “… Contextual Sparsity …”.
questions: questions: Please refer to Weaknesses.
limitations: limitations: There are no discussions about the proposed method's limitations and potential societal impact.
paper_summary: paper_summary: This paper aims to improve contextual sparsity (CS) approaches: LLM sparsification / parameter pruning methods where the sparsification strategy is conditioned on the input sequence / prompt itself. The paper begins by reproducing contextual sparsity baselines, but applied to modern LLMs: LLama 2 and 3 (7B & 8B respectively), and verifying that their reproduction successfully replicates previous CS techniques on summarization and question answering tasks. The authors then demonstrate that the technique is much less effective on more challenging reasoning tasks such as GSM8K and MMLU-CoT. Through investigation the authors observe that often, by correcting a small amount of tokens during the generation (~10%), the sparse models can significantly improve their performance to be on-par with the full dense model. This motivates their main proposed technique: SIRIUS, which employs a speculative decoding like technique to use the full dense model to score every k tokens (k=16-24 in practice) during the decoding process in order to rollback and use the dense model to correct a token when necessary. By doing so, the method is able to mitigate the losses on reasoning tasks while maintaining most of the efficiency of the sparsification method.
strength_weakness: strength_weakness: 1. The main spirit of this paper is whether we can use some small amount of additional compute budget over the sparse model alone (e.g. via limited calls to the full model) to improve the decoding from the sparse model. However, the authors do not compare against any baseline approaches to do this same thing, such as speculative decoding. Even though the authors claim in Related Work that speculative decoding does not suit this problem since it’s not efficient enough, the authors do not provide any empirical evidence of such.

1. This paper is very much an efficiency paper, which tries to introduce some small amount of corrections from the full model to the dense model, while not doing too much as to nullify the gains from the sparse method. However, the authors only analyze Average Parameters Used Per Token (APU), a sparsity measure, instead of any other efficiency metric (importantly wall-time). 
    - The authors make an argument for why this is in Section 4.1, using findings from previous papers. While this may be a reasonable intuitive argument, the method in this paper has significant differences like KV-cache correction that still makes this valuable to empirically verify. 
    - Moreover, even if I intuitively believe from the author’s argument that the wall time latency of this approach would be somewhere between the sparse and full model’s wall time, it is unclear where on this spectrum it lies, how it compares to other techniques like speculative decoding, or how the tradeoff plays with varying sparsity levels without such measurement and comparison. 
    - To place this into context, the CATS paper, which is one of the methods this work experiments with (FSparse) directly states “Activation sparsity of a model is not sufficient to directly enable wall-clock time inference
Speedups”, and provides wall-clock time analysis.

1. The paper is poorly written, with various spelling errors (in the title), grammatical errors (for example: line 1, Section 4.2 title, and more), placeholders for citations (line 114), and difficult to read citation formatting. In many ways the presentation of this work is significantly not publication ready. 

1. Without the code release, the description of the method is not very detailed, even though Algorithm 1 is provided, and would make it very hard for the reader to understand how this method is implemented from reading the paper alone, which hurts reproducibility. 
    1. For example: the description of the correction algorithm is only described in one paragraph in Section 4.3 with insufficient detail on how the KV cache is updated. 
    1. Algorithm 1 has undefined variables with respect to the KV cache such as “cachetracker”. The description in 4.3 states that the KV cache is directly written to by the full model to correct the past, however Algorithm 1 does not describe this process other than  “Update C_s based on j” is unclear what kind of update is happening to the cache.
    1. The important LIKELIHOOD function is undefined, which uses the full model’s likelihoods to judge whether the sparse model’s output is incorrect. There are no details on what threshold is used here. 

1. Some additional claims are not sound:
    - Line 270: “KV Cache can also be looked as an implicit knowledge distillation method to pass dark information from the LLM to the sparse counterparts” What is dark information?
questions: questions: What were the motivations behind the differences between Sirius (Algorithm 1 / Section 4.3) and speculative decoding? Why not directly use speculative decoding, with the sparse model being the drafting model? (which maybe has more guarantees)

Why is the method named Sirius?
limitations: limitations: The authors briefly describe some limitations but the work is generally lacking discussion here. Mainly the authors mention that the method does not work well under extreme sparsity (to my knowledge almost no sparsity method does in the general case – this is more of an open question than a limitation.) The authors could be more forthcoming here. It would be valuable to elaborate on whether the authors believe the two CS approaches are representative of the space of approaches, whether the experiments are adequate, whether they expect the results to change at different model scales, or for tasks beyond the ones studied here.
paper_summary: paper_summary: The paper introduces SIRIUS, a novel method designed to enhance the efficiency and accuracy of sparse Large Language Models in reasoning and deduction tasks. SIRIUS employs contextual sparsity to reduce computational costs and incorporates an efficient correction mechanism that recovers sparse model performance by correcting only a minimal number of key tokens. Experimental results demonstrate that SIRIUS significantly improves the performance of sparse models on complex reasoning tasks while maintaining efficiency.
strength_weakness: strength_weakness: 1.	There are too few experiments,  so no experiments were conducted on other LLMs， and no experiments were conducted on more contextual sparsity methods
2.	The Mc annotation in pseudocode is unclear
questions: questions: If the correction requires the use of a full model, won't the speed of correction slow down compared to directly using the sparse model? Is the memory usage the same as using a full model directly? If the memory usage is the same as using a full model directly, there is no advantage in memory savings
limitations: limitations: The author improved the performance of the sparse model in reasoning and understanding tasks by adding a full model correction, which requires the use of the entire full model. Although the average parameters used per token increase is minimal, the memory usage is the same as using the full model directly
paper_summary: paper_summary: The paper introduces a sparse LLM correction mechanism (SIRIUS) designed to improve the inference efficiency of large language models through contextual sparsity. Contextual sparsity reduces computational cost by pruning parameters that are less relevant based on the input context. However, this approach degrades performance in tasks requiring high-level reasoning and deduction. SIRIUS addresses this by selectively correcting key tokens, using the full model to rewrite the sparse model's Key-Value cache. Experiments show improvements in strict-match EM scores, particularly on mathematical reasoning tasks.
strength_weakness: strength_weakness: - The KV cache rewriting mechanism requires an available full LLM to be of the same architecture as the sparse model.
- The criteria for when to trigger KV cache rewriting are not clearly detailed, making it difficult to assess the practicality and reliability of the approach.
- It is unclear whether using the full model’s KV cache to rewrite the sparse model guarantees that the rewritten KV cache can be accurately interpreted and utilised by the sparse model, due to the differences in the sparse and full model.
- The hypothesis that the gap between the full and sparse models can be bridged by correcting a few key tokens (L44-45) is not sufficiently validated beyond the provided examples. It remains unclear whether this hypothesis holds across a broader range of tasks and datasets.
- The writing needs to be improved for clarity, for example, 
     - there are a few missing citations (e.g., L114. L284)
     - the tables can be clarified with the best performing variant per task in boldface, and a caption which describes the metrics in the table.
     - Algorithm 1's functions can be better described, for example, the descriptions in "normal forward function FORWARD, forward function with direct cache rewrite KVFORWARD, likelihood judge function of LIKELIHOOD" are unclear.
questions: questions: - How are the likelihood thresholds for detecting and correcting errors empirically tuned? Can the authors provide more details on this process and its impact on performance?
- How does SIRIUS handle long-term dependencies and corrections over extended sequences of text? Are there scenarios where the KV cache rewriting might not be sufficient to correct extended errors?
limitations: limitations: not observed negative societal impact to my understanding.

Comments: title: Thank you for your comment
Thank you for your comments
Thank you for your careful reading and raising the score
A Call for AC's Discretion (Thanks to all the reviewer for your attention on Sirius)
Follow-up on Previous Response
Response to reviewer
Response by Reviewer ArEw
Kindly Asking to Reconsider Your Score
Kindly Asking to Reconsider Your Score
Updated rating
comment: We would like to express our deep appreciation for your time reading our work. Your insightful comments on KV cache rewrite mechanism really help us reflect on our previous description of Sirius, and it motivates us to refine it for more readers to understand more easily. Thank you for your comments.
We would like to express our deep appreciation for your time reading our work. The comment on lacks of empirical results help us make our study on Contextual Sparsity correction more concrete.
We would like to express our deep appreciation for your careful reading and insightful comments. Your comments hit home on the important insight of Sirius. We really appreciate your time and effort spent on our work.
Dear Area Chair, 

We are blessed to receive a group of supportive reviewers and an engaging discussion. We believe that we have effectively addressed most of the reviewer's genuine concerns, leading our score to increase from `4345` to `4656`. **However, despite most reviewers looking positively at our work, there is still a lingering concern that our last-minute response doesn’t wait for its feedback.** Thus, we are genuinely looking for AC’s discretion to judge our work’s value. To facilitate AC’s review process on our lengthy discussion, we here provide a one-stop overview and summarize the problem we solve, the method Sirius, the concerns, and how we address them. 

**Problem and Sirius** 

Contextual Sparsity (CS) refers to the sparsity methods that dynamically change their sparsity pattern during inference based on the context. For LLMs, these methods are especially favorable to static sparsity because CS tends to achieve a higher sparsity level (more efficient) than static methods while maintaining the original full model’s performance. Despite their success on text summarization and conversation tasks, where generation depends on prompts, we show contextual sparsity incurs significant degradation in asks that require LLM’s reasoning and world common sense knowledge. 

We present Sirius for the challenge. Since the full weights are natively loaded in GPU memory during CS generation, Sirius efficiently corrects the sparse model by infrequently calling the full model to rewrite the sparse model’s KV Cache, interleaving the full model’s decoded token, and infrequently rolling back unlikely tokens. We show Sirius strong correction with minimal cost through extensive empirical experiments. 

**Main Concerns we faced** 

*Difference with Speculative Decoding* (`rbBk`) 

We appreciate the brilliant question. We show that typical LLMs (Llama-3-8B) tend to achieve a very high acceptance rate (0.89) under the SD setting, but still not enough for the verification period to go past 5 because of the plateauing effect. However, increasing the verification period is the key to amortizing the full model's verification overhead, the longer the period, the more efficient the model is. Sirius can achieve an average advance length of over 13.6 out of a period of 16, compared to SD's less than 9, bringing the average parameter used per token to be lower than 0.76 compared to 0.86 minimum by SD. 

*Practical Speedup from Sirius* (`a8uY` and `rbBk`)

To address the concern that Sirius lacks practical speedup in the paper. We further implement the system for two mainstream models (Llama-3-8B and Llama-3-70B) under two different settings (on-chip and offloading). We show that for 8B model under the on-chip setting, we correct the performance from 36% to 73% for GSM8K-COT with a latency ratio from 0.65 to 0.78. For 70B under offloading, we correct from 76% to 87% with a latency ratio from 0.62 to 0.64. Both experiments show that Sirius maintains a strong tradeoff between performance and accuracy. 

*Miscommunication on Sirius method* (`a8uY`,`ArEw`,`BB9w`) 

We carefully answer the reviewer's confusion and take in their suggestions in our further refined version of the paper. Reviewers raising the concern reflect that their concerns are addressed during further exchange. 

*Lack of empirical evaluations* (`ArEw`) 

We further evaluate Sirius across 6 different models on 7 different tasks ranging from arithmetic reasoning, commonsense reasoning, and coding. The experiments have shown that Sirius is effective in general settings. The review expressed the concern is effectively "alleviated". 

*Given the similar parameter count, the more well-trained the model is, the more degradation would be* (`a8uY`) 

The original claim isn't clearly defined, which is corrected: given the similar parameter count, the more well-trained (powerful) the model is, the more degradation contextual sparsity would bring to the sparse model. To support the claim, we present experiments comparing Llama-2-7B to Llama-3-8B and Llama-2-70B to Llama-3-70B. We show clearly that given the similar parameter size, Llama-3 models that generally score higher than Llama-2, lose accuracy much more from CS under different sparsity levels. We believe that the experiment concretely validates the claim. Also, the observation makes the challenge more urgent when models now become more well-trained given the same parameter size. We deeply appreciate this concern raised. 

**However, since the results are presented at the last minute, we worry that the reviewer doesn't have enough time to go through and provide feedback.** 

---

We wish that the overview we presented is concise for reviewers and AC. Also, we hope that the overlooked and important problem our work revealed will be attended to by the efficiency community, and Sirius will be the pioneering step in bridging the gap between contextually sparse models and the full models. 

Sincerely, 

Paper20976 Authors
We again appreciate the review’s suggestions and inquiries on the efficiency of the claim in our paper. We follow up on our previous message with more experiment results on the comparisons on Llama-2-70B-Chat to provide more empirical studies on the claim. In the table below, we compare two pairs of models on GSM-8K-COT with similar parameters and the degradation after applying the contextual sparse methods (due to time limit, we subsample 20% from the entire dataset for 70B model). Also, we vary the sparsity level to have 50%, 40%, 30% of non-zero values in the weights, where lower than 30% would lead to Llama-3-8B-Instruct to have 0 accuracy). 

| Llama-3-70B-Instruct | Accuracy | Degradation | Llama-3-8B-Instruct | Accuracy | Degradation | 
|----------------------|----------|-------------|---------------------|----------|-------------|
| Full | 0.9205 | | Full | 0.7462 | |
| Csparse 50% | 0.7652 | 0.1553 | Csparse 50% | 0.3636 | 0.3826 | 
| Csparse 40% | 0.6023 | 0.3182 | Csparse 40% | 0.1856 | 0.5606 |
| Csparse 30% | 0.3144 | 0.6061 | Csparse 30% | 0.0644 | 0.6818 | 
| Fsparse 50% | 0.8864 | 0.0341 | Fsparse 50% | 0.6477 | 0.0985 |
| Fsparse 40% | 0.8485 | 0.072 | Fsparse 40% | 0.4053 | 0.3409 | 
| Fsparse 30% | 0.7386 | 0.1819 | Fsparse 30% | 0.0265 | 0.7197 |
| **Llama-2-70B-Chat** | **Accuracy** | **Degradation** | **Llama-2-7B-Chat** | **Accuracy** | **Degradation** | 
| Full | 0.4508 | | Full | 0.1856 | | 
| Csparse 50% | 0.3939 | 0.0569 | Csparse 50% | 0.1515 | 0.0341 | 
| Csparse 40% | 0.3447 | 0.1061 | Csparse 40% | 0.1098 | 0.0758 | 
| Csparse 30% | 0.2689 | 0.1819 | Csparse 30% | 0.072 | 0.1136 | 
| Fsparse 50% | 0.3864 | 0.0644 | Fsparse 50% | 0.1629 | 0.0227 | 
| Fsparse 40% | 0.3902 | 0.0606 | Fsparse 40% | 0.1364 | 0.0492 | 
| Fsparse 30% | 0.2689 | 0.1819 | Fsparse 30% | 0.1212 | 0.0644 |

From the table, we can clearly see that given similar parameter size, contextual sparsity brings more degradation to Llama-3 family models than to Llama-2 models (Please read the table vertically to compare between Llama-3-70B-Instruct with Llama-2-70B-Chat and Llama-3-8B-Instruct with Llama-2-7B-Chat). 


Surprisingly, we also notice another interesting phenomenon where the models with larger parameter sizes seem to be more resilient to the contextual sparsity methods. Please read the table horizontally to compare models within the Llama-3 family models and Llama-2 family models. The trend is as expected since the model with a larger parameter size often has more redundancy in parameters. 


Furthermore, the contextual sparsity’s weakness is in full display in the above table. We can see that for Llama-3-70B-Instruct, even at 50% sparsity, the performance on GSM-8K-COT is comparable to Llama-3-8B-Instruct. Given that the sparse 70B model still has over 40B parameter size, the performance degradation is unacceptable, let alone a sparse 70B model with lower sparsity levels (40% and 30%). Sirius corrects the sparse model to have 87% accuracy and incur negligible overhead during the offloading setting, again showing its effectiveness in helping contextual sparsity methods. 


Also, as a follow-up on the efficiency-accuracy tradeoff concern, we would emphasize another set of results that was previously overlooked. Please be aware that the accuracy after Sirius correction is dependent on the sparsity level and the sparsity method. Previously, we looked at the coarse-grained sparsity methods where the sparsity pattern is determined for every input prompt and fixed throughout the generation. For the more flexible fine-grained sparsity, where the sparsity pattern changes for different decoded tokens, the sparse method alone achieved 59.68% accuracy on GSM8K-COT, versus the full model’s 75%, Sirius can correct the fine-grained sparse models to 74% accuracy with 0.775 theoretical Average Parameter Used to full model. 

Thank you again for your additional thoughtful and acute comments. If you think that your concerns have been addressed, please consider adjusting your score.
We thank the reviewer again for additional feedback and suggestions. We agree that our previous claim “the stronger the model, the larger the quality degradation would be” is not well-stated without enough context and quantifications. Also, the experiments we presented in the first round of rebuttal is not clear.

We here clarify the claim as: **given the same number of parameters**, the more well-trained (powerful) the model is, the more performance degradation when applying contextual sparsity method would be. In the paper, we present the experiments of Llama-3-8B-Instruct and Llama-2-7B-Chat. Now, we further present the comparison between running Llama-3-70B-Instruct and Llama-2-70B-Chat on GSM8K COT dataset. The experiments are now running and will be soon added. Again, we are deeply grateful for the reviewer’s insightful suggestions on the relationship between performance degradation from contextual sparsity and full model performance.

On the speedup portion, we would like to point the reader to the speedup ratio between the plain sparse and dense models measured on different devices from the CSparse methods on Llama-3-8B-In (In the table, we show A40, L40, and A100).  Theoretically, the CSparse method only prunes the model by 45% of parameters on Llama-3-8B-In. Therefore, the method will at most achieve 0.65 of the latency as the original dense model, given the LLM inference is bounded by memory. We found that our implementation is very close to optimal for A40 and L40, as the sparse-to-dense ratio on these two devices is close to 0.65 theoretical value, and on top of that, Sirius incurs an additional 10% of the dense model latency as the overhead of correction. **Compared to the sparse model’s original accuracy on GSM-8K-COT 36%, Sirius now corrects it to 73%, which is more than doubled. We argue that Sirius achieves a reasonable efficiency-accuracy tradeoff on A40 and L40.** Moreover, compared to A100, L40 and A40 are much cheaper commodity hardware, which is closer to the users of sparse models that are more resource-limited. 

Admittedly, our current implementation is slightly not as optimal on more high-end GPUs like A100, since the raw sparse-to-dense ratio is close to 0.72 already, which is 0.06 higher than the theoretical value. We found that the slightly suboptimal ratio is caused by extremely high-end GPUs like A100 that are much more demanding on the attention kernels we use. However, building an optimal attention kernel for a relatively short context is beyond the scope of our project. Once the next-generation more optimized attention kernel is rolled out, we will have a similar speedup ratio on A100 as on other slightly slower GPUs. Again, we argue that A100 is much more expensive to rent, making it further away from the target audiences of sparse models. 

Furthermore, we would like to point readers to the accuracy and efficiency tradeoff we have on the Llama-3-70B-In in the offloading setting (part of the weights are loaded on GPU's on-chip memory, the rest is offloaded on CPU RAM), which is one of the only ways a normal practitioner would run inference on the 70B model without high-end GPU clusters. We found that Sirius corrects the CSparse 70B model from 0.76 to 0.87 in accuracy, while only increasing the latency by roughly 2% of the original model latency. (Please note that there is a typo in the offloading 70B table, on the Performance row, the sparse and dense data should be swapped). **Again, we argue that the Sirius method achieves a reasonable tradeoff between Efficiency and Accuracy for the 70B model as well.** 

Thank you again for your additional thoughtful and acute comments. If you think that your concerns have been addressed, please consider adjusting your score.
The rebuttal addresses some of my concerns. I have increase my score to 6.
Thank you for your detailed explanation and rich experimental supplements. It alleviated my concerns to a certain extent. I raise my score to 5.
Thanks for your detailed response. It has addressed some of my concerns. However, I am still concerned that 1) the claim “the stronger the model, the larger the quality degradation would be” since it does not match the experimental results on more SoTA models like Llama-3-70B-In vs. Llama-3-8B-In in the response. 2) The speed-up is slight (e.g., A100: 11.8 ms -> 13.3 ms) compared to the degradation of performance (0.7612 -> 0.7309). Thus, I will keep my score unchanged.
Thank you again for your feedback.

We have added the necessary material to the author's rebuttal to address the points you raised. Given that we only have four reviewers, each reviewer's score significantly impacts the overall assessment of our work. We believe that the current overall assessment does not adequately reflect the contribution of our work. Therefore, we kindly request you reconsider your score. Thank you again for your time and effort in reviewing our paper.
Thank you again for your feedback.

We have added the necessary material to the author's rebuttal to address the points you raised. Given that we only have four reviewers, each reviewer's score significantly impacts the overall assessment of our work. We believe that the current overall assessment does not adequately reflect the contribution of our work. Therefore, we kindly request you reconsider your score. Thank you again for your time and effort in reviewing our paper.
Thank you for providing your extensive clarifications and updated results. In particular, the latency evaluations on real hardware, additional ablations, and revised presentation changes my impression of this paper a lot. I have increased my score to 6. 

Nit: there is still a typo in the title "Sparisty", and other places like Figure 3 title "Efficientcy". Please continue to refine for the future versions.