CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation

Authors: Wang Qinsi, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, Yiran Chen
Primary area: foundation or frontier models, including LLMs
Keywords: Sparse Activation, Sparse Inference, Efficient LLMs, LLMs Inference Acceleration

Abstract:
Large language models (LLMs) with billions of parameters have sparked a new wave of exciting AI applications. However, their high computational costs and memory demands during inference pose significant challenges. Dynamic sparse activation inference, which activates only a small number of neurons for each token, offers a novel way to accelerate model inference without degrading performance, showing great potential for resource-constrained hardware devices. Nevertheless, existing methods predict activated neurons based on individual tokens with additional MLP, which involve frequent changes in activation maps and resource calls, limiting the acceleration benefits of sparse activation.
In this paper, we introduce **CoreInfer**, an MLP-free adaptive sparse activation inference method based on sentence-level prediction. Specifically, we propose the concept of sentence-wise core neurons,  which refers to the subset of neurons most critical for a given sentence, and empirically demonstrate its effectiveness. To determine the core neurons, we explore the correlation between core neurons and the sentence's semantics. Remarkably, we discovered that core neurons exhibit both stability and similarity in relation to the sentence's semantics—an insight overlooked by previous studies. Building on this finding, we further design two semantic-based methods for predicting core neurons to fit different input scenarios. In CoreInfer, the core neurons are determined during the pre-filling stage and fixed during the encoding stage, enabling zero-cost sparse inference. We evaluated the model generalization and task generalization of CoreInfer across various models and tasks. Notably, on an NVIDIA TITAN XP GPU, CoreInfer achieved a 10.33 $\times$ and 2.72 $\times$ speedup compared to the Huggingface implementation and PowerInfer, respectively.

Decision: reject
Reviewer scores: 3, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730332441525

thread_type:
official_review

<thread_object>
creator: reviewer_7ifw

<content>
summary:
This paper proposes an MLP-free adaptive sparse activation inference method based on
sentence-level prediction.

strengths:
1. The paper is easy to follow, with clear writing and presentation.
2. Evaluation results are comprehensive.

weaknesses:
1. In lines 93-94, it should be 'nearly lossless generation'
2. How would the method scale to larger models such as LLaMA-70B in multi-GPU settings?
3. Would this method be compatible with quantization methods?
4. The authors should also discuss related works on sparse KV cache [1-4], another important direction to accelerate LLM inference.

[1] Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference, MLSys 2024.

[2] Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache, MLSys 2024.

[3] ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching, ISCA 2024.

[4] InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management, OSDI 2024.

questions:
Please see the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671431328

thread_type:
official_review

<thread_object>
creator: reviewer_2da3

<content>
summary:
This work proposes an activation sparsity method to accelerate LLM inference. Specifically, the method identifies core neurons at the sentence level during the pre-fill stage when the input sentence exhibits stable semantics. Furthermore, by analyzing the relationship between core neurons and semantics, the method can be adapted to tasks with variable sentences by clustering samples into distinct groups. Experimental results demonstrate a significant inference speedup with an acceptable accuracy drop.

strengths:
1. This work has a clear motivation. In related work, token-wise sparsification and the inclusion of an additional learnable predictor both present challenges, as they lead to extra inference overhead.

2. The proposed concept of sentence-wise core neurons is novel. It's intriguing that core neurons can be directly identified in the pre-fill phase without the need for a additional predictor.

3. In the experiments, the proposed methods shows significant inference speedup with acceptable accuracy drop.

weaknesses:
1. The proposed method involves three manually tuned hyper-parameters (alpha, beta, gamma) to determine final sentence-wise core neurons, that maybe limit its practical usage. 
 
2.  The claim for Fig.2(C) needs to be further justified.  I agree with the authors that Fig.2(C) shows some of the activations are close to each other for the same sentence. However, Fig.2(C) also exists some of group neurons with same color (e.g., same sentence) are located superlatively. How do the authors claim that "the distribution of core neurons of tokens in the same sentence is always closer".  

3. In the experiments, some of the results are unclear. (Please see the detailed comments in the Question below)


Others:

A. It may be inappropriate to refer to the proposed method as "zero-cost sparse inference" in the Abstract, as sparse inference inherently involves some cost.

questions:
Overall, I find the proposed method well-motivated and innovative. However, the current version has several areas that require further justification. Specifically, please address the following questions:

1.  Weakness.2 as shown above.

2. The speedup gain is consistent across OPT-6.7 and other LLMs. In Table 3, the proposed method achieves a 10.33x speedup over the original transformer on OPT-6.7B; however, the speedup is notably lower on other LLMs, as shown in Fig. 7. For instance, on Llama2-7B with a sequence length of 256, the speedup is around 1.6x.

3. The system memory usage needs further clarification. According to Fig. 7, the proposed method shows no system memory consumption, while the original transformer uses substantial system memory. My understanding is that the proposed method uses activation sparsity to accelerate inference in the original transformer without any system-level modifications. Could you clarify this aspect?

4. The work lacks a comparison of perplexity (PPL) or zero/few-shot accuracy. The authors focus only on hardware performance comparisons with previous works, such as Deja Vu and PowerInfer. How does the accuracy of the proposed method compare to these works? Additionally, what is the sparsity level of these other methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706027421

thread_type:
official_review

<thread_object>
creator: reviewer_aw95

<content>
summary:
CoreInfer determines the top Beta neurons activated (pooling across tokens in the prefill) and then uses just these for the decoding, if these are found to be stable (if this group stays approximately the same). If these are not found to be the CoreInfer then defers to a "similarity-guided" prediction, that takes core-neurons found to be semantically similar (available in some datasets like ag_news, or found via K-means clustering).

The result is an MLP-free method of actively reducing the effective size of the FFN, only reusing these "core neurons", causing speed-ups observed up to around 10x for decoding from saved calculations and reduced need for memory movement. Similar results and accuracy is observed in several cases.

strengths:
# Strengths

This paper investigates prediction of which neurons will be activated, stemming from an observation that similar semantics (especially per sentence) tend to have a similar set of neurons activated. The amount of savings in inference time is quite large, suggesting a high degree of sparsity in the FFN -- especially when using ReLU activation functions -- which unblock running large language models efficiently on smaller GPUs and edge-devices.

## Significance:

The FFN holding the greatest number of weights as the model increases in size, and hardware needing to begin loading early (to prevent bubbles in pipelining), this provides an excellent recipe efficient hardware decoding for LLMs.

The majority of energy being spent on off-chip DRAM f  Might be interesting to check what the average performance would be omitting this fall-back in the unstable case, while increasing the size of the accepted core-neuron group.
etching (see "Melting Point" and "MobileLLM" papers), this also provides an excellent route for running LLMs on constrained hardware.

## Clarity and Originality

The paper is very clear in it's motivations and methods, and how it differs from related works.  Prior works tend to use MLPs and learned parameters, which this paper claims add overhead which reduce the inference savings. This work focuses on two techniques for on-the-fly decoding acceleration, without additional learned parameters (maximizing speed ups and energy/memory savings of the technique in edge-inference)

weaknesses:
The semantic stability is noted by the paper to sometimes be not strong enough to warrant restricting to neurons characterized in the prefill stage.

When this happens the authors explored finding semantic similarity to core-neuron sets already discovered, allowing for time studied into inner understanding of the LLM's "core-neuron sets" to make gains in inference speed.

Adding more clear labelling would benefit a few of the figures.  Perhaps Asterisks on Table 2 indicating on which benchmarks used the Stable vs Unstable methods.  Figure 6 preferably would have titles, to clarify which graph refers to which method, and should either be on the same graph or use the same x and y scales to make it easier to compare.

questions:
Sentence length was found to be a distinguishing factor for when the semantics are stable enough to dynamically determine the core-neurons, would there be any data for how this transfers across very different looking tasks (e.g. programming where sentences are not as well defined) and languages with different average sentence lengths and syntax?

How does this correlate with tokenization? Namely, if the language is caught in the byte-fallback of the tokenizer (e.g. Mandarin characters for r50k Tiktoken) would the number of core neurons increase?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730824889411

thread_type:
official_review

<thread_object>
creator: reviewer_55rr

<content>
summary:
This work proposes a new sparse activation inference method based on the semantics of input sentences. It finds that the activated neurons remain relatively stable for semantically consistent sentences and thus proposes to only activate the frequently used neurons based on the input context. Experiments validate the effectiveness of the proposed method in improving throughput compared to previous predictor-based sparse activation methods.

strengths:
1. The paper is clearly written and easy to follow.

2. The proposed method is motivated by an analysis of core neurons and semantics.

3. Experiments demonstrate that the proposed method achieves improved throughput compared to previous predictor-based methods.

weaknesses:
1. The major concern with this paper is the general applicability of the proposed method. For user input contexts and questions, especially for long-context reasoning, it is difficult to ensure and inaccurate to assume that all inputs are semantically stable and full of relevant information. Thus, the assumption underlying the proposed method is too strong and may not be applicable in real-world applications or more challenging questions.

Additionally, for similarity-guided prediction, core neurons are identified by finding the closest semantic groups. If this is the case, the insight of this work is generalized to the following one: similar topics will always activate the same set of core neurons, which is also a too strong assumption. Similarly, the input context may not contain only one topic.

2. As a follow-up to the above concern, the proposed method is evaluated on a limited number of tasks. It would be highly desirable for it to be evaluated on a larger set of commonsense reasoning tasks and long-context tasks, such as LongBench, to validate whether the sparsified neurons can handle more complex input contexts.

3. The proposed method is not benchmarked against predictor-based sparse activation methods in terms of task accuracy.

4. Some design aspects of the proposed method need more clarification: (1) it is unclear how semantic similarity is computed; (2) it also remains unclear how to determine whether the input is stable.

questions:
My questions are listed in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732126992957

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Statement

comment:
Dear Reviewers,

We sincerely appreciate all of you for your valuable comments and suggestions. We are grateful for the opportunity to address some concerns and improve the manuscript, with changes highlighted in blue (text) and orange (figures and tables). Specifically, we made the following major changes:

**1. Testing Performance on Additional Long-Context Datasets.**  Based on the reviewers' suggestions, we evaluated our method on more challenging tasks from the LongBench dataset using the Llama2-7B-chat-4k model. We tested 21 tasks covering key long-text applications such as single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. We found that CoreInfer achieved near-lossless performance across different tasks while activating only 20% of neurons. Notably, CoreInfer led to improved performance on QA tasks, increasing the F1 score from 25.38 to 29.36 on MultiFieldQA-en, and from 9.21 to 12.86 on MultiFieldQA-zh. *Detailed results can be found in Appendix A.4.1.*

**2.** **Verifying Applicability to Diverse Inputs (Including Chinese and Code Programming).** Per the reviewer’s suggestions,  we explored the behavior of core neurons when the input was in different languages. We found that core neurons tended to stabilize as the input length increased, regardless of the language. However, the token length required to reach stability varied depending on the language, as different languages require different numbers of tokens to express semantics. We verified that CoreInfer performs well on Chinese text QA and code prediction (Java/C#/Python) tasks. For example, on the Lcc and RepoBench-P datasets, CoreInfer experienced less than 3% performance loss even with a 20% neuron sparsity level. *Detailed exploration and experimental results can be found in Appendices A.2.3 and A.4.1.*

**3.** **Adding Comparisons with Predictor-Based Methods.** Based on the reviewers' suggestions, we added a comparison of CoreInfer's performance with the state-of-the-art predictor-based method PowerInfer on three different models and four classic commonsense reasoning tasks. We found that both CoreInfer and PowerInfer achieved near-lossless performance across all models and tasks. However, the size of the predictor required by PowerInfer increased with the model size, leading to higher training and inference costs. For example, for OPT-30b, PowerInfer requires approximately 10.45GB for the predictors, which accounts for about one-sixth of the total model size. *Detailed results are presented in Appendix A.4.2.*

**4.** **Verifying Adaptability to Quantization.** As suggested by the reviewers, we combined CoreInfer with 4-bit quantization (including NF4 and FP4). We found that CoreInfer achieved lossless sparse inference even with applying 4-bit quantization, as the activation of neurons was not affected by quantization. After combining CoreInfer with the FP4 and NF4 quantization formats, the average accuracy improved from 69.98 and 68.83 to 69.99 and 68.89, respectively. Therefore, CoreInfer can be combined with state-of-the-art quantization methods for further acceleration. *Detailed results are presented in Appendix A.4.3.*

We sincerely appreciate all of your suggestions, which have led us to a deeper thinking. We welcome any further feedback from you and are more than willing to address any questions or concerns you may have. 

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:** The paper introduces CoreInfer, a method for accelerating LLM inference through semantics-inspired adaptive sparse activation. CoreInfer predicts core neurons critical for sentence semantics during the pre-filling stage and removes other neurons during subsequent inference, thus improving efficiency without relying on on-the-fly predictors. Experimental results across various tasks validate the effectiveness of the strategy.

**Strength:** 

1. Intriguing observations regarding sentence semantics: The observations regarding the stability and similarity of core neurons with respect to sentence semantics are insightful.  

2. Boosted efficiency: CoreInfer can achieve non-trivial inference speedups on real devices while maintaining accuracy on most tasks.  

**Weakness:**

1. Too strong assumptions: The method assumes stable semantics and neuron activation patterns across the entire input context, which may not generalize to complex inputs.

2. Insufficient experiments: Limited benchmarks against state-of-the-art static and dynamic pruning methods (e.g., Deja Vu) reduce confidence in its comparative advantages.

3. Accuracy drops on key tasks: CoreInfer showed notable accuracy declines on tasks like ARC-E and MMLU, raising concerns about its robustness under high sparsity.

**Reasons for the decision:**

While CoreInfer presents a novel approach to sparse activation and demonstrates efficiency gains, its assumptions about semantic stability and the lack of sufficient comparisons with key baselines limit its generalizability. Additionally, significant accuracy drops on critical tasks under high sparsity weaken its overall contribution.  

Although only Reviewer 55rr provided a negative rating initially, during the reviewer discussion stage, they further emphasized the above concerns, which were agreed upon by Reviewer 7ifw. Both reviewers leaned towards rejecting the submission, and the other two reviewers did not strongly advocate for the work. Therefore, my overall recommendation is to reject.