Title: D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models

Dynamic Inference

Inference Acceleration

Adaptive Computing Resource Allocation
Abstract: Large language models have shown an impressive societal impact owing to their excellent understanding and logical reasoning skills. However, such strong ability relies on a huge amount of computing resources, which makes it difficult to deploy LLMs on computing resource-constrained platforms. Currently, LLMs process each token equivalently, but we argue that not every word is equally important. Some words should not be allocated excessive computing resources, particularly for dispensable terms in simple questions. In this paper, we propose a novel dynamic inference paradigm for LLMs, namely D-LLMs, which adaptively allocate computing resources in token processing. We design a dynamic decision module for each transformer layer that decides whether a network unit should be executed or skipped. Moreover, we tackle the issue of adapting D-LLMs to real-world applications, specifically concerning the missing KV-cache when layers are skipped. To overcome this, we propose a simple yet effective eviction policy to exclude the skipped layers from subsequent attention calculations. The eviction policy not only enables D-LLMs to be compatible with prevalent applications but also reduces considerable storage resources. Experimentally, D-LLMs show superior performance, in terms of computational cost and KV storage utilization. It can reduce up to 45\% computational cost and KV storage on Q\&A, summarization, and math solving tasks, 50\% on commonsense reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes D-LLMs, a novel dynamic inference framework for large language models (LLMs) that adaptively allocates computing resources based on the importance of individual tokens. By introducing a decision module for each transformer layer, D-LLMs can decide whether to execute or skip specific layers for each token, optimizing computational efficiency. The framework also includes a KV-cache eviction strategy to maintain compatibility with existing acceleration techniques and reduce storage overhead. Experimental results demonstrate that D-LLMs can significantly reduce computational costs and KV-cache storage by up to 50% without compromising performance across various tasks, including Q&A, summarization, and commonsense reasoning.\n\nSTRENGTHS:\nThe paper demonstrates significant originality by introducing D-LLMs, a dynamic inference framework that adaptively allocates computational resources in large language models. This novel approach addresses a critical challenge in deploying LLMs on resource-constrained platforms, offering a creative solution that dynamically determines the necessity of executing each transformer layer based on token importance. The quality of the work is evident in the thorough design and implementation of the decision modules and the KV-cache eviction strategy, which ensure compatibility with existing acceleration techniques. The clarity of the paper is commendable, as it systematically explains the motivation, methodology, and experimental setup, making it accessible to a broad audience. The significance of this research lies in its potential to drastically reduce computational costs and storage requirements for LLMs, making high-performance language models more practical for a wider range of applications. This work not only advances the field by improving efficiency but also opens up new avenues for optimizing model deployment in real-world scenarios.\n\nWEAKNESSES:\nOne key limitation of this work is the lack of detailed discussion on the granularity of tokens. The effectiveness of the D-LLMs framework heavily relies on accurately assessing the importance of each token during the inference process. However, the paper does not elaborate on how tokenization is handled, whether different granularities were considered, or the impact of token granularity on the model's performance and computational efficiency. This omission leaves questions about the generalizability and robustness of the proposed method across different types of tokens and tasks.\n\nAdditionally, the conditions under which the decision module determines to skip or execute layers are not clearly defined, leaving some ambiguity about the decision-making process. The paper would benefit from a more detailed explanation of the hyper-parameters used in the experiments, as this information is crucial for reproducing the results and understanding the model's behavior. Furthermore, there are instances of inconsistent terminology, such as the use of \"dynamic decision module\" versus \"execution decision module,\" which can cause confusion.\n\nEmpirical analysis or ablation studies comparing different levels of token granularity, along with more precise definitions of the decision criteria and detailed hyper-parameter settings, would have strengthened the validation of the approach and provided clearer guidance for practical implementation.\n\nQUESTIONS:\n1. **Token Granularity:**\n   - How does the framework handle tokenization? Have different levels of granularity (e.g., subwords, words, characters) been considered, and how do they impact the model's performance and efficiency?\n   - Can you provide empirical results or ablation studies that compare different levels of token granularity to validate the chosen approach?\n\n2. **Decision Module Criteria:**\n   - What specific criteria or metrics does the decision module use to determine whether a transformer layer should be executed or skipped for a given token?\n   - Are there any thresholds or heuristics involved in this decision-making process? If so, how are they set and optimized?\n\n3. **Hyper-Parameters:**\n   - Can you provide more details on the hyper-parameters used in your experiments, such as those related to the dynamic decision module and the KV-cache eviction strategy?\n   - How sensitive is the performance of D-LLMs to these hyper-parameters, and are there any guidelines for tuning them?\n\n4. **Compatibility with Different LLM Architectures:**\n   - While the paper demonstrates the framework's effectiveness on certain LLMs (e.g., LLaMA, GPT), how well does the approach generalize to other architectures, especially newer ones?\n   - Have you tested D-LLMs on any other LLM architectures or benchmarks not mentioned in the paper? If so, can you share the results?\n\n5. **Impact on Latency:**\n   - Does the dynamic decision-making process introduce any latency during inference? If so, how significant is this latency, and what are its implications for real-time applications?\n   - Are there any optimizations or future directions planned to minimize this latency?\n\n6. **Comprehensive Evaluation:**\n   - The experiments focus on specific benchmarks. Have you considered evaluating D-LLMs on a broader set of tasks to better understand its generalizability and robustness?\n   - Are there any plans to conduct further experiments that encompass a wider variety of NLP tasks and datasets?\n\n7. **Implementation Details:**\n   - Can you provide more detailed implementation guidelines, including pseudocode or a step-by-step description of the D-LLMs framework?\n   - Are there any particular challenges or considerations to be aware of when implementing this framework on different hardware or software environments?\n\nLIMITATIONS:\nThe authors have partially addressed the limitations of their work, primarily focusing on the technical aspects of the D-LLMs framework. However, there are several areas where the discussion of limitations could be expanded and more detailed. Additionally, the paper does not explicitly address potential negative societal impacts, which is an important consideration for any work involving large language models.\n\n1. The paper does not discuss the impact of token granularity on the effectiveness of the D-LLMs framework. Given that token granularity is crucial for the dynamic allocation of computational resources, a more thorough examination is needed. The authors should include an analysis of different granularity levels (e.g., subwords, words, characters) and their impact on model performance and efficiency.\n\n2. The evaluation is limited to specific benchmarks. A broader set of tasks and datasets would provide a more comprehensive understanding of the framework's generalizability and robustness.\n\n3. Dynamic inference mechanisms might inadvertently amplify biases present in the training data, as decisions on token importance might favor certain types of content over others. The authors should discuss how they mitigate such biases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces D-LLMs, a dynamic inference paradigm for large language models that adaptively allocates computing resources based on token importance. D-LLMs reduce computational costs and KV-cache storage by up to 45% and 50% on various tasks.\n\nSTRENGTHS:\n1.\tThe concept of dynamically adjusting the execution of transformer layers at the token level is both novel and impactful, providing a promising direction for reducing computational costs in LLMs.\n2.\tThe paper addresses the practical challenge of integrating dynamic inference with KV-cache methods, which is essential for real-world applications.\n3.\tThe authors conducted extensive experiments, demonstrating significant reductions in computational resources without sacrificing model performance across diverse NLP tasks.\n\nWEAKNESSES:\n1.\tWhile the experimental results are impressive, the paper lacks a deep theoretical analysis of the dynamic decision module's behavior and its impact on model performance.\n2.\tThe computational overhead introduced by the dynamic decision module itself is not sufficiently analyzed.\n3.\tThe manuscript requires a deeper examination of the impact of various components of the loss function. Specifically, a thorough analysis is needed to investigate how varying the value of 𝛼 affects the performance of model. \n4.\tAlthough the paper demonstrates the effectiveness of the overall approach, it would benefit from more detailed ablation studies to isolate the impact of different components of the dynamic decision module and eviction policy.\n5.\tThe manuscript necessitates a more elaborate description of the evaluation metrics. For instance, a detailed explanation of the computation methodology for FLOPs is required to provide readers with a comprehensive understanding of the evaluation process.\n\nQUESTIONS:\nSee the above\n\nLIMITATIONS:\nSee the above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes D-LLM, a novel layer-skipping framework for LLM inference to reduce computation costs. It designs and trains (fine-tuning) the additional decision module per transformer layer and implements KV-cache eviction by adjusting self-attention masks upon the execution decision. The framework presents ~50% reduction in computing costs.\n\nSTRENGTHS:\n- The study defines a clear problem to solve, which is important for current trends in model inference\n- The consideration of token-level importance and resource allocation, which results in a layer-skipping mechanism, is commendable\n- D-LLM carefully contemplates several hyperparameters, such as the number of preserved tokens, to achieve reasonable accuracy compared to the baselines\n- Evaluation contains measurements in diverse aspects\n\nWEAKNESSES:\n- D-LLM requires additional computation costs/resources to train (fine-tune) the decision modules and adaptors (LoRA modules). Presenting such additional overheads from the decision module training would be interesting.\n- Baselines do not include “dynamic inference mechanisms,” which are more closely related to D-LLM\n- D-LLM introduces the concept of a “user-defined target acceleration rate;” however, the experimental results do not include its use case or the changing behavior depending on the user-defined rate.\n- D-LLM claims that recent research shows preserving initial tokens is crucial, but I recommend the authors add references or clear evidence to support this statement (line 210).\n- The paper’s writing needs to be carefully revisited. For example, AdaInfer, MoD, and Shorten-LlaMa (baselines) have two duplicated citations (e.g., 14 and 15), which could be confusing.\n\nQUESTIONS:\n- What is the difference between 'dynamic pruning' (line 99) and 'dynamic inference mechanism' (line 105)? I understand the explanations in the paper, but the structure of the current paper distinguishes the related work into 1) LLMs acceleration methods and 2) dynamic inference mechanisms. I wonder why the above dynamic pruning belongs to the first.\n- The idea of inserting a decision module as proposed by D-LLM seems to be applicable even without a PEFT like LoRA. Discussion on any specific use of LoRA in this study or applying the module in other situations would be interesting.\n- If the LoRA modules (low-rank adaptors) are not trained together, will the desired accuracy not be achieved?\n- As mentioned in the weaknesses section, additional training is required to achieve similar performance to the baseline. How much overhead is this additional training? Is it small enough that it doesn't need to be shown in the paper?\n- If we want to infer on a completely new dataset, does D-LLM need to train further? If we use the decision module and low-rank adaptor already trained on a different dataset without additional training, how will the performance change?\n- How much does the target acceleration rate differ from the actual LLM inference acceleration rate (skipped-layers ratio)?\n- In equation 11, why is the average acceleration rate calculated with b^tilde rather than using b?\n- The evaluation parts measure performance mainly by FLOPS. Can FLOPS be interpreted for various purposes of evaluation? For example, how do FLOPS (instead of actual memory consumption) explain the overhead in KV-cache?\n\nLIMITATIONS:\n- Although the authors mentioned that the difficulty has not been modeled, a precise definition of the concept is required to generalize this design to be applied to other contexts\n- Currently, D-LLM is only applicable to parameter-efficient fine-tuning workloads that use LoRA (even if LoRA is a widely adopted PEFT algorithm) and requires additional training for new datasets or benchmarks", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
This manuscript introduces a new dynamic inference paradigm for LLMs called D-LLMs, which adaptively allocates computing resources in token processing. With the dynamic decision module, the network unit is decided to be executed or skipped on the fly. The KV-cache eviction policy is proposed to exclude skipped layers from subsequent calculations, reducing storage overhead while maintaining compatibility with KV-cache methods. The experimental results demonstrate the notable computation reduction and KV-cache storage utilization.

STRENGTHS:
1. The assumption that "not every word is equally important ..." is reasonable.
2. The experiments showcase the notable reduction in computation cost.

WEAKNESSES:
1. Highlighting how D-LLMs differ significantly from or improve upon existing methods like AdaInfer and SkipNet would strengthen the novelty claim. Providing a more detailed comparison with these methods in terms of theoretical underpinnings or empirical performance could clarify the unique contributions of D-LLMs.
2. The ablation on hyper-parameter \alpha in Eq. 14 is not provided. Please discuss its influence on task performance and computation reduction.
3. The font in Fig. 2-4 is too small for visualization.

QUESTIONS:
Please see weakness part.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13